How to prepare data for statistical analysis (R, STATISTICA)
Experienced analysts half-jokingly say that 80% of the time in analysis is tidying data, and only 20% the actual computing. The better you prepare the data, the less pain when loading it into R, STATISTICA or Python - and the less risk that the result will be wrong because of a mess at the input.
The principle: “tidy data”
The most convenient for analysis is the long format:
- one row = one observation (one measurement),
- one column = one variable (date, station, indicator, value, unit),
- consistent column names, no merged cells and no “two-storey” headers.
Such an arrangement loads into any tool without manual rework (see also long vs wide format).
What to watch out for in particular
- Consistent units - one indicator = one unit across the whole dataset; otherwise averages make no sense.
- Number format - for CSV use a dot decimal and UTF-8; a comma decimal breaks the import in many tools.
- Missing data - mark it unambiguously (e.g. an empty cell /
NA), not with zero - zero is a value. - Values below the limit of quantification - set and describe a convention (this is a common problem in environmental data); the statistics depend on it.
- Dates - one unambiguous format (preferably
YYYY-MM-DD), treated as time.
Repeatability
- Keep the raw data separate from the transformed - you can always go back to the source.
- Record what and how was computed (even in a note), so the analysis can be reproduced.
- CSV / UTF-8 is the safest format for exchange between programs.
In practice
The fewest errors arise when the data is tidy from the start - in fixed stations, sessions and units. LimnoLog keeps measurements in this structure and exports them to Excel ready for further processing, while simpler analyses and charts you can do straight in the app - reaching for R or STATISTICA for heavier statistics only when you really need it.
← Knowledge base