Verifying measuring devices - method comparison

Knowledge base · June 27, 2026

You buy a new online probe, switch laboratories, or want to replace an expensive lab analysis with a cheaper field measurement. Each case raises the same question: can the new method be trusted as much as the old one? The answer isn’t a gut feeling or a correlation coefficient - it’s a structured method comparison.

Correlation is not agreement

The most common mistake: “the sensor correlates with the lab at 0.99, so it’s fine.” It isn’t enough. A high Pearson r only tells you that both methods rise and fall together - yet the sensor may be systematically high by a fixed amount or a fixed percentage. Correlation won’t catch that. Assessing interchangeability of methods calls for different tools.

The toolkit that actually answers the question

• Deming regression - a fit line that, unlike ordinary OLS regression, assumes error on both axes. When neither method is the absolute truth, this is the right choice. A slope ≈ 1 and intercept ≈ 0 mean no proportional and no constant error. LimnoLog assumes the two methods have comparable error (error-variance ratio = 1).
• Identity line (y = x) - the “perfect agreement” reference. The closer the points lie to it, the better.
• Bias (mean difference) and bias % - how far, on average, the test method departs from the reference. This is the measure of systematic error.
• Bland-Altman plot - the difference between methods against their mean, with limits of agreement (mean difference ± 1.96 SD of the differences). It shows whether the discrepancy is constant or grows with concentration, and how large the differences realistically get.
• RMSE - the typical error magnitude in measurement units.
• N - the number of paired measurements; without enough pairs the conclusions are fragile.

⚠️ This whole toolkit only works for two methods measuring THE SAME quantity. Deming regression and Bland-Altman assume both numbers are attempts to estimate the same thing (BOD₅ from a probe and BOD₅ from the lab). If you put different quantities side by side - turbidity ↔ suspended solids, UV254/SAC ↔ TOC, conductivity ↔ TDS - the bias and limits of agreement mean nothing, because there is nothing to compare: there is something to convert. The right tool is then a predictive regression (surrogate calibration) with a prediction error and prediction band, not an agreement assessment. In LimnoLog this is the Goal: Agreement / Calibration switch inside the same mode.

What it looks like in practice

Below is an example from LimnoLog (demonstration data): the BOD₅ reported by an online optical probe (test method; the probe does not incubate the sample, it estimates BOD₅ from an optical measurement) set against the laboratory BOD₅ determination (reference method) in the same treated-wastewater samples:

How to read it: in this example (15 pairs, BOD₅ of about 7-14 mg O₂/L) the Deming slope is 1.01 (ideal = 1) and the bias is -0.15 mg O₂/L - on average the probe reads about 1.5% lower than the laboratory, with no visible proportional error. The limits of agreement (-0.68 … +0.38 mg O₂/L) by construction contain about 95% of the differences. Interchangeability is decided only by whether a difference of this size is immaterial for the purpose of the measurement (e.g. when comparing with a limit value from a permit) and whether the conclusion stays within the range of concentrations and the matrix that were studied. Had the points been tilted or the limits wide, the cheaper method could not safely stand in for the reference.

The decision: when are methods interchangeable

There is no single threshold - it depends on what the data are used for. Helpful questions:

• Are the limits of agreement narrower than a difference that would matter in practice (e.g. a decision threshold or a standard)?
• Is the bias small enough to ignore - and if not, is it constant and therefore correctable?
• Does the discrepancy not grow with concentration (no tilt on the Bland-Altman plot)?

If the answers are “yes”, you can use the cheaper or faster method within the range and matrix studied, provided that regulations or the permit do not require a specific reference method.

Sources

• Difference plot and limits of agreement: Bland J.M., Altman D.G. (1986), Statistical methods for assessing agreement between two methods of clinical measurement, The Lancet 327(8476), 307-310 - DOI.
• Deming regression and the choice of regression procedure: Linnet K. (1993), Evaluation of regression procedures for methods comparison studies, Clinical Chemistry 39(3), 424-432 - DOI.
• How many paired measurements you need - not a single number; it depends on the range and on the error of both methods: Linnet K. (1999), Necessary sample size for method comparison studies based on regression analysis, Clinical Chemistry 45(6), 882-894 - DOI.

(Addresses checked on 18 August 2026.)

In LimnoLog

The app handles this in Charts → Method comparison. All you need is to record both the test and the reference result in the same sample (session + station) - for example a telemetry-probe parameter alongside a lab determination. LimnoLog pairs the measurements automatically and computes the Deming regression, bias, limits of agreement and Bland-Altman. It’s a natural complement to sound QA/QC practice and broader statistical analysis; for continuous measurements themselves, see Telemetry and online sensors.

Start a project in LimnoLog and verify your equipment on your own data.