An Overview of Measurement System Variability
Measurement System Variation
Since it is a process itself, the act of measuring is subject to variability as are all processes. It is extremely important to understand measurement variation since many decisions may be made based on measurement results. Some basic questions we will try to answer are:
1. What are the basic sources of variation?
2. Is the system statistically stable over time?
3. How close to the “truth” are measured results? How is this quantified?
4. What are some means of the quantifying or characterizing the variation in a measurement system.
Types of Variability
Variability in measurement, of course, involves special and common causes. Variability (or errors) can be divided into three categories: human errors, systematic errors, and random errors.
Human errors are the most elusive type to attempt to control. They occur at random, intermittently, and can be either large or small. Misreading instruments or equipment, transposition of numbers, inputting the wrong values into a computer or calculator, and measuring the wrong sample are examples. Most are impossible to control and correct as carelessness is usually the principal cause.
Systematic errors or assignable errors are always of the same sign, either positive or negative. They are constant regardless of the number of measurements made. These are errors due to bias, as defined in the following paragraphs. As such they can usually be identified. After identification, they can be eliminated or negated through correction factors. Elimination is always preferred over correction as a control method.
Random errors represent the common cause variability of the measurement system. They are both positive and negative in effect and occur by chance. Some examples are the slight variations that may exist in the sample injection techniques for a gas chromatograph or minor temperature of a drying oven or the sensitivity limitations of a pH electrode.
While they cannot be totally eliminated, they can be reduced. They can be estimated statistically and used to validate measurement results.
Our objective must be to control, monitor, and estimate the variability in measurement results, and to eliminate the effects of systematic errors.
There are some terms that have widespread use when dealing with measurements. Before proceeding, these need to be discussed.
Stability refers to the total variation in measurement obtained with the same equipment on the same standard over an extended period of time. Statistical stability of a measurement system implies that the test is predictable over time. Without this, any analysis of measurement variability is only applicable to the study time period. Statistical stability permits the results to be used to characterize future performance. Unless there is objective evidence of the measurement systems statistical stability, do not use results from a measurement variability study to predict future performance of the tests/equipment.
The means of demonstrating statistical stability is the control chart. Charting standards on average and range or individual and moving range charts not only depict the stability of the measurements but also serve as indicators that calibration is required. Calibration while the system still indicates an in control condition will generally only serve to increase the measurement systems variation.
Statistical stability, or statistical control, does not mean the measurement process has been optimized. Several different organizations may use similar measurement methods with each in statistical control, but their performance can differ notably.
Accuracy, Bias, and Precision
Accuracy is the closeness of agreement between a test result and the “true” or accepted reference value. In other words, how close are we to the “truth.” To better define accuracy, two additional terms are used.
Bias refers to a systematic error that contributes to the difference between a population mean of the measurements or test results and an accepted reference or true value.
Precision is the closeness of agreement between randomly selected individual measurements or test results obtained under prescribed conditions. An accurate method is one capable of producing unbiased and precise results. With measurements, we evaluate inaccuracy; we attempt to quantify the bias and the imprecision.
Accepted Reference Value is a value that serves as an agreed upon reference for comparison and which is derived as:
· a theoretical or established value based on scientific principles,
· an assigned value based on experimental work such as NIST or
· a consensus value, based on collaborative experimental work (such as the ASTM Inter-lab Crosscheck Sample Exchange Program.)
ASTM D6299 provides an accepted methodology for statistically determining an accepted reference value.
Standard Deviation is a mathematically calculated quantity that measure precision or “noise” of a process,
· σ, commonly referred to as ‘sigma’
· Estimated from historical and current data using statistical techniques
· A measure of variation
The standard deviation of the measurement error may be used as a measure of precision, or actually “imprecision.”
Calibration, or re-calibration, can improve the accuracy of a measurement by reducing the error or bias. However, calibration does not necessarily have any effect on the precision of the measurements.
Measurement System Variability
The accuracy, bias and precision of a measurement system can be partitioned into a portion that is attributable to the equipment or apparatus and that associated with different people or laboratories performing the test. Special terms for these constituents of precision are as follows:
Repeatability of a measurement process implies that the test variation is consistent. It is a measure of the degree of agreement between independent test results obtained within a short time interval with the same test method in the same laboratory by the same operator using the same equipment and the same sample(s). By keeping so many factors the same, repeatability represents the inherent variability in the test equipment or apparatus.
Reproducibility is a measure of the degree of agreement between test results obtained in different labs with the same test method using the same sample(s). It includes the differences such as operators, equipment, and supervision that will exist between labs. As a result, it can never be less than the repeatability of a test. ASTM uses this definition and that for repeatability to characterize test method performance for any lab.
There are differences in terminology because AIAG does not use the ASTM definitions. While their definition of repeatability is essentially the same, AIAG methodology uses reproducibility to mean variability associated with the operators. Their equivalent of ASTM reproducibility is called R & R, or the combination of equipment and operator variability.
You should be aware of the terminology used by your customers.
Sources of Variability
The systematic and random errors that can influence measurement results can come from a multitude of sources. Generally, these can be summarized into the following categories:
The equipment, whether it is a sophisticated automated electronic analyzer or glassware, has been manufactured to certain tolerances. The variation inherent to the equipment specifications will be reflected in the test results. Component wear, failure, or inadequate maintenance will increase the variation in test results. Any inconsistencies in calibration verification and/or recalibration will also effect the consistency of the results obtained from the equipment.
People are almost always a contributor to variation simply because none of us are exactly alike. We differ in dexterity, reaction times, color sensitivity, and other ways. Even the same operators can perform differently at different times due to degrees of mental and physical alertness. Some degree of operator differences are practically unavoidable. Of course, some tests are more sensitive to the effects of operator differences. Incomplete or inexplicit test methods open the door to another difference in operators, “interpretation” of the requirements.
Some samples and equipment may be susceptible to temperature, humidity, atmospheric pressure, and other environmental factors. Because these cannot be controlled perfectly within or between labs, they provide some contribution to the variation of test results.
Any non-uniformity of the sample can add to the variation in the test results. When conducting studies to determine testing variability, special effort must be made to obtain test samples that are as uniform or similar as possible.
All of the previously mentioned sources of variation can themselves change with time. In measurement studies, efforts are usually made to keep the time span as short as practical.
Measurement Systems Analysis
A number of different techniques are useful for analyzing measurement system variability. These include Measurement Variability Studies (both short and long), control charts, designed experiments, and analysis of variance. Donald Wheeler’s book, “Evaluating the Measurement Process,” does an excellent job of presenting the control chart approach. Please refer to this for a detailed discussion of the topic. The AIAG, MSA Manual, 4th edition is the ‘bible’ for the automotive industry. To be compliant with the IATF 16949:2016 standard, all MSA studies must conform to the methodology outlined in the MSA manual.
Inter-laboratory versus Intra-laboratory Studies
Establishing the repeatability of a method is accomplished about as well in one laboratory as in another. Usually the differences in results between labs are due not so much to differences in precision, but in systematic errors or biases.
Inter-laboratory studies (between labs) can establish the relative magnitudes of the biases and the precision. They do not offer much assistance in uncovering the assignable causes for the biases.
To gain the necessary information to identify the assignable causes and eliminating their effects, studies on the measurement system must be conducted in a single laboratory. (Intra-laboratory) These studies may involve independent verification of a laboratory’s results.
Independent verification activities:
· Blind Sample Programs
· Inter-lab Crosschecks
1. What are some sources of measurement variation?
2. Explain the differences between bias and precision.
3. How do you judge whether a measurement system is statistically stable?
4. List some different techniques for analyzing measurement variability.
5. Explain what repea