The precision of a value describes the number of digits that are used to express that value. In a scientific setting this would be the total number of digits (sometimes called the significant figures or significant digits) or, less commonly, the number of fractional digits or decimal places (the number of digits following the point). This second definition is useful in financial and engineering applications where the number of digits in the fractional part has particular importance.
In both cases, the term precision can be used to describe the position at which an inexact result will be rounded. For example, in floating point arithmetic, a result is rounded to a given or fixed precision, which is the length of the resulting significand. In financial calculations, a number is often rounded to a given number of places (for example, to two places after the decimal separator for many world currencies).
As an illustration, the decimal quantity 12.345 can be expressed with various numbers of significant digits or decimal places. If insufficient precision is available then the number is rounded in some manner to fit the available precision. The following table shows the results for various total precisions and decimal places rounded to the nearest value using the round-to-even method.
| Precision |
Rounded to significant digits |
Rounded to decimal places |
|---|---|---|
| Five | 12.345 | 12.34500 |
| Four | 12.34 | 12.3450 |
| Three | 12.3 | 12.345 |
| Two | 12 | 12.34 |
| One | 1 × 101 | 12.3 |
| Zero | n/a | 12 |
Note that it is often not appropriate to display a figure with more digits than that which can be measured. For instance, if a device measures to the nearest gram and gives a reading of 12.345 kg, it would create false precision if you were to express this measurement as 12.34500 kg.
The representation of a positive number x to a precision of p significant digits has a numerical value that is given by the formula
- round(10−n·x)·10n, where n = floor(log10 x) + 1 − p.
For a negative number, the numerical value is minus that of the absolute value. The number 0, to any precision, can be taken to be 0.
Famous quotes containing the words arithmetic and/or precision:
“Your discovery of the contradiction caused me the greatest surprise and, I would almost say, consternation, since it has shaken the basis on which I intended to build my arithmetic.... It is all the more serious since, with the loss of my rule V, not only the foundations of my arithmetic, but also the sole possible foundations of arithmetic seem to vanish.”
—Gottlob Frege (1848–1925)
“We are often struck by the force and precision of style to which hard-working men, unpracticed in writing, easily attain when required to make the effort. As if plainness and vigor and sincerity, the ornaments of style, were better learned on the farm and in the workshop than in the schools. The sentences written by such rude hands are nervous and tough, like hardened thongs, the sinews of the deer, or the roots of the pine.”
—Henry David Thoreau (1817–1862)