#5483 closed defect (fixed)
Standard deviation non numerically stable
| Reported by: | Even Rouault | Owned by: | Even Rouault |
|---|---|---|---|
| Priority: | normal | Milestone: | 2.0.0 |
| Component: | GDAL_Raster | Version: | unspecified |
| Severity: | normal | Keywords: | standard deviation |
| Cc: |
Description
The "naive" algorithm used in GDALRasterBand::ComputeStatistics(), relying on the difference of the weighted sum of squared values with the square of the mean, is rather sensitive to numerical precision issues, particularly when operating on large absolute values. For example it might return a non-zero value if operating on uniform values of the order of one million or more.
Change History (2)
comment:1 by , 10 years ago
| Milestone: | → 2.0 |
|---|---|
| Resolution: | → fixed |
| Status: | new → closed |
Note:
See TracTickets
for help on using tickets.
trunk r27382 "ComputeStatistics(): use Welford algorithm to avoid numerical precision issues when computing standard deviation (#5483)"