Opened 7 years ago

Closed 7 years ago

Last modified 6 years ago

#5483 closed defect (fixed)

Standard deviation non numerically stable

Reported by: Even Rouault Owned by: Even Rouault
Priority: normal Milestone: 2.0.0
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords: standard deviation
Cc:

Description

The "naive" algorithm used in GDALRasterBand::ComputeStatistics?(), relying on the difference of the weighted sum of squared values with the square of the mean, is rather sensitive to numerical precision issues, particularly when operating on large absolute values. For example it might return a non-zero value if operating on uniform values of the order of one million or more.

Change History (2)

comment:1 Changed 7 years ago by Even Rouault

Milestone: 2.0
Resolution: fixed
Status: newclosed

trunk r27382 "ComputeStatistics?(): use Welford algorithm to avoid numerical precision issues when computing standard deviation (#5483)"

comment:2 Changed 6 years ago by Even Rouault

Milestone: 2.02.0.0

Milestone renamed

Note: See TracTickets for help on using tickets.