Opened 11 years ago

Closed 11 years ago

#163 closed defect (fixed)

ntv2 grids should be fully cached for performance reasons

Reported by: ravenAtSafe Owned by: steffen
Priority: major Milestone:
Component: Library Version:
Keywords: Cc:

Description

After upgrading to CS-Map 13.10 (r2144) from 12.02 (r1964), FME was observed to slow down by two orders of magnitude while reprojecting ~44M points involving a DHDN BeTa2007 NTv2 datum transformation. This slowdown only occurred on Windows; Linux remained performant.

Profiling indicated that:

  • all time was being spent within fseek/fread calls
  • we appeared to be doing an appropriate amount of work for each point
  • whereas previous versions of FME registered almost no disk I/O during this operation, the current FME shows massive disk I/O (as observed in Process Explorer)

Tweaking setvbuf cache sizes had no observable impact on performance. Diff inspection of the upgrade did not find any changes obviously related to the degraded performance. Extensive Google searches did not result in any recommendations of how to encourage Windows to return to its previous caching behavior.

Erratic performance seemingly tied to file system caching has been observed within FME before, with Japanese grids (CScalcTokyoToJgd2k).

One thing not explicitly tested that might have been interesting: Can a slowdown of similar magnitude be observed in the CS-Map testbed? This may have indicated whether the way FME interacts with CS-Map, or the way FME is managing other resources, is a factor.

Testing of patch:

  • Logs of the Test and TestCpp programs were collected with and without the NTv2 caching patch (random seed of '1' used in all runs). Diffs indicate no change in behavior.
  • FME's own testsuite did not detect behavior changes during NTv2 reprojection

Change History (2)

comment:1 by steffen, 11 years ago

Owner: changed from hugueswski to steffen

comment:2 by steffen, 11 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.