NTv2 Performance Problem

[Norm Olsen]

Context: The NAD27_to_NAD83 Geodetic Transformation definition lists the Canadian NTv2 file first. We are converting coordinates outside the coverage to the NTv2 file.

Problem: With NTv2 first on the list, performance is is 25% slower than when it is placed last on the list.

Seems that since the coordinates are outside the coverage of the entire NTv2 file, the performance penalty should not be so large.

Added overall file extents to the main NTv2 transformation object. Now the CStestNTv2 function can see if the point in question is within the overall coverage of the file rather quickly. Sub-grids are examined only in the case where the query point is within the coverage of the overall file.

This is especially helpful in the case of NAD83, where most of the US48 geography is not covered by the Canadian NTv2 data file.

Fix incorporation produced revision 2444.

[Norm Olsen]

The new Geodetic Transformation facility introduced by RFC-2 removed the total file coverage elements from the cs_Ntv2_ overall control structure. Thus, now all sub-grids are examined to determine if the file does indeed cover the point to be converted. This is nice design, but makes the CScntv2T significantly slower than it needs to be.

I suspect that this was done this way as the Spanish file does not conform to the standard wherein the it has separate sub-files which overlap and the combination does not result in single cohesive rectangle of coverage. Need to check into that. If sop, we probably should, at run-time, identify that a file is of this variety, and enable the faster coverage check for all files which are not strange in this way. The performance improvement may turn out to be very significant.

I believe we can change the cs_Ntv2_ structure without any repercussions outside of CS-MAP so that it contains the coverage of the entire NTv2 file.

[ravenAtSafe]

Is this before or after the NTv2 performance fix on http://trac.osgeo.org/csmap/ticket/163? Feels like it could have had an impact?

[Norm Olsen]

I have come to this issue based on incredibly poor WKT processing on Windows 7 using VC++ 2012. A process (csTestJ.cpp) that completes on Linux in about 60 seconds runs for hours in this environment on the very same box. Indications are the same: fseek, fread problems. I suspect that Microsoft has run amuck with its security paranoia; problem is probably in the CRT delivered with the compilers. This is the only explaination I can think of given that there is a drastic difference between VC++ 2008 and VC++ 2012 on the exact same system (OS & box).

[ravenAtSafe]

For what it is worth, we found issue 163 while building with the Visual Studio 2010 Platform Toolset (v100).

[Norm Olsen]

The performance issue I address here is unrelated to the disk I/O issues which I have investigated rather thoroughly. I'll deal with that in another ticket.

The issue I intend to deal with here is that since the NTv2 file is included in the NAD83 definition, and is usually listed first on the list, the Canadian file is tested for coverage for all US data points being converted. Currently, this test for coverage does a scan of all the sub-grids. In the Canadian file there are a hundred sub-grids. Thus, this represents a significant performance hit.

I propose that the cs_NTv2_ structure be modified to include a coverage indication which represents the entire file. The sub-grids would be interrogated only if the point in question is within that region of coverage. Thus, the performance penalty of having the Canadian file first will be greatly diminished.

[Norm Olsen]

The performance issue I'm chasing here is completely different from the streaming issue. I added Trac Ticket 177 which isolates the stream problem related to WKT. My testing did not find any problem with direct use of FILE type streaming in Windows 7, regardless of compiler. So the cause behind Ticket 163 remains a mystery.

Discovered on thing of interest in correcting this ticket. Using VC++, I have found that using a ">=" comparision on two "double"s is very much slower than a simple "<" or ">". Much more so than I would have ever thought. I have not determined if gcc behaves similarly.