Opened 15 years ago

Closed 13 years ago

#841 closed defect (fixed)

transformTo() memory leak

Reported by: hobu Owned by: warmerdam
Priority: high Milestone:
Component: OGR_SRS Version: unspecified
Severity: normal Keywords:
Cc:

Description (last modified by hobu)

IRC Log:

hobu_work	any ideas why this would just continue to consume gobs of memory?
hobu_work	http://rafb.net/paste/results/ILVkKn59.html
hobu_work	what am I forgetting to destroy?
FrankW	hobu_work: I don't see any obvious source of problems. Perhaps there 
        is an issue in transformTo.
hobu_work	I modified it to this http://rafb.net/paste/results/o6fBo256.html 
            and took out all of the Clone()'s and it still grows just as fast 
            (this is a 300k oracle table I'm dumping out to WKT)
FrankW	Can you dump the table to shapefile, and run against that instead? 
        That would help isolate whether it is an oracle problem or not.
FrankW	Another quick test would be to try removing the transformTo() call 
        and see if that makes a difference.
hobu_work	removing the TransformTo does not slow the memory consumption down...
hobu_work	I take that back... removing the TransformTo (both of them) 
            *does* limit the memory consumption
hobu_work	like so... http://rafb.net/paste/results/bPsGrA42.html
FrankW	Ah, I didn't notice you were calling get_zone() each time.
FrankW	If you enable the main transformTo is there still a leak?
hobu_work	yes
FrankW	Are these lines? Polygons? Points?
hobu_work	Poly's
hobu_work	with small 4-40 pts in them
hobu_work	is looking at OGRErr OGRGeometry::transformTo( OGRSpatialReference *poSR )
FrankW	I looked through transform() which it calls, and it seems OK.
FrankW	Ahh
FrankW	The problem may be with cleaning up the transformer which is 
        created and destroyed each time.
hobu_work	the delete poCT; statement?
FrankW	Yes, but it is possible that the OGRCoordianteTransformation 
        destructor isn't as clean as it ought to be.
FrankW	Yes, I think I see.
FrankW	the new srs is assigned to the geometries, but the ~OGRGeometry() method 
        does not delete the SRS if the reference count falls to zero, though it 
        does dereference it.
FrankW	This is because some driver implementations personally destroy the 
        OGRSpatialReference for themselves (which they assign to their geometries) 
        assuming that no one else could be still referencing them.
FrankW	So I broke the normal reference counting rules for geometries to avoid 
        causing crashes in esoteric circumstances.
FrankW	You are welcome to file a bug on this (include my text above), but I'm 
        not likelyi to fix it right away.
FrankW	Of course, there could be more going on. I'll likely need to prepare a 
        small c++ test program I can easily run under valgrind to be sure. 

Python Code:
import sys
#sys.path.insert(0,r'D:\cvs\gdal\gdal\pymod\ng\build\lib.win32-2.3')
sys.path.insert(0,r'D:\cvs\gdal\gdal\pymod\\')
import os
os.environ['GDAL_DATA']=r'd:\cvs\gdal\gdal\data'
os.environ['PROJ_LIB']=r'c:\proj\nad'
import ogr
import osr
 
 
zones = {10:[-126,-120],
         11:[-120,-114],
         12:[-114,-108],
         13:[-108,-102],
         14:[-102,-96],
         15:[-96,-90],
         16:[-90,-84],
         17:[-84,-78],
         18:[-78,-72],
         19:[-72,-66],
         20:[-66,-60]
        }
 
ref = osr.SpatialReference()
ref.ImportFromEPSG(4326)
 
def get_zone(point):
 
    #point2 = point.Clone()
    point.TransformTo(ref)
    minx = maxx = point.GetEnvelope()[0]
    #point.Destroy()
 
    for i in zones:
        min,max = __builtins__.map(float,zones[i])
        if minx > min and minx < max:
            min_utmzone = 26900+i
        if maxx > min and maxx < max:
            max_utmzone = 26900+i
    if min_utmzone == max_utmzone:
        return min_utmzone
 
 
esri_lines =
['PROJCS["USA_Contiguous_Albers_Equal_Area_Conic_USGS_version",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-96.0],PARAMETER["Standard_Parallel_1",29.5],PARAMETER["Standard_Parallel_2",45.5],PARAMETER["Latitude_Of_Origin",23.0],UNIT["Meter",1.0]]']
in_ref = osr.SpatialReference()
in_ref.ImportFromESRI(esri_lines)
out_ref = osr.SpatialReference()
 
f = open(r'd:\scott_psu.txt','wb')
 
lyr = ogr.Open('OCI:username/pw@oranri_design.cssm.iastate.edu:SEGMENT').GetLayer(0)
feat1 = lyr.GetNextFeature()
 
while feat1:
    geom = feat1.GetGeometryRef()
    geom.AssignSpatialReference(in_ref)
 
    #geom.AssignSpatialReference(in_ref)
 
   # zone =  get_zone(geom)
    zone = 26910
    out_ref.ImportFromEPSG(zone)
 
    #geom.TransformTo(out_ref)
 
    outstring = '%s,%s,%s,%s,%s\n'
    outstring = outstring %(feat1.STATE, feat1.COUNTY, feat1.PSU, zone,
geom.ExportToWkt())
 
    f.write(outstring)
    feat1.Destroy()
 
    feat1 = lyr.GetNextFeature()
 
 
 
f.close()

Change History (4)

comment:1 Changed 15 years ago by hobu

pre-generating the refs doesn't slow the memory consumption down (turning off
both the call to get_zone and the geom.TransformTo(out_ref) does, however)


import sys
#sys.path.insert(0,r'D:\cvs\gdal\gdal\pymod\ng\build\lib.win32-2.3')
sys.path.insert(0,r'D:\cvs\gdal\gdal\pymod\\')
import os
os.environ['GDAL_DATA']=r'd:\cvs\gdal\gdal\data'
os.environ['PROJ_LIB']=r'c:\proj\nad'
import ogr
import osr
 
 
zones = {10:[-126,-120],
         11:[-120,-114],
         12:[-114,-108],
         13:[-108,-102],
         14:[-102,-96],
         15:[-96,-90],
         16:[-90,-84],
         17:[-84,-78],
         18:[-78,-72],
         19:[-72,-66],
         20:[-66,-60]
        }
 
ref4326 = osr.SpatialReference()
ref4326.ImportFromEPSG(4326)
 
refs = {}
for i in zones:
    aref= osr.SpatialReference()
    aref.ImportFromEPSG(26900+i)
    refs[i] = aref
 
def get_zone(point):
 
    point.TransformTo(ref4326)
    minx = maxx = point.GetEnvelope()[0]
 
    for i in zones:
        min,max = __builtins__.map(float,zones[i])
        if minx > min and minx < max:
            min_utmzone = i
        if maxx > min and maxx < max:
            max_utmzone = i
    if min_utmzone == max_utmzone:
        return refs[min_utmzone]
 
 
esri_lines =
['PROJCS["USA_Contiguous_Albers_Equal_Area_Conic_USGS_version",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-96.0],PARAMETER["Standard_Parallel_1",29.5],PARAMETER["Standard_Parallel_2",45.5],PARAMETER["Latitude_Of_Origin",23.0],UNIT["Meter",1.0]]']
in_ref = osr.SpatialReference()
in_ref.ImportFromESRI(esri_lines)
out_ref = osr.SpatialReference()
 
f = open(r'd:\scott_psu.txt','wb')
 
lyr = ogr.Open('OCI:username/ps@oranri_design.cssm.iastate.edu:SEGMENT').GetLayer(0)
feat1 = lyr.GetNextFeature()
 
while feat1:
    geom = feat1.GetGeometryRef()
    geom.AssignSpatialReference(in_ref)
 
    out_ref= get_zone(geom)
 
    geom.TransformTo(out_ref)
 
    outstring = '%s,%s,%s,%s,%s\n'
    outstring = outstring %(feat1.STATE, feat1.COUNTY, feat1.PSU, 1,
geom.ExportToWkt())
 
    f.write(outstring)
    feat1.Destroy()
 
    feat1 = lyr.GetNextFeature()
 
 
 
f.close()

comment:2 Changed 15 years ago by hobu

Tried with a shapefile, and it leaks just as fast

comment:3 Changed 15 years ago by warmerdam

Problem confirmed with the following program under Valgrind.  The leak 
is one OGRSpatialReference per shape processed.  The problem is that the
OGRCoordinateTransformation clones the spatial reference, it is attached to
the geometry, and then coordinate transformation is deleted before the geometry.
But the geometry will not actually delete the spatial reference, it just 
dereferences it.  We can't "fix" this without a careful pass through all the
OGR drivers to make them honour reference counting on spatial references too.


#include <stdio.h>
#include "cpl_conv.h"
#include "ogrsf_frmts.h"


int main()

{
    OGRRegisterAll();

    OGRDataSource       *poDS;
        
    poDS = OGRSFDriverRegistrar::Open( "data/esri/shape/world/world.shp", 
                                       FALSE );

    
    OGRLayer *poLayer = poDS->GetLayer(0);

    OGRSpatialReference *poWGS84;

    poWGS84 = new OGRSpatialReference();
    poWGS84->SetFromUserInput( "WGS84" );

    OGRFeature *poFeat;

    while( (poFeat = poLayer->GetNextFeature()) != NULL )
    {
        OGRGeometry *poGeom = poFeat->GetGeometryRef();
        OGRSpatialReference *poER;
    
        poER = new OGRSpatialReference();
        poER->SetFromUserInput( "+proj=eqc +datum=WGS84" );
        
        poGeom->assignSpatialReference( poWGS84 );
        poGeom->transformTo( poER );

        delete poFeat;

        delete poER;
    }

    delete poWGS84;
    delete poDS;

    delete OGRSFDriverRegistrar::GetRegistrar();
    CPLFinderClean();
}

comment:5 Changed 13 years ago by hobu

Description: modified (diff)
Resolution: fixed
Status: assignedclosed

I'm pretty sure Frank did a pass through the drivers about 1.5 years ago and has cleaned this issue up. Closing for now.

Note: See TracTickets for help on using tickets.