gdalwarp memory allocation error

[alsaleem]

I am tiling a large vrt file (~850 files, total 2.3T, 3.2G/file) using command line gdalwarp. The command is

gdalwarp -ts 256 256 -te 3799837.550113 3228700.074766 3801060.542565 3229923.067218 /mnt/hdd/ksa/ksa.vrt /mnt/files/aa.tif

then I got an error

ERROR 2: VSIMalloc3(): Out of memory allocating 4926998016 bytes.

ERROR 2: VSIMalloc3(): Out of memory allocating 4926998016 bytes.
I debugged gdalwrap and found error appear when calling GDALWarpCreateOutput -> GDALSuggestedWarpOutput2

I used to use smaller set of these files (~20 files, 3G each) and had no error. Some files are tiled and some are strips.

Could this be a bug? or because of large number of files in vrt?

[Even Rouault]

If your input VRT is very huge (which seems to be the case), I guess you hit the following code :

#define N_PIXELSTEP 50
    int nSteps = (int) (double(MIN(nInYSize, nInXSize)) / N_PIXELSTEP + .5);
    if (nSteps < 20)
        nSteps = 20;

retry:
    int nSampleMax = (nSteps + 1)*(nSteps + 1);
    int *pabSuccess = NULL;
    double *padfX, *padfY, *padfZ;
    double *padfXRevert, *padfYRevert, *padfZRevert;

    double dfRatio = 0.0;
    double dfStep = 1. / nSteps;

    pabSuccess = (int *) VSIMalloc3(sizeof(int), nSteps + 1, nSteps + 1);
    padfX = (double *) VSIMalloc3(sizeof(double) * 3, nSteps + 1, nSteps + 1);
    padfXRevert = (double *) VSIMalloc3(sizeof(double) * 3, nSteps + 1, nSteps + 1);
    if (pabSuccess == NULL || padfX == NULL || padfXRevert == NULL)
    {
        CPLFree( padfX );
        CPLFree( padfXRevert );
        CPLFree( pabSuccess );
        if (nSteps > 20)
        {
            nSteps = 20;
            goto retry;
        }
        return CE_Failure;
    }

The error should not be fatal. A retry should be made with a much smaller value of nSteps = 20. Does gdalwarp abort or does it go on ? If it is go on, do not worry about the error. Warping such a huge dataset is likely to take a huge amount of time too. I'd suggest creating a tiled geotiff with -co TILED=YES

What struck me in your command line is the use of "-ts 256 256" : do you really want to create a dataset that is only 256x256 pixels ?

If you don't do any reprojection, you might have a better chance trying gdal_translate instead of gdalwarp

Possible idea for optimization for GDAL developers: with both -ts and -te specified, we could likely skip the call to GDALSuggestWarpOutput2()

[Jukka Rahkonen]

You wrote you are tiling. Does it mean that you want to split the whole dataset into 256x256 pixel sized tiles with some automatic script?

If that is the case and if only getting the job done is important, it could be faster to create the tiles with Mapserver. And my experience is that Mapserver is faster with shapefile tileindex created with gdaltindex than with .vrt file if there are tons of files in the index. There is also gdal2tiles ​http://www.gdal.org/gdal2tiles.html which is especially made for cutting tiles.

[Even Rouault]

Ah, I read too fast and missed the mention to tiling. So -ts 256 256 with a sub extent specified with -te makes sense and my mention to -co TILED=YES isn't really relevant ;-)

gdal2tiles or gdal_retile are certainly tools best adapted for tiling.

[alsaleem]

gdal2tiles is very very slow. So, I am tiling for WMS with my own script (C++).
gdalwarp goes on but not sure if it gives the right output.
The error message is annoying. Is there away to suppress it without recompiling?
gdal_translate does not give the option to set re-sampling so I have to go with gdalwarp.
I am servicing the tiles with openlayers.WMS so I am not using mapserver.

Thanks.

[Jukka Rahkonen]

I was meaning that you could use Mapserver for seeding, you do not need to use it for serving the ready made tiles. TileCache, MapCache, MapProxy, GeoWebCache etc. are pretty clever in tiling, they can request metatiles and split them to final dimensions which can be faster than creating tiles one by one. One more idea: If big vrt files are slow, which is also my experience, and you have written your own script, why not to use many smaller vrt files as input? You are interested in small area at a time so you could create for example 20 vrt files with 40 images in each and feed them for your script.

[Even Rouault]

I've no reason to believe that gdalwarp would lead to bad output due even if this error message is emitted. As I wrote, the values computed by GDALSuggestWarpOutput2() are likely immediately overriden afterwards by the value specified by -ts and -te. Suppressing the error message requires a fix in GDAL code.

I believe you could still use gdal_translate for your purpose by using the -projwin and -outsize values. If your request involves resampling, the resampling algorithm used by gdal_translate will be nearest neighbour (but this is also the case in the gdalwarp command line you've posted above, unless you use an additional resampling flag)

[alsaleem]

I am planning to use re-sampling if it goes well as re-sampling will slow the initial setup/dry run.
I've taken your words that the output is still good, otherwise I have to manually check image with a one I made on previous runs with smaller set.
Also, the reason I am not using any of these progs is that I can still use my script for non-WMS tiles (such as TatukGis's pixelstore).

Splitting vrt into smaller sets will create problems later at edges of vrt worlds as these tiff files overlap (I am combining with nodata). Besides I have to manually group these sets.

Am I left with re-compile?

Thanks.

[Even Rouault]

Although I'm confident that the error message can be ignored, you're not obliged to trust me blindly, so I'd encourage you to check however on a few samples ;-)

For the moment, no need to re-compile anything because there's no fix committed (unless you intend to fix it by yourself of course). The error message is a bit annoying of course, but I'm not sure why you consider this as "blocker" if the end result works for you.

[alsaleem]

I was not sure if the output will be ok with this error (without consulting you).
Let me check manually/visually with a few samples to see if the error does not affect output.

Regards,

[alsaleem]

I tested the output with previously generated one (from different vrt file) and found identical.
Now, how do I suppress this annoying error message?

Regards,

[Even Rouault]

trunk(r24929) : gdalwarp: don't call GDALSuggestedWarpOutput2() when (-tr and -te) or (-ts and -te) are specified (#4804)