[raster] Unable to open more than 1024 out-db files in a single connection

[Bborie Park]

On a large query spanning every day of many years, I get the following error:

rt_band_load_offline_data: Cannot open offline raster: SOME_RASTER_FILE

In the PostgreSQL logs…

ERROR 4: `SOME_RASTER_FILE' not recognised as a supported file format.

I know these files are all valid rasters (as per GDAL).

Pruning the query to access ⇐ 1024 raster files permits the query to complete. This does have the assumption that the connection is new and there-fore the process has no open out-db raster files.

Googling (can't find link) indicated that GDAL limits the number of open files to 1024. Just in case this was not true, I did alter the soft and hard nofile limits of the system for the postgres user with no luck.

The immediate solution is to not use GDALOpenShared for opening out-db rasters but instead GDALOpen. A better solution is keep track of the out-db rasters opened in the process and then close those as needed.

[Bborie Park]

Fixed in trunk as of r12185. Fixed in -2.1 as of r12186. Fixed in -2.0 as of r12187.

Fixed by replacing GDALOpenShared with GDALOpen. Need to explore how to get GDALOpenShared back by keeping a cache of open GDAL datasets and when the cache count gets close to the max permitted (~1000), close out the some of the oldest datasets.