Opened 6 years ago

Closed 6 years ago

#2607 closed defect (fixed)

[raster] Unable to open more than 1024 out-db files in a single connection

Reported by: Bborie Park Owned by: Bborie Park
Priority: critical Milestone: PostGIS 2.0.6
Component: raster Version: 2.0.x
Keywords: history Cc:

Description

On a large query spanning every day of many years, I get the following error:

rt_band_load_offline_data: Cannot open offline raster: SOME_RASTER_FILE

In the PostgreSQL logs...

ERROR 4: `SOME_RASTER_FILE' not recognised as a supported file format.

I know these files are all valid rasters (as per GDAL).

Pruning the query to access <= 1024 raster files permits the query to complete. This does have the assumption that the connection is new and there-fore the process has no open out-db raster files.

Googling (can't find link) indicated that GDAL limits the number of open files to 1024. Just in case this was not true, I did alter the soft and hard nofile limits of the system for the postgres user with no luck.

The immediate solution is to not use GDALOpenShared for opening out-db rasters but instead GDALOpen. A better solution is keep track of the out-db rasters opened in the process and then close those as needed.

Change History (2)

comment:1 Changed 6 years ago by Bborie Park

Status: newassigned

comment:2 Changed 6 years ago by Bborie Park

Keywords: history added
Resolution: fixed
Status: assignedclosed

Fixed in trunk as of r12185. Fixed in -2.1 as of r12186. Fixed in -2.0 as of r12187.

Fixed by replacing GDALOpenShared with GDALOpen. Need to explore how to get GDALOpenShared back by keeping a cache of open GDAL datasets and when the cache count gets close to the max permitted (~1000), close out the some of the oldest datasets.

Note: See TracTickets for help on using tickets.