#2782 closed defect (fixed)
Failure reading BSB image
Reported by: | avar | Owned by: | chaitanya |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | GDAL_Raster | Version: | |
Severity: | normal | Keywords: | BSB |
Cc: | warmerdam, avar |
Description
I have a collection of BSB files in .NOS/.GEO format. Some of which gdal_translate can decode but others fail:
avar@aoeu:~/src/dl/gdal-1.6.0$ for i in IS1612*NOS; do echo $i && ./apps/gdal_translate -of PNG $i $i.png; done IS1612_1.NOS Input file size is 5677, 7272 0...10...20...30...40...50...60...70...80...90...100 - done. IS1612_4.NOS Input file size is 6606, 8542 0...10...20...30...Warning 1: Got 1010 pixels when looking for 6606 pixels. ERROR 1: IReadBlock failed at X offset 0, Y offset 3390 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3390
Attached is a tarball with the two files in question. IS1612_1.NOS is decoded successfully while IS1612_4.NOS triggers an error.
Attachments (1)
Change History (18)
comment:1 by , 16 years ago
comment:2 by , 16 years ago
I also reproduce the problem with IS1612_4.NOS. I've also tried bsb2tif from libbsb and it shows the same behaviour as the GDAL BSB driver (libbsb error is "Warning: Short row for row 3391 written=1010 width=6606"). Provided that one code has not been inspired by the other one, it could be a sign that the file is corrupted, or that both codes have missed some part of the spec ?
follow-up: 4 comment:3 by , 16 years ago
I'm pretty sure that they aren't corrupt and were readable by a proprietary program. But I'll seek confirmation of this.
follow-ups: 6 8 comment:4 by , 16 years ago
comment:5 by , 15 years ago
Cc: | added |
---|---|
Component: | default → GDAL_Raster |
Keywords: | BSB added |
Owner: | changed from | to
Summary: | BSB (Maptech/NOAA BSB Nautical Chart Format) driver fails for gdal_translate with IReadBlock/GetBlockRef error → Failure reading BSB image |
Based on Even's report, I'm inclined to agree it is corrupt. However, given that GPSNavX can read it, we should likely consider whether we can work around the problem.
Chaitanya, could you examine the specifics of what is going wrong with the line in question and whether there is an obvious recovery strategy?
BSB files normally contain a table of offsets to scanlines at the end of the file. We normally ignore this and try to read the scanlines in order so that things will work even if the offsets table is somehow corrupt. I suspect we would need to read the offsets table to avoid this problem but I'm uncertain what problems that might lead to.
follow-up: 7 comment:6 by , 15 years ago
Cc: | added |
---|
comment:7 by , 15 years ago
Replying to chaitanya:
Replying to avar:
I've confirmed that the charts are openable in GPSNavX which is the program originally used to open them.
avar, Will you be able to provide the IS1612_4.NOS data in some other format (lossless and at the same dimensions) for testing my code.
I don't have access (or know of) any other program than gdal or libbsb that could convert them, so no. I can't provide them in a format that has already been decoded from the .NOS/.GEO files since I have no way to decode them.
What I can provide you with is:
- A collection (~100) files that can be decoded my gdal/libbsb from the same map set
- A complete set of the files that can't be decoded (~8 files)
- Confirmation that they can be opened in GPSNavX
I contacted the GPSNavX devs on January 19th asking them if they could provide decoded copies in some other format but haven't yet heard back.
comment:8 by , 15 years ago
Replying to avar:
Replying to avar:
I'm pretty sure that they aren't corrupt and were readable by a proprietary program. But I'll seek confirmation of this.
I've confirmed that the charts are openable in GPSNavX which is the program originally used to open them.
FWIW I've discovered that these files were not originally intended to be openable by GPSNavX but by Nobeltec Visual Navigation Suite 4.0. Users of GPSNavX later found that they could simply drop the .NOS/.GEO files into GPSNavX if they wanted to open them on OSX.
So these files aren't just readable due to the idiosyncrasies of one isolated program but are openable by at least two distinct commercial mapping products that read BSB files.
by , 15 years ago
Attachment: | bsb_read.c.patch added |
---|
Code made more tolerant of corruption in lines or offset index of the file.
follow-up: 11 comment:9 by , 15 years ago
With the attached patch by chaitanya I can decode all of the images in question:
avar@aoeu:~/Documents/Kort/Charts/corrupt-bsb$ for i in *NOS; do ~/src/gdal/apps/gdal_translate -of PNG $i $i.PNG; done Input file size is 6606, 8542 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6559, 8588 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 5616, 7328 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6457, 8590 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 5599, 7312 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6592, 8590 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6618, 8642 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 5599, 7271 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 5634, 7317 0...10...20...30...40...50...60...70...80...90...100 - done.
But with [16174] only 4/9 of them:
avar@aoeu:~/Documents/Kort/Charts/corrupt-bsb$ for i in *NOS; do ~/src/gdal/apps/gdal_translate -of PNG $i $i.PNG; done Input file size is 6606, 8542 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6559, 8588 0...10...20...30...Warning 1: Got 6521 pixels when looking for 6559 pixels ERROR 1: IReadBlock failed at X offset 0, Y offset 3293 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3293 Input file size is 5616, 7328 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 6457, 8590 0...10...20...30...40..Warning 1: Got 6079 pixels when looking for 6457 pixels ERROR 1: IReadBlock failed at X offset 0, Y offset 4060 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4060 Input file size is 5599, 7312 0.Warning 1: Got 5568 pixels when looking for 5599 pixels ERROR 1: IReadBlock failed at X offset 0, Y offset 242 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 242 Input file size is 6592, 8590 0...10...20...30...40...50.Warning 1: Got 6580 pixels when looking for 6592 pixels ERROR 1: IReadBlock failed at X offset 0, Y offset 4623 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4623 Input file size is 6618, 8642 0...10...20...30...40...50...60...70...80...90...100 - done. Input file size is 5599, 7271 0...10...20.Warning 1: Got 5594 pixels when looking for 5599 pixels ERROR 1: IReadBlock failed at X offset 0, Y offset 1655 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 1655 Input file size is 5634, 7317 0...10...20...30...40...50...60...70...80...90...100 - done.
comment:10 by , 15 years ago
In r16174 (and additional fix in r16176 for above issues reported by avar):
* Fix several issues with BSB files reading (#2782) - Enable to read IS1612_4.NOS product which has a \0 character in the middle of row data (derived from Chaitanya's patch) - Allow reading the index table at the end of a KAP file, that point to the start of each row, when it's present and valid (derived from Chaitanya's patch) - Make BSBGetc set an error flag when I/O error (like file truncation) to avoid endless loops when reading corrupted/hostile files - Avoid crashes with invalid color table index - Avoid reading images with negative dimensions - Enable to read some NOAA charts, like 1115A_1.KAP, that declare a height which is one row more than available data
I've modified chaitanya's patch to enable reading BSB images without offset table, like the one in bsb.py in autotest. The strange thing was that VSIFSeekL always succeed, even if the offset is not in the range of the file, so we cannot rely on that. I've modified the tests to check that the offset table is valid by checking the offsets are within the file, and that at the beginning of each scanline, we find the expected scanline number.
comment:11 by , 15 years ago
comment:12 by , 15 years ago
Milestone: | → 1.5.5 |
---|---|
Resolution: | → fixed |
Status: | new → closed |
Version: | 1.6.0 |
comment:13 by , 15 years ago
Resolution: | fixed |
---|---|
Status: | closed → reopened |
rouault,
The latest fix could make the code to read past the current scanline. Please apply this patch to make it stop there and fill the remaining space with zeros.
Index: bsb_read.c =================================================================== --- bsb_read.c (revision 16179) +++ bsb_read.c (working copy) @@ -818,8 +818,16 @@ while ( iPixel < psInfo->nXSize && (nScanline == psInfo->nYSize-1 || psInfo->panLineOffset[nScanline+1] == -1 || - VSIFTellL( fp ) - psInfo->nBufferSize + psInfo->nBufferOffset <= psInfo->panLineOffset[nScanline+1]) ); + VSIFTellL( fp ) - psInfo->nBufferSize + psInfo->nBufferOffset < psInfo->panLineOffset[nScanline+1]) ); +/* -------------------------------------------------------------------- */ +/* If the line buffer is not filled after reading the line in the */ +/* file upto the next line offset, just fill it with zeros. */ +/* (The last pixel value from nPixValue could be a better value?) */ +/* -------------------------------------------------------------------- */ + while( iPixel < psInfo->nXSize ) + pabyScanlineBuf[iPixel++] = 0; + /* -------------------------------------------------------------------- */ /* Remember the start of the next line. */ /* But only if it is not already known. */
follow-up: 15 comment:14 by , 15 years ago
A regression has been introduced since [16174], I can't decode this file with either current trunk ([16179]) or with the patch pasted in comment 13 by chaitanya.
comment:15 by , 15 years ago
Replying to avar:
A regression has been introduced since [16174], I can't decode this file with either current trunk ([16179]) or with the patch pasted in comment 13 by chaitanya.
Actually it seems like I had two versions of that file around, the one I referenced in the above post appears to be truncated, gdal can still read the original file.
So no regression then.
comment:16 by , 15 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
chaitanya,
I've applied your latest patch that was effectively necessary.
I've also added some logic that helps reading short rows when the offset table is not available. In that case, it doesn't get always 100% success, but it helps reading a few images we could not before, and shouldn't cause regressions.
Fixed in trunk in r16180, in branches/1.6 in r16181 and in branches/1.5 in r16182
I can't attach files over 1MB. Here's a tarball with the files: http://u.nix.is/gdal/IS1612.tar.gz