Failure reading BSB image

[avar]

I have a collection of ​BSB files in .NOS/.GEO format. Some of which gdal_translate can decode but others fail:

avar@aoeu:~/src/dl/gdal-1.6.0$ for i in IS1612*NOS; do echo $i && ./apps/gdal_translate -of PNG $i $i.png; done
IS1612_1.NOS
Input file size is 5677, 7272
0...10...20...30...40...50...60...70...80...90...100 - done.
IS1612_4.NOS
Input file size is 6606, 8542
0...10...20...30...Warning 1: Got 1010 pixels when looking for 6606 pixels.
ERROR 1: IReadBlock failed at X offset 0, Y offset 3390
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3390

Attached is a tarball with the two files in question. IS1612_1.NOS is decoded successfully while IS1612_4.NOS triggers an error.

[avar]

I can't attach files over 1MB. Here's a tarball with the files: ​http://u.nix.is/gdal/IS1612.tar.gz

[Even Rouault]

I also reproduce the problem with IS1612_4.NOS. I've also tried bsb2tif from ​libbsb and it shows the same behaviour as the GDAL BSB driver (libbsb error is "Warning: Short row for row 3391 written=1010 width=6606"). Provided that one code has not been inspired by the other one, it could be a sign that the file is corrupted, or that both codes have missed some part of the spec ?

[avar]

I'm pretty sure that they aren't corrupt and were readable by a proprietary program. But I'll seek confirmation of this.

[avar]

Replying to avar:

I'm pretty sure that they aren't corrupt and were readable by a proprietary program. But I'll seek confirmation of this.

I've confirmed that the charts are openable in ​GPSNavX which is the program originally used to open them.

[warmerdam]

Based on Even's report, I'm inclined to agree it is corrupt. However, given that GPSNavX can read it, we should likely consider whether we can work around the problem.

Chaitanya, could you examine the specifics of what is going wrong with the line in question and whether there is an obvious recovery strategy?

BSB files normally contain a table of offsets to scanlines at the end of the file. We normally ignore this and try to read the scanlines in order so that things will work even if the offsets table is somehow corrupt. I suspect we would need to read the offsets table to avoid this problem but I'm uncertain what problems that might lead to.

[chaitanya]

Replying to avar:

I've confirmed that the charts are openable in ​GPSNavX which is the program originally used to open them.

avar, Will you be able to provide the IS1612_4.NOS data in some other format (lossless and at the same dimensions) for testing my code.

[avar]

Replying to chaitanya:

Replying to avar:

I've confirmed that the charts are openable in ​GPSNavX which is the program originally used to open them.

avar, Will you be able to provide the IS1612_4.NOS data in some other format (lossless and at the same dimensions) for testing my code.

I don't have access (or know of) any other program than gdal or libbsb that could convert them, so no. I can't provide them in a format that has already been decoded from the .NOS/.GEO files since I have no way to decode them.

What I can provide you with is:

• A collection (~100) files that can be decoded my gdal/libbsb from the same map set
• A complete set of the files that can't be decoded (~8 files)
• Confirmation that they can be opened in GPSNavX

I contacted the GPSNavX devs on January 19th asking them if they could provide decoded copies in some other format but haven't yet heard back.

[avar]

Replying to avar:

Replying to avar:

I'm pretty sure that they aren't corrupt and were readable by a proprietary program. But I'll seek confirmation of this.

I've confirmed that the charts are openable in ​GPSNavX which is the program originally used to open them.

FWIW I've discovered that these files were not originally intended to be openable by GPSNavX but by ​Nobeltec ​Visual Navigation Suite 4.0. Users of GPSNavX later found that they could simply drop the .NOS/.GEO files into GPSNavX if they wanted to open them on OSX.

So these files aren't just readable due to the idiosyncrasies of one isolated program but are openable by at least two distinct commercial mapping products that read BSB files.

[chaitanya]

Code made more tolerant of corruption in lines or offset index of the file.

[avar]

With the attached patch​ by chaitanya I can decode all of the images in question:

avar@aoeu:~/Documents/Kort/Charts/corrupt-bsb$ for i in *NOS; do ~/src/gdal/apps/gdal_translate -of PNG $i $i.PNG; done
Input file size is 6606, 8542
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6559, 8588
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 5616, 7328
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6457, 8590
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 5599, 7312
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6592, 8590
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6618, 8642
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 5599, 7271
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 5634, 7317
0...10...20...30...40...50...60...70...80...90...100 - done.

But with [16174] only 4/9 of them:

avar@aoeu:~/Documents/Kort/Charts/corrupt-bsb$ for i in *NOS; do ~/src/gdal/apps/gdal_translate -of PNG $i $i.PNG; done
Input file size is 6606, 8542
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6559, 8588
0...10...20...30...Warning 1: Got 6521 pixels when looking for 6559 pixels
ERROR 1: IReadBlock failed at X offset 0, Y offset 3293
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 3293
Input file size is 5616, 7328
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 6457, 8590
0...10...20...30...40..Warning 1: Got 6079 pixels when looking for 6457 pixels
ERROR 1: IReadBlock failed at X offset 0, Y offset 4060
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4060
Input file size is 5599, 7312
0.Warning 1: Got 5568 pixels when looking for 5599 pixels
ERROR 1: IReadBlock failed at X offset 0, Y offset 242
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 242
Input file size is 6592, 8590
0...10...20...30...40...50.Warning 1: Got 6580 pixels when looking for 6592 pixels
ERROR 1: IReadBlock failed at X offset 0, Y offset 4623
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4623
Input file size is 6618, 8642
0...10...20...30...40...50...60...70...80...90...100 - done.
Input file size is 5599, 7271
0...10...20.Warning 1: Got 5594 pixels when looking for 5599 pixels
ERROR 1: IReadBlock failed at X offset 0, Y offset 1655
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 1655
Input file size is 5634, 7317
0...10...20...30...40...50...60...70...80...90...100 - done.

[Even Rouault]

In r16174 (and additional fix in r16176 for above issues reported by avar):

* Fix several issues with BSB files reading (#2782)
- Enable to read IS1612_4.NOS product which has a \0 character in the middle of row data (derived from Chaitanya's patch)
- Allow reading the index table at the end of a KAP file, that point to the start of each row, when it's present and valid (derived from Chaitanya's patch)
- Make BSBGetc set an error flag when I/O error (like file truncation) to avoid endless loops when reading corrupted/hostile files
- Avoid crashes with invalid color table index
- Avoid reading images with negative dimensions
- Enable to read some NOAA charts, like 1115A_1.KAP, that declare a height which is one row more than available data

I've modified chaitanya's patch to enable reading BSB images without offset table, like the one in bsb.py in autotest. The strange thing was that VSIFSeekL always succeed, even if the offset is not in the range of the file, so we cannot rely on that. I've modified the tests to check that the offset table is valid by checking the offsets are within the file, and that at the beginning of each scanline, we find the expected scanline number.

r16174 and r16176 backported in branches/1.6 in r16177

[avar]

Replying to avar:

With the attached patch​ by chaitanya I can decode all of the images in question: [...] But with [16174] only 4/9 of them:

[...]

Replying to rouault:

In r16174 (and additional fix in r16176 for above issues reported by avar):

I can confirm that [16176] fixed this in trunk.

[Even Rouault]

Partial backport in branches/1.5 in r16178 (the reading of the offset table is not backported, but we don't need it to read IS1612_4.NOS successfully).

Tests added in r16179

[chaitanya]

rouault,

The latest fix could make the code to read past the current scanline. Please apply this patch to make it stop there and fill the remaining space with zeros.

Index: bsb_read.c

===================================================================

--- bsb_read.c	(revision 16179)

+++ bsb_read.c	(working copy)

@@ -818,8 +818,16 @@

     while ( iPixel < psInfo->nXSize &&
             (nScanline == psInfo->nYSize-1 ||
              psInfo->panLineOffset[nScanline+1] == -1 ||
-             VSIFTellL( fp ) - psInfo->nBufferSize + psInfo->nBufferOffset <= psInfo->panLineOffset[nScanline+1]) );
+             VSIFTellL( fp ) - psInfo->nBufferSize + psInfo->nBufferOffset < psInfo->panLineOffset[nScanline+1]) );
+/* -------------------------------------------------------------------- */
+/*      If the line buffer is not filled after reading the line in the  */
+/*      file upto the next line offset, just fill it with zeros.        */
+/*      (The last pixel value from nPixValue could be a better value?)  */
+/* -------------------------------------------------------------------- */
+    while( iPixel < psInfo->nXSize )
+        pabyScanlineBuf[iPixel++] = 0;
 
+
 /* -------------------------------------------------------------------- */
 /*      Remember the start of the next line.                            */
 /*      But only if it is not already known.                            */

[avar]

A regression has been introduced since [16174], I can't decode ​this file with either current trunk ([16179]) or with the patch pasted in comment 13 by chaitanya.

[avar]

Replying to avar:

A regression has been introduced since [16174], I can't decode ​this file with either current trunk ([16179]) or with the patch pasted in comment 13 by chaitanya.

Actually it seems like I had two versions of that file around, the one I referenced in the above post appears to be truncated, gdal can still read the original file.

So no regression then.

[Even Rouault]

chaitanya,

I've applied your latest patch that was effectively necessary.

I've also added some logic that helps reading short rows when the offset table is not available. In that case, it doesn't get always 100% success, but it helps reading a few images we could not before, and shouldn't cause regressions.

Fixed in trunk in r16180, in branches/1.6 in r16181 and in branches/1.5 in r16182

[Even Rouault]

Milestone 1.5.5 deleted