Opened 2 months ago

Closed 2 months ago

Last modified 4 weeks ago

#7081 closed defect (fixed)

JP2KAK: Abort in unit tests (v7.10.2 / 7.A.2)

Reported by: Robert Coup Owned by: Robert Coup
Priority: normal Milestone:
Component: default Version: 2.2.2
Severity: normal Keywords: JP2KAK
Cc: Even Rouault

Description

Getting the following unit test fail with Kakadu 7.A.2 (GDAL 2.2.x@9647f4b4, Ubuntu 14.04 amd64).

root@c98fa7b8dc25:/source/autotest/gdrivers# CPL_TIMESTAMP=ON CPL_DEBUG=ON python jp2kak.py
...
  TEST: jp2kak_3 ... Tue Oct 10 15:06:16 2017: JP2KAK: Using 1 threads.
Tue Oct 10 15:06:16 2017: JP2KAK: Cuse_precincts=1, PreferNonPersistentReads=0
Tue Oct 10 15:06:16 2017: JP2KAK: order=PCRL
Tue Oct 10 15:06:16 2017: JP2KAK: nResCount=6
Tue Oct 10 15:06:16 2017: GDALJP2Metadata: Got projection from GeoJP2 (geotiff) box (0): PROJCS["NAD27 / UTM zone 11N",GEOGCS["NAD27",DATUM["North_American_Datum_1927",SPHEROID["Clarke 1866",6378206.4,294.9786982138982,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-117],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","26711"]]
Tue Oct 10 15:06:16 2017: GDAL: GDALOpen(data/byte.jp2, this=0x138e770) succeeds as JP2KAK.
Tue Oct 10 15:06:16 2017: JP2KAK: IReadBlock(0,0) on band 1.
Tue Oct 10 15:06:16 2017: JP2KAK: DirectRasterIO() for 0,0,100,100 -> 100x100 (no intermediate)
Tue Oct 10 15:06:16 2017: GDAL: QuietDelete(tmp/byte.jp2.tst) invoking Delete()
Tue Oct 10 15:06:16 2017: JP2KAK: Cannot read JP2 boxes
Tue Oct 10 15:06:16 2017: JP2KAK: Final JPEG2000 Tile Size is 100P x 100L.
Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(Corder=PCRL)
Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(Cprecincts={512,512},{256,512},{128,512},{64,512},{32,512},{16,512},{8,512},{4,512},{2,512})
Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(ORGgen_plt=yes)
terminate called after throwing an instance of 'kdu_cpl_error_message::JP2KAKException'
Aborted

in GDB:

  TEST: jp2kak_3 ... [New Thread 0x7fffe525f700 (LWP 31)]
Tue Oct 10 18:37:33 2017: JP2KAK: Using 1 threads.
Tue Oct 10 18:37:33 2017: JP2KAK: Cuse_precincts=1, PreferNonPersistentReads=0
Tue Oct 10 18:37:33 2017: JP2KAK: order=PCRL
Tue Oct 10 18:37:33 2017: JP2KAK: nResCount=6
Tue Oct 10 18:37:33 2017: GDALJP2Metadata: Got projection from GeoJP2 (geotiff) box (0): PROJCS["NAD27 / UTM zone 11N",GEOGCS["NAD27",DATUM["North_American_Datum_1927",SPHEROID["Clarke 1866",6378206.4,294.9786982138982,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-117],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","26711"]]
Tue Oct 10 18:37:33 2017: GDAL: GDALOpen(data/byte.jp2, this=0xc29ab0) succeeds as JP2KAK.
Tue Oct 10 18:37:33 2017: JP2KAK: IReadBlock(0,0) on band 1.
Tue Oct 10 18:37:33 2017: JP2KAK: DirectRasterIO() for 0,0,100,100 -> 100x100 (no intermediate)
Tue Oct 10 18:37:33 2017: GDAL: QuietDelete(tmp/byte.jp2.tst) invoking Delete()
Tue Oct 10 18:37:33 2017: JP2KAK: Cannot read JP2 boxes
Tue Oct 10 18:37:33 2017: JP2KAK: Final JPEG2000 Tile Size is 100P x 100L.
Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(Corder=PCRL)
Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(Cprecincts={512,512},{256,512},{128,512},{64,512},{32,512},{16,512},{8,512},{4,512},{2,512})
Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(ORGgen_plt=yes)
terminate called after throwing an instance of 'kdu_cpl_error_message::JP2KAKException'

Program received signal SIGABRT, Aborted.
0x00007ffff7829c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) thread apply all bt
Thread 6 (Thread 0x7fffe525f700 (LWP 31)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00007ffff635f804 in kdu_core::kdu_thread_entity::process_jobs(kdu_core::kdu_thread_entity_condition*) () from /usr/lib/libgdal.so.20
#2  0x00007ffff635ed7f in kdu_core::worker_startproc(void*) () from /usr/lib/libgdal.so.20
#3  0x00007ffff635ebc9 in kdu_core::kd_thread_create_entry_point(void*) () from /usr/lib/libgdal.so.20
#4  0x00007ffff7bc4184 in start_thread (arg=0x7fffe525f700) at pthread_create.c:312
#5  0x00007ffff78f0ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fe6740 (LWP 23)):
#0  0x00007ffff7829c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff782d028 in __GI_abort () at abort.c:89
#2  0x00007ffff48d8535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff48d66d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff48d6703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff48d6922 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff5888901 in kdu_cpl_error_message::flush (this=0xc5b430, end_of_message=true) at jp2kakdataset.h:219
#7  0x00007ffff6313596 in kdu_core::kdu_error::~kdu_error() () from /usr/lib/libgdal.so.20
#8  0x00007ffff6235386 in kdu_supp::jp2_target::open_codestream(bool) () from /usr/lib/libgdal.so.20
#9  0x00007ffff5885f35 in JP2KAKCreateCopy (pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", poSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0,
    pfnProgress=0x7ffff61c2f10 <GDALDummyProgress(double, char const*, void*)>, pProgressData=0xc47ab0) at jp2kakdataset.cpp:2542
#10 0x00007ffff5bbc6e7 in GDALDriver::CreateCopy (this=0xabdf30, pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", poSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0,
    pfnProgress=0x7ffff61c2f10 <GDALDummyProgress(double, char const*, void*)>, pProgressData=0xc47ab0) at gdaldriver.cpp:852
#11 0x00007ffff5bbc88f in GDALCreateCopy (hDriver=0xabdf30, pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", hSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0, pfnProgress=0x0, pProgressData=0xc47ab0)
    at gdaldriver.cpp:899
#12 0x00007ffff6bd35d9 in GDALDriverShadow_CreateCopy (self=0xabdf30, utf8_path=0x7fffe52cfa9c "tmp/byte.jp2.tst", src=0xc29ab0, strict=0, options=0xc794e0, callback=0x0, callback_data=0xc47ab0)
    at extensions/gdal_wrap.cpp:3966
#13 0x00007ffff6be3c19 in _wrap_Driver_CreateCopy (args=0x7fffe58ab230, kwargs=0x7fffe5267050) at extensions/gdal_wrap.cpp:10406
#14 0x000000000052714b in PyEval_EvalFrameEx ()
#15 0x0000000000555551 in PyEval_EvalCodeEx ()
#16 0x0000000000525560 in PyEval_EvalFrameEx ()
#17 0x0000000000555551 in PyEval_EvalCodeEx ()
#18 0x0000000000524338 in PyEval_EvalFrameEx ()
#19 0x00000000005247ea in PyEval_EvalFrameEx ()
#20 0x00000000005247ea in PyEval_EvalFrameEx ()
#21 0x00000000005247ea in PyEval_EvalFrameEx ()
#22 0x0000000000567d14 in ?? ()
#23 0x0000000000465bf4 in PyRun_FileExFlags ()
#24 0x000000000046612d in PyRun_SimpleFileExFlags ()
#25 0x0000000000466d92 in Py_Main ()
#26 0x00007ffff7814f45 in __libc_start_main (main=0x466e50 <main>, argc=2, argv=0x7fffffffe718, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe708)
    at libc-start.c:287
#27 0x0000000000577c2e in _start ()

Attachments (4)

7081-1.patch (2.8 KB) - added by Robert Coup 2 months ago.
791-prefix.txt (14.1 KB) - added by Robert Coup 2 months ago.
Kakadu 7.9.1 before the patch
791-postfix.txt (14.1 KB) - added by Robert Coup 2 months ago.
Kakadu 7.9.1 after the patch
7A2-postfix.txt (14.1 KB) - added by Robert Coup 2 months ago.
Kakadu 7.A.2 after the patch

Download all attachments as: .zip

Change History (14)

comment:1 Changed 2 months ago by Robert Coup

Cc: Even Rouault added
Owner: changed from warmerdam to Robert Coup
Status: newassigned

aha, debugger reveals the following. I think since the GDAL JP2KAKException is fatal, the Kakadu errors should probably be logged through CPL as fatal too? But if CPL_LOG_ERRORS=ON is set, the following is logged as it dies:

CPLError: Error in Kakadu File Format Support:
You may not call `jp2_target::open_codestream' until after you have called `jp2_target::open' and `jp2_target::write_header'.

comment:2 Changed 2 months ago by Even Rouault

I think since the GDAL JP2KAKException is fatal, the Kakadu errors should probably be logged through CPL as fatal too?

The kdu_cpl_error_message class "catches" Kakadu errors, emit them as CPLError(CE_Failure, ) and re-throw them as JP2KAKException. So the crash is due to a JP2KAKException not handled by a try {} catch {} block. Using CPLError(CE_Fatal, ...) causes the GDAL process to be halted, which is rarely desirable. But anyway, this is just the symptom of the issue. The core issue is some apparent misuse of the Kakadu API.

Changed 2 months ago by Robert Coup

Attachment: 7081-1.patch added

comment:3 Changed 2 months ago by Robert Coup

7081-1.patch swaps to using the jp2_family_tgt object rather than the jp2_output_box object for adding the metadata boxes. This seems to match what kdu_compress.cpp in 7.A.2 is doing more closely. But I don't actually understand JP2 enough to say whether it produces effectively the same output or not? (ie. is the metadata now attached to the wrong place). Is there an easy way to test that?

comment:4 Changed 2 months ago by Even Rouault

You can try comparing the output of "python swig/python/samples/dump_jp2.py output.jp2"

  • with output.jp2 generated by the JP2KAK driver before your changes with Kakadu < 7.A
  • with output.jp2 generated by the JP2KAK driver after your changes with Kakadu < 7.A
  • with output.jp2 generated by the JP2KAK driver after your changes with Kakadu >= 7.A

They should be identical (excluding the COM marker that contains the Kakadu version)

comment:5 Changed 2 months ago by Robert Coup

7.9.1 pre- & post-fix output match 7.10.2 differs slightly, mostly just the box offsets and I think some unrelated changes in Kakadu?

$ diff -u 791-prefix.txt 7A2-postfix.txt
--- 791-prefix.txt	2017-10-16 16:18:53.000000000 +0100
+++ 7A2-postfix.txt	2017-10-16 15:49:37.000000000 +0100
@@ -1,4 +1,4 @@
-<JP2File filename="791-prefix.jp2">
+<JP2File filename="7A2-postfix.jp2">
   <JP2Box name="jP  " box_offset="0" box_length="12" data_offset="8" data_length="4">
     <BinaryContent>0D0A870A</BinaryContent>
   </JP2Box>
@@ -106,7 +106,7 @@
       </VRTDataset>
     </DecodedGeoTIFF>
   </JP2Box>
-  <JP2Box name="jp2c" box_offset="2417" box_length="9886" data_offset="2425" data_length="9878">
+  <JP2Box name="jp2c" box_offset="2417" box_length="9883" data_offset="2425" data_length="9875">
     <JP2KCodeStream>
       <Marker name="SOC" offset="2425" length="2" />
       <Marker name="SIZ" offset="2427" length="43">
@@ -142,36 +142,36 @@
         <Field name="SPcod_Precincts5" type="uint8" description="PPx=9 PPy=9: 512x512">153</Field>
       </Marker>
       <Marker name="QCD" offset="2490" length="21" />
-      <Marker name="COM" offset="2511" length="19">
+      <Marker name="COM" offset="2511" length="20">
         <Field name="Rcom" type="uint16" description="LATIN1">1</Field>
-        <Field name="COM" type="string" size="13">Kakadu-v7.9.1</Field>
+        <Field name="COM" type="string" size="14">Kakadu-v7.10.2</Field>
       </Marker>
-      <Marker name="COM" offset="2530" length="281">
+      <Marker name="COM" offset="2531" length="281">
         <Field name="Rcom" type="uint16" description="LATIN1">1</Field>
         <Field name="COM" type="string" size="275">Kdu-Layer-Info: log_2{Delta-D(squared-error)/Delta-L(bytes)}, L(bytes)
- -14.1,  4.2e+02
- -15.3,  4.4e+02
- -16.5,  4.7e+02
- -17.7,  5.8e+02
- -18.8,  8.2e+02
- -20.0,  1.6e+03
- -21.2,  2.5e+03
- -22.3,  3.5e+03
- -23.5,  4.1e+03
- -24.7,  5.1e+03
+ -13.9,  4.2e+02
+ -15.1,  4.4e+02
+ -16.2,  4.7e+02
+ -17.4,  5.6e+02
+ -18.6,  7.0e+02
+ -19.7,  1.3e+03
+ -20.9,  2.5e+03
+ -22.1,  3.4e+03
+ -23.3,  4.1e+03
+ -24.4,  4.9e+03
  -26.9,  6.8e+03
 -192.0,  9.8e+03
 </Field>
       </Marker>
-      <Marker name="SOT" offset="2811" length="12">
+      <Marker name="SOT" offset="2812" length="12">
         <Field name="Isot" type="uint16">0</Field>
-        <Field name="Psot" type="uint32">9490</Field>
+        <Field name="Psot" type="uint32">9486</Field>
         <Field name="TPsot" type="uint8">0</Field>
         <Field name="TNsot" type="uint8">1</Field>
       </Marker>
-      <Marker name="PLT" offset="2823" length="91" />
-      <Marker name="SOD" offset="2914" length="9387" />
-      <Marker name="EOC" offset="12301" length="2" />
+      <Marker name="PLT" offset="2824" length="90" />
+      <Marker name="SOD" offset="2914" length="9384" />
+      <Marker name="EOC" offset="12298" length="2" />
     </JP2KCodeStream>
   </JP2Box>
 </JP2File>

Changed 2 months ago by Robert Coup

Attachment: 791-prefix.txt added

Kakadu 7.9.1 before the patch

Changed 2 months ago by Robert Coup

Attachment: 791-postfix.txt added

Kakadu 7.9.1 after the patch

Changed 2 months ago by Robert Coup

Attachment: 7A2-postfix.txt added

Kakadu 7.A.2 after the patch

comment:6 Changed 2 months ago by Robert Coup

Commands run to generate those were:

CPL_DEBUG=ON GDAL_SKIP=JP2OpenJPEG gdal_translate -of JP2KAK -co QUALITY=100 ../autotest/gdrivers/data/byte.jp2 7A2-postfix.jp2
python2 ../gdal/swig/python/samples/dump_jp2.py 7A2-postfix.jp2  > 7A2-postfix.txt

That matches the failing unit test jp2kak_3()

comment:7 Changed 2 months ago by Even Rouault

Resolution: fixed
Status: assignedclosed

In 40451:

JP2KAK: fix crashing bug when writing JP2 boxes with Kakadu 7.10 (patch by Robert Coup, fixes #7081)

comment:8 Changed 2 months ago by Robert Coup

Found another one, occurs in gcore/misc.py:misc_6, but not on every platform/build... Patch reverts to the previous behaviour of not calling jp2_out.set_target_size().

CPLError: Error in Kakadu File Format Support:
Attempting to close an output JP2 box whose length was defined ahead of time, having written less bytes than indicated by that length value.
diff --git a/gdal/frmts/jp2kak/jp2kakdataset.cpp b/gdal/frmts/jp2kak/jp2kakdataset.cpp
index 19e07a9541..70dd6bae33 100644
--- a/gdal/frmts/jp2kak/jp2kakdataset.cpp
+++ b/gdal/frmts/jp2kak/jp2kakdataset.cpp
@@ -1762,13 +1762,10 @@ static void JP2KAKWriteBox( jp2_family_tgt *jp2_family, GDALJP2Box *poBox )
     memcpy(&nBoxType, poBox->GetType(), sizeof(nBoxType));
     CPL_MSBPTR32(&nBoxType);

-    int length = static_cast<int>(poBox->GetDataLength());
-
     // Write to a box on the JP2 file.
     jp2_out.open(jp2_family, nBoxType);
-    jp2_out.set_target_size(length);
     jp2_out.write(const_cast<kdu_byte *>(poBox->GetWritableData()),
-                   length);
+                  static_cast<int>(poBox->GetDataLength()));
     jp2_out.close();

     delete poBox;
Last edited 2 months ago by Robert Coup (previous) (diff)

comment:9 Changed 2 months ago by Even Rouault

I've applied an alternated patch for this issue: r40510

comment:10 Changed 4 weeks ago by Even Rouault

In 40766:

JP2KAK: fix crashing bug when writing JP2 boxes with Kakadu 7.10 (patch by Robert Coup + myself, backport trunk r40451+r40510, fixes #7081)

Note: See TracTickets for help on using tickets.