#7081 closed defect (fixed)
JP2KAK: Abort in unit tests (v7.10.2 / 7.A.2)
Reported by: | Robert Coup | Owned by: | Robert Coup |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | default | Version: | 2.2.2 |
Severity: | normal | Keywords: | JP2KAK |
Cc: | Even Rouault |
Description
Getting the following unit test fail with Kakadu 7.A.2 (GDAL 2.2.x@9647f4b4
, Ubuntu 14.04 amd64).
root@c98fa7b8dc25:/source/autotest/gdrivers# CPL_TIMESTAMP=ON CPL_DEBUG=ON python jp2kak.py ... TEST: jp2kak_3 ... Tue Oct 10 15:06:16 2017: JP2KAK: Using 1 threads. Tue Oct 10 15:06:16 2017: JP2KAK: Cuse_precincts=1, PreferNonPersistentReads=0 Tue Oct 10 15:06:16 2017: JP2KAK: order=PCRL Tue Oct 10 15:06:16 2017: JP2KAK: nResCount=6 Tue Oct 10 15:06:16 2017: GDALJP2Metadata: Got projection from GeoJP2 (geotiff) box (0): PROJCS["NAD27 / UTM zone 11N",GEOGCS["NAD27",DATUM["North_American_Datum_1927",SPHEROID["Clarke 1866",6378206.4,294.9786982138982,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-117],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","26711"]] Tue Oct 10 15:06:16 2017: GDAL: GDALOpen(data/byte.jp2, this=0x138e770) succeeds as JP2KAK. Tue Oct 10 15:06:16 2017: JP2KAK: IReadBlock(0,0) on band 1. Tue Oct 10 15:06:16 2017: JP2KAK: DirectRasterIO() for 0,0,100,100 -> 100x100 (no intermediate) Tue Oct 10 15:06:16 2017: GDAL: QuietDelete(tmp/byte.jp2.tst) invoking Delete() Tue Oct 10 15:06:16 2017: JP2KAK: Cannot read JP2 boxes Tue Oct 10 15:06:16 2017: JP2KAK: Final JPEG2000 Tile Size is 100P x 100L. Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(Corder=PCRL) Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(Cprecincts={512,512},{256,512},{128,512},{64,512},{32,512},{16,512},{8,512},{4,512},{2,512}) Tue Oct 10 15:06:16 2017: JP2KAK: parse_string(ORGgen_plt=yes) terminate called after throwing an instance of 'kdu_cpl_error_message::JP2KAKException' Aborted
in GDB:
TEST: jp2kak_3 ... [New Thread 0x7fffe525f700 (LWP 31)] Tue Oct 10 18:37:33 2017: JP2KAK: Using 1 threads. Tue Oct 10 18:37:33 2017: JP2KAK: Cuse_precincts=1, PreferNonPersistentReads=0 Tue Oct 10 18:37:33 2017: JP2KAK: order=PCRL Tue Oct 10 18:37:33 2017: JP2KAK: nResCount=6 Tue Oct 10 18:37:33 2017: GDALJP2Metadata: Got projection from GeoJP2 (geotiff) box (0): PROJCS["NAD27 / UTM zone 11N",GEOGCS["NAD27",DATUM["North_American_Datum_1927",SPHEROID["Clarke 1866",6378206.4,294.9786982138982,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",-117],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","26711"]] Tue Oct 10 18:37:33 2017: GDAL: GDALOpen(data/byte.jp2, this=0xc29ab0) succeeds as JP2KAK. Tue Oct 10 18:37:33 2017: JP2KAK: IReadBlock(0,0) on band 1. Tue Oct 10 18:37:33 2017: JP2KAK: DirectRasterIO() for 0,0,100,100 -> 100x100 (no intermediate) Tue Oct 10 18:37:33 2017: GDAL: QuietDelete(tmp/byte.jp2.tst) invoking Delete() Tue Oct 10 18:37:33 2017: JP2KAK: Cannot read JP2 boxes Tue Oct 10 18:37:33 2017: JP2KAK: Final JPEG2000 Tile Size is 100P x 100L. Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(Corder=PCRL) Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(Cprecincts={512,512},{256,512},{128,512},{64,512},{32,512},{16,512},{8,512},{4,512},{2,512}) Tue Oct 10 18:37:33 2017: JP2KAK: parse_string(ORGgen_plt=yes) terminate called after throwing an instance of 'kdu_cpl_error_message::JP2KAKException' Program received signal SIGABRT, Aborted. 0x00007ffff7829c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) thread apply all bt Thread 6 (Thread 0x7fffe525f700 (LWP 31)): #0 sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85 #1 0x00007ffff635f804 in kdu_core::kdu_thread_entity::process_jobs(kdu_core::kdu_thread_entity_condition*) () from /usr/lib/libgdal.so.20 #2 0x00007ffff635ed7f in kdu_core::worker_startproc(void*) () from /usr/lib/libgdal.so.20 #3 0x00007ffff635ebc9 in kdu_core::kd_thread_create_entry_point(void*) () from /usr/lib/libgdal.so.20 #4 0x00007ffff7bc4184 in start_thread (arg=0x7fffe525f700) at pthread_create.c:312 #5 0x00007ffff78f0ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7ffff7fe6740 (LWP 23)): #0 0x00007ffff7829c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007ffff782d028 in __GI_abort () at abort.c:89 #2 0x00007ffff48d8535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007ffff48d66d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007ffff48d6703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff48d6922 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007ffff5888901 in kdu_cpl_error_message::flush (this=0xc5b430, end_of_message=true) at jp2kakdataset.h:219 #7 0x00007ffff6313596 in kdu_core::kdu_error::~kdu_error() () from /usr/lib/libgdal.so.20 #8 0x00007ffff6235386 in kdu_supp::jp2_target::open_codestream(bool) () from /usr/lib/libgdal.so.20 #9 0x00007ffff5885f35 in JP2KAKCreateCopy (pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", poSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0, pfnProgress=0x7ffff61c2f10 <GDALDummyProgress(double, char const*, void*)>, pProgressData=0xc47ab0) at jp2kakdataset.cpp:2542 #10 0x00007ffff5bbc6e7 in GDALDriver::CreateCopy (this=0xabdf30, pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", poSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0, pfnProgress=0x7ffff61c2f10 <GDALDummyProgress(double, char const*, void*)>, pProgressData=0xc47ab0) at gdaldriver.cpp:852 #11 0x00007ffff5bbc88f in GDALCreateCopy (hDriver=0xabdf30, pszFilename=0x7fffe52cfa9c "tmp/byte.jp2.tst", hSrcDS=0xc29ab0, bStrict=0, papszOptions=0xc794e0, pfnProgress=0x0, pProgressData=0xc47ab0) at gdaldriver.cpp:899 #12 0x00007ffff6bd35d9 in GDALDriverShadow_CreateCopy (self=0xabdf30, utf8_path=0x7fffe52cfa9c "tmp/byte.jp2.tst", src=0xc29ab0, strict=0, options=0xc794e0, callback=0x0, callback_data=0xc47ab0) at extensions/gdal_wrap.cpp:3966 #13 0x00007ffff6be3c19 in _wrap_Driver_CreateCopy (args=0x7fffe58ab230, kwargs=0x7fffe5267050) at extensions/gdal_wrap.cpp:10406 #14 0x000000000052714b in PyEval_EvalFrameEx () #15 0x0000000000555551 in PyEval_EvalCodeEx () #16 0x0000000000525560 in PyEval_EvalFrameEx () #17 0x0000000000555551 in PyEval_EvalCodeEx () #18 0x0000000000524338 in PyEval_EvalFrameEx () #19 0x00000000005247ea in PyEval_EvalFrameEx () #20 0x00000000005247ea in PyEval_EvalFrameEx () #21 0x00000000005247ea in PyEval_EvalFrameEx () #22 0x0000000000567d14 in ?? () #23 0x0000000000465bf4 in PyRun_FileExFlags () #24 0x000000000046612d in PyRun_SimpleFileExFlags () #25 0x0000000000466d92 in Py_Main () #26 0x00007ffff7814f45 in __libc_start_main (main=0x466e50 <main>, argc=2, argv=0x7fffffffe718, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe708) at libc-start.c:287 #27 0x0000000000577c2e in _start ()
Attachments (4)
Change History (14)
comment:1 by , 6 years ago
Cc: | added |
---|---|
Owner: | changed from | to
Status: | new → assigned |
comment:2 by , 6 years ago
I think since the GDAL JP2KAKException is fatal, the Kakadu errors should probably be logged through CPL as fatal too?
The kdu_cpl_error_message class "catches" Kakadu errors, emit them as CPLError(CE_Failure, ) and re-throw them as JP2KAKException. So the crash is due to a JP2KAKException not handled by a try {} catch {} block. Using CPLError(CE_Fatal, ...) causes the GDAL process to be halted, which is rarely desirable. But anyway, this is just the symptom of the issue. The core issue is some apparent misuse of the Kakadu API.
by , 6 years ago
Attachment: | 7081-1.patch added |
---|
comment:3 by , 6 years ago
7081-1.patch swaps to using the jp2_family_tgt
object rather than the jp2_output_box
object for adding the metadata boxes. This seems to match what kdu_compress.cpp in 7.A.2 is doing more closely. But I don't actually understand JP2 enough to say whether it produces effectively the same output or not? (ie. is the metadata now attached to the wrong place). Is there an easy way to test that?
comment:4 by , 6 years ago
You can try comparing the output of "python swig/python/samples/dump_jp2.py output.jp2"
- with output.jp2 generated by the JP2KAK driver before your changes with Kakadu < 7.A
- with output.jp2 generated by the JP2KAK driver after your changes with Kakadu < 7.A
- with output.jp2 generated by the JP2KAK driver after your changes with Kakadu >= 7.A
They should be identical (excluding the COM marker that contains the Kakadu version)
comment:5 by , 6 years ago
7.9.1 pre- & post-fix output match 7.10.2 differs slightly, mostly just the box offsets and I think some unrelated changes in Kakadu?
$ diff -u 791-prefix.txt 7A2-postfix.txt --- 791-prefix.txt 2017-10-16 16:18:53.000000000 +0100 +++ 7A2-postfix.txt 2017-10-16 15:49:37.000000000 +0100 @@ -1,4 +1,4 @@ -<JP2File filename="791-prefix.jp2"> +<JP2File filename="7A2-postfix.jp2"> <JP2Box name="jP " box_offset="0" box_length="12" data_offset="8" data_length="4"> <BinaryContent>0D0A870A</BinaryContent> </JP2Box> @@ -106,7 +106,7 @@ </VRTDataset> </DecodedGeoTIFF> </JP2Box> - <JP2Box name="jp2c" box_offset="2417" box_length="9886" data_offset="2425" data_length="9878"> + <JP2Box name="jp2c" box_offset="2417" box_length="9883" data_offset="2425" data_length="9875"> <JP2KCodeStream> <Marker name="SOC" offset="2425" length="2" /> <Marker name="SIZ" offset="2427" length="43"> @@ -142,36 +142,36 @@ <Field name="SPcod_Precincts5" type="uint8" description="PPx=9 PPy=9: 512x512">153</Field> </Marker> <Marker name="QCD" offset="2490" length="21" /> - <Marker name="COM" offset="2511" length="19"> + <Marker name="COM" offset="2511" length="20"> <Field name="Rcom" type="uint16" description="LATIN1">1</Field> - <Field name="COM" type="string" size="13">Kakadu-v7.9.1</Field> + <Field name="COM" type="string" size="14">Kakadu-v7.10.2</Field> </Marker> - <Marker name="COM" offset="2530" length="281"> + <Marker name="COM" offset="2531" length="281"> <Field name="Rcom" type="uint16" description="LATIN1">1</Field> <Field name="COM" type="string" size="275">Kdu-Layer-Info: log_2{Delta-D(squared-error)/Delta-L(bytes)}, L(bytes) - -14.1, 4.2e+02 - -15.3, 4.4e+02 - -16.5, 4.7e+02 - -17.7, 5.8e+02 - -18.8, 8.2e+02 - -20.0, 1.6e+03 - -21.2, 2.5e+03 - -22.3, 3.5e+03 - -23.5, 4.1e+03 - -24.7, 5.1e+03 + -13.9, 4.2e+02 + -15.1, 4.4e+02 + -16.2, 4.7e+02 + -17.4, 5.6e+02 + -18.6, 7.0e+02 + -19.7, 1.3e+03 + -20.9, 2.5e+03 + -22.1, 3.4e+03 + -23.3, 4.1e+03 + -24.4, 4.9e+03 -26.9, 6.8e+03 -192.0, 9.8e+03 </Field> </Marker> - <Marker name="SOT" offset="2811" length="12"> + <Marker name="SOT" offset="2812" length="12"> <Field name="Isot" type="uint16">0</Field> - <Field name="Psot" type="uint32">9490</Field> + <Field name="Psot" type="uint32">9486</Field> <Field name="TPsot" type="uint8">0</Field> <Field name="TNsot" type="uint8">1</Field> </Marker> - <Marker name="PLT" offset="2823" length="91" /> - <Marker name="SOD" offset="2914" length="9387" /> - <Marker name="EOC" offset="12301" length="2" /> + <Marker name="PLT" offset="2824" length="90" /> + <Marker name="SOD" offset="2914" length="9384" /> + <Marker name="EOC" offset="12298" length="2" /> </JP2KCodeStream> </JP2Box> </JP2File>
comment:6 by , 6 years ago
Commands run to generate those were:
CPL_DEBUG=ON GDAL_SKIP=JP2OpenJPEG gdal_translate -of JP2KAK -co QUALITY=100 ../autotest/gdrivers/data/byte.jp2 7A2-postfix.jp2 python2 ../gdal/swig/python/samples/dump_jp2.py 7A2-postfix.jp2 > 7A2-postfix.txt
That matches the failing unit test jp2kak_3()
comment:8 by , 6 years ago
Found another one, occurs in gcore/misc.py:misc_6
, but not on every platform/build... Patch reverts to the previous behaviour of not calling jp2_out.set_target_size()
.
CPLError: Error in Kakadu File Format Support: Attempting to close an output JP2 box whose length was defined ahead of time, having written less bytes than indicated by that length value.
diff --git a/gdal/frmts/jp2kak/jp2kakdataset.cpp b/gdal/frmts/jp2kak/jp2kakdataset.cpp index 19e07a9541..70dd6bae33 100644 --- a/gdal/frmts/jp2kak/jp2kakdataset.cpp +++ b/gdal/frmts/jp2kak/jp2kakdataset.cpp @@ -1762,13 +1762,10 @@ static void JP2KAKWriteBox( jp2_family_tgt *jp2_family, GDALJP2Box *poBox ) memcpy(&nBoxType, poBox->GetType(), sizeof(nBoxType)); CPL_MSBPTR32(&nBoxType); - int length = static_cast<int>(poBox->GetDataLength()); - // Write to a box on the JP2 file. jp2_out.open(jp2_family, nBoxType); - jp2_out.set_target_size(length); jp2_out.write(const_cast<kdu_byte *>(poBox->GetWritableData()), - length); + static_cast<int>(poBox->GetDataLength())); jp2_out.close(); delete poBox;
aha, debugger reveals the following. I think since the GDAL JP2KAKException is fatal, the Kakadu errors should probably be logged through CPL as fatal too? But if
CPL_LOG_ERRORS=ON
is set, the following is logged as it dies: