Opened 6 years ago

Closed 6 years ago

#3693 closed defect (worksforme)

Issue with G76 in docker: creating a location fails

Reported by: neteler Owned by: grass-dev@…
Priority: normal Milestone: 7.4.3
Component: Startup Version: svn-releasebranch76
Keywords: docker g.proj Cc: carmenTawalika
CPU: x86-64 Platform: Linux

Description

while we meanwhile use GRASS GIS 7.6.svn in docker for months we ran into this issue:

root@52e6935c3247:/grassdb# grass76 --text c 'EPSG:4326'
/actinia_core/grassdb/latlongtest
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <latlongtest>...
Traceback (most recent call last):
  File "/usr/local/bin/grass76", line 2162, in <module>
    main()
  File "/usr/local/bin/grass76", line 2075, in main
    geofile=params.geofile, create_new=True)
  File "/usr/local/bin/grass76", line 1005, in set_mapset
    create_location(gisdbase, location_name, geofile)
  File "/usr/local/bin/grass76", line 807, in create_location
    epsg=epsg, datum_trans=datum_trans)
  File "/usr/local/grass-7.6.svn/etc/python/grass/script/core.py",
line 1538, in create_location
    encoding='utf-8', mode='w')
  File "/usr/lib/python2.7/codecs.py", line 898, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory:
'/actinia_core/grassdb/latlongtest/PERMANENT/MYNAME'

Thanks to Vaclav (https://lists.osgeo.org/pipermail/grass-dev/2018-November/090480.html) I now understand that the mkdir part in lib/python/script/core.py fails.

Not sure how relevant but the volume is mounted into docker:

df -h /actinia_core/grassdb/
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/wandel--app--vg-opt   19G   13G  5.3G  71% /actinia_core/grassdb

The used docker image doesn't show this behaviour on a different machine.

Attached a patch which fixes it for me; it is based on the code snippet proposed here: https://stackoverflow.com/a/273227

Perhaps a better code style is possible.

Attachments (1)

lib_python_script_core.diff (814 bytes ) - added by neteler 6 years ago.
Patch of lib/python/script/core.py to overcome location creation problems on docker with externally mounted volumes

Download all attachments as: .zip

Change History (7)

by neteler, 6 years ago

Attachment: lib_python_script_core.diff added

Patch of lib/python/script/core.py to overcome location creation problems on docker with externally mounted volumes

in reply to:  description ; comment:1 by mmetz, 6 years ago

Replying to neteler:

while we meanwhile use GRASS GIS 7.6.svn in docker for months we ran into this issue:

root@52e6935c3247:/grassdb# grass76 --text c 'EPSG:4326'
/actinia_core/grassdb/latlongtest
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <latlongtest>...
Traceback (most recent call last):
  File "/usr/local/bin/grass76", line 2162, in <module>
    main()
  File "/usr/local/bin/grass76", line 2075, in main
    geofile=params.geofile, create_new=True)
  File "/usr/local/bin/grass76", line 1005, in set_mapset
    create_location(gisdbase, location_name, geofile)
  File "/usr/local/bin/grass76", line 807, in create_location
    epsg=epsg, datum_trans=datum_trans)
  File "/usr/local/grass-7.6.svn/etc/python/grass/script/core.py",
line 1538, in create_location
    encoding='utf-8', mode='w')
  File "/usr/lib/python2.7/codecs.py", line 898, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory:
'/actinia_core/grassdb/latlongtest/PERMANENT/MYNAME'

Thanks to Vaclav (https://lists.osgeo.org/pipermail/grass-dev/2018-November/090480.html) I now understand that the mkdir part in lib/python/script/core.py fails.

I disagree, the mapset should have been created by g.proj: https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_7_6/lib/python/script/core.py#L1515

Attached a patch which fixes it for me; it is based on the code snippet proposed here: https://stackoverflow.com/a/273227

The patch does not create a valid mapset, it creates only an empty directory: the files DEFAULT_WIND, PROJ_EPSG, PROJ_INFO, PROJ_UNITS, WIND are missing.

You need to find out why g.proj called by create_location() has not created the location with mapset PERMANENT (/actinia_core/grassdb/latlongtest/PERMANENT)

in reply to:  1 ; comment:2 by neteler, 6 years ago

Cc: carmenTawalika added
Component: PythonStartup
CPU: Unspecifiedx86-64
Keywords: g.proj added
Platform: UnspecifiedLinux

(I am a bit verbose here in order to show the debugging within docker)

Replying to mmetz:

I disagree, the mapset should have been created by g.proj: https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_7_6/lib/python/script/core.py#L1515

... You need to find out why g.proj called by create_location() has not created the location with mapset PERMANENT (/actinia_core/grassdb/latlongtest/PERMANENT)

I see, thanks for the hint. So I get (wow!):

GRASS 7.6.svn (latlongtest3):/grassdb > g.proj -t epsg=4326 location=location
Illegal instruction (core dumped)

Trying with gdb in docker:

GRASS 7.6.svn (latlongtest3):/grassdb > gdb g.proj
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git

(gdb) r -t epsg=4326 location=location
Starting program: /usr/local/grass-7.6.svn/bin/g.proj -t epsg=4326 location=location
warning: Error disabling address space randomization: Operation not permitted
warning: Could not trace the inferior process.
Error: 
warning: ptrace: Operation not permitted
During startup program exited with code 127.

Following http://visualgdb.com/gdbreference/commands/set_disable-randomization I get

(gdb) set disable-randomization off
(gdb) r -t epsg=4326 location=location
Starting program: /usr/local/grass-7.6.svn/bin/g.proj -t epsg=4326 location=location
warning: Could not trace the inferior process.
Error: 
warning: ptrace: Operation not permitted
During startup program exited with code 127.

Normally one has to use docker run --cap-add=SYS_PTRACE ... but we use docker-compose here, so that goes into the related .yml file (hint).

Now debugging is enabled:

(gdb) r -t epsg=4326 location=location
Starting program: /usr/local/grass-7.6.svn/bin/g.proj -t epsg=4326 location=location
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x00007ffff7bced1e in read_datum_table () from /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so
(gdb) bt full
#0  0x00007ffff7bced1e in read_datum_table () from /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so
No symbol table info available.
#1  0x00007ffff7bcde4b in GPJ_osr_to_grass () from /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so
No symbol table info available.
#2  0x0000555555557dab in input_epsg ()
No symbol table info available.
#3  0x0000555555556d57 in main ()
No symbol table info available.

Trying with strace:

GRASS 7.6.svn (latlongtest3):/grassdb > strace g.proj -t epsg=4326 location=location
...
futex(0x7fecb07d73a4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
stat("/usr/share/gdal/2.2/gcs.csv", {st_mode=S_IFREG|0644, st_size=53326, ...}) = 0
openat(AT_FDCWD, "/usr/share/gdal/2.2/gcs.csv", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=53326, ...}) = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 4096) = 4096
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 4096) = 4096
brk(0x5632c9f8b000)                     = 0x5632c9f8b000
fstat(3, {st_mode=S_IFREG|0644, st_size=53326, ...}) = 0
lseek(3, 53248, SEEK_SET)               = 53248
read(3, "Helena Geodetic Datum 2015,1174,"..., 4096) = 78
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 53248) = 53248
read(3, "Helena Geodetic Datum 2015,1174,"..., 4096) = 78
close(3)                                = 0
stat("/usr/share/gdal/2.2/gcs.override.csv", {st_mode=S_IFREG|0644, st_size=513, ...}) = 0
openat(AT_FDCWD, "/usr/share/gdal/2.2/gcs.override.csv", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=513, ...}) = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 4096) = 513
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 4096) = 513
fstat(3, {st_mode=S_IFREG|0644, st_size=513, ...}) = 0
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\"COORD_REF_SYS_CODE\",\"COORD_REF_"..., 4096) = 513
lseek(3, 513, SEEK_SET)                 = 513
close(3)                                = 0
stat("/usr/share/gdal/2.2/ellipsoid.csv", {st_mode=S_IFREG|0644, st_size=14365, ...}) = 0
openat(AT_FDCWD, "/usr/share/gdal/2.2/ellipsoid.csv", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=14365, ...}) = 0
read(3, "ellipsoid_code,ellipsoid_name,se"..., 4096) = 4096
lseek(3, 0, SEEK_SET)                   = 0
read(3, "ellipsoid_code,ellipsoid_name,se"..., 4096) = 4096
fstat(3, {st_mode=S_IFREG|0644, st_size=14365, ...}) = 0
lseek(3, 12288, SEEK_SET)               = 12288
read(3, "been used in Danish work.,\"Kort "..., 4096) = 2077
lseek(3, 0, SEEK_SET)                   = 0
read(3, "ellipsoid_code,ellipsoid_name,se"..., 12288) = 12288
read(3, "been used in Danish work.,\"Kort "..., 4096) = 2077
close(3)                                = 0
stat("/usr/share/gdal/2.2/gdal_datum.csv", {st_mode=S_IFREG|0644, st_size=234839, ...}) = 0
openat(AT_FDCWD, "/usr/share/gdal/2.2/gdal_datum.csv", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=234839, ...}) = 0
read(3, "\"DATUM_CODE\",\"DATUM_NAME\",\"DATUM"..., 4096) = 4096
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\"DATUM_CODE\",\"DATUM_NAME\",\"DATUM"..., 4096) = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
lseek(3, 4096, SEEK_SET)                = 4096
read(3, "seau_Geodesique_de_la_RDC_2005\"\n"..., 4096) = 4096
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
lseek(3, 8192, SEEK_SET)                = 8192
read(3, "1982-01-01,7024,8901,3228,Topogr"..., 4096) = 4096
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
lseek(3, 12288, SEEK_SET)               = 12288
read(3, "al Reference Frame,geodetic,ITRF"..., 4096) = 4096
lseek(3, 8192, SEEK_SET)                = 8192
...
read(3, "Santa Maria de Ipire,engineering"..., 4096) = 1367
lseek(3, 229376, SEEK_SET)              = 229376
read(3, ". Latitude: 40.835864 grads N, l"..., 4096) = 4096
read(3, "Santa Maria de Ipire,engineering"..., 4096) = 1367
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
lseek(3, 234839, SEEK_SET)              = 234839
read(3, "", 4096)                       = 0
lseek(3, 0, SEEK_CUR)                   = 234839
read(3, "", 4096)                       = 0
lseek(3, 0, SEEK_CUR)                   = 234839
close(3)                                = 0
openat(AT_FDCWD, "/usr/local/grass-7.6.svn/etc/proj/datum.table", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=5725, ...}) = 0
read(3, "# Datum Transformation Parameter"..., 4096) = 4096
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x7fecb0a2ed1e} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

Quite confusing for me. The file starts like this:

head /usr/local/grass-7.6.svn/etc/proj/datum.table
# Datum Transformation Parameter Lists for GRASS
# (single 3-parameter datum transformation))
#
# This file contains datum shift parameters,
# it supplies a datum database table for GRASS
...

The version used here is:

GRASS 7.6.svn (latlongtest3):/grassdb > g.version -rbe
GRASS 7.6.svn (2018)

 ./configure  --enable-largefile --with-cxx --with-nls --with-readline --with-sqlite --with-bzlib --with-zstd --with-cairo --with-cairo-ldflags=-lfontconfig --with-freetype --with-freetype-includes=/usr/include/freetype2/ --with-fftw --with-netcdf --with-liblas --with-liblas-config=/usr/bin/liblas-config --with-pdal --with-proj --with-proj-share=/usr/share/proj --with-geos=/usr/bin/geos-config --with-postgres --with-postgres-includes=/usr/include/postgresql --with-opengl-libs=/usr/include/GL
libgis Revision: 72327 
libgis Date: 2018-03-06 12:12:44 +0100 (Tue, 06 Mar 2018) 
PROJ.4: 4.9.3
GDAL/OGR: 2.2.3
GEOS: 3.6.2
SQLite: 3.22.0

uname -a
Linux d30387cd7649 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Honestly, I have no idea what's wrong.

in reply to:  2 ; comment:3 by mmetz, 6 years ago

Replying to neteler:

Trying with strace:

GRASS 7.6.svn (latlongtest3):/grassdb > strace g.proj -t epsg=4326 location=location
...
openat(AT_FDCWD, "/usr/local/grass-7.6.svn/etc/proj/datum.table", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=5725, ...}) = 0
read(3, "# Datum Transformation Parameter"..., 4096) = 4096
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x7fecb0a2ed1e} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

...

Honestly, I have no idea what's wrong.

This does not seem to be a GRASS-related error. Considering that the same (?) docker image works on a different system (docker images are specific to each system, right?) and that g.proj works outside docker, I suggest to rebuild the docker image causing trouble.

in reply to:  3 comment:4 by neteler, 6 years ago

Replying to mmetz:

This does not seem to be a GRASS-related error. Considering that the same (?) docker image works on a different system (docker images are specific to each system, right?) and that g.proj works outside docker, I suggest to rebuild the docker image causing trouble.

I did so, no difference. When I delete "datum.table" then I get:

(gdb) r -t epsg=4326 location=location
Starting program: /usr/local/grass-7.6.svn/bin/g.proj -t epsg=4326 location=location
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
WARNING: Unable to open datum table file
         </usr/local/grass-7.6.svn/etc/proj/datum.table>
WARNING: Datum <WGS_1984> not recognised by GRASS and no parameters found

Program received signal SIGILL, Illegal instruction.
0x00007ffff7bce060 in GPJ_osr_to_grass () from /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so
(gdb) bt full
#0  0x00007ffff7bce060 in GPJ_osr_to_grass () from /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so
No symbol table info available.
#1  0x0000555555557dab in input_epsg ()
No symbol table info available.
#2  0x0000555555556d57 in main ()
No symbol table info available.

GRASS 7.6.svn (latlongtest3):/grassdb > ldd /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so | grep gdal
	libgdal.so.20 => /usr/lib/libgdal.so.20 (0x00007f8ec76c3000)
GRASS 7.6.svn (latlongtest3):/grassdb > ls -la /usr/lib/libgdal.so.20
lrwxrwxrwx 1 root root 17 Feb  6  2018 /usr/lib/libgdal.so.20 -> libgdal.so.20.3.2
GRASS 7.6.svn (latlongtest3):/grassdb > ls -la /usr/lib/libgdal.so.20.3.2 
-rw-r--r-- 1 root root 16559240 Feb  6  2018 /usr/lib/libgdal.so.20.3.2

GRASS 7.6.svn (latlongtest3):/grassdb > ldd /usr/local/grass-7.6.svn/lib/libgrass_gproj.7.6.svn.so | grep proj
	libproj.so.12 => /usr/lib/x86_64-linux-gnu/libproj.so.12 (0x00007fac75934000)
GRASS 7.6.svn (latlongtest3):/grassdb > ldd /usr/lib/x86_64-linux-gnu/libproj.so.12
	linux-vdso.so.1 (0x00007ffdd7ffb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4c8142a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4c8108c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4c80c9b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4c818b2000)

GRASS 7.6.svn (latlongtest3):/grassdb > ls -la /usr/lib/x86_64-linux-gnu/libproj.so.12
lrwxrwxrwx 1 root root 17 Jun 18  2017 /usr/lib/x86_64-linux-gnu/libproj.so.12 -> libproj.so.12.0.0
GRASS 7.6.svn (latlongtest3):/grassdb > ls -la /usr/lib/x86_64-linux-gnu/libproj.so.12.0.0 
-rw-r--r-- 1 root root 430016 Jun 18  2017 /usr/lib/x86_64-linux-gnu/libproj.so.12.0.0
GRASS 7.6.svn (latlongtest3):/grassdb > cs2cs -v
Using from definition: 
Rel. 4.9.3, 15 August 2016

I will check tomorrow if my attempt to rebuild the docker image went wrong.

comment:5 by neteler, 6 years ago

Problem solved: update of docker-compose from v1.8.0 to v1.23.1 (what a waste of time to figure out this one...).

Explanations:

Program received signal SIGILL, Illegal instruction.

This means that a CPU instruction is called which is not present in the CPU running the docker. In our case, we run it on a KVM virtualization. Apparently docker-compose v1.23 (at least, later than v1.8 addresses the problem of the missing instruction). w00t!

Closing.

comment:6 by neteler, 6 years ago

Resolution: worksforme
Status: newclosed
Note: See TracTickets for help on using tickets.