Opened 3 months ago

Last modified 2 months ago

#3979 new defect

G79: problem with i.sentinel.import and importer.write_metadata()

Reported by: neteler Owned by: grass-dev@…
Priority: normal Milestone: 7.8.3
Component: Imagery Version: svn-trunk
Keywords: i.sentinel.import, image collections Cc:
CPU: Unspecified Platform: Unspecified

Description

I was trying to import Sentinel-2 (S2-L1C) data into G78 in order to make use of image collections:

i.sentinel.import input=/mnt/geodata/openeo_bolzano/ register_output=s2_t_register.csv
[...]
Processing <T32TPS_20180608T101021_B02>...
Importing raster map <T32TPS_20180608T101021_B02>...
 100%
Processing <T32TPS_20180608T101021_B12>...
Importing raster map <T32TPS_20180608T101021_B12>...
 100%
Processing <T32TPS_20180608T101021_B07>...
Importing raster map <T32TPS_20180608T101021_B07>...
 100%
Processing <T32TPS_20180608T101021_B11>...
Importing raster map <T32TPS_20180608T101021_B11>...
 100%
Processing <T32TPS_20180608T101021_B10>...
Importing raster map <T32TPS_20180608T101021_B10>...
 100%
Processing <T32TPS_20180608T101021_B03>...
Importing raster map <T32TPS_20180608T101021_B03>...
 100%
Writing metadata to maps...
Traceback (most recent call last):
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 504, in <module>
    sys.exit(main())
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 488, in main
    importer.write_metadata()
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 407, in write_metadata
    meta = ip_meta[ip]
KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'

I am using a rasther fresh version:

g.version -rge
version=7.9.dev
date=2019
revision=exported
build_date=2019-11-19
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj=4.9.3
gdal=2.2.3
geos=3.6.2
sqlite=3.22.0

Change History (12)

comment:1 Changed 3 months ago by sbl

Please test: https://github.com/OSGeo/grass-addons/pull/65 I could not reproduce the error with that particular S2 scene, but got deprecation warnings from xml on Fedora 30.

comment:2 Changed 3 months ago by neteler

Thanks for your efforts but still...:

Importing raster map <T32TPS_20180608T101021_B03>...
 100%
Writing metadata to maps...
Traceback (most recent call last):
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 503, in <module>
    sys.exit(main())
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 487, in main
    importer.write_metadata()
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 406, in write_metadata
    meta = ip_meta[ip]
KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'
GRASS 7.9.dev (utm32n_32632): >

comment:3 Changed 3 months ago by neteler

I added the pdb debugger there:

> /root/.grass7/addons/scripts/i.sentinel.import(407)write_metadata()
-> meta = ip_meta[ip]
(Pdb) ip
'S2A_MSIL1C_20180608T101021_N0206_R022_T32TPS_20180608T135059'
(Pdb) ip_meta[ip]
{'timestamp': datetime.datetime(2018, 6, 8, 10, 17, 26, 568000), 'SATELLITE': 'S2A', 'CLOUDY_PIXEL_PERCENTAGE': '63.2363', 'DEGRADED_MSI_DATA_PERCENTAGE': '0', 'ZENITH_ANGLE_0': '6.87463083682177', 'AZIMUTH_ANGLE_0': '106.678480338394', 'ZENITH_ANGLE_9': '6.92095256322843', 'AZIMUTH_ANGLE_9': '106.766469213864', 'ZENITH_ANGLE_10': '6.6711378900598', 'AZIMUTH_ANGLE_10': '106.075220597716', 'ZENITH_ANGLE_1': '6.60472703969214', 'AZIMUTH_ANGLE_1': '105.531721669131', 'ZENITH_ANGLE_2': '6.64725248700114', 'AZIMUTH_ANGLE_2': '105.80797975942', 'ZENITH_ANGLE_3': '6.69114850385045', 'AZIMUTH_ANGLE_3': '106.033552825819', 'ZENITH_ANGLE_4': '6.72593186902208', 'AZIMUTH_ANGLE_4': '106.198282849613', 'ZENITH_ANGLE_5': '6.76337836610513', 'AZIMUTH_ANGLE_5': '106.292239966037', 'ZENITH_ANGLE_6': '6.79969827918603', 'AZIMUTH_ANGLE_6': '106.408451382071', 'ZENITH_ANGLE_7': '6.62125461003938', 'AZIMUTH_ANGLE_7': '105.678569152871', 'ZENITH_ANGLE_8': '6.83898070776319', 'AZIMUTH_ANGLE_8': '106.505396826378', 'ZENITH_ANGLE_11': '6.74365826507417', 'AZIMUTH_ANGLE_11': '106.342330082536', 'ZENITH_ANGLE_12': '6.83298026030892', 'AZIMUTH_ANGLE_12': '106.566283022332', 'MEAN_SUN_ZENITH_GRID_ANGLE': 26.24232325141777, 'MEAN_SUN_AZIMUTH_GRID_ANGLE': 149.15294328922496, 'MEAN_SUN_ZENITH_ANGLE': '26.2423240875397', 'MEAN_SUN_AZIMUTH_ANGLE': '149.1529063561'}

No idea - I don't see the issue

comment:4 Changed 3 months ago by sbl

That`s odd indeed. I cannot reproduce the error with exactly the same scene and both trunk and 7.8 on Fedora 30.

With an older version of trunk:

g.version -rge
version=7.9.dev
date=2019
revision=400349708
build_date=2019-08-31
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj4=5.2.0
gdal=2.3.2
geos=3.7.1
sqlite=3.26.0

as well as a fresh compiled one and 7.8.1:

g.version -rge
version=7.8.1RC1
date=2019
revision=exported
build_date=2019-11-11
build_platform=x86_64-redhat-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj=5.2.0
gdal=2.3.2
geos=3.7.1
sqlite=3.26.0

I have no issues with i.sentinel.import with that scene.

Maybe the download was not 100% OK? Do you have other scenes in that directory?

BTW, does 7.8 support band references?

comment:5 Changed 3 months ago by neteler

My system here is based on the mundialis GRASS GIS 7.9/PDAL/Python 3 docker image (Ubuntu).

I have added, since it is a key error, added a print statement:

WARNING: Raster map <T32TPS_20180608T101021_B03> already exists and will be
         overwritten
Importing raster map <T32TPS_20180608T101021_B03>...
 100%
Writing metadata to maps...
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
[...]
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', 'AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIMUTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUTH_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
Traceback (most recent call last):
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 505, in <module>
    sys.exit(main())
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 489, in main
    importer.write_metadata()
  File "/root/.grass7/addons/scripts/i.sentinel.import", line 407, in write_metadata
    meta = ip_meta[ip]
KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'

PS: No, 7.8.x does not contain image collection support.

comment:6 Changed 3 months ago by martinl

What is content of /mnt/geodata/openeo_bolzano/ dir?

comment:7 in reply to:  6 Changed 3 months ago by neteler

Replying to martinl:

What is content of /mnt/geodata/openeo_bolzano/ dir?

root@d07d7993f8ce:/src/actinia_core# ls -la /mnt/geodata/openeo_bolzano/
total 5083276
drwxr-xr-x  7 root root      4096 Nov 21 10:42 .
drwxr-xr-x 13 1002 1002      4096 Nov 19 12:09 ..
-rw-r--r--  1 1002 1002     98304 Nov 19 12:07 openeo_bolzano.gpkg
-rw-r--r--  1 root root 778891069 Nov 19 12:10 S2A_MSIL1C_20180608T101021_N0206_R022_T32TPS_20180608T135059.zip
-rw-r--r--  1 root root 666971053 Nov 19 12:10 S2A_MSIL1C_20180611T102021_N0206_R065_T32TPS_20180611T123241.zip
drwxr-xr-x  7 root root      4096 Nov 19 12:31 S2A_MSIL1C_20180618T101021_N0206_R022_T32TPS_20180618T135619.SAFE
-rw-r--r--  1 root root 855084777 Nov 19 12:10 S2A_MSIL1C_20180618T101021_N0206_R022_T32TPS_20180618T135619.zip
drwxr-xr-x  7 root root      4096 Nov 19 12:31 S2A_MSIL1C_20180621T102021_N0206_R065_T32TPS_20180621T140615.SAFE
-rw-r--r--  1 root root 800185372 Nov 19 12:10 S2A_MSIL1C_20180621T102021_N0206_R065_T32TPS_20180621T140615.zip
drwxr-xr-x  7 root root      4096 Nov 19 12:31 S2B_MSIL1C_20180606T102019_N0206_R065_T32TPS_20180606T172808.SAFE
-rw-r--r--  1 root root 730499111 Nov 19 12:10 S2B_MSIL1C_20180606T102019_N0206_R065_T32TPS_20180606T172808.zip
drwxr-xr-x  5 root root      4096 Nov 19 12:34 S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213.SAFE
-rw-r--r--  1 root root 574563063 Nov 19 12:11 S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213.zip
drwxr-xr-x  7 root root      4096 Nov 19 12:31 S2B_MSIL1C_20180616T102019_N0206_R065_T32TPS_20180616T154713.SAFE
-rw-r--r--  1 root root 798910322 Nov 19 12:11 S2B_MSIL1C_20180616T102019_N0206_R065_T32TPS_20180616T154713.zip
-rw-r--r--  1 root root       434 Nov 19 11:58 s2_L1C_scenes.csv

comment:8 Changed 3 months ago by sbl

Hm, I think I have a hunch. Can you try to 1) delete all SAFE directories (or use a different unzip dir) 2) run i.sentinel.import again and see if the error remains?

If you don`t link (l-flag) the unpacked SAFE dirs should actually be removed automatically...

If they are present, i.sentinel.import does not unpack again (anymore after my changes). If an import is interrupted it may lead to issues like yours...

comment:9 in reply to:  8 Changed 3 months ago by neteler

Replying to sbl:

Hm, I think I have a hunch. Can you try to 1) delete all SAFE directories (or use a different unzip dir) 2) run i.sentinel.import again and see if the error remains?

Bingo! Now it ran through, finally.

If you don`t link (l-flag) the unpacked SAFE dirs should actually be removed automatically...

I didn't use -l this time.

If they are present, i.sentinel.import does not unpack again (anymore after my changes). If an import is interrupted it may lead to issues like yours...

I see. May I suggest that, if present, the SAFE directory be actually removed before unpacking? Less confusion :-)

Rationale: it may happen then a i.sentinel.import job is interrupted, be by the user, for disk-full reasons or whatever else might occur...

comment:10 Changed 3 months ago by sbl

OK, I see three options: 1) revert to old behavior 2) add a flag for not unziping again 3) use GDALs /vsizip/ virtual file system (no unzipping)

comment:11 Changed 3 months ago by sbl

OK, for the time being I solved it with a new flag (-n) for not unzipping again. So old behavior is restored (see updated PR: https://github.com/OSGeo/grass-addons/pull/65).

However, I would prefer using /vsizip and reading from zip as a default instead of unpacking at all. That would save disk space and be more efficient in many (most?) cases. Unless one wants to link data and use them frequently (e.g. linking L2A products and use them without any further pre-processing) repeatedly...

comment:12 Changed 2 months ago by neteler

Milestone: 7.8.3
Note: See TracTickets for help on using tickets.