Opened 4 years ago
Last modified 4 years ago
#3979 new defect
G79: problem with i.sentinel.import and importer.write_metadata()
| Reported by: | neteler | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | 7.8.3 |
| Component: | Imagery | Version: | svn-trunk |
| Keywords: | i.sentinel.import, image collections | Cc: | |
| CPU: | Unspecified | Platform: | Unspecified |
Description
I was trying to import Sentinel-2 (S2-L1C) data into G78 in order to make use of image collections:
i.sentinel.import input=/mnt/geodata/openeo_bolzano/ register_output=s2_t_register.csv [...] Processing <T32TPS_20180608T101021_B02>... Importing raster map <T32TPS_20180608T101021_B02>... 100% Processing <T32TPS_20180608T101021_B12>... Importing raster map <T32TPS_20180608T101021_B12>... 100% Processing <T32TPS_20180608T101021_B07>... Importing raster map <T32TPS_20180608T101021_B07>... 100% Processing <T32TPS_20180608T101021_B11>... Importing raster map <T32TPS_20180608T101021_B11>... 100% Processing <T32TPS_20180608T101021_B10>... Importing raster map <T32TPS_20180608T101021_B10>... 100% Processing <T32TPS_20180608T101021_B03>... Importing raster map <T32TPS_20180608T101021_B03>... 100% Writing metadata to maps... Traceback (most recent call last): File "/root/.grass7/addons/scripts/i.sentinel.import", line 504, in <module> sys.exit(main()) File "/root/.grass7/addons/scripts/i.sentinel.import", line 488, in main importer.write_metadata() File "/root/.grass7/addons/scripts/i.sentinel.import", line 407, in write_metadata meta = ip_meta[ip] KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'
I am using a rasther fresh version:
g.version -rge version=7.9.dev date=2019 revision=exported build_date=2019-11-19 build_platform=x86_64-pc-linux-gnu build_off_t_size=8 libgis_revision=00000 libgis_date="?" proj=4.9.3 gdal=2.2.3 geos=3.6.2 sqlite=3.22.0
Change History (12)
comment:1 by , 4 years ago
comment:2 by , 4 years ago
Thanks for your efforts but still...:
Importing raster map <T32TPS_20180608T101021_B03>...
100%
Writing metadata to maps...
Traceback (most recent call last):
File "/root/.grass7/addons/scripts/i.sentinel.import", line 503, in <module>
sys.exit(main())
File "/root/.grass7/addons/scripts/i.sentinel.import", line 487, in main
importer.write_metadata()
File "/root/.grass7/addons/scripts/i.sentinel.import", line 406, in write_metadata
meta = ip_meta[ip]
KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'
GRASS 7.9.dev (utm32n_32632): >
comment:3 by , 4 years ago
I added the pdb debugger there:
> /root/.grass7/addons/scripts/i.sentinel.import(407)write_metadata()
-> meta = ip_meta[ip]
(Pdb) ip
'S2A_MSIL1C_20180608T101021_N0206_R022_T32TPS_20180608T135059'
(Pdb) ip_meta[ip]
{'timestamp': datetime.datetime(2018, 6, 8, 10, 17, 26, 568000), 'SATELLITE': 'S2A', 'CLOUDY_PIXEL_PERCENTAGE': '63.2363', 'DEGRADED_MSI_DATA_PERCENTAGE': '0', 'ZENITH_ANGLE_0': '6.87463083682177', 'AZIMUTH_ANGLE_0': '106.678480338394', 'ZENITH_ANGLE_9': '6.92095256322843', 'AZIMUTH_ANGLE_9': '106.766469213864', 'ZENITH_ANGLE_10': '6.6711378900598', 'AZIMUTH_ANGLE_10': '106.075220597716', 'ZENITH_ANGLE_1': '6.60472703969214', 'AZIMUTH_ANGLE_1': '105.531721669131', 'ZENITH_ANGLE_2': '6.64725248700114', 'AZIMUTH_ANGLE_2': '105.80797975942', 'ZENITH_ANGLE_3': '6.69114850385045', 'AZIMUTH_ANGLE_3': '106.033552825819', 'ZENITH_ANGLE_4': '6.72593186902208', 'AZIMUTH_ANGLE_4': '106.198282849613', 'ZENITH_ANGLE_5': '6.76337836610513', 'AZIMUTH_ANGLE_5': '106.292239966037', 'ZENITH_ANGLE_6': '6.79969827918603', 'AZIMUTH_ANGLE_6': '106.408451382071', 'ZENITH_ANGLE_7': '6.62125461003938', 'AZIMUTH_ANGLE_7': '105.678569152871', 'ZENITH_ANGLE_8': '6.83898070776319', 'AZIMUTH_ANGLE_8': '106.505396826378', 'ZENITH_ANGLE_11': '6.74365826507417', 'AZIMUTH_ANGLE_11': '106.342330082536', 'ZENITH_ANGLE_12': '6.83298026030892', 'AZIMUTH_ANGLE_12': '106.566283022332', 'MEAN_SUN_ZENITH_GRID_ANGLE': 26.24232325141777, 'MEAN_SUN_AZIMUTH_GRID_ANGLE': 149.15294328922496, 'MEAN_SUN_ZENITH_ANGLE': '26.2423240875397', 'MEAN_SUN_AZIMUTH_ANGLE': '149.1529063561'}
No idea - I don't see the issue
comment:4 by , 4 years ago
That`s odd indeed. I cannot reproduce the error with exactly the same scene and both trunk and 7.8 on Fedora 30.
With an older version of trunk:
g.version -rge version=7.9.dev date=2019 revision=400349708 build_date=2019-08-31 build_platform=x86_64-pc-linux-gnu build_off_t_size=8 libgis_revision=00000 libgis_date="?" proj4=5.2.0 gdal=2.3.2 geos=3.7.1 sqlite=3.26.0
as well as a fresh compiled one and 7.8.1:
g.version -rge version=7.8.1RC1 date=2019 revision=exported build_date=2019-11-11 build_platform=x86_64-redhat-linux-gnu build_off_t_size=8 libgis_revision=00000 libgis_date="?" proj=5.2.0 gdal=2.3.2 geos=3.7.1 sqlite=3.26.0
I have no issues with i.sentinel.import with that scene.
Maybe the download was not 100% OK? Do you have other scenes in that directory?
BTW, does 7.8 support band references?
comment:5 by , 4 years ago
My system here is based on the mundialis GRASS GIS 7.9/PDAL/Python 3 docker image (Ubuntu).
I have added, since it is a key error, added a print statement:
WARNING: Raster map <T32TPS_20180608T101021_B03> already exists and will be
overwritten
Importing raster map <T32TPS_20180608T101021_B03>...
100%
Writing metadata to maps...
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', '
AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIM
UTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUT
H_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
[...]
dict_keys(['timestamp', 'SATELLITE', 'CLOUDY_PIXEL_PERCENTAGE', 'DEGRADED_MSI_DATA_PERCENTAGE', 'ZENITH_ANGLE_0', 'AZIMUTH_ANGLE_0', 'ZENITH_ANGLE_9', 'AZIMUTH_ANGLE_9', 'ZENITH_ANGLE_10', 'AZIMUTH_ANGLE_10', 'ZENITH_ANGLE_1', 'AZIMUTH_ANGLE_1', 'ZENITH_ANGLE_2', 'AZIMUTH_ANGLE_2', 'ZENITH_ANGLE_3', 'AZIMUTH_ANGLE_3', 'ZENITH_ANGLE_4', 'AZIMUTH_ANGLE_4', 'ZENITH_ANGLE_5', 'AZIMUTH_ANGLE_5', 'ZENITH_ANGLE_6', 'AZIMUTH_ANGLE_6', 'ZENITH_ANGLE_7', 'AZIMUTH_ANGLE_7', 'ZENITH_ANGLE_8', 'AZIMUTH_ANGLE_8', 'ZENITH_ANGLE_11', 'AZIMUTH_ANGLE_11', 'ZENITH_ANGLE_12', 'AZIMUTH_ANGLE_12', 'MEAN_SUN_ZENITH_GRID_ANGLE', 'MEAN_SUN_AZIMUTH_GRID_ANGLE', 'MEAN_SUN_ZENITH_ANGLE', 'MEAN_SUN_AZIMUTH_ANGLE'])
Traceback (most recent call last):
File "/root/.grass7/addons/scripts/i.sentinel.import", line 505, in <module>
sys.exit(main())
File "/root/.grass7/addons/scripts/i.sentinel.import", line 489, in main
importer.write_metadata()
File "/root/.grass7/addons/scripts/i.sentinel.import", line 407, in write_metadata
meta = ip_meta[ip]
KeyError: 'S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213'
PS: No, 7.8.x does not contain image collection support.
comment:7 by , 4 years ago
Replying to martinl:
What is content of
/mnt/geodata/openeo_bolzano/dir?
root@d07d7993f8ce:/src/actinia_core# ls -la /mnt/geodata/openeo_bolzano/ total 5083276 drwxr-xr-x 7 root root 4096 Nov 21 10:42 . drwxr-xr-x 13 1002 1002 4096 Nov 19 12:09 .. -rw-r--r-- 1 1002 1002 98304 Nov 19 12:07 openeo_bolzano.gpkg -rw-r--r-- 1 root root 778891069 Nov 19 12:10 S2A_MSIL1C_20180608T101021_N0206_R022_T32TPS_20180608T135059.zip -rw-r--r-- 1 root root 666971053 Nov 19 12:10 S2A_MSIL1C_20180611T102021_N0206_R065_T32TPS_20180611T123241.zip drwxr-xr-x 7 root root 4096 Nov 19 12:31 S2A_MSIL1C_20180618T101021_N0206_R022_T32TPS_20180618T135619.SAFE -rw-r--r-- 1 root root 855084777 Nov 19 12:10 S2A_MSIL1C_20180618T101021_N0206_R022_T32TPS_20180618T135619.zip drwxr-xr-x 7 root root 4096 Nov 19 12:31 S2A_MSIL1C_20180621T102021_N0206_R065_T32TPS_20180621T140615.SAFE -rw-r--r-- 1 root root 800185372 Nov 19 12:10 S2A_MSIL1C_20180621T102021_N0206_R065_T32TPS_20180621T140615.zip drwxr-xr-x 7 root root 4096 Nov 19 12:31 S2B_MSIL1C_20180606T102019_N0206_R065_T32TPS_20180606T172808.SAFE -rw-r--r-- 1 root root 730499111 Nov 19 12:10 S2B_MSIL1C_20180606T102019_N0206_R065_T32TPS_20180606T172808.zip drwxr-xr-x 5 root root 4096 Nov 19 12:34 S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213.SAFE -rw-r--r-- 1 root root 574563063 Nov 19 12:11 S2B_MSIL1C_20180613T101019_N0206_R022_T32TPS_20180613T122213.zip drwxr-xr-x 7 root root 4096 Nov 19 12:31 S2B_MSIL1C_20180616T102019_N0206_R065_T32TPS_20180616T154713.SAFE -rw-r--r-- 1 root root 798910322 Nov 19 12:11 S2B_MSIL1C_20180616T102019_N0206_R065_T32TPS_20180616T154713.zip -rw-r--r-- 1 root root 434 Nov 19 11:58 s2_L1C_scenes.csv
follow-up: 9 comment:8 by , 4 years ago
Hm, I think I have a hunch. Can you try to 1) delete all SAFE directories (or use a different unzip dir) 2) run i.sentinel.import again and see if the error remains?
If you don`t link (l-flag) the unpacked SAFE dirs should actually be removed automatically...
If they are present, i.sentinel.import does not unpack again (anymore after my changes). If an import is interrupted it may lead to issues like yours...
comment:9 by , 4 years ago
Replying to sbl:
Hm, I think I have a hunch. Can you try to 1) delete all SAFE directories (or use a different unzip dir) 2) run i.sentinel.import again and see if the error remains?
Bingo! Now it ran through, finally.
If you don`t link (l-flag) the unpacked SAFE dirs should actually be removed automatically...
I didn't use -l this time.
If they are present, i.sentinel.import does not unpack again (anymore after my changes). If an import is interrupted it may lead to issues like yours...
I see. May I suggest that, if present, the SAFE directory be actually removed before unpacking? Less confusion :-)
Rationale: it may happen then a i.sentinel.import job is interrupted, be by the user, for disk-full reasons or whatever else might occur...
comment:10 by , 4 years ago
OK, I see three options: 1) revert to old behavior 2) add a flag for not unziping again 3) use GDALs /vsizip/ virtual file system (no unzipping)
comment:11 by , 4 years ago
OK, for the time being I solved it with a new flag (-n) for not unzipping again. So old behavior is restored (see updated PR: https://github.com/OSGeo/grass-addons/pull/65).
However, I would prefer using /vsizip and reading from zip as a default instead of unpacking at all. That would save disk space and be more efficient in many (most?) cases. Unless one wants to link data and use them frequently (e.g. linking L2A products and use them without any further pre-processing) repeatedly...
comment:12 by , 4 years ago
| Milestone: | → 7.8.3 |
|---|
Please test: https://github.com/OSGeo/grass-addons/pull/65 I could not reproduce the error with that particular S2 scene, but got deprecation warnings from xml on Fedora 30.