Opened 4 years ago

Closed 4 years ago

#2892 closed defect (fixed)

t.rast.aggregate: error when input space time raster dataset store many rasters

Reported by: tmsz Owned by: grass-dev@…
Priority: normal Milestone: 7.0.4
Component: Temporal Version: 7.0.2
Keywords: wingrass, t.rast.aggregate, g.message Cc:
CPU: x86-64 Platform: MSWindows 7

Description

Steps to reproduce (follow example from Soren Gebbert presentation: The temporal GRASS GIS framework, page 53):

  • import netCDF files
    • r.in.gdal -oe input=tg_0.25deg_reg_1950-1964_v5.0.nc output=temperature_mean offset=0
    • r.in.gdal -oe input=tg_0.25deg_reg_1965-1979_v12.0.nc output=temperature_mean offset=5480
    • r.in.gdal -oe input=tg_0.25deg_reg_1980-1994_v12.0.nc output=temperature_mean offset=10959
    • r.in.gdal -oe input=tg_0.25deg_reg_1995-2015_v12.0.nc output=temperature_mean offset=16439

total count of temperature_mean rasters = 23925

  • t.create type=strds output=temperature_mean_1950_2015_daily temporal=absolute title="European mean temperature 1950-2015" description="The European daily mean temperature"
  • create text file with all temperature_mean rasters

python code:

import os
map_list = os.path.join(os.path.expanduser('~'), 'map_list.txt')
with open(map_list, 'w') as f:
    for i in range(23925):
        f.write('temperature_mean.{0}\n'.format(i + 1))
  • t.register -i type=rast input=temperature_mean_1950_2015_daily file="path to the map_list.txt file" start=1950-01-01 increment="1 days"
  • g.region -p rast=temperature_mean.1
  • t.rast.aggregate input=temperature_mean_1950_2015_daily method=average output=temperature_mean_1950_2015_monthly base=temperature_mean_monthly granularity="1 months"

Error message:

Traceback (most recent call last):
  File "C:\Program Files (x86)\GRASS GIS
7.0.2/scripts/t.rast.aggregate.py", line 215, in <module>
    main()
  File "C:\Program Files (x86)\GRASS GIS
7.0.2/scripts/t.rast.aggregate.py", line 185, in main
    nprocs))
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 580, in info
    message(msg, flag='i')
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 551, in message
    run_command("g.message", flags=flag, message=msg,
errors='ignore')
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 396, in
run_command
    ps = start_command(*args, **kwargs)
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 364, in
start_command
    return Popen(args, **popts)
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\etc\python\grass\script\core.py", line 67, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\Python27\lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "C:\Program Files (x86)\GRASS GIS
7.0.2\Python27\lib\subprocess.py", line 953, in
_execute_child
    startupinfo)
WindowsError: [Error 206] The filename or extension is too
long

Error appear in the core.py file, class Popen (row 67). I try debug subprocess.py file. Error realated with creating process (subprocess.py file, method _subprocess.CreateProcess?, row 946) when g.message.exe command is callled (created process) with long message argument as args parameter in method _subprocess.CreateProcess? (attached file with args argument).

On Windows OS platform is command-line string limitation for total length of the following command line https://support.microsoft.com/en-us/kb/830473.

Attachments (1)

args.tar.xz (17.4 KB) - added by tmsz 4 years ago.
args argument text file

Download all attachments as: .zip

Change History (7)

Changed 4 years ago by tmsz

Attachment: args.tar.xz added

args argument text file

comment:1 Changed 4 years ago by veroandreo

Hi,

I followed your workflow but I wasn't able to reproduce the error you get. I'm using grass71 (r67703) under Fedora 23. Have you tried with the new grass gis stable version 7.0.3?

Btw, your offsets in r.in.gdal didn't work for me, they left one map out and then it was not possible to register maps. So, here is how I changed them (and the total count is now 23922):

r.in.gdal -oe input=tg_0.25deg_reg_1950-1964_v12.0.nc output=temperature_mean offset=0
r.in.gdal -oe input=tg_0.25deg_reg_1965-1979_v12.0.nc output=temperature_mean offset=5479 --o
r.in.gdal -oe input=tg_0.25deg_reg_1980-1994_v12.0.nc output=temperature_mean offset=10957 --o
r.in.gdal -oe input=tg_0.25deg_reg_1995-2015_v12.0.nc output=temperature_mean offset=16436 --o

For the rest, the monthly aggregation works just fine here and no error regarding filename or extension. Have you tried shrinking names? Something like: basename=month_tmean?

comment:2 in reply to:  1 Changed 4 years ago by tmsz

Hi,

This error appear on OS Windows platform only. Offset for import rasters can be different on OS GNU/Linux. I think problem is not with raster name length.

When I run t.rast.aggregate, which inner create process (in the source code for create new process is called python method _subprocess.CreateProcess?() on OS Windows platform) with argument which is command-line calling g.message command with argument message which is very long. On Windows OS platform is command-line string limitation for total length of the following command line (last previous link).

short preview, how is g.message called on OS Windows:

 "C:\Program Files (x86)\GRASS GIS 7.0.2\bin\g.message.EXE" -i "message=granularity_list: [<grass.temporal.space_time_datasets.RasterDataset object at 0x040968F0>, ...

Check the text file (complete g.message command calling) in the attached previous archive.

comment:3 in reply to:  1 Changed 4 years ago by tmsz

Hi again,

I appologize for my bad offset value. Your offset value are correct.

comment:4 Changed 4 years ago by veroandreo

Hi

No problem... But that was not the issue anyway, right? So, I really recommend to update to the last stable release and try again (perhaps there has been some change with regards to the version you are using), and maybe also try with shorter names.

I'm not a programmer, so unfortunately I cannot further help here... we need a specialist :)

comment:5 in reply to:  4 Changed 4 years ago by tmsz

Hi again,

I tried last stable grass gis (wingrass) 7.0.3, code revision 67691 and t.rast.aggregate work correctly now. Thank you for your help and time :).

comment:6 Changed 4 years ago by tmsz

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.