Python OSError: [Errno 7] Argument list too long when running grass.pipe_command repeatedly

[maxnova]

Code to replicate here: ​https://gist.github.com/max-nova/5cd876bc7bfefe5b7c9e

Running on:

lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:        14.04
Codename:       trusty

uname -a
Linux 35d9d88be0b6 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

When running grass.pipe_command many times in a row, I get:

...
Iteration 2843
Iteration 2844
Iteration 2845
Iteration 2846
Iteration 2847
Iteration 2848
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-094d639353cc> in <module>()
----> 1 grass_demo.run()

/tmp/grass_demo.py in run()
     40
     41         out_fn = os.path.join(work_dir, 'rast.tif')
---> 42         export(loc_name, rast_name, out_fn)
     43
     44         shutil.rmtree(work_dir)

/tmp/grass_demo.py in export(loc, name, out_fn)
     27         'output': out_fn
     28     }
---> 29     process = grass.pipe_command(cmd, **kwargs)
     30     stdout, stderr = process.communicate()
     31

/usr/lib/grass64/etc/python/grass/script/core.pyc in pipe_command(*args, **kwargs)
    213     """
    214     kwargs['stdout'] = PIPE
--> 215     return start_command(*args, **kwargs)
    216
    217 def feed_command(*args, **kwargs):

/usr/lib/grass64/etc/python/grass/script/core.pyc in start_command(prog, flags, overwrite, quiet, verbose, **kwargs)
    175         sys.stderr.flush()
    176
--> 177     return Popen(args, **popts)
    178
    179 def run_command(*args, **kwargs):

/usr/lib/grass64/etc/python/grass/script/core.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
     54                                   preexec_fn, close_fds, shell,
     55                                   cwd, env, universal_newlines,
---> 56                                   startupinfo, creationflags)
     57
     58 PIPE = subprocess.PIPE

/usr/lib/python2.7/subprocess.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
    708                                 p2cread, p2cwrite,
    709                                 c2pread, c2pwrite,
--> 710                                 errread, errwrite)
    711         except Exception:
    712             # Preserve original exception in case os.close raises.

/usr/lib/python2.7/subprocess.pyc in _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
   1325                         raise
   1326                 child_exception = pickle.loads(data)
-> 1327                 raise child_exception
   1328
   1329

OSError: [Errno 7] Argument list too long

[glynn]

Replying to maxnova:

When running grass.pipe_command many times in a row, I get:

OSError: [Errno 7] Argument list too long

The error message (which comes from strerror) is misleading. From the documentation for ​execve:

[E2BIG]
    The number of bytes used by the new process image's argument list and environment list is greater than the system-imposed limit of {ARG_MAX} bytes.

In this case, it's the environment which is too large, not the argument list.

This is caused by calling grass.script.setup.init() for each iteration. That function appends the GRASS-specific directories to PATH, LD_LIBRARY_PATH (or equivalent) and PYTHONPATH, even if they're already present. So those variables' values get larger with each iteration.

This isn't a bug in grass.pipe_command (etc); you would have exactly the same problem if you invoked subprocess.Popen() (or os.system(), or any other interface for executing commands).

The solution is not to invoke grass.script.setup.init() for each iteration. Instead, just create a new gisrc file with grass.script.setup.write_gisrc() and set os.environGISRC to the filename. IOW, duplicate the last two lines of grass.script.setup.init().

[maxnova]

Replying to glynn:

Brilliant! For anyone else that runs across this issue, I've updated the demonstration code with the fix described above: ​https://gist.github.com/max-nova/487a82de00651a33f2c2

Many thanks to you glynn - let me know where to send the case of beer!

[wenzeslaus]

Replying to maxnova:

For anyone else that runs across this issue, I've updated the demonstration code with the fix described above: ​https://gist.github.com/max-nova/487a82de00651a33f2c2

Since the link to code appeared again on mailing list, it is worth noting that that the main issue was calling init function multiple times instead of just once (for initialization). (The ticket should have been ideally closed as invalid or at least wontfix.)

The linked code is potentially doing same mistake, setting GISRC variable and file multiple times. If this is an actual mistake depends on how the code is used, but in general, it is enough to set GISRC just once and change the file content to change the Mapset.

In some cases it is also advantageous to use os.environ.copy() to copy environment to dictionary, modify it (e.g. env['GISRC'] = ...) and then pass it as ​env parameter of a function from grass.script.start_command() family.

To learn more about this topic:

• ​grass.script.setup documentation (recently updated)
• ​--exec parameter of grass command (available in trunk, upcoming 7.1 release, #2579)
• user wiki: ​Working with GRASS without starting it explicitly
• grass-dev: ​How GRASS should behave when another session is already active? (​Gmane, ​Nabble)