g.parser does not allow newline character in option value for Python scripts

[wenzeslaus]

g.parser mechanism does not allow newline character when newline is in option value. This applies to Python modules:

m.proj input=test.txt proj_out="+proj=lcc..." proj_in="+proj=merc...+to_meter=1
"

C modules are fine:

g.proj proj4="
"

(error is Can't parse PROJ.4-style parameter string which means that parser allowed module to continue)

When newline is in parameter, g.parser -s module/name/or/path will output (for the above example):

@ARGS_PARSED@
flag_i=0
flag_o=0
flag_d=0
flag_e=0
flag_c=0
opt_input=-
opt_output=
opt_separator= , 
opt_proj_in=+proj=merc...+to_meter=1

opt_proj_out=+proj=lcc...+to_meter=1

I'm not sure how newlines should be handled, I haven't found a documentation of this format.

Anyway, the problem is in _parse_opts()​ function which stops when line is empty:

if not line:
    break

If the option with newline is not the last one, Python script will get the options dictionary incomplete which causes unexpected error.

In my case I was running m.proj in a Python script with the parameter proj_in obtained by g.proj:

proj_in = gcore.read_command('g.proj', flags='jf')
...
proc = gcore.start_command('m.proj', input='-', separator=' , ',
                           proj_in=from_proj, proj_out=to_proj,
                           stdin=gcore.PIPE,
                           stdout=gcore.PIPE, stderr=gcore.PIPE)

And the error I obtained was:

Traceback (most recent call last):
  File "/home/vasek/dev/grass/trunk_clang/dist.i686-pc-linux-gnu/scripts/m.proj", line 292, in <module>
    main()
  File "/home/vasek/dev/grass/trunk_clang/dist.i686-pc-linux-gnu/scripts/m.proj", line 121, in main
    proj_out = options['proj_out']
KeyError: 'proj_out'

Simple proj_in.strip() solves the problem in my case because the output of g.proj is ended with a newline.

However, this is a general problem and if g.parser Python mechanism is not able to handle newlines g.parser itself should give an error message when user tries to input some newline. If g.parser Python mechanism should allow newlines (anywhere in the option value) for all modules, the output and/or the reader (_parse_opts() function) should be fixed accordingly.

Maybe the _parse_opts() function should check the lines by regular expressions opt_.*= and lines which does not match should be merged with above option value (I'm not sure about last line, it might be empty always, if so, it should not be merged).

Are there other suggestions, e.g. for some fast (in any sense) implementation?

[glynn]

Replying to wenzeslaus:

Are there other suggestions, e.g. for some fast (in any sense) implementation?

r58339 adds a -n switch to g.parser to separate options with a null character rather an newline, and modifies grass.script.parser() to use that option.

This needs to be tested on Windows.

[wenzeslaus]

I just tried one MS Windows computer with test command above and it worked (command fails in wrong way, see #2140, but the parameters are passed).

Thanks for the fix. Closing the ticket, reopen when it will be failing on other MS Windows machines.

To test this in correct way generate test file:

echo "100 200" > test.txt

Run test:

m.proj input=test.txt proj_out="+proj=lcc +lat_1=36.16666666666666 +lat_2=34.33333333333334 +lat_0=33.75 +lon_0=-79 +x_0=609601.22 +y_0=0 +no_defs +a=6378137 +rf=298.257222101 +towgs84=0.000,0.000,0.000 +to_meter=1" proj_in="+proj=merc +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +no_defs +a=6378137 +rf=298.257223563 +towgs84=0.000,0.000,0.000 +to_meter=1
"

Check output:

10024493.03|-20933.49|0.00