g.extension: option to specify proxy server

[hamish]

Hi,

for folks who have to go through an auth'd proxy server to get a connection to the internet g.extension reports a failure of svn+urllib(.py) (or svn+wget,curl in 6.x) to download the files.

Something like 6.x's r.in.wms's wgetoptions= parameter could work around this, or maybe urllib(.py) has some nicer way.

6.x work-around for wget/curl: export the http_proxy enviro variable before starting GRASS. see the wget/curl man pages.

7.x work-around for urllib: same http_proxy enviro var as above, see also ​http://docs.python.org/library/urllib.html

work-around for 'svn co', all versions: edit the "servers" file in ~/.subversion, or in MS Windows %APPDATA%\Subversion\.

Hamish

[hamish]

see also ​http://subversion.apache.org/faq.html#proxy

[neteler]

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

[zarch]

g.extension diff to add support to proxies

[zarch]

Replying to neteler:

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

Ok, I've changed the code, now the proxy should be supported... see attachement: g.extension.diff

(Un)fortunately I'm not behind a proxy, therefore I cannot test...

If someone can test the changes, then I can commit...

Note: I changed the global variables from lower to capitals (PEP8).

[hamish]

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

thanks, Hamish

[zarch]

Replying to hamish:

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

# Use proxies from environment - both versions are equivalent
filehandle = urllib.urlopen(some_url, proxies=None)

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

IMHO, if the user specify the http_proxy parameter on the module this should have the precedence.

[hamish]

Replying to zarch:

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

> # Use proxies from environment - both versions are equivalent
> filehandle = urllib.urlopen(some_url, proxies=None)

seems like an easy string to set in the Preferences menu? (General tab?) Rather than lots of input boxes maybe just a single one with a tooltip saying something like:

 USAGE: http://[username[:password]@]hostname.example.org:portnumber
 EXAMPLE: http://jdoe:secret@proxy.example.com:8181

?

stuck behind a firewall, Hamish

[neteler]

While the GRASS GIS 7's version of g.extension comes with proxy support, this crashes when the proxy requires authorization:

(Tue Oct 13 11:51:20 2015)                                                      
g.extension -s extension=r.seg operation=add proxy=http=http://proxy.unige.it:8080
Downloading precompiled GRASS Addons <r.seg>...
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1692, in <module>
    sys.exit(main())
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1675, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 637, in install_extension
    ret += install_extension_win(module)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 895, in install_extension_win
    url_file = urlopen(zfile, proxies=PROXIES)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 87, in
urlopen
    return opener.open(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 208,
in open
    return getattr(self, name)(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 359,
in open_http
    return self.http_error(url, fp, errcode, errmsg,
headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 372,
in http_error
    result = method(url, fp, errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 715,
in http_error_407
    errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 381,
in http_error_default
    raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 407, 'Proxy Authentication
Required', <httplib.HTTPMessage instance at 0x024A2DA0>)
(Tue Oct 13 11:51:21 2015) Command finished (0 sec)

Can this please turned into a proper error message? Thanks.

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after 7.0.3 milestone closed

[martinl]

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

[neteler]

Replying to martinl:

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

Could anyone please try g.extension with (broken) proxy on Windows?

On Linux I get some partial success:

GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add   proxy=http=http://proxynotthere.osgeo.org:8080
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
ERROR: Unable to read addons metadata file from the remote server
WARNING: No addons metadata available
Installation of <r.vif> successfully finished

# ---> shouldn't it completely fail when using a non-existing proxy server?


GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add 
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
Installation of <r.vif> successfully finished

# ... that works fine as before.

[martinl]

Milestone renamed

[neteler]

Proxy ticket also in #3179

[martinl]

Closing, feel free to re-open if needed.

[mmetz]

Replying to martinl:

Closing, feel free to re-open if needed.

From #3179:

According to Python docs, "Proxies which require authentication for use are not currently supported; this is considered an implementation limitation" in urllib, and g.extension uses urllib, not urllib2 to open urls.

[neteler]

Ticket retargeted after milestone closed

[martinl]

All enhancement tickets should be assigned to 7.6 milestone.

[martinl]

Ticket retargeted after milestone closed

[martinl]

Ticket retargeted after milestone closed

[sbl]

Is this still relevant in more recent versions of GRASS?