Opened 6 years ago

Last modified 5 weeks ago

#1434 new enhancement

g.extension: option to specify proxy server

Reported by: hamish Owned by: grass-dev@…
Priority: major Milestone: 7.4.0
Component: Installation Version: svn-trunk
Keywords: g.extension, urllib, wget, r.in.wms Cc:
CPU: All Platform: All

Description

Hi,

for folks who have to go through an auth'd proxy server to get a connection to the internet g.extension reports a failure of svn+urllib(.py) (or svn+wget,curl in 6.x) to download the files.

Something like 6.x's r.in.wms's wgetoptions= parameter could work around this, or maybe urllib(.py) has some nicer way.

6.x work-around for wget/curl: export the http_proxy enviro variable before starting GRASS. see the wget/curl man pages.

7.x work-around for urllib: same http_proxy enviro var as above, see also http://docs.python.org/library/urllib.html

work-around for 'svn co', all versions: edit the "servers" file in ~/.subversion, or in MS Windows %APPDATA%\Subversion\.

Hamish

Attachments (1)

g.extension.diff (9.8 KB) - added by zarch 4 years ago.
g.extension diff to add support to proxies

Download all attachments as: .zip

Change History (17)

comment:2 Changed 4 years ago by neteler

Priority: normalmajor

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

Changed 4 years ago by zarch

Attachment: g.extension.diff added

g.extension diff to add support to proxies

comment:3 in reply to:  2 Changed 4 years ago by zarch

Replying to neteler:

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

Ok, I've changed the code, now the proxy should be supported... see attachement: g.extension.diff

(Un)fortunately I'm not behind a proxy, therefore I cannot test...

If someone can test the changes, then I can commit...

Note: I changed the global variables from lower to capitals (PEP8).

comment:4 Changed 4 years ago by hamish

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

thanks, Hamish

comment:5 in reply to:  4 ; Changed 4 years ago by zarch

Replying to hamish:

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

# Use proxies from environment - both versions are equivalent
filehandle = urllib.urlopen(some_url, proxies=None)

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

IMHO, if the user specify the http_proxy parameter on the module this should have the precedence.

comment:6 in reply to:  5 Changed 4 years ago by hamish

Milestone: 7.0.06.4.4

Replying to zarch:

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

> # Use proxies from environment - both versions are equivalent
> filehandle = urllib.urlopen(some_url, proxies=None)

seems like an easy string to set in the Preferences menu? (General tab?) Rather than lots of input boxes maybe just a single one with a tooltip saying something like:

 USAGE: http://[username[:password]@]hostname.example.org:portnumber
 EXAMPLE: http://jdoe:secret@proxy.example.com:8181

?

stuck behind a firewall, Hamish

comment:7 Changed 2 years ago by neteler

Milestone: 6.4.47.0.2

While the GRASS GIS 7's version of g.extension comes with proxy support, this crashes when the proxy requires authorization:

(Tue Oct 13 11:51:20 2015)                                                      
g.extension -s extension=r.seg operation=add proxy=http=http://proxy.unige.it:8080
Downloading precompiled GRASS Addons <r.seg>...
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1692, in <module>
    sys.exit(main())
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1675, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 637, in install_extension
    ret += install_extension_win(module)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 895, in install_extension_win
    url_file = urlopen(zfile, proxies=PROXIES)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 87, in
urlopen
    return opener.open(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 208,
in open
    return getattr(self, name)(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 359,
in open_http
    return self.http_error(url, fp, errcode, errmsg,
headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 372,
in http_error
    result = method(url, fp, errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 715,
in http_error_407
    errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 381,
in http_error_default
    raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 407, 'Proxy Authentication
Required', <httplib.HTTPMessage instance at 0x024A2DA0>)
(Tue Oct 13 11:51:21 2015) Command finished (0 sec)

Can this please turned into a proper error message? Thanks.

comment:8 Changed 2 years ago by neteler

Milestone: 7.0.27.0.3

Ticket retargeted after milestone closed

comment:9 Changed 21 months ago by neteler

Milestone: 7.0.3

Ticket retargeted after milestone closed

comment:10 Changed 21 months ago by neteler

Milestone: 7.0.4

Ticket retargeted after 7.0.3 milestone closed

comment:11 in reply to:  7 ; Changed 19 months ago by martinl

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

comment:12 in reply to:  11 Changed 19 months ago by neteler

Replying to martinl:

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

Could anyone please try g.extension with (broken) proxy on Windows?

On Linux I get some partial success:

GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add   proxy=http=http://proxynotthere.osgeo.org:8080
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
ERROR: Unable to read addons metadata file from the remote server
WARNING: No addons metadata available
Installation of <r.vif> successfully finished

# ---> shouldn't it completely fail when using a non-existing proxy server?


GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add 
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
Installation of <r.vif> successfully finished

# ... that works fine as before.

comment:13 Changed 18 months ago by martinl

Milestone: 7.0.47.0.5

comment:14 Changed 14 months ago by martinl

Milestone: 7.0.57.3.0

comment:15 Changed 14 months ago by martinl

Milestone: 7.3.07.4.0

Milestone renamed

comment:16 Changed 5 weeks ago by neteler

Proxy ticket also in #3179

Note: See TracTickets for help on using tickets.