Opened 13 years ago

Last modified 3 years ago

#1434 reopened enhancement

g.extension: option to specify proxy server

Reported by: hamish Owned by: grass-dev@…
Priority: major Milestone: 7.6.2
Component: Installation Version: svn-trunk
Keywords: g.extension, urllib, wget, r.in.wms Cc:
CPU: All Platform: All

Description

Hi,

for folks who have to go through an auth'd proxy server to get a connection to the internet g.extension reports a failure of svn+urllib(.py) (or svn+wget,curl in 6.x) to download the files.

Something like 6.x's r.in.wms's wgetoptions= parameter could work around this, or maybe urllib(.py) has some nicer way.

6.x work-around for wget/curl: export the http_proxy enviro variable before starting GRASS. see the wget/curl man pages.

7.x work-around for urllib: same http_proxy enviro var as above, see also http://docs.python.org/library/urllib.html

work-around for 'svn co', all versions: edit the "servers" file in ~/.subversion, or in MS Windows %APPDATA%\Subversion\.

Hamish

Attachments (1)

g.extension.diff (9.8 KB ) - added by zarch 11 years ago.
g.extension diff to add support to proxies

Download all attachments as: .zip

Change History (25)

comment:2 by neteler, 11 years ago

Priority: normalmajor

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

by zarch, 11 years ago

Attachment: g.extension.diff added

g.extension diff to add support to proxies

in reply to:  2 comment:3 by zarch, 11 years ago

Replying to neteler:

Increasing importance, enhancement asked in Italian GRASS GIS mailing list.

Ok, I've changed the code, now the proxy should be supported... see attachement: g.extension.diff

(Un)fortunately I'm not behind a proxy, therefore I cannot test...

If someone can test the changes, then I can commit...

Note: I changed the global variables from lower to capitals (PEP8).

comment:4 by hamish, 11 years ago

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

thanks, Hamish

in reply to:  4 ; comment:5 by zarch, 11 years ago

Replying to hamish:

Hi,

please also check if the http_proxy, ftp_proxy, etc. environment variables exist, that's quite widely supported convention and quite handy for those that need it set for everything.

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

# Use proxies from environment - both versions are equivalent
filehandle = urllib.urlopen(some_url, proxies=None)

I'm not sure which should have precedence if it is set multiple ways, I guess the command line option before the envrio. variable.

IMHO, if the user specify the http_proxy parameter on the module this should have the precedence.

in reply to:  5 comment:6 by hamish, 11 years ago

Milestone: 7.0.06.4.4

Replying to zarch:

If the environment variables exist and user does not specify new one with: http_proxy, ftp_proxy, etc. the environment variables are used. Looking at the python docs:

> # Use proxies from environment - both versions are equivalent
> filehandle = urllib.urlopen(some_url, proxies=None)

seems like an easy string to set in the Preferences menu? (General tab?) Rather than lots of input boxes maybe just a single one with a tooltip saying something like:

 USAGE: http://[username[:password]@]hostname.example.org:portnumber
 EXAMPLE: http://jdoe:secret@proxy.example.com:8181

?

stuck behind a firewall, Hamish

comment:7 by neteler, 8 years ago

Milestone: 6.4.47.0.2

While the GRASS GIS 7's version of g.extension comes with proxy support, this crashes when the proxy requires authorization:

(Tue Oct 13 11:51:20 2015)                                                      
g.extension -s extension=r.seg operation=add proxy=http=http://proxy.unige.it:8080
Downloading precompiled GRASS Addons <r.seg>...
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1692, in <module>
    sys.exit(main())
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 1675, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 637, in install_extension
    ret += install_extension_win(module)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn/scripts/g.extens
ion.py", line 895, in install_extension_win
    url_file = urlopen(zfile, proxies=PROXIES)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 87, in
urlopen
    return opener.open(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 208,
in open
    return getattr(self, name)(url)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 359,
in open_http
    return self.http_error(url, fp, errcode, errmsg,
headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 372,
in http_error
    result = method(url, fp, errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 715,
in http_error_407
    errcode, errmsg, headers)
  File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 381,
in http_error_default
    raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 407, 'Proxy Authentication
Required', <httplib.HTTPMessage instance at 0x024A2DA0>)
(Tue Oct 13 11:51:21 2015) Command finished (0 sec)

Can this please turned into a proper error message? Thanks.

comment:8 by neteler, 8 years ago

Milestone: 7.0.27.0.3

Ticket retargeted after milestone closed

comment:9 by neteler, 8 years ago

Milestone: 7.0.3

Ticket retargeted after milestone closed

comment:10 by neteler, 8 years ago

Milestone: 7.0.4

Ticket retargeted after 7.0.3 milestone closed

in reply to:  7 ; comment:11 by martinl, 8 years ago

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

in reply to:  11 comment:12 by neteler, 8 years ago

Replying to martinl:

Replying to neteler:

Can this please turned into a proper error message? Thanks.

please try r68228.

Could anyone please try g.extension with (broken) proxy on Windows?

On Linux I get some partial success:

GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add   proxy=http=http://proxynotthere.osgeo.org:8080
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
ERROR: Unable to read addons metadata file from the remote server
WARNING: No addons metadata available
Installation of <r.vif> successfully finished

# ---> shouldn't it completely fail when using a non-existing proxy server?


GRASS 7.0.4svn (demolocation):~ > g.extension -s extension=r.vif operation=add 
Fetching <r.vif> from GRASS-Addons SVN repository (be patient)...
Compiling...
Installing...
Updating addons metadata file...
Installation of <r.vif> successfully finished

# ... that works fine as before.

comment:13 by martinl, 8 years ago

Milestone: 7.0.47.0.5

comment:14 by martinl, 8 years ago

Milestone: 7.0.57.3.0

comment:15 by martinl, 8 years ago

Milestone: 7.3.07.4.0

Milestone renamed

comment:16 by neteler, 7 years ago

Proxy ticket also in #3179

comment:17 by martinl, 6 years ago

Resolution: fixed
Status: newclosed

Closing, feel free to re-open if needed.

in reply to:  17 comment:18 by mmetz, 6 years ago

Resolution: fixed
Status: closedreopened

Replying to martinl:

Closing, feel free to re-open if needed.

From #3179:

According to Python docs, "Proxies which require authentication for use are not currently supported; this is considered an implementation limitation" in urllib, and g.extension uses urllib, not urllib2 to open urls.

comment:19 by neteler, 6 years ago

Milestone: 7.4.07.4.1

Ticket retargeted after milestone closed

comment:20 by neteler, 6 years ago

Milestone: 7.4.17.4.2

comment:21 by martinl, 6 years ago

Milestone: 7.4.27.6.0

All enhancement tickets should be assigned to 7.6 milestone.

comment:22 by martinl, 5 years ago

Milestone: 7.6.07.6.1

Ticket retargeted after milestone closed

comment:23 by martinl, 5 years ago

Milestone: 7.6.17.6.2

Ticket retargeted after milestone closed

comment:24 by sbl, 3 years ago

Is this still relevant in more recent versions of GRASS?

Note: See TracTickets for help on using tickets.