Opened 3 years ago

Last modified 2 weeks ago

#3179 new defect

g.extension not installing addons behind proxy

Reported by: madi Owned by: grass-dev@…
Priority: normal Milestone:
Component: Default Version: unspecified
Keywords: g.extension, proxy, addons Cc:
CPU: Unspecified Platform: Unspecified

Description

Related to https://lists.osgeo.org/pipermail/grass-user/2016-October/075337.html

when i try to use g.extension behind a proxy, following the example from the manual, it gives me the following error:

g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
Traceback (most recent call last):
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in <module>
    sys.exit(main())
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in main
    for ptype, purl in (p.split('=') for p in options['proxy'].split(',')):
ValueError: need more than 1 value to unpack

Using the form suggested by Pietro:

proxy="http=<value>,ftp=<value>"
g.extension ext=i.segment.hierarchical proxy="http=http://xx.xxx.xxx.xx:xxxx"

gives:

Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
patient)...
ERROR: Extension <i.segment.hierarchical> not found

which is exactly the same as if I don't indicate the proxy at all.

FWIW, I normally use urllib2 to handle the proxy. However I have some difficulties to apply this simple method to the 1700+ rows code (handling several different addons sources for several os's) of g.extension, but I'm behind a proxy and I'm available to test if you have a patch.

Here's what I do in python:

import urllib2
import os
proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')

Thanks

Change History (20)

comment:1 Changed 20 months ago by neteler

Proxy ticket also in #1434

comment:2 Changed 16 months ago by neteler

Please indicate the operating system, the output of g.version -reg and the Python version.

comment:3 Changed 16 months ago by madi

Can't check this now, as I'm currently not behind a proxy. Operating system at the time of reporting was CentOS.

comment:4 in reply to:  description Changed 16 months ago by mmetz

Replying to madi:

Related to https://lists.osgeo.org/pipermail/grass-user/2016-October/075337.html

when i try to use g.extension behind a proxy, following the example from the manual, it gives me the following error:

g.extension ext=i.segment.hierarchical proxy="http://xx.xxx.xxx.xx:xxxx"
Traceback (most recent call last):
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1730, in <module>
    sys.exit(main())
  File "/usr/local/grass-7.3.svn/scripts/g.extension", line 1675, in main
    for ptype, purl in (p.split('=') for p in options['proxy'].split(',')):
ValueError: need more than 1 value to unpack

Using the form suggested by Pietro:

proxy="http=<value>,ftp=<value>"
g.extension ext=i.segment.hierarchical proxy="http=http://xx.xxx.xxx.xx:xxxx"

gives:

Fetching <i.segment.hierarchical> from GRASS GIS Addons repository (be
patient)...
ERROR: Extension <i.segment.hierarchical> not found

which is exactly the same as if I don't indicate the proxy at all.

According to Python docs, "Proxies which require authentication for use are not currently supported; this is considered an implementation limitation" in urllib, and g.extension uses urllib, not urllib2 to open urls.

FWIW, I normally use urllib2 to handle the proxy. However I have some difficulties to apply this simple method to the 1700+ rows code (handling several different addons sources for several os's) of g.extension, but I'm behind a proxy and I'm available to test if you have a patch.

Here's what I do in python:

import urllib2
import os
proxy = urllib2.ProxyHandler({'http':os.environ['HTTP_PROXY']})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.google.com')

That should also work for g.extension, including authentification.

comment:5 Changed 16 months ago by neteler

Milestone: 7.4.07.4.1

Ticket retargeted after milestone closed

comment:6 Changed 11 months ago by neteler

Milestone: 7.4.17.4.2

comment:7 Changed 7 months ago by neteler

Milestone: 7.4.27.4.3

Ticket retargeted after milestone closed

comment:8 Changed 5 months ago by martinl

Milestone: 7.4.37.4.4

Bump milestone to 7.4.4

comment:9 Changed 4 months ago by neteler

Milestone: 7.4.47.4.5

Ticket retargeted after milestone closed

comment:10 Changed 3 months ago by neteler

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

comment:11 in reply to:  10 ; Changed 3 months ago by mmetz

Replying to neteler:

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Do we have a list of required python 2 components and required python 3 components that can be updated? Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

comment:12 in reply to:  11 ; Changed 3 months ago by neteler

Replying to mmetz:

Replying to neteler:

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

comment:13 in reply to:  12 ; Changed 3 months ago by mlennert

Replying to neteler:

Replying to mmetz:

Replying to neteler:

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

I would plead for reworking this to use urllib2 (Python 2) which is already imported in the module and which seems to work according to madi. IIUC, authenticated proxy handling is part of urllib in Python 3, or ?

Let's try to avoid the endless dependency hell just out of easy availability of small helper packages. See the experience with npm in the Javascript world as a case in point. I would plead to keep most modules down to standard python library dependency, unless there really is no other way.

comment:14 in reply to:  13 Changed 3 months ago by mmetz

Replying to mlennert:

Replying to neteler:

Replying to mmetz:

Replying to neteler:

Fix attempt via https://github.com/GRASS-GIS/grass-ci/pull/6 submitted as r74116. Note the different proxy syntax in the manual page.

If ok, to be backported.

Note, this is introducing a new python dependency, on RHEL 7 and derivates, this is python-requests.

Yes, I realized that today.

Do we have a list of required python 2 components and required python 3 components that can be updated?

Also yes. See r74121, it is REQUIREMENTS.html.

Ideally such a list would also indicate which python components are required only for the GUI (systems such as HPC may use GRASS without GUI).

That's more or less indicated in that file.

So, I am undecided now about the backport.

I would plead for reworking this to use urllib2 (Python 2) which is already imported in the module and which seems to work according to madi.

+1

Let's try to avoid the endless dependency hell just out of easy availability of small helper packages. See the experience with npm in the Javascript world as a case in point. I would plead to keep most modules down to standard python library dependency, unless there really is no other way.

+1

comment:15 Changed 5 weeks ago by annakrat

In 74374:

g.extension: avoid python requests import, see #3179

comment:16 Changed 5 weeks ago by annakrat

If somebody could test it with proxy, that would be great, I couldn't do that now. This should work with Python 2/3, it uses six package (which we need to require unlike requests).

If this works, we need to revert r74121.

As a side note, since we are moving to GitHub? and we expect to get more pull requests, we should be more strict about what we accept. People typically care about getting a specific issue fixed, but this will often introduce more mess in the code or even break other functionality.

comment:17 Changed 5 weeks ago by pmav99

requests need to be removed from REQUIREMENTS.html and the Dockerfile, too.

comment:18 Changed 3 weeks ago by martinl

Component: AddonsDefault

comment:19 Changed 3 weeks ago by martinl

Milestone: 7.4.5

Remove Milestone from Addons bugreports.

comment:20 in reply to:  17 Changed 2 weeks ago by mmetz

Replying to pmav99:

requests need to be removed from REQUIREMENTS.html and the Dockerfile, too.

Done in trunk r74454

Note: See TracTickets for help on using tickets.