Opened 9 months ago

Closed 9 months ago

#3719 closed enhancement (fixed)

prepare OSGeo4W environment for winGRASS 7.8 with python 3

Reported by: hellik Owned by: grass-dev@…
Priority: critical Milestone: 7.8.0
Component: Python Version: svn-trunk
Keywords: python3, py3 Cc:
CPU: All Platform: MSWindows

Description

there is an ongoing discussion about GRASS and python 3 support, e.g. GRASS + Python3 + QGIS (vs Python2), experimental Python 3 support in trunk and other threads.

to bring the python 3 support a step forward, the OSGeo4W environment has to be assessed if it's ready for winGRASS 7.8 with python 3.

actually available py3 packages in OSGeo4W, taken from OSGeo4W download

pyqt5/
python3-certifi/ 
python3-chardet/- 	 
python3-core/
python3-coverage/	 
python3-cycler/ 
python3-decorator/ 
python3-devel/	 
python3-exifread/ 
python3-future/
python3-h5py/	 
python3-help/
python3-httplib2/
python3-idna/ 
python3-ipython-genutils/
python3-jinja2/ 
python3-jsonschema/ 
python3-jupyter-core/ 
python3-kiwisolver/ 
python3-markupsafe/ 
python3-matplotlib/ 
python3-mock/ 
python3-nbformat/ 
python3-networkx/ 
python3-nose2/ 
python3-numpy/ 
python3-owslib/
python3-pandas/
python3-pbr/
python3-pip/
python3-plotly/ 
python3-ply/ 
python3-psycopg2/
python3-pygments/
python3-pyodbc/ 	 
python3-pyparsing/ 
python3-pypiwin32/ 	 
python3-pyproj/	 
python3-pypubsub/	 
python3-pysal/ 
python3-python-dateutil/ 
python3-pytz/
python3-pywin32/ 	 
python3-pyyaml/
python3-qscintilla/
python3-requests/ 
python3-retrying/ 
python3-scipy/
python3-setuptools/ 
python3-shapely/ 
python3-simplejson/	 
python3-six/ 
python3-tcltk/
python3-test/	 
python3-tools/	 
python3-traitlets/ 
python3-urllib3/ 
python3-wx/	 
python3-xlrd/	 
python3-xlwt/
sip-qt5/

is there any list which py3-packages are needed for winGRASS 7.8 with python 3?

Attachments (4)

osgeo4w_setup_hint_python3.patch (2.1 KB) - added by hellik 9 months ago.
patch of osgeo4w setup.hint for python 3 support of wingrass
osgeo4w_set_python3_environment.patch (1007 bytes) - added by hellik 9 months ago.
patch to set python 3 environment in OSGeo4W
GRASS-Packager.bat.tmpl_supportpython3.patch (963 bytes) - added by hellik 9 months ago.
patch to bundle python3 for standalone winGRASS
env.bat_set_python3_standalone_wingrass.patch (481 bytes) - added by hellik 9 months ago.
patch to set python 3 environment in standalone winGRASS

Download all attachments as: .zip

Change History (32)

comment:1 Changed 9 months ago by hellik

OSGeo4W dependencies for winGRASS are handled by setup_x86.hint.tmpl and setup_x86_64.hint.tmpl

32 bit:

1	sdesc: "GRASS GIS - daily builds of development version"
2	ldesc: "Geographic Resources Analysis Support System (GRASS GIS) - daily builds of SVN trunk"
3	category: Desktop
4	requires: liblas avce00 gpsbabel gs gdal-python matplotlib msvcrt msvcrt2013 msvcrt2015 iconv libtiff gdal proj freetype-mingw pdcurses zlib fftw libpng python-wx python-numpy python-pil python-pip python-ply pyopengl cairo psycopg2 rbatch zstd zstd-bin
5	maintainer: MartinLanda
6	curr: @GRASS_VERSION_NUMBER@-1
7	prev: @GRASS_VERSION_NUMBER@-1

64 bit:

1	sdesc: "GRASS GIS - daily builds of development version"
2	ldesc: "Geographic Resources Analysis Support System (GRASS GIS) - daily builds of SVN trunk"
3	category: Desktop
4	requires: liblas avce00 gpsbabel gs gdal-python matplotlib msvcrt msvcrt2012 msvcrt2013 msvcrt2015 iconv libtiff gdal proj pdcurses zlib fftw libpng python-wx python-numpy python-pil python-pip python-ply pyopengl cairo psycopg2 rbatch zstd zstd-bin
5	maintainer: MartinLanda
6	curr: @GRASS_VERSION_NUMBER@-1
7	prev: @GRASS_VERSION_NUMBER@-1

these files have to be screened and updated accordingly with the needed python 3 packages

comment:2 in reply to:  1 ; Changed 9 months ago by hellik

Replying to hellik:

OSGeo4W dependencies for winGRASS are handled by setup_x86.hint.tmpl and setup_x86_64.hint.tmpl

matching python 2 packages with python 3 packages available in OSGeo4W:

32 bit:

requires: 

gdal-python		=> 		python3-gdal
matplotlib 		=>		python3-matplotlib/
python-wx 		=>		python3-wx/
python-numpy 	        =>		python3-numpy/
python-pil 		=>		MISSING
python-pip 		=>		python3-pip/
python-ply		=>		python3-ply/ 
pyopengl 		=>		MISSING  
psycopg2		=>		python3-psycopg2/

no dependency	        =>		python3-six/

64 bit:

gdal-python		=> 		python3-gdal 
matplotlib		=>		python3-matplotlib/ 
python-wx 		=>		python3-wx/ 
python-numpy 	        =>		python3-numpy/ 
python-pil 		=>		MISSING 
python-pip 		=>		python3-pip/
python-ply		=>		python3-ply/  
pyopengl 		=>		MISSING 
psycopg2		=>		python3-psycopg2/

no dependency	        =>		python3-six/

six is a new dependency for GRASS with python 3 support, see thread about experimental Python 3 support in trunk

it seems, two python 2 packages are missing on the python 3 side in OSGeo4W

python-pil
pyopengl

are these packages still needed?

comment:3 Changed 9 months ago by hellik

anything else missing?

comment:4 in reply to:  2 ; Changed 9 months ago by martinl

Replying to hellik:

it seems, two python 2 packages are missing on the python 3 side in OSGeo4W

python-pil
pyopengl

are these packages still needed?

PIL is required

ui/wxpython/mapwin/decorations.py:    from PIL import Image
gui/wxpython/core/utils.py:        from PIL import Image
gui/wxpython/psmap/utils.py:    from PIL import Image as PILImage
gui/wxpython/animation/controller.py:                    from PIL import Image
gui/wxpython/animation/utils.py:    from PIL import Image
lib/python/imaging/operations.py:    from PIL import Image
lib/python/imaging/operations.py:        from PIL import PILLOW_VERSION  # pylint: disable=unused-import
lib/python/imaging/images2gif.py:    from PIL import Image
lib/python/imaging/images2gif.py:        from PIL import PILLOW_VERSION  # test if user has Pillow or PIL
lib/python/imaging/images2gif.py:    from PIL.GifImagePlugin import getheader, getdata

I am not sure pyopengl.

comment:5 in reply to:  3 ; Changed 9 months ago by martinl

Replying to hellik:

anything else missing?

build scripts must be updated grass-addons/tools/wingrass-packager since grass77+ with python3 support will need to built in separated environment.

comment:6 in reply to:  4 ; Changed 9 months ago by martinl

Replying to martinl:

PIL is required

well, PIL has been updated since 2006, https://pypi.org/project/PIL/. Pillow should be used instead, https://pypi.org/project/Pillow/.

comment:7 Changed 9 months ago by martinl

I would suggest to change wingrass build process as below:

  • grass74/76 daily builds untouched
  • grass77 daily builds switched to python3 environment

It means that grass77 daily builds can be unstable for some time, but it's the only way how to prepare for next stable release (7.8) with full Python3 support.

Any comments?

Last edited 9 months ago by martinl (previous) (diff)

comment:8 in reply to:  6 ; Changed 9 months ago by hellik

Replying to martinl:

Replying to martinl:

PIL is required

well, PIL has been updated since 2006, https://pypi.org/project/PIL/. Pillow should be used instead, https://pypi.org/project/Pillow/.

just opened an OSGEo4W ticket to install pillow and pyopengl.

comment:9 in reply to:  5 Changed 9 months ago by hellik

Replying to martinl:

Replying to hellik:

anything else missing?

build scripts must be updated grass-addons/tools/wingrass-packager since grass77+ with python3 support will need to built in separated environment.

additionally some environment settings are needed to use python3; something like in python-qgis.bat, e.g.

[...]
call py3_env.bat
[...]
set PYTHONPATH=%OSGEO4W_ROOT%\apps\qgis\python;%PYTHONPATH%
"%PYTHONHOME%\python" %*

the content of py3_env.bat

SET PYTHONPATH=
SET PYTHONHOME=%OSGEO4W_ROOT%\apps\Python37
PATH %OSGEO4W_ROOT%\apps\Python37;%OSGEO4W_ROOT%\apps\Python37\Scripts;%PATH%
Last edited 9 months ago by hellik (previous) (diff)

comment:10 in reply to:  7 Changed 9 months ago by hellik

Replying to martinl:

I would suggest to change wingrass build process as below:

  • grass74/76 daily builds untouched
  • grass77 daily builds switched to python3 environment

strong +1 here for grass77 daily switch to python 3

It means that grass77 daily builds can be unstable for some time, but it's the only way how to prepare for next stable release (7.8) with full Python3 support.

yes

comment:11 in reply to:  8 Changed 9 months ago by hellik

Replying to hellik:

Replying to martinl:

Replying to martinl:

PIL is required

well, PIL has been updated since 2006, https://pypi.org/project/PIL/. Pillow should be used instead, https://pypi.org/project/Pillow/.

just opened an OSGEo4W ticket to install pillow and pyopengl.

for the record, osgeo4w Python 3 stack is updated, See

https://trac.osgeo.org/osgeo4w/ticket/586#comment:1

next steps like switching build environment can be started.

comment:12 in reply to:  1 Changed 9 months ago by hellik

Replying to hellik:

OSGeo4W dependencies for winGRASS are handled by setup_x86.hint.tmpl and setup_x86_64.hint.tmpl

32 bit:

1	sdesc: "GRASS GIS - daily builds of development version"
2	ldesc: "Geographic Resources Analysis Support System (GRASS GIS) - daily builds of SVN trunk"
3	category: Desktop
4	requires: liblas avce00 gpsbabel gs gdal-python matplotlib msvcrt msvcrt2013 msvcrt2015 iconv libtiff gdal proj freetype-mingw pdcurses zlib fftw libpng python-wx python-numpy python-pil python-pip python-ply pyopengl cairo psycopg2 rbatch zstd zstd-bin
5	maintainer: MartinLanda
6	curr: @GRASS_VERSION_NUMBER@-1
7	prev: @GRASS_VERSION_NUMBER@-1

64 bit:

1	sdesc: "GRASS GIS - daily builds of development version"
2	ldesc: "Geographic Resources Analysis Support System (GRASS GIS) - daily builds of SVN trunk"
3	category: Desktop
4	requires: liblas avce00 gpsbabel gs gdal-python matplotlib msvcrt msvcrt2012 msvcrt2013 msvcrt2015 iconv libtiff gdal proj pdcurses zlib fftw libpng python-wx python-numpy python-pil python-pip python-ply pyopengl cairo psycopg2 rbatch zstd zstd-bin
5	maintainer: MartinLanda
6	curr: @GRASS_VERSION_NUMBER@-1
7	prev: @GRASS_VERSION_NUMBER@-1

these files have to be screened and updated accordingly with the needed python 3 packages

patch of osgeo4w setup.hint for python 3 support of wingrass added to ticket

Changed 9 months ago by hellik

patch of osgeo4w setup.hint for python 3 support of wingrass

comment:13 Changed 9 months ago by martinl

In 73886:

force Python3 for compilation, see #3719

comment:14 Changed 9 months ago by martinl

In 73887:

wingrass: update setup.hint to use python3 dependencies, thanks hellik for the patch, see #3719

Changed 9 months ago by hellik

patch to set python 3 environment in OSGeo4W

comment:15 Changed 9 months ago by hellik

osgeo4w_set_python3_environment.patch​ sets the python3 environment in OSGeo4W-winGRASS trunk.

TODO adapt the procedure for standalone winGRASS.

Changed 9 months ago by hellik

patch to bundle python3 for standalone winGRASS

comment:16 Changed 9 months ago by hellik

GRASS-Packager.bat.tmpl_supportpython3.patch​ to bundle python3 for standalone winGRASS

Changed 9 months ago by hellik

patch to set python 3 environment in standalone winGRASS

comment:17 Changed 9 months ago by hellik

added env.bat_set_python3_standalone_wingrass.patch​ to set python 3 environment in standalone winGRASS

comment:18 in reply to:  15 ; Changed 9 months ago by hellik

Replying to hellik:

TODO adapt the procedure for standalone winGRASS.

now added patches to set python 3 environment also for standalone winGRASS.

review needed before applying.

comment:19 Changed 9 months ago by martinl

In 73893:

set python 3 environment in OSGeo4W
bundle python3 for standalone winGRASS
python 3 environment in standalone winGRASS
patches provided by hellik, see #3719

comment:20 in reply to:  18 Changed 9 months ago by martinl

Replying to hellik:

review needed before applying.

thanks for providing the patches! I took liberty to apply them to trunk in r73893.

comment:21 Changed 9 months ago by martinl

there is a new issue: python3-wx package must be updated (current version refers to Python 3.6). I will do it ASAP. Then I will also setup a new building environment. But next build will be probably broken anyway.

Last edited 9 months ago by martinl (previous) (diff)

comment:22 Changed 9 months ago by martinl

In 73895:

wingrass: switch package environment to python3-based, see #3719

comment:23 in reply to:  21 ; Changed 9 months ago by martinl

Replying to martinl:

there is a new issue: python3-wx package must be updated (current version refers to Python

done, new version (4.0.3-2) uploaded.

comment:24 in reply to:  23 Changed 9 months ago by hellik

Replying to martinl:

Replying to martinl:

there is a new issue: python3-wx package must be updated (current version refers to Python

done, new version (4.0.3-2) uploaded.

thanks.

locally tested compilation of winGRASS with python 3 support. compilation seems to work.

should we close the ticket then? for python 3 issues of winGRASS itself, another ticket should be opened.

comment:25 Changed 9 months ago by martinl

let's close ticket when wingrass daily builds (osgeo4w and standalone) will be functional.

comment:26 in reply to:  25 Changed 9 months ago by hellik

Replying to martinl:

let's close ticket when wingrass daily builds (osgeo4w and standalone) will be functional.

see #3723 for first detected python3 issues in winGRASS

comment:27 in reply to:  25 Changed 9 months ago by hellik

Replying to martinl:

let's close ticket when wingrass daily builds (osgeo4w and standalone) will be functional.

daily compilation and building seems to work, see

https://wingrass.fsv.cvut.cz/grass77/x86_64/logs/log-r73895-361/error.log

GRASS GIS 7.7.svn r73895 compilation log
--------------------------------------------------
Started compilation: Tue Jan  1 16:24:31     2019
--
Errors in:
No errors detected.
--
Finished compilation: Tue Jan  1 17:27:28     2019

comment:28 Changed 9 months ago by martinl

Resolution: fixed
Status: newclosed

Tested both OSGeo4W and standalone installer. This issue seems to be solved.

Note: See TracTickets for help on using tickets.