Opened 16 months ago

Closed 9 months ago

Last modified 9 months ago

#3883 closed defect (fixed)

GRASS master does not compile with Python 3 on Mac

Reported by: cmbarton Owned by: grass-dev@…
Priority: major Milestone: 7.8.3
Component: Default Version: svn-trunk
Keywords: Cc:
CPU: Unspecified Platform: MacOSX

Description

The problem seems to be with ctypes. I am attaching error and build logs (thanks to Anna Petrasova). I am using the most current Mac build environment (OS X 10.14.6 and Xcode 10.3 with command line tools). May be an nviz issue based on ctype build errors.

Attachments (1)

ctypes_build output.zip (334.3 KB) - added by cmbarton 16 months ago.
ctype build output and errors

Download all attachments as: .zip

Change History (32)

Changed 16 months ago by cmbarton

Attachment: ctypes_build output.zip added

ctype build output and errors

comment:1 Changed 16 months ago by neteler

I see in your log eg.:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/Headers/ATSTypes.h:24:10: fatal error: 'CarbonCore/CarbonCore.h' file not found
#include <CarbonCore/CarbonCore.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~

Does the file exist anywhere on your machine?

comment:2 in reply to:  1 Changed 16 months ago by cmbarton

Replying to neteler:

I see in your log eg.:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/Headers/ATSTypes.h:24:10: fatal error: 'CarbonCore/CarbonCore.h' file not found
#include <CarbonCore/CarbonCore.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~

Does the file exist anywhere on your machine?

I checked. CarbonCore?.h is NOT included in the most recent Mac OSX SDK for v. 10.14. However, it IS included in earlier versions, which I have.

/MacOSX10.13.sdk has CarbonCore?.h but does not have some other frameworks that nviz wants--maybe because Apple dropped default support for OpenGL. However, everything needed seems available in /MacOSX10.12.sdk on back, which I also have.

In the configure string, I can specify --with-macosx-sdk=/Developer/SDKs/MacOSX10.12.sdk.

BUT ctypes appears to ignore this and I get the same error that you spotted. It is looking in the default SDK for 10.14 rather than in the one I pointed to in configure.

Last edited 16 months ago by cmbarton (previous) (diff)

comment:3 Changed 15 months ago by neteler

Milestone: 7.8.07.8.1

Ticket retargeted after milestone closed

comment:4 Changed 13 months ago by neteler

Milestone: 7.8.17.8.2

Ticket retargeted after milestone closed

comment:5 Changed 12 months ago by cmbarton

So what to do? This is not something I know how to solve. Will GRASS drop support for the Mac?

comment:6 Changed 12 months ago by hcho

I think the problem is not many developers have a Mac? Personally, I'd like to support the Mac, but I cannot afford (or don't want?) to buy one just for compiling GRASS.

comment:7 Changed 12 months ago by neteler

Milestone: 7.8.2

Ticket retargeted after milestone closed

comment:8 Changed 12 months ago by neteler

Milestone: 7.8.3

comment:9 Changed 10 months ago by guano

I'm facing similar issues. Compiling on OSX 10.14 Mojave, with python 3.7.4 (pyenv). I'm not using the Mac SDK, just the Mac Command Line Tools. Compilation breaks with this error:

Errors in:
/Volumes/MacintoshHD2/grass/lib/python/ctypes
/Volumes/MacintoshHD2/grass/raster/r.in.lidar
/Volumes/MacintoshHD2/grass/raster3d/r3.in.lidar
/Volumes/MacintoshHD2/grass/vector/v.out.lidar
/Volumes/MacintoshHD2/grass/vector/v.in.lidar
--
In case of errors please change into the directory with error and run 'make'.
If you get multiple errors, you need to deal with them in the order they
appear in the error log. If you get an error building a library, you will
also get errors from anything which uses the library.
--
Finished compilation: Tue Jan 21 12:42:28 -03 2020
make: *** [default] Error 1

So I cd into ctypes and run make:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 173, in parse
    ppout, pperr = pp.communicate()
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 1721, in _communicate
    self.stdout.errors)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 816, in _translate_newlines
    data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 1400362: invalid start byte
make[1]: *** [OBJ.x86_64-apple-darwin18.7.0/nviz.py] Error 1
make: *** [default] Error 2

So is this NVIZ-related?

comment:10 in reply to:  9 Changed 10 months ago by cmbarton

Replying to guano:

I'm facing similar issues. Compiling on OSX 10.14 Mojave, with python 3.7.4 (pyenv). I'm not using the Mac SDK, just the Mac Command Line Tools. Compilation breaks with this error:

Errors in:
/Volumes/MacintoshHD2/grass/lib/python/ctypes
/Volumes/MacintoshHD2/grass/raster/r.in.lidar
/Volumes/MacintoshHD2/grass/raster3d/r3.in.lidar
/Volumes/MacintoshHD2/grass/vector/v.out.lidar
/Volumes/MacintoshHD2/grass/vector/v.in.lidar
--
In case of errors please change into the directory with error and run 'make'.
If you get multiple errors, you need to deal with them in the order they
appear in the error log. If you get an error building a library, you will
also get errors from anything which uses the library.
--
Finished compilation: Tue Jan 21 12:42:28 -03 2020
make: *** [default] Error 1

So I cd into ctypes and run make:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 173, in parse
    ppout, pperr = pp.communicate()
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 1721, in _communicate
    self.stdout.errors)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 816, in _translate_newlines
    data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 1400362: invalid start byte
make[1]: *** [OBJ.x86_64-apple-darwin18.7.0/nviz.py] Error 1
make: *** [default] Error 2

So is this NVIZ-related?

Maybe but I don't think so. GRASS compiled under Mohave until August/September? 2019. Then this error popped up. So I don't think it is a change in the Mac OS, but something in the GRASS code.

comment:11 Changed 10 months ago by guano

Searching for this error I found it to be quite common with python2 code being ported to python3, as with python3 all strings are unicode.

>>> b'\x9c'.decode('cp1252')
'œ'
>>> b'\x9c'.decode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte

comment:12 Changed 9 months ago by nila

While not getting to the root of the problem, there is a workaround that seem to work:

adding encoding='latin1' to subprocess.Popen at (lib/python/ctypes/ctypesgencore/parser/preprocessor.py:170)

enables compilation, seemingly without issues.

---
 lib/python/ctypes/ctypesgencore/parser/preprocessor.py | 1 +
 1 file changed, 1 insertion(+)
 mode change 100644 => 100755 lib/python/ctypes/ctypesgencore/parser/preprocessor.py

diff --git a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
old mode 100644
new mode 100755
index 18999483e..1d0dd8749
--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -168,6 +168,7 @@ class PreprocessorParser(object):
         pp = subprocess.Popen(cmd,
                               shell=True,
                               universal_newlines=True,
+                              encoding='latin1',
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
-- 
2.23.0

comment:13 Changed 9 months ago by guano

I can confirm that compilation was successfull with the modification. (OS Mojave, python 3.7.4 pyenv)

comment:14 in reply to:  12 Changed 9 months ago by cmbarton

Replying to nila:

This is great news. I'll try it out.

While not getting to the root of the problem, there is a workaround that seem to work:

adding encoding='latin1' to subprocess.Popen at (lib/python/ctypes/ctypesgencore/parser/preprocessor.py:170)

enables compilation, seemingly without issues.

---
 lib/python/ctypes/ctypesgencore/parser/preprocessor.py | 1 +
 1 file changed, 1 insertion(+)
 mode change 100644 => 100755 lib/python/ctypes/ctypesgencore/parser/preprocessor.py

diff --git a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
old mode 100644
new mode 100755
index 18999483e..1d0dd8749
--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -168,6 +168,7 @@ class PreprocessorParser(object):
         pp = subprocess.Popen(cmd,
                               shell=True,
                               universal_newlines=True,
+                              encoding='latin1',
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
-- 
2.23.0

comment:15 Changed 9 months ago by nila

I believe I have tracked down the bad one. There is a (one) macOS system file encoded with Mac Roman (mac_roman):

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk\
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/HIServices.framework/Headers/InternetConfig.h

being parsed. Remains to be seen how this can be fixed...

comment:16 Changed 9 months ago by annakrat

I would say, the better way to workaround it is to ignore the decoding errors for stderr:

--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -162,15 +162,18 @@ class PreprocessorParser(object):
 
         self.cparser.handle_status(cmd)
 
+        cmd = cmd.encode()
+
         if sys.platform == 'win32':
-            cmd = ['sh.exe', '-c', cmd]
+            cmd = [b'sh.exe', b'-c', cmd]
 
         pp = subprocess.Popen(cmd,
                               shell=True,
-                              universal_newlines=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
+        ppout = ppout.decode()
+        pperr = pperr.decode(errors='replace')
 
         for line in pperr.split("\n"):
             if line:

Alternatively, just pass errors='replace' to Popen, but that is not available in Python 3.5

comment:17 Changed 9 months ago by nila

Thanks @annakrat! I put up a PR https://github.com/OSGeo/grass/pull/385, following roughly what you proposed.

comment:18 Changed 9 months ago by cmbarton

I just tried Nila's fix and it does compile. Anna's does not. Unfortunately, even though the first fix compiles successfully, the GRASS GUI will not launch. Here is the error:

Launching <wxpython> GUI in the background, please wait...
GRASS 7.9.dev (nc_spm_08_grass7):~ > Traceback (most recent call last):
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/wxgui.py", line 105, in OnInit
    from lmgr.frame import GMFrame
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/lmgr/frame.py", line 51, in <module>
    from lmgr.layertree import LayerTree, LMIcons
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/lmgr/layertree.py", line 38, in <module>
    from mapdisp.frame import MapFrame
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/mapdisp/frame.py", line 33, in <module>
    from mapdisp.toolbars import MapToolbar, NvizIcons
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/mapdisp/toolbars.py", line 22, in <module>
    from nviz.main import haveNviz
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/main.py", line 24, in <module>
    from nviz import mapwindow
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/mapwindow.py", line 42, in <module>
    from nviz.workspace import NvizSettings
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/workspace.py", line 23, in <module>
    from nviz import wxnviz
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/wxnviz.py", line 32, in <module>
    from numpy import matrix
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/core/__init__.py", line 107, in <module>
    from . import _internal
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/core/_internal.py", line 20, in <module>
    IS_PYPY = platform.python_implementation() == 'PyPy'
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/platform.py", line 1264, in python_implementation
    return _sys_version()[0]
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/platform.py", line 1225, in _sys_version
    repr(sys_version))
ValueError: failed to parse CPython sys.version: '3.7.0 (default, Jun 28 2018, 07:39:16) \n[Clang 4.0.1 (tags/RELEASE_401/final)]'
OnInit returned false, exiting...

I'm using OSX Mojave 10.14.6 with python 3.7.6 I'm compiling from master source updated in the past hour or so.

v.in.pdal also does not compile. But that is another issue.

Last edited 9 months ago by cmbarton (previous) (diff)

comment:19 Changed 9 months ago by nila

I'm confident my fix will solve this particular issue.

What you experienced is the problem with python 3.7.6 (see: https://trac.osgeo.org/grass/ticket/4018). I deliberately work with 3.7.5. I may note, there will be/are a bunch of bugs ahead of us for macOS even after this issue is fixed. But we have to start somewhere.

comment:20 Changed 9 months ago by cmbarton

OK. Are you saying this only works with one version of Python: 3.7.5?

comment:21 Changed 9 months ago by nila

No, I'm saying this fix will work for this particular issue for any version of python 3.

But, there are completely different and (to this issue) independent issues introduced with python 3.7.6, affecting most if not all platforms (see the link above for issue #4018). That is why I downgraded to 3.7.5, but any pre 3.7.5 python3 version will probably do too.

comment:22 Changed 9 months ago by cmbarton

What about Python 3.6 or 3.8? I am having difficulty downgrading to 3.7..5 or 3.7.4 and keeping needed dependencies.

comment:23 Changed 9 months ago by nila

I have not tested, but any 3.6 should work and 3.8.2 as well.

comment:24 Changed 9 months ago by cmbarton

Has your PR been merged yet?

comment:25 Changed 9 months ago by cmbarton

I manually entered the new code from your PR https://github.com/OSGeo/grass/commit/9f3883f414a2f61ddd0a1a6e64bb18004abc9346 and tried compiling with Python 3.6.7. It failed to compile with the ctypes error. I guess I'll wait until all the updating is done and try again, maybe with Python 3.8.2 if I can.

comment:26 Changed 9 months ago by annakrat

I merged it now. Later should be merged to 7.8 I guess.

comment:27 Changed 9 months ago by nila

Thanks! I just tested with official python 3.8.2 (python.org) distribution too and it compiles and runs.

Yes, it should be backported to 7.8 as it is vital for macOS compilation.

comment:28 Changed 9 months ago by cmbarton

I should be able to test this tomorrow. I'll try to make a Python 3.8.2 environment in Anaconda and do it there rather than dropping back to 3.6.

comment:29 Changed 9 months ago by nila

Fix to this has been committed to master with ee75516

and backported to 7.8 with 7a9ce5d.

As I see it, this issue may be closed as fixed.

comment:30 Changed 9 months ago by annakrat

Resolution: fixed
Status: newclosed

comment:31 Changed 9 months ago by cmbarton

I can confirm that I finally got GRASS to compile and run with this fix. Great news. I did have a problem with the installation that I had to fix manually and hope I can find a way to deal with it in the build process. Maybe a result of the difficulty in setting up a build environment that does not use Python 3.7.6 (current stable) so I can create a Mac app package. I will work on it more on the coming weeks to see if I can turn out some useable new packages. Hopefully it will now be easier to finally get to some of the other annoying to serious bugs. Thanks to all who worked on this

Note: See TracTickets for help on using tickets.