GRASS master does not compile with Python 3 on Mac

[cmbarton]

The problem seems to be with ctypes. I am attaching error and build logs (thanks to Anna Petrasova). I am using the most current Mac build environment (OS X 10.14.6 and Xcode 10.3 with command line tools). May be an nviz issue based on ctype build errors.

[cmbarton]

ctype build output and errors

[neteler]

I see in your log eg.:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/Headers/ATSTypes.h:24:10: fatal error: 'CarbonCore/CarbonCore.h' file not found
#include <CarbonCore/CarbonCore.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~

Does the file exist anywhere on your machine?

[cmbarton]

Replying to neteler:

I see in your log eg.:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/ApplicationServices.framework/Frameworks/ATS.framework/Headers/ATSTypes.h:24:10: fatal error: 'CarbonCore/CarbonCore.h' file not found
#include <CarbonCore/CarbonCore.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~

Does the file exist anywhere on your machine?

I checked. CarbonCore.h is NOT included in the most recent Mac OSX SDK for v. 10.14. However, it IS included in earlier versions, which I have.

/MacOSX10.13.sdk has CarbonCore.h but does not have some other frameworks that nviz wants--maybe because Apple dropped default support for OpenGL. However, everything needed seems available in /MacOSX10.12.sdk on back, which I also have.

In the configure string, I can specify --with-macosx-sdk=/Developer/SDKs/MacOSX10.12.sdk.

BUT ctypes appears to ignore this and I get the same error that you spotted. It is looking in the default SDK for 10.14 rather than in the one I pointed to in configure.

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[cmbarton]

So what to do? This is not something I know how to solve. Will GRASS drop support for the Mac?

[hcho]

I think the problem is not many developers have a Mac? Personally, I'd like to support the Mac, but I cannot afford (or don't want?) to buy one just for compiling GRASS.

[neteler]

Ticket retargeted after milestone closed

[guano]

I'm facing similar issues. Compiling on OSX 10.14 Mojave, with python 3.7.4 (pyenv). I'm not using the Mac SDK, just the Mac Command Line Tools. Compilation breaks with this error:

Errors in:
/Volumes/MacintoshHD2/grass/lib/python/ctypes
/Volumes/MacintoshHD2/grass/raster/r.in.lidar
/Volumes/MacintoshHD2/grass/raster3d/r3.in.lidar
/Volumes/MacintoshHD2/grass/vector/v.out.lidar
/Volumes/MacintoshHD2/grass/vector/v.in.lidar
--
In case of errors please change into the directory with error and run 'make'.
If you get multiple errors, you need to deal with them in the order they
appear in the error log. If you get an error building a library, you will
also get errors from anything which uses the library.
--
Finished compilation: Tue Jan 21 12:42:28 -03 2020
make: *** [default] Error 1

So I cd into ctypes and run make:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 173, in parse
    ppout, pperr = pp.communicate()
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 1721, in _communicate
    self.stdout.errors)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 816, in _translate_newlines
    data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 1400362: invalid start byte
make[1]: *** [OBJ.x86_64-apple-darwin18.7.0/nviz.py] Error 1
make: *** [default] Error 2

So is this NVIZ-related?

[cmbarton]

Replying to guano:

I'm facing similar issues. Compiling on OSX 10.14 Mojave, with python 3.7.4 (pyenv). I'm not using the Mac SDK, just the Mac Command Line Tools. Compilation breaks with this error:

Errors in:
/Volumes/MacintoshHD2/grass/lib/python/ctypes
/Volumes/MacintoshHD2/grass/raster/r.in.lidar
/Volumes/MacintoshHD2/grass/raster3d/r3.in.lidar
/Volumes/MacintoshHD2/grass/vector/v.out.lidar
/Volumes/MacintoshHD2/grass/vector/v.in.lidar
--
In case of errors please change into the directory with error and run 'make'.
If you get multiple errors, you need to deal with them in the order they
appear in the error log. If you get an error building a library, you will
also get errors from anything which uses the library.
--
Finished compilation: Tue Jan 21 12:42:28 -03 2020
make: *** [default] Error 1

So I cd into ctypes and run make:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/Volumes/MacintoshHD2/grass/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 173, in parse
    ppout, pperr = pp.communicate()
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 1721, in _communicate
    self.stdout.errors)
  File "/Users/guano/.pyenv/versions/3.7.4/lib/python3.7/subprocess.py", line 816, in _translate_newlines
    data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 1400362: invalid start byte
make[1]: *** [OBJ.x86_64-apple-darwin18.7.0/nviz.py] Error 1
make: *** [default] Error 2

So is this NVIZ-related?

Maybe but I don't think so. GRASS compiled under Mohave until August/September 2019. Then this error popped up. So I don't think it is a change in the Mac OS, but something in the GRASS code.

[guano]

Searching for this error I found it to be quite common with python2 code being ported to python3, as with python3 all strings are unicode.

>>> b'\x9c'.decode('cp1252')
'œ'
>>> b'\x9c'.decode('utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte

[nila]

While not getting to the root of the problem, there is a workaround that seem to work:

adding encoding='latin1' to subprocess.Popen at (lib/python/ctypes/ctypesgencore/parser/preprocessor.py:170)

enables compilation, seemingly without issues.

---
 lib/python/ctypes/ctypesgencore/parser/preprocessor.py | 1 +
 1 file changed, 1 insertion(+)
 mode change 100644 => 100755 lib/python/ctypes/ctypesgencore/parser/preprocessor.py

diff --git a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
old mode 100644
new mode 100755
index 18999483e..1d0dd8749
--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -168,6 +168,7 @@ class PreprocessorParser(object):
         pp = subprocess.Popen(cmd,
                               shell=True,
                               universal_newlines=True,
+                              encoding='latin1',
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
-- 
2.23.0

[guano]

I can confirm that compilation was successfull with the modification. (OS Mojave, python 3.7.4 pyenv)

[cmbarton]

Replying to nila:

This is great news. I'll try it out.

While not getting to the root of the problem, there is a workaround that seem to work:

adding encoding='latin1' to subprocess.Popen at (lib/python/ctypes/ctypesgencore/parser/preprocessor.py:170)

enables compilation, seemingly without issues.

---
 lib/python/ctypes/ctypesgencore/parser/preprocessor.py | 1 +
 1 file changed, 1 insertion(+)
 mode change 100644 => 100755 lib/python/ctypes/ctypesgencore/parser/preprocessor.py

diff --git a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
old mode 100644
new mode 100755
index 18999483e..1d0dd8749
--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -168,6 +168,7 @@ class PreprocessorParser(object):
         pp = subprocess.Popen(cmd,
                               shell=True,
                               universal_newlines=True,
+                              encoding='latin1',
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
-- 
2.23.0

[nila]

I believe I have tracked down the bad one. There is a (one) macOS system file encoded with Mac Roman (mac_roman):

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk\
/System/Library/Frameworks/ApplicationServices.framework/Frameworks/HIServices.framework/Headers/InternetConfig.h

being parsed. Remains to be seen how this can be fixed...

[annakrat]

I would say, the better way to workaround it is to ignore the decoding errors for stderr:

--- a/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
+++ b/lib/python/ctypes/ctypesgencore/parser/preprocessor.py
@@ -162,15 +162,18 @@ class PreprocessorParser(object):
 
         self.cparser.handle_status(cmd)
 
+        cmd = cmd.encode()
+
         if sys.platform == 'win32':
-            cmd = ['sh.exe', '-c', cmd]
+            cmd = [b'sh.exe', b'-c', cmd]
 
         pp = subprocess.Popen(cmd,
                               shell=True,
-                              universal_newlines=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)
         ppout, pperr = pp.communicate()
+        ppout = ppout.decode()
+        pperr = pperr.decode(errors='replace')
 
         for line in pperr.split("\n"):
             if line:

Alternatively, just pass errors='replace' to Popen, but that is not available in Python 3.5

[nila]

Thanks @annakrat! I put up a PR ​https://github.com/OSGeo/grass/pull/385, following roughly what you proposed.

[cmbarton]

I just tried Nila's fix and it does compile. Anna's does not. Unfortunately, even though the first fix compiles successfully, the GRASS GUI will not launch. Here is the error:

Launching <wxpython> GUI in the background, please wait...
GRASS 7.9.dev (nc_spm_08_grass7):~ > Traceback (most recent call last):
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/wxgui.py", line 105, in OnInit
    from lmgr.frame import GMFrame
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/lmgr/frame.py", line 51, in <module>
    from lmgr.layertree import LayerTree, LMIcons
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/lmgr/layertree.py", line 38, in <module>
    from mapdisp.frame import MapFrame
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/mapdisp/frame.py", line 33, in <module>
    from mapdisp.toolbars import MapToolbar, NvizIcons
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/mapdisp/toolbars.py", line 22, in <module>
    from nviz.main import haveNviz
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/main.py", line 24, in <module>
    from nviz import mapwindow
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/mapwindow.py", line 42, in <module>
    from nviz.workspace import NvizSettings
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/workspace.py", line 23, in <module>
    from nviz import wxnviz
  File "/Applications/GRASS-7.9.app/Contents/Resources/gui/wxpython/nviz/wxnviz.py", line 32, in <module>
    from numpy import matrix
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/core/__init__.py", line 107, in <module>
    from . import _internal
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/site-packages/numpy/core/_internal.py", line 20, in <module>
    IS_PYPY = platform.python_implementation() == 'PyPy'
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/platform.py", line 1264, in python_implementation
    return _sys_version()[0]
  File "/Applications/GRASS-7.9.app/Contents/Resources/lib/python3.7/platform.py", line 1225, in _sys_version
    repr(sys_version))
ValueError: failed to parse CPython sys.version: '3.7.0 (default, Jun 28 2018, 07:39:16) \n[Clang 4.0.1 (tags/RELEASE_401/final)]'
OnInit returned false, exiting...

I'm using OSX Mojave 10.14.6 with python 3.7.6 I'm compiling from master source updated in the past hour or so.

v.in.pdal also does not compile. But that is another issue.

[nila]

I'm confident my fix will solve this particular issue.

What you experienced is the problem with python 3.7.6 (see: ​https://trac.osgeo.org/grass/ticket/4018). I deliberately work with 3.7.5. I may note, there will be/are a bunch of bugs ahead of us for macOS even after this issue is fixed. But we have to start somewhere.

[cmbarton]

OK. Are you saying this only works with one version of Python: 3.7.5?

[nila]

No, I'm saying this fix will work for this particular issue for any version of python 3.

But, there are completely different and (to this issue) independent issues introduced with python 3.7.6, affecting most if not all platforms (see the link above for issue #4018). That is why I downgraded to 3.7.5, but any pre 3.7.5 python3 version will probably do too.

[cmbarton]

What about Python 3.6 or 3.8? I am having difficulty downgrading to 3.7..5 or 3.7.4 and keeping needed dependencies.

[nila]

I have not tested, but any 3.6 should work and 3.8.2 as well.

[cmbarton]

Has your PR been merged yet?

[cmbarton]

I manually entered the new code from your PR ​https://github.com/OSGeo/grass/commit/9f3883f414a2f61ddd0a1a6e64bb18004abc9346 and tried compiling with Python 3.6.7. It failed to compile with the ctypes error. I guess I'll wait until all the updating is done and try again, maybe with Python 3.8.2 if I can.

[annakrat]

I merged it now. Later should be merged to 7.8 I guess.

[nila]

Thanks! I just tested with official python 3.8.2 (python.org) distribution too and it compiles and runs.

Yes, it should be backported to 7.8 as it is vital for macOS compilation.

[cmbarton]

I should be able to test this tomorrow. I'll try to make a Python 3.8.2 environment in Anaconda and do it there rather than dropping back to 3.6.

[nila]

Fix to this has been committed to master with ​ee75516

and backported to 7.8 with ​7a9ce5d.

As I see it, this issue may be closed as fixed.

[cmbarton]

I can confirm that I finally got GRASS to compile and run with this fix. Great news. I did have a problem with the installation that I had to fix manually and hope I can find a way to deal with it in the build process. Maybe a result of the difficulty in setting up a build environment that does not use Python 3.7.6 (current stable) so I can create a Mac app package. I will work on it more on the coming weeks to see if I can turn out some useable new packages. Hopefully it will now be easier to finally get to some of the other annoying to serious bugs. Thanks to all who worked on this