problems compiling map swipe, animation, modeler, and r.li.setup

[cmbarton]

A recent problem has emerged with compiling several wxPython related modules, including map swipe, animation, modeler and r.li.setup. At least map swipe and modeler compiled a month ago. This may be primarily a Mac issue, but I don't know if there are problems on Window or not.

Here is the error (r.li.setup example):

anthgradpc7:g.version cmbarton$ cd /Users/Shared/grass_dev/grass7_dev/gui/wxpython/rlisetup
anthgradpc7:rlisetup cmbarton$ make
if [ "/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/scripts/g.gui.rlisetup" != "" ] ; then GISRC=/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/demolocation/.grassrc70 GISBASE=/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0 PATH="/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/bin:/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/bin:$PATH" PYTHONPATH="/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/etc/python:/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/etc/python:$PYTHONPATH" DYLD_LIBRARY_PATH="/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/bin:/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/lib:/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/lib:" LC_ALL=C /Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/scripts/g.gui.rlisetup --html-description < /dev/null | grep -v '</body>\|</html>' > g.gui.rlisetup.tmp.html ; fi
Traceback (most recent call last):
 File "/Users/Shared/grass_dev/grass7_dev/dist.x86_64-apple-darwin12.2.0/scripts/g.gui.rlisetup", line 32, in <module>
   import  wx
 File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/__init__.py", line 45, in <module>
   from wx._core import *
 File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 4, in <module>
   import _core_
ImportError: /usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core_.so: no appropriate 64-bit architecture (see "man python" for running in 32-bit mode)
make: *** [g.gui.rlisetup.tmp.html] Error 1
rm g.gui.rlisetup.tmp.html

This may or may not be limited to current OS X systems (10.8 and/or 10.7). No test yet on older OS X 10.6

[kyngchaos]

I'm pretty sure this is a Mac only problem. Something is not working with your external python 32bit wrapper - make sure it's in your PATH before configuration. I have no problems on 10.6 or 10.7.

[cmbarton]

OK. Will try again tomorrow or Tuesday. Hope you are right.

Michael

[cmbarton]

I did make distclean, svn up, configure and make. I double checked my 32bit python hack. It was missing for some reason, so I redid it. I still get the same error. Is there a way to check to make sure that python is in 32 bit mode in my terminal window with the 32bit script named "python" running?

Michael

[kyngchaos]

"type python" should verify which python is found first in your PATH. If you run Python in the terminal with the PATH set, Activity Monitor shows 32/64bit mode of apps. These tell you what should happen in make, but it's hard to check what really runs since python runs so quickly and quits that you probably won't see it in Activity Monitor.

[cmbarton]

Before I go changing the setup I have for compiling--which worked fine up through r53641--it would be helpful to know what changed in the GUI for mapswipe and modeler (also affecting the new animation and r.li.setup) subsequent to that release.

Michael

[wenzeslaus]

If we say that g.gui.* modules should not load anything from wx-related (and thus also anything from wxGUI) before parser() function is called and we implement #2142, these problems might disappear. However, the actual problem will remain (which is probably wrong environment set during the compilation).

Not importing non-trivial dependencies before parser() call might be useful in more cases, see:

• ​http://lists.osgeo.org/pipermail/grass-dev/2015-February/073702.html
• ​http://lists.osgeo.org/pipermail/grass-dev/2015-February/073713.html
• ​http://lists.osgeo.org/pipermail/grass-dev/2015-February/073734.html
• source:grass-addons/grass7/raster/r.flexure/r.flexure.py?rev=64624#L171​

[cmbarton]

Solving this issue may well fix other compiling issues on the Mac, as well as avoiding yet other problems down the road

Michael

[wenzeslaus]

Replying to wenzeslaus:

If we say that g.gui.* modules should not load anything from wx-related (and thus also anything from wxGUI) before parser() function is called...

Implemented in r64664 (trunk). Probably safe to backport but I would not backport it before it is tested by other people and before we know that it actually helps with the compilation on Mac.

See also comment:7:ticket:2561.

[cmbarton]

OK. I'll compile trunk now and let you know.

Michael

[cmbarton]

This solved all the compiler errors except for g.gui.tplot and the failure to make the menu tree xml files. Here is the plot error:

Errors in:
/Users/cmbarton/grass_source/grass_trunk/gui/wxpython/tplot
--
In case of errors please change into the directory with error and run 'make'.
If you get multiple errors, you need to deal with them in the order they
appear in the error log. If you get an error building a library, you will
also get errors from anything which uses the library.
--
Finished compilation: Tue Feb 17 15:29:34 MST 2015
make: *** [default] Error 1
cmb-imaccsdc:grass_trunk cmbarton$ cd /Users/cmbarton/grass_source/grass_trunk/gui/wxpython/tplot
cmb-imaccsdc:tplot cmbarton$ make
/Applications/Xcode.app/Contents/Developer/usr/bin/make /Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/docs/html/g.gui.tplot.html
GISRC=/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/demolocation/.grassrc71 GISBASE=/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0 PATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts:$PATH" PYTHONPATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/etc/python:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/gui/wxpython:$PYTHONPATH" DYLD_LIBRARY_PATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/lib:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/lib:" LC_ALL=C /Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts/g.gui.tplot --html-description < /dev/null | grep -v '</body>\|</html>' > g.gui.tplot.tmp.html
Traceback (most recent call last):
  File "/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts/g.gui.tplot", line 48, in <module>
    import  wx
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/__init__.py", line 45, in <module>
    from wx._core import *
  File "/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core.py", line 4, in <module>
    import _core_
ImportError: /usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7/site-packages/wx-2.8-mac-unicode/wx/_core_.so: no appropriate 64-bit architecture (see "man python" for running in 32-bit mode)
make[1]: *** [g.gui.tplot.tmp.html] Error 1
rm g.gui.tplot.tmp.html
make: *** [guiscript] Error 2
cmb-imaccsdc:tplot cmbarton$ 

Michael

[cmbarton]

Some more information. Let me know if this should be filed under a different ticket.

Because of the failure for some Python code to create the menu tree files during compilation, I have to launch a GRASS session, cd to the ../gui/wxpython directory, make clean, and make. This also 'fixed' the errors in animation, map swipe, etc.

The tplot error is persisting, even when I make from within a GRASS session. It has a different error from within a GRASS session however. Perhaps it will be helpful.

rm -f g.gui.*.tmp.html
/Applications/Xcode.app/Contents/Developer/usr/bin/make /Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/docs/html/wxGUI.timeline.html
make[3]: `/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/docs/html/wxGUI.timeline.html' is up to date.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C tplot || echo /Users/cmbarton/grass_source/grass_trunk/gui/wxpython/tplot >> /Users/cmbarton/grass_source/grass_trunk/error.log
/Applications/Xcode.app/Contents/Developer/usr/bin/make /Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/docs/html/g.gui.tplot.html
GISRC=/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/demolocation/.grassrc71 GISBASE=/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0 PATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts:$PATH" PYTHONPATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/etc/python:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/gui/wxpython:$PYTHONPATH" DYLD_LIBRARY_PATH="/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/bin:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/lib:/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/lib:/Applications/GRASS/GRASS-7.0.app/Contents/MacOS/lib:/Users/cmbarton/Library/GRASS/7.0/Modules/lib:/Library/GRASS/7.0/Modules/lib" LC_ALL=C /Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts/g.gui.tplot --html-description < /dev/null | grep -v '</body>\|</html>' > g.gui.tplot.tmp.html
Traceback (most recent call last):
  File "/Users/cmbarton/grass_source/grass_trunk/dist.x86_64-apple-darwin14.1.0/scripts/g.gui.tplot", line 51, in <module>
    from core.utils import GuiModuleMain
ImportError: cannot import name GuiModuleMain
make[3]: *** [g.gui.tplot.tmp.html] Error 1
rm g.gui.tplot.tmp.html
make[2]: *** [guiscript] Error 2

[wenzeslaus]

Replying to cmbarton:

This solved all the compiler errors except for g.gui.tplot

I simply forgot to change g.gui.tplot because I was looking at the list in comment:7:ticket:2561. Fixed in r64674.

Replying to cmbarton:

and the failure to make the menu tree xml files.

Replying to cmbarton:

Because of the failure for some Python code to create the menu tree files during compilation, I have to launch a GRASS session, cd to the ../gui/wxpython directory, make clean, and make. This also 'fixed' the errors in animation, map swipe, etc.

That's the part which should be fixed once #2142 is fixed. Unfortunately, I'm not able to fix it for the 7.0.0 release. It requires a lot of design decisions and testing.

And as I said, these changes make sense in general but basically they are all workarounds. The actual issue is wrong runtime environment during compilation where 64 and 32 bit libraries are mixed.

This ticket should be closed once r64664 and r64674 are backported to release branch 7.0.

[wenzeslaus]

Replying to wenzeslaus:

That's the part which should be fixed once #2142 is fixed. Unfortunately, I'm not able to fix it for the 7.0.0 release. It requires a lot of design decisions and testing.

A quick fix implemented in r64677. It is far from solving #2142 and it is quite dirty but it could solve (workaround) the compilation issue with wx, 32/64 bit and toolboxes.

And as I said, these changes make sense in general but basically they are all workarounds. The actual issue is wrong runtime environment during compilation where 64 and 32 bit libraries are mixed.

[cmbarton]

Which items are fixed in r64677? I can recompile on Thursday but need to know what to look for.

On the Mac, because of problems running wxPython for 64 bit, the compiling environment is forced to 32 bit for Python. That should not be a problem and indeed is not for all compiling except these modules, all of which use a new windowing class developed initially for map swipe. Perhaps they are trying to force the import of 64 bit wx, which is incorrect for current Mac compiling, and would conflict with the 32 bit Python environment set as default for compiling. That seems to be what is going on with the failure to correctly create the menu xml files too. The errors are the same--they cannot import wx.It acts like there is a hardwired call to Python or wxPython somewhere instead of calling it using the system default path.

[wenzeslaus]

Replying to cmbarton:

Which items are fixed in r64677? I can recompile on Thursday but need to know what to look for.

toolboxes.py is called (as a script) during compilation to build the XMLs for toolboxes. toolboxes.py before r64677 required wxPython even when executed as a script. The environment to run GRASS modules (and toolboxes.py script) during compilation is somehow incomplete. The wrong environment cases wrong wxPython/wxWidgets to load. Then import fails and the thus toolboxes.py script fails to create the XMLs.

r64677 removes the need to import wxPython when toolboxes.py is executed as a script. Thus the script does not need the build environment to be perfect. Thus it should work on Mac OS X and build the XMLs.

This is unrelated to the original topic of this ticket. This relates to XML, toolboxes and menu problem mentioned in comment:10.

On the Mac, because of problems running wxPython for 64 bit, the compiling environment is forced to 32 bit for Python. That should not be a problem and indeed is not for all compiling except these modules, all of which use a new windowing class developed initially for map swipe.

I have no idea about which class you are talking about. What the directories which were failing has in common is that they contain g.gui.* modules. All modules (r.*, v.*, ..., g.gui.*) needs to be executed during compilation to build manual pages. The issue is that g.gui.* required wxPython to be imported. Thus hitting the same issue with incomplete build environment as mentioned above. The commits r64664 and r64674 remove the need to import wxPython packages in g.gui.* when only grass.script.core.parser() is called to obtain the interface description (execution ends with parser() call when the module is asked just for the interface using --interface-description). The trick in r64664 and r64674 is that imports are done only after parser() call in the body of main() function where the packages are needed (and not using the standard import at the begging of the file).

Perhaps they are trying to force the import of 64 bit wx, which is incorrect for current Mac compiling, and would conflict with the 32 bit Python environment set as default for compiling.

They are not trying to do that, at least there is nothing in GRASS GIS source code which would indicate that (please check me). I cannot say what is in wxPython itself (do you think that there is some class?). Supposing that wxPython is all right, the compilation environment must be wrong.

That seems to be what is going on with the failure to correctly create the menu xml files too. The errors are the same--they cannot import wx.

They are indeed the same and thus r64664, r64674 and r64677 are using the same approach of avoiding global imports of wx-related packages.

It acts like there is a hardwired call to Python or wxPython somewhere instead of calling it using the system default path.

If there is something wrong in GRASS GIS wxGUI source code or source code of wxPython itself and you would find it, it would be very useful. I don't know where is the problem but I haven't seen anything like this in wxGUI source code. And wxGUI and modules are running when started from regular GRASS session. Therefore I think that the problem is in build environment, i.e. GRASS Makefiles.

[cmbarton]

Replying to wenzeslaus:

Replying to cmbarton:

This solved all the compiler errors except for g.gui.tplot

I simply forgot to change g.gui.tplot because I was looking at the list in comment:7:ticket:2561. Fixed in r64674.

Replying to cmbarton:

and the failure to make the menu tree xml files.

Replying to cmbarton:

Because of the failure for some Python code to create the menu tree files during compilation, I have to launch a GRASS session, cd to the ../gui/wxpython directory, make clean, and make. This also 'fixed' the errors in animation, map swipe, etc.

That's the part which should be fixed once #2142 is fixed. Unfortunately, I'm not able to fix it for the 7.0.0 release. It requires a lot of design decisions and testing.

And as I said, these changes make sense in general but basically they are all workarounds. The actual issue is wrong runtime environment during compilation where 64 and 32 bit libraries are mixed.

This ticket should be closed once r64664 and r64674 are backported to release branch 7.0.

The compiling errors for mapswipe, etc (including tplot) are all now fixed. The automatic menu building is not fixed.

[cmbarton]

Replying to wenzeslaus:

Replying to cmbarton:

Which items are fixed in r64677? I can recompile on Thursday but need to know what to look for.

toolboxes.py is called (as a script) during compilation to build the XMLs for toolboxes. toolboxes.py before r64677 required wxPython even when executed as a script. The environment to run GRASS modules (and toolboxes.py script) during compilation is somehow incomplete. The wrong environment cases wrong wxPython/wxWidgets to load. Then import fails and the thus toolboxes.py script fails to create the XMLs.

r64677 removes the need to import wxPython when toolboxes.py is executed as a script. Thus the script does not need the build environment to be perfect. Thus it should work on Mac OS X and build the XMLs.

This is unrelated to the original topic of this ticket. This relates to XML, toolboxes and menu problem mentioned in comment:10.

On the Mac, because of problems running wxPython for 64 bit, the compiling environment is forced to 32 bit for Python. That should not be a problem and indeed is not for all compiling except these modules, all of which use a new windowing class developed initially for map swipe.

I have no idea about which class you are talking about. What the directories which were failing has in common is that they contain g.gui.* modules. All modules (r.*, v.*, ..., g.gui.*) needs to be executed during compilation to build manual pages. The issue is that g.gui.* required wxPython to be imported. Thus hitting the same issue with incomplete build environment as mentioned above. The commits r64664 and r64674 remove the need to import wxPython packages in g.gui.* when only grass.script.core.parser() is called to obtain the interface description (execution ends with parser() call when the module is asked just for the interface using --interface-description). The trick in r64664 and r64674 is that imports are done only after parser() call in the body of main() function where the packages are needed (and not using the standard import at the begging of the file).

Perhaps they are trying to force the import of 64 bit wx, which is incorrect for current Mac compiling, and would conflict with the 32 bit Python environment set as default for compiling.

They are not trying to do that, at least there is nothing in GRASS GIS source code which would indicate that (please check me). I cannot say what is in wxPython itself (do you think that there is some class?). Supposing that wxPython is all right, the compilation environment must be wrong.

That seems to be what is going on with the failure to correctly create the menu xml files too. The errors are the same--they cannot import wx.

They are indeed the same and thus r64664, r64674 and r64677 are using the same approach of avoiding global imports of wx-related packages.

It acts like there is a hardwired call to Python or wxPython somewhere instead of calling it using the system default path.

If there is something wrong in GRASS GIS wxGUI source code or source code of wxPython itself and you would find it, it would be very useful. I don't know where is the problem but I haven't seen anything like this in wxGUI source code. And wxGUI and modules are running when started from regular GRASS session. Therefore I think that the problem is in build environment, i.e. GRASS Makefiles.

I am pretty sure that this must be either in a wxPython class created for mapswipe and reused for other modules OR something in the make/build environment used for part of the wxPython code. Or perhaps a combination of these. So far, I've tried to track it down, working with Anna but have failed so far.

I hesitate to suggest a workaround instead of a fix. But one possibility that might be helpful is to automatically run the Python code to build the menu xml files each time GRASS is launched. This would eliminate the effects of this compiler problem (thought not the source). It could also have the benefit of providing an updated menu if there are any additions or subtractions to the modules in bin or scripts.

Michael

[wenzeslaus]

Michael, can you please update what works from this ticket and from #2498 in 7.0.0 release and in trunk? I lost track what works for you. All should compile well and GUI should start without any addition work thanks to the workarounds (increased robustness in r64664, r64674 and r64677). The changes should be backported but not without knowing that they are actually solving the Mac 64bit issue.

[cmbarton]

I was able to compile GRASS 7 for the first time in a while without having to recompile from within a GRASS session to get the menu xml files to build. But I don't remember if the 'bogus' errors about mapswipe, etc still showed up. I can check my compiler log next time I'm at the office where I compiled it and let you know. But I can say that it is compelling better now.

[cmbarton]

I was able to look at the compiler logs for my 24 February builds. There were no errors. So these recent fixes made the previous errors go away it seems. I still have to do a somewhat complicated set up of the compiling environment to force it to compile in 32 bit mode for wxPython. But when this is done, I can compile with out any errors.

Michael

[wenzeslaus]

Replying to cmbarton:

I was able to look at the compiler logs for my 24 February builds. There were no errors. So these recent fixes made the previous errors go away it seems.

r64664, r64674, r64677 and r64678 backported in r64851 and r64852. Closing. (Compilation issue fixed but there is something strange related to 31-64bit issues in compilation environment.)

I still have to do a somewhat complicated set up of the compiling environment to force it to compile in 32 bit mode for wxPython. But when this is done, I can compile with out any errors.

Is the setup described at ​GRASS Wiki: Compiling on MacOSX?