Framework builds for OSX

[hobu]

Time to straighten out my notes and get this going.  These are ideas on what to do for a GDAL framework, and a few specific patches (but not comprehensive).  This will probably be a bit disjointed (and I noted a couple bugs that I should get off my butt and report).  My plan is not just a purely OSX framework, but something that can be used by unix software without changes to that software, thus there are some additions to the standard framework structure.

- Basic structure (* unix compatibility additions):

GDAL.framework
    GDAL -> Versions/Current/GDAL
    Headers -> Versions/Current/Headers
    Programs -> Versions/Current/Programs
    Resources -> Versions/Current/Resources
    unix * -> Versions/Current/unix
    Versions
        [M.m]
            GDAL [=libgdal.dylib]
            Headers [=include]
            Programs [=bin]
            Resources [=share]
            unix *
                bin -> ../Programs
                include -> ../Headers
                lib
                    libgdal.dylib -> ../../GDAL
        Current -> [M.m]

(I went with unix versions with Major.minor, so currently it would be 1.3.  Apple generally uses a default A, B, C...)

In my current GDAL framework I also have a Libraries folder for all the support dependencies that aren't in the system or other frameworks.  This isn't necessary for a framework build, it's up to the user to decide what to do with those.

- shared-only - no such thing as a static framework.

- plugins - a framework should not be changed to extend it, ie with plugins.  Plugins should be installed externally to the framework.  So I decided to put plugins in the standard Application Support folders for OS X: /Library/Application Support/GDAL/PlugIns and ~/Library/Application Support/GDAL/PlugIns.  This required patching the plugin loading to add these as default locations.  I conditionalized it on MACOSX_FRAMEWORK.  Of course a user can still add to that in the usual way.

Affected files: gcore/gdaldrivermanager.cpp, ogr/ogrsf_frmts/generic/ogrsfdriverregistrar.cpp

NOTE: bug (non-framework) in ogrsfdriverregistrar.cpp: the home plugin searchpath should be 'lib/gdalplugins', not 'lib'.

- configure - add --enable-macosx-framework, which should do:

I'm conditionalizing framework stuff on MACOSX_FRAMEWORK, so this should be set.  Since it's used in both code and the makefile, it needs to be added to GDALmake.opt.in and cpl_config.h.in.  I have patches for those bits, but not the configure setup (I just hacked configure, I don't know how to change configure.in).

Also, the install prefix for frameworks is fixed, in /Library/Frameworks/GDAL.framework/Versions/M.m, though it's a bit more complicated because the framework base is just /Library/Frameworks/GDAL.framework, and the symlinks must be setup there.  Plus there is the unix compatibility version of prefix.

Library install_name and GDAL_PREFIX should be the full, versioned prefix.  But prefix in gdal_config should be the unix compatibility prefix.  I used two vars for the FW prefixes: 

fwprefix = /Library/Frameworks/GDAL.framework
fwverprefix = ${fwprefix}/Versions/$(GDAL_MMVER)

and a major.minor version var is needed for this (GDAL_MMVER).  Then set the INST_* as (override base --*dir= options and defaults):

INST_PREFIX = ${fwprefix}
INST_INCLUDE = ${fwverprefix}/Headers
INST_DATA = ${fwverprefix}/Resources/gdal
INST_LIB = ${fwverprefix}
INST_BIN = ${fwverprefix}/Programs
INST_DOCS = ${fwverprefix}/Resources/doc
INST_MAN = ${fwverprefix}/Resources/man

NOTE: bug/wish (non-framework) - currently in gdalmake-opt.in INST_DOCS = @exec_prefix@/doc, this makes it impossible to override the doc dir and forces it outside the share dir (OSX docs [man/doc/info] are normally inside share).  Configure needs a --with-docdir option.

The normal prefix for the build should be the versioned FW prefix.

- gdal_config - framework use should never need to look at gdal_config.  gdal_config is for a normal unix build.  So, it needs the unix prefix, and have -lgdal and -I and -L flags pointing to the unix folder.  But this shouldn't interfere with building the framework with the framework prefix.  So basically, set prefix here as /Library/Frameworks/GDAL.framework/Versions/M.m/unix.

- makefile - setup framework dirs, install and rename lib.

install-actions target: create framework folder structure.

install-lib target: completely separate install procedure for framework - bypass libtool because lib symlinks and .la not needed, and lib is renamed.


NOTE: bug - install-docs target is missing $(DESTDIR) on copying html files:

-cp html/*.* $(INST_DOCS)/gdal
+cp html/*.* $(DESTDIR)$(INST_DOCS)/gdal


- headers - I was worried that a major change to GDAL was needed to reorganize the headers.  Happily, no changes needed.  In case you don't know, locating headers in OSX frameworks involves a bit of CPP magic.  There are builtin framework dirs and these are searched automatically (new ones are added with -F flag).  Without adding to the include search path, when one specifies <framework_name/header.h>, CPP looks in the framework's Headers folder.  So "#include <GDAL/gdal.h>" will find gdal.h without adding any -I flags.  And with the rest of the GDAL headers using the quoted include form, they are also found.


Patch files:

Here are patches for the sources and makefile, mentioned above.  I don't have a patch for configure.in or other autoconfig stuff, so you'll have to figure out what's needed as mentioned above.  They shouldn't get in the way of the current build, and will be dormant until the configure stuff is added.

For the makefile, install part, I had to bypass the libtool install because: I don't want the .la file, I don't want the symlinks, and I do want to rename it on install.  If there are libtool install options to do these, that would make it simpler.


Ask questions.  I probably missed something.  Or mis-described something...

[hobu]

Zip of original patches

[hobu]

William,

I have committed these changes with some modifications.  To build a Framework build, use --with-macosx-framework.  This will only do stuff when uname |grep Darwin returns something ;)

Additionally, I was able to link the python modules against this framework to verify that it works.  I will probably patch the setup.py file in the next-gen bindings to take advantage of this build option with some user input.

[kyngchaos@…]

First problem - configure doesn't set MACOSX_FRAMEWORK in confdefs.h, just for gdalmake.opt.

cat >>confdefs.h <<_ACEOF
#define MACOSX_FRAMEWORK 1
_ACEOF

should be added to the enabled block in configure.

(an odd bit that doesn't seem to affect the framework test - there is a bogus empty --with- check between the darwin test and the framework test.)

I see you left out the driver/file finding patches.  Those need to be added, since the directory structure is so different in a framework.

Installation needs some work.  Partly my fault, since I could only figure out a hack around the libtool install.  I need to think about this more, and I'll follow up later.  One thing is that the prefix should probably be forced to the framework path, and maybe even have a separate block for the various INST_* variables so the correct subfolders are used.


As a side note, I was thinking an Xcode project would be better, but there are so many configurable and optional things to handle.  I recently worked out a configuration setup for the OSSIM Xcode project that might work for a GDAL Xcode project.  It would make it MUCH easier to deal with libtool install weirdness.  I'll take a look at the feasability of this, now that I'm starting to get the hang of Xcode.

Another option for libtool is to work on making the non-libtool build work for OSX.  That would make installation (and part of the build process) easier to manage.

[hobu]

I'll work on this some more tonight.  Any chance you can get on IRC on #gdal to hack on it with me?

(In reply to comment #3)
> First problem - configure doesn't set MACOSX_FRAMEWORK in confdefs.h, just for
> gdalmake.opt.
> 
> cat >>confdefs.h <<_ACEOF
> #define MACOSX_FRAMEWORK 1
> _ACEOF
> 
> should be added to the enabled block in configure.
> 
> (an odd bit that doesn't seem to affect the framework test - there is a bogus
> empty --with- check between the darwin test and the framework test.)
> 

oops

> I see you left out the driver/file finding patches.  Those need to be added,
> since the directory structure is so different in a framework.

Part of the problem is that Frank didn't want to propagate the GetHome bits from GDAL into OGR.  They aren't fully baked yet, and we're not sure where exactly it should go (if at all).  The first somewhat compelling use case for looking up home directories to find dynamic drivers is the framework builds, I think. 

> Installation needs some work.  Partly my fault, since I could only figure out a
> hack around the libtool install.  I need to think about this more, and I'll
> follow up later.  One thing is that the prefix should probably be forced to the
> framework path, and maybe even have a separate block for the various INST_*
> variables so the correct subfolders are used.
> 
> 
> As a side note, I was thinking an Xcode project would be better, but there are
> so many configurable and optional things to handle.  I recently worked out a
> configuration setup for the OSSIM Xcode project that might work for a GDAL
> Xcode project.  It would make it MUCH easier to deal with libtool install
> weirdness.  I'll take a look at the feasability of this, now that I'm starting
> to get the hang of Xcode.

GDAL is fairly anti-project files as far as MSVC and XCode goes.  They're nice, but they're really hard to maintain in parallel to everything else, and the only thing that's guaranteed to be maintained is the makefiles (and tested with the buildbot, which is another crucial bit).


 
> Another option for libtool is to work on making the non-libtool build work for
> OSX.  That would make installation (and part of the build process) easier to
> manage.
> 

Dos the non-libtool build currently not work for OSX?  I haven't tried in a while, but I thought --without-libtool was the way to build GDAL if you wanted a  python module that worked....  maybe things have changed.

[kyngchaos@…]

(In reply to comment #4)
> I'll work on this some more tonight.  Any chance you can get on IRC on #gdal to
> hack on it with me?

I don't do IRC.  Tried many times, not for me.  I have AIM for occassional chatting.

> > I see you left out the driver/file finding patches.  Those need to be added,
> > since the directory structure is so different in a framework.
> 
> Part of the problem is that Frank didn't want to propagate the GetHome bits
> from GDAL into OGR.  They aren't fully baked yet, and we're not sure where
> exactly it should go (if at all).  The first somewhat compelling use case for
> looking up home directories to find dynamic drivers is the framework builds, I
> think. 

The Home parts can be left out.  But the others are for builtin plugin locations in the framework and addon (App Support).

> GDAL is fairly anti-project files as far as MSVC and XCode goes.  They're nice,
> but they're really hard to maintain in parallel to everything else, and the
> only thing that's guaranteed to be maintained is the makefiles (and tested with
> the buildbot, which is another crucial bit).

Does the builtbot do OSX?  Even if it did, a framework build might not even be buildbot-friendly.  I'll still take a look at Xcode - good practice, and there might be good parallel maintenance features there.

> Dos the non-libtool build currently not work for OSX?  I haven't tried in a
> while, but I thought --without-libtool was the way to build GDAL if you wanted
> a  python module that worked....  maybe things have changed.

Last time I tried it (month or 2 ago), it gave me .so linux stuff, and the compile and link flags were all wrong.  I didn't see any flexibility in configure or the makefile for anything but Linux.

[hobu]

(In reply to comment #5)
> (In reply to comment #4)
> > I'll work on this some more tonight.  Any chance you can get on IRC on #gdal to
> > hack on it with me?
> 
> I don't do IRC.  Tried many times, not for me.  I have AIM for occassional
> chatting.

My AIM handle is HCButlerIA, but I won't be on at my desk until sometime after 6pm Central.

> 
> > > I see you left out the driver/file finding patches.  Those need to be added,
> > > since the directory structure is so different in a framework.
> > 
> > Part of the problem is that Frank didn't want to propagate the GetHome bits
> > from GDAL into OGR.  They aren't fully baked yet, and we're not sure where
> > exactly it should go (if at all).  The first somewhat compelling use case for
> > looking up home directories to find dynamic drivers is the framework builds, I
> > think. 
> 
> The Home parts can be left out.  But the others are for builtin plugin
> locations in the framework and addon (App Support).

Hmm... I thought I did the hardlink stuff... Maybe I forgot to commit this. Will check when I get home.

> > GDAL is fairly anti-project files as far as MSVC and XCode goes.  They're nice,
> > but they're really hard to maintain in parallel to everything else, and the
> > only thing that's guaranteed to be maintained is the makefiles (and tested with
> > the buildbot, which is another crucial bit).
> 
> Does the builtbot do OSX?  Even if it did, a framework build might not even be
> buildbot-friendly.  I'll still take a look at Xcode - good practice, and there
> might be good parallel maintenance features there.
> 

Buildbot does build on my PPC buildslave.  http://buildbot.osgeo.org or http://gdal.builds.hobu.net/ It isn't up right now, but will be soon.

> > Dos the non-libtool build currently not work for OSX?  I haven't tried in a
> > while, but I thought --without-libtool was the way to build GDAL if you wanted
> > a  python module that worked....  maybe things have changed.
> 
> Last time I tried it (month or 2 ago), it gave me .so linux stuff, and the
> compile and link flags were all wrong.  I didn't see any flexibility in
> configure or the makefile for anything but Linux.

I'll see what it takes to clean this up a bit more.  I reworked the SO_NAME macros in MapServer to do the right thing on OSX.  Maybe we just need to port them over to GDAL. 

[kyngchaos@…]

(In reply to comment #6)
> My AIM handle is HCButlerIA, but I won't be on at my desk until sometime after
> 6pm Central.

kyngchaos (of course), I usually forget to login or I'm distracted.  I can try to remember.  I'm on central time also, after 7pm.

> Buildbot does build on my PPC buildslave.  http://buildbot.osgeo.org or
> http://gdal.builds.hobu.net/ It isn't up right now, but will be soon.

There might even be a CLI interface to Xcode (just a guess) that could make an automated build possible.

[kyngchaos@…]

additional patches

[kyngchaos@…]

After our chat, I have a couple more things I missed in the original batch of patches.  Both cpl_csv.cpp and cpl_findfile.cpp have some paths that need framework conditionals.  Patches included.

Also, in the gdal/ogr driver managers, the App Supp plugins dir should be outside the GDAL_PREFIX block - it should be included regardless of whether GDAL_PREFIX is set or not. Patches included.

One last item, the main gnumakefile, install-actions target.  The mv command I added should have a -f flag, just in case it is installed over an existing framework.  No patch included, should be simple enough (I'm lazy ^_^).

[kyngchaos@…]

Looks good now.  I can't think of anything else needed right now except to think of a better way to work around the libtool install (cleaning up unneeded symlinks and embedded library paths).  If the non-libtool build is fixed to work on OSX, that's one option.  I'll putter around with the Xcode project idea.  Maybe there are more precise options for libtool to control library build names and install mechanics.

Documenting the new option on the build and install instructions page would be a good idea.

I'll do some more testing (kinda relaxed) to see if there is any other problems.  And see if others find anything.

[kyngchaos@…]

Argh!  I just found a problem - in cnfigure, the framework option sets prefix to ${OSX_VERSION_FRAMEWORK_PREFIX}, which expands and works fine in the makefiles.  But later in configure, prefix is used to set GDAL_PREFIX in cpl_config.h, and that is not expanded.  All the plugin and other data paths in the binary don't get set right.

So, the framework prefix must be fully expanded in configure.  In configure.in, change the Mac OS X prefix to:

dnl MacOS X Framework Build
...
    prefix="/Library/Frameworks/GDAL.framework/Versions/${GDAL_VERSION_MAJOR}.${GDAL_VERSION_MINOR}"

That will get the makefile prefix and cpl_config GDAL_PREFIX both correct.

[hobu]

William,

Can we close this one and declare ourselves good for 1.4?

Howard

[kyngchaos@…]

If you have done my change from msg 11, it should be good.  It wasn't in the last snapshot I have from Nov 16, and the maptools downloads aren't responding right now, so I can't check today's snapshot.

[hobu]

Closing this one down then.