MapGuide RFC 149 - Lightweight MapGuide Source Tree

This page contains a change request (RFC) for the MapGuide Open Source project. More MapGuide RFCs can be found on the RFCs page.

Status

RFC Template Version	(1.0)
Submission Date	28 May 2015
Last Modified	2 Jun 2015
Author	Jackie Ng
RFC Status	draft
Implementation Status	pending
Proposed Milestone	3.1
Assigned PSC guide(s)	(when determined)
Voting History	(vote date)
+1
+0
-0
-1
no vote

Overview

This RFC proposes to separate OEM from the MgDev source tree, with some major structural changes to the build system on Windows and Linux to reduce svn checkout footprint and to simplify/speed up the MapGuide build process

Motivation

The MapGuide OEM tree is something that rarely changes between MapGuide releases, except when a certain component needs to be upgraded to a newer version.

Having this included with the main MapGuide source adds unnecessary overheads to our build process and workflows:

• Initial checkout is *expensive*. The OEM source tree currently weighs in at 1.9GB of source code.
• Working copy operations (switch, commit, revert) also are slow due to having to process the OEM directory that never changes.
• A major fraction of the overall build time in MapGuide is spent building OEM components that do not change.

By separating OEM out of the MgDev tree, we have

• An OEM source tree that only needs to be built once per release branch of MapGuide
• OEM component updates can happen independently of the MapGuide source
• Initial checkout time of the MapGuide source is significantly reduced as it will no longer contain OEM component source. Combined with a pre-built OEM components for that particular release of MapGuide. The main build of MapGuide itself will be significantly faster, lending itself to the feasibility of having MapGuide under continuous integration and delivery whether through an in-house CI/CD system or through 3rd party services like Travis/AppVeyor/etc.
• Lends itself to allowing our OEM components to be replaced with system-installed versions when building for Linux

Solution

The majority of the Oem tree will be moved up one level to become a sibling of MgDev.

The proposed tree layout is as follows (only applicable directories shown for brevity):

/trunk
   /Oem
      /ACE
      /CppUnit-1.9.14
      /CsMap [svn external to CS-Map SVN repo]
      /dbxml
      /DWFTK
      /FDO
      /gd
      /geos
      /httpd
      /jsoncpp
      /LinuxApt
      /php
      /SQLite
      /SWIGEx
      /WebTools [moved from MgDev/BuildTools/WebTools]
   /MgDev
      /Oem
         /agg-2.4
         /fusion [svn external to fusion SVN repo]
         /fusionMG

As can be seen, not all of Oem is moved up. The reasons for keeping some directories within MgDev/Oem are:

• agg-2.4: AGG is not built as a library on Windows. Its source files are referenced by the Renderers project on Windows at the source level. Moving this out of Oem is unnecessary given is negligible size.
• fusion: Fusion is not so much an external library, but rather a MapGuide-specific application framework that happens to sit in an external repository, so moving this out is also unnecessary. It is also somewhat negligible in size.
• fusionMG: Same "external library" and size reason as fusion

The release branching strategy under this new layout is to make a branch-specific root directory and branch the respective Oem and MgDev directories from trunk into this branch-specific root directory. The respective Fusion and CS-Map external references will be updated to point to their respective release-specific branches

For example, a 3.1 release branch would look like this (again, relevant directories shown for brevity):

/branches/3.1
    /Oem [branched from /trunk/Oem]
        /CsMap [svn external pointing to release-specific branch of CS-Map]
    /MgDev [branched from /trunk/MgDev]
        /Oem
            /fusion [svn external pointing to release-specific branch of Fusion]

A new build script will be added to Oem that will build all Oem compoents and package the resulting headers/libs/dlls into a "buildpack" archive. To keep this archive as small as possible, anything that is not a header, import library or dll (so's in Linux) is omitted.

A new script will be added to Oem that will set the appropriate Oem component include/linker paths as environment variables. This will be bundled with the "buildpack" archive. Such buildpack archives can be made available for download so other developers can go straight to building the MapGuide source by using the OEM buildpack instead of having to checkout/build their own OEM source.

The existing Server/Web/Desktop projects will be modified to use our Oem component environment variables.

On Linux, the main configure script will be modified to accept individual --with options for specific oem component paths, along with a general --with-oem option which when pointed to the root path of an oem buildpack will automatically set the paths of all required oem components. All existing Makefile.am files will be modified to use parameterized include/linker paths based on the new OEM component variables defined by the configure script.

As mentioned above, agg is kept inside MgDev/Oem as it is source-referenced by the Renderers project on Windows. On Linux however, agg is compiled as a library and linked by Renderers. As a result, a prebuild.sh script is included in the MgDev root to build agg and Fusion for Linux. prebuild.sh accepts the same parameters as build_oem.sh. build_oem.sh will move to the new Oem location. It is expected that prebuild.sh is executed for the desired configuration/platform before the main configure script is run.

Where possible and convenient, we no longer keep binaries of key dependencies and opt for a download-and-cache approach. Such dependencies include:

• PHP: If required we can download the official distribution from the PHP windows download site.
• Apache HTTPD: This is an apachelounge distribution, but apachelounge do not maintain download archives. We will use our OSGeo download location as a local mirror for this purpose.

This will reduce the checkout footprint of the installer components.

The new build workflows that are possible under this scheme are outlined as follows

Workflow 1: Full build from fresh checkout (headless)

This is the longest as it requires a full checkout and build of both MapGuide and Oem (ie. What we currently do)

1. Checkout $SVN_ROOT/MgDev to $MG_SRC
2. Checkout $SVN_ROOT/Oem to $MG_OEM_SRC
3. Checkout $SVN_ROOT/Installer to $MG_INST_SRC
4. Run $MG_OEM_SRC/build.bat to compile the Oem tree for $ARCH and $CONFIG, which produces a mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive
5. Extract mapguide-buildpack-$VERSION-$ARCH-$PLATFORM archive to $MG_OEM_LIBS
6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables
7. Run $MG_SRC/setenviroment[64].bat to set additional environment variables
8. Run $MG_SRC/build.bat to build to the MapGuide source
9. Run $MG_SRC/build.bat -a=install to build the MapGuide source and copy binaries to $MG_STAGING_AREA
10. Run $MG_INST_SRC/build.bat -a=prepare to add additional binaries to $MG_STAGING_AREA
11. Run $MG_INST_SRC/build.bat -a=generate to generate the installer file listing $MG_STAGING_AREA
12. Run $MG_INST_SRC/build.bat to build the windows installer using the files from $MG_STAGING_AREA

Workflow 2: IDE build/debug from fresh checkout

1. Checkout $SVN_ROOT/MgDev to $MG_SRC
2. Checkout $SVN_ROOT/Oem to $MG_OEM_SRC
3. Checkout $SVN_ROOT/Installer to $MG_INST_SRC
4. Run $MG_OEM_SRC/build.bat to compile the Oem tree for $ARCH and $CONFIG, which produces a mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive
5. Extract mapguide-buildpack-$VERSION-$ARCH-$PLATFORM archive to $MG_OEM_LIBS
6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables
7. Run $MG_SRC/setenviroment[64].bat to set additional environment variables
8. Run Visual Studio (devenv) from that same command line session
9. Open server solution and build/debug in $CONFIG|$ARCH
10. Open web solution and build/debug in $CONFIG|$ARCH

Workflow 3: IDE build/debug with pre-built OEM components

Given a pre-existing mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive and a FDO binary SDK of the correct bitness

1. Checkout $SVN_ROOT/MgDev to $MG_SRC
2. Extract mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive to $MG_OEM_LIBS
3. Extract a pre-existing FDO binary SDK into the FDO directory in $MG_OEM_LIBS
4. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables
5. Run $MG_SRC/setenviroment[64].bat to set additional environment variables
6. Run Visual Studio (devenv) from that same command line session
7. Open server solution and build/debug in $CONFIG|$ARCH
8. Open web solution and build/debug in $CONFIG|$ARCH

Benefits gained:

• No OEM checkout overhead
• No OEM build overhead as the buildpack archive already contains the headers/libs/dlls
• $MG_OEM_LIBS can stay for the duration of needing to build that particular release branch. It only needs to be replaced if building for a different release branch of MapGuide that has different required OEM component versions

Workflow 4: headless build with pre-built OEM components

Given a pre-existing mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive and a FDO binary SDK of the correct bitness

1. Checkout $SVN_ROOT/MgDev to $MG_SRC
2. Extract mapguide-buildpack-$VERSION-$ARCH-$CONFIG archive to $MG_OEM_LIBS
3. Extract a pre-existing FDO binary SDK into the FDO directory in $MG_OEM_LIBS
4. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables
5. Run $MG_SRC/setenviroment[64].bat to set additional environment variables
6. Run $MG_OEM_LIBS/setenvironment $ARCH $CONFIG to set OEM component environment variables
7. Run $MG_SRC/build.bat to build to the MapGuide source
8. Run $MG_SRC/build.bat -a=install to build the MapGuide source and copy binaries to $MG_STAGING_AREA
9. Run $MG_INST_SRC/build.bat -a=prepare to add additional binaries to $MG_STAGING_AREA
10. Run $MG_INST_SRC/build.bat -a=generate to generate the installer file listing $MG_STAGING_AREA
11. Run $MG_INST_SRC/build.bat to build the windows installer using the files from $MG_STAGING_AREA

Benefits gained:

• No OEM checkout overhead
• No OEM build overhead as the buildpack archive already contains the headers/libs/dlls
• $MG_OEM_LIBS can stay for the duration of needing to build that particular release branch. It only needs to be replaced if building for a different release branch of MapGuide that has different required OEM component versions

This workflow makes continuous integration feasible as it avoids the significant build/checkout overhead of having to build OEM components

Implications

This change will significantly change current developer workflows. Refer to any of the example workflows above that match yours and adjust as necessary.

The Visual Studio IDE is tied to the environment variables set by the buildpack setenvironment.bat. It is not possible to switch Platform/Configuration and build. Switching Platform/Configuration would require exiting Visual Studio, invoking setenvironment.bat on the appropriate buildpack for the given Platform/Configuration, restart Visual Studio under the new environment variables, set Platform/Configuration to match and then build.

Care must also be taken with regards to future OEM component changes/updates. Any such changes should ensure that the buildpacks produced by the OEM build scripts will contain the necessary headers and libraries of your updated OEM components.

Although this RFC enables the possibility of replacing our OEM components with system-installed versions on Linux, the scope of this RFC will only extend to the point that the Linux build process can build off of a pre-existing OEM buildpack (and running a configure script --with-oem pointing to this buildpack location)

Test Plan

Ensure existing test suites pass under this new build workflow

Funding / Resources

Community