Using Git to maintain GDAL workflow

This article describes how to maintain GDAL development workflow with  Git against GDAL Subversion repository.

Git repository setup using svn2git

 svn2git is a tool written in Ruby which makes it easy to mirror SVN repository in Git, especially if one wants to mirror complete SVN repository based on the standard layout with: turnk, tags and branches. svn2git also makes it easy to update the Git mirror pulling latest changes from SVN.

Here is example of initial import of GDAL repository from SVN to Git:

$ svn2git http://svn.osgeo.org/gdal/ --exclude sandbox --exclude spike

It will import content of trunk, tags and branches.

Next, one may want to frequently update the Git repository pulling latest changes from SVN:

$ svn2git --rebase

The local repository can be published on remote server like  GitHub this way:

$ git remote add origin git@github.com:GITHUB_USERNAME/REPO_NAME.git
$ git push origin master

Then, it is ready to keep the Git mirror in remote location (e.g.  GitHub) updated:

$ svn2git --rebase
$ git push origin master

Git repository setup using pygit-svn-mirror

 pygit-svn-mirror is Python port of Ruby tool  git-svn-mirror. It allows to create and maintain mirror of SVN repository through Git bare repository:

Creating mirror of Subversion repository of PROJ.4 project at GitHub? involves the following commands:

$ mkdir /path/to/proj4/mirror
$ cd /path/to/proj4/mirror
$ git-svn-mirror.py init --from=https://svn.osgeo.org/metacrs/proj/ --to=git@github.com:<USRNAME>/proj.4.git

Then, update (synchronise) the mirror from its workbench directory:

cd /path/to/proj4/mirror
git-svn-mirror.py update

or from any folder but with workbench location pointed explicitly:

git-svn-mirror.py update -w /path/to/proj4/mirror

Git repository setup using git-svn

Note, this is not a complete guide of how to use git-svn, but a quick list of basic operations. Requirements:  git and  git-svn

First, initialize local Git repository for copy of trunk form Subversion:

$ git svn init --prefix=svn --trunk https://svn.osgeo.org/gdal/trunk/

The option --trunk explicitly specifies intention of this operation. We use --prefix so remote svn branches are named eg. 'svn/trunk' rather than 'trunk', since otherwise you get unhelpful name clashes when doing cross-branch operations.

It is also possible to copy complete repository, including trunk, branches and tags modules. Learn about --stdlayout option dedicated for this purpose. However, note that this initial fetch is a time consuming process. (fetching of SVN trunk takes nearly one hour).

Next, fetch complete copy of GDAL trunk:

$ git svn --authors-file=/path/to/gdal-git-authors.txt fetch

The file gdal-git-authors.txt (attached) is used by Git to translate names of SVN committers to Git names which uses format "Full name <e-mail>". Specifying the --authors-file option is optional, but recommended. You can also do this once via git config --add svn.authorsfile /path/to/gdal-git-authors.txt, then you don't have to specify it when you're running other git svn commands as described below.

It may be a good idea to compress Git repository in order to save some space. For example, SVN trunk occupies nearly 140 MB of disk space. Git can compress it to about 100 MB. Git provides a dedicated command for this purpose  git-gc:

$ git gc

You can bring your local Git repository up to date against GDAL trunk any time.

$ git svn --authors-file=/path/to/gdal-git-authors.txt rebase

The rebase command is equivalent to svn update. Learn more about rebase in  git-svn manual.

Development cycle

==> Caution ! : I (E. Rouault) have had problems with the workflow detailed below, especially when there's a conflict between one of your local change in GIT and a change in SVN. The solution seems to create a GIT branch where you do your local changes, and keep your GIT master in sync with SVN. Then you rebase your master on to your local branch, resolve the conflict in it, commit it, and then (forward) merge the branch into master. At that point, you can git svn dcommit. Workflow inspired by  http://stackoverflow.com/questions/629048/git-svn-dcommit-error-restart-the-commit

After this step, you maintain your own local repository of GDAL trunk. You can proceed with your own development, make changes and commit them to your local repository.

$ git commit -m "my big feature"

If you are GDAL committer, you can push your changes to GDAL Subversion repository directly from your local Git repository of GDAL trunk:

$ git svn dcommit --dry-run
$ git svn dcommit

You can also push the local copy to your own Git remote repository to share your development experiments, then others will be able to clone it. For example, you can push it to  GitHub:

$ git push --all git@github.com:mloskot/gdal.git

Update cycle

Once local clone of a git-svn enabled repository is ready for the cycle of svn update and git push, the routine of updating mirror includes the following two commands only:

$ git svn --authors-file=/path/to/gdal-git-authors.txt rebase
$ git push git@github.com:mloskot/gdal.git

=> Note from user etiennesky: I had to use the following command after the initial fetch:

git reset --hard remotes/trunk

GitHub? Mirror

There is an up-to-date Git repository with mirror of GDAL Subversion trunk, branches and tags available at  https://github.com/OSGeo/gdal maintained by Even Rouault using the  pygit-svn-mirror tool (see section above).

Note, if you are GDAL committer, clone of this repository will not allow you to issue git svn dcommit command to push your changes back to GDAL trunk. If you expect this functionality, you need to maintain Git repository of GDAL trunk on your own according to the steps described above. Quoting one of  best practices listed here:

For the sake of simplicity and interoperating with SVN, it is recommended that all git-svn users clone, fetch and dcommit directly from the SVN server (the remote SVN repository that is), and avoid all git-clone/pull/merge/push operations between git repositories and branches which are either retrieved via git svn clone and which are also used to push back changesets into the remote SVN repository.

This process may be time consuming, so an alternative is to use rsync to copy existing GDAL tree enabled with git-svn from another committer.

Resources

Attachments