[[TOC]] = Migrating GRASS code repository from SVN to git = == Background and aims == SVN has served the GRASS project well as a version control system for numerous years now. The project has established routines and infrastructure (code repository, ticketing system, developer wiki) connected to SVN. However, with an increasing number of Open Source developers using git (and here especially the success of !GitHub), interest in moving from SVN to git has been expressed. Reasons to consider moving from SVN to git are to: * increase the number of contributors and thus contributions due of the popularity of git * make features and functionality of git available for developing GRASS and thus * make the current GRASS developers happy that prefer git (even if git can already be used against SVN: https://trac.osgeo.org/grass/wiki/HowToMaintainGrassWorkflowUsingGit) Other OSGeo projects already moved (even if some only partly) * **GDAL** moved to github (https://trac.osgeo.org/gdal/wiki/rfc71_github_migration) * **QGIS** code has been on github for a while issues are in redmine, discussion of moving also issues (including moving everything to gitlab) ongoing (https://github.com/qgis/QGIS/wiki/QGISbugtracker and https://github.com/qgis/QGIS/wiki/QGIS-Platform-migration-plan) * **Proj.4** moved to github (https://github.com/OSGeo/proj.4) * **GEOS** is on gitea (https://git.osgeo.org/gitea/geos/geos.git, provided by OSGeo, with github mirror) * ... * See e.g.: * https://git.osgeo.org/gitea/strk/trac2gogs Other relevant projects: * **Python**.org moved to github, see https://www.python.org/dev/peps/pep-0512/ * **Spring** Framework moved to github, see https://spring.io/blog/2019/01/15/spring-framework-s-migration-from-jira-to-github-issues GRASS GIS has already a git-mirror-repository for CI: * https://github.com/GRASS-GIS (used for Travis CI, in sync, see https://travis-ci.org/GRASS-GIS/grass-ci) In addition GRASS has a (yet unused) team within the OSGeo organisation on github.com: * https://github.com/orgs/OSGeo/teams/grass-gis Possible migration of GRASS GIS source code to git has been **discussed** at community sprints in 2017 and 2018. At the community Sprint in Bonn 2018 first tools for moving content from trac (SVN) to git were developed (mainly by Martin Landa): https://trac.osgeo.org/grass/browser/grass-addons/tools/svn2git?order=name See also: https://trac.osgeo.org/grass/ticket/3722 == Choosing a git platform == Moving to git involves the question which git platform to move to. So, in addition to technical work, strategical decisions have to be made. === Available options === Most common options / git platforms are: * github (still the most popular, yet proprietary, system with online hosting service) * gitlab (another platform with online hosting service and increasing popularity esp. after Microsoft acquired github) * bitbucket (another popular platform with online hosting service) * gitea/gogs ("painlessly selfhosted" platform, hosted by OSGeo SAC) Yet, there are plenty of other options: https://wiki.osgeo.org/wiki/GitHostingSoftware And there are several comparisons of available git platforms available online: * https://wiki.osgeo.org/wiki/GitInfrastructureComparison * https://ethercalc.org/choose-the-git-host.html (August 2016) * https://www.reddit.com/r/devops/comments/8nro34/gitlab_vs_github_whats_the_right_hosting_platform/ * https://www.linuxjournal.com/content/opinion-github-vs-gitlab * https://about.gitlab.com/devops-tools/github-vs-gitlab.html !GitLab compared to other !DevOps tools * https://about.gitlab.com/devops-tools/ === Risk of "vendor lock in" === See this gitlab-ticket regarding migration from trac to gitlab: https://gitlab.com/gitlab-com/support-forum/issues/2765 And see this manual for moving from SVN to gitlab: https://docs.gitlab.com/ee/user/project/import/ It seems also possible / relatively straight forward to (e.g. later) move from gitlab.com to a selfhosted gitlab instance: https://docs.gitlab.com/ee/user/project/import/gitlab_com.html === Requirements and criteria === Anyway, a first step in order to chose from the available options is to define a list of criteria (here requirements and features) of the git platform to move to. For OSGeo a list of requirements has been compiled: https://wiki.osgeo.org/wiki/GitServiceRequirement Yet, that list does not necessarily reflect all or the most important requirements of the GRASS project and the motivation for moving to git... * Sign in using OSGeo Userid * Autonomously create and manage teams * Autonomously create and manage repositories * Create private repositories (for software vulnerability testing, etc.) * Import tickets from Trac * Import tickets from Redmine * Comment tickets via email * Comment/close tickets via commit log * SVN->GIT sync - (Mirroring existing SVN repository) see:/ticket/1654 * Integration with CI service (hosted eg. Travis-CI, !AppVeyor, !GitLab CI), self-hosted (e.g. !Buildbot, ?)) Extra link: * ~~ New !GitHub API for importing issues: https://gist.github.com/jonmagic/5282384165e0f86ef105 ~~ (we didn't know about it in time) == Steps done == === User Survey === 5 Feb 2019: * User survey posted at https://docs.google.com/forms/d/1BoTFyZRNebqVX98A3rh5GpUS2gKFfmuim78gbradDjc * deadline 12 Feb 2019; deadline later extended to 15 Feb 2019 20 Feb 2019: * Results available here: https://docs.google.com/forms/d/1BoTFyZRNebqVX98A3rh5GpUS2gKFfmuim78gbradDjc/viewanalytics === Git Test Migration === 5 Feb 2019: * svn -> git test migration ongoing, see #3722 === PSC vote === * RFC 6 * https://trac.osgeo.org/grass/wiki/RFC/6_MigrationGitHub * Motion on RFC 6: 18 Apr 2019 * https://lists.osgeo.org/pipermail/grass-psc/2019-April/002026.html * Result: motion carried 22 Apr 2019 * https://lists.osgeo.org/pipermail/grass-psc/2019-April/002034.html == Implementation of migration from OSGeo SVN and trac to !GitHub == === Migration plan (in action) === ==== Roadmap ==== 1. source code migration - **done 17-19 May 2019** 1. migration of source code will be announced on grass-dev ML few days before day D - [x] 2. svn system will be switched to read-only mode - [x] 3. Git `grass` repo (https://github.com/grass-svn2git/grass) will be created from scratch - [x] 4. source code migration will be launched (https://trac.osgeo.org/grass/browser#grass-addons/tools/svn2git; will take 1-2 hours) - [x] 5. push `grass` repo under [https://github.com/OSGeo/ Github OSGeo organization] - [x] 2. trac ticket migration - **planned for ~~May 2019~~ June? 2019** 1. migration of issues will be announced on grass-dev ML few days before day D 2. trac ticket system (only tickets, wiki will be still editable) will be switched to read-only mode 3. git `grass` repository under OSGeo organization will be switched to private mode 4. migration of trac issue will be launched (target: `grass` repo; will take few hours) 5. switch `grass` repo back to public mode Similar procedure will be performed for `grass-addons` repo. **New repositories:** * GRASS GIS core: https://github.com/OSGeo/grass * GRASS GIS legacy: https://github.com/OSGeo/grass-legacy * GRASS GIS Add-ons: https://github.com/OSGeo/grass-addons * GRASS GIS promotional material: https://github.com/OSGeo/grass-promo * (GRASS GIS Website (future site): https://github.com/OSGeo/grass-website) === Enforce-pull-request policy === ~~In !GitHub ([https://github.com/OSGeo/grass/settings/branch_protection_rules/new here]), we have set "Require pull request reviews before merging" and "Include administrators" for master and all branch to enforce this policy.~~ No direct committing to master, whatsoever, please. Go through feature branches and pull requests! See [[wiki:HowToGit]] === Authorship: Recognizing the GRASS GIS contributors during and after migration === In the !GitHub repository the "contributors" list contains only the authors who have a !GitHub account which was known to us during the migration. We used the lookup tables `AUTHORS.txt` + `svn2git_users.csv` (in [https://trac.osgeo.org/grass/browser/grass-addons/tools/svn2git addons/tools/svn2git/]) which map SVN account to the respective !GitHub account. We additionally mapped multiple CVS/SVN accounts to the respective !GitHub account (as some developers changed their account name in the transition from CVS to SVN). And here is the list of [https://id.osgeo.org/ldap/group?group=grass OSGeo LDAP enabled SVN contributors]. Importantly, the contributors list still covers only a tiny fraction of the [https://grass.osgeo.org/home/credits/ real authorship]! Now, if you/someone isn't listed yet under contributors: * please get a !GitHub account and/or communicate it to us (name + related email) * Or, simply, to match your commits to your !GitHub account, just add your associated email address(es) to your account in order to [https://github.com/settings/emails claim your contributions]. Updating is always possible and easy, so nothing is lost. === Migration of trac issues === * Ongoing implementation of a script to convert all trac issues >= GRASS GIS 7.4 to github * draft: https://github.com/landam/grass-gis-git-migration-test/issues * Issue labels: * draft: https://github.com/landam/grass-gis-git-migration-test/labels === New labels in the !GitHub issue tracker === * c.f. also current QGIS cleanup "Migrating issues from Redmine to !GitHub" * https://github.com/qgis/QGIS-Enhancement-Proposals/issues/141 * initial set of labels will be created during the trac issue migration === Migration of trac wiki === * Suggestion: yes, we migrate the pages but only relevant stuff * This trac wiki to markdown converter can support us in this regard: https://gist.github.com/mitio/a3d54e32c341ae523314 === Aftermath cleanup === Numerous changes are needed after successful migration from SVN to !GitHub: ==== Needed source code updates ==== In various places "svn" is coded and needs to be replaced: * scripts/g.extension/g.extension.py * replace SVN tags * `$Id$` * `$Date$` * question: replace with...? * probably !GitHub commit webhook to replace svn Date * DONE Addons: link to source code in tools/addons/compile.sh * DONE configure.in/configure grass.pc.in Dockerfile * DONE https://github.com/OSGeo/grass/pull/9 * DONE include/VERSION general/g.version/main.c general/g.version/Makefile general/g.version/g.version.html * DONE INSTALL REQUIREMENTS.html ==== Infrastructure updates ==== * **Travis** CI/CD update to use new repo * https://travis-ci.org/GRASS-GIS --> https://travis-ci.org/OSGeo/grass ?? * https://github.com/GRASS-GIS/grass-ci --> https://github.com/OSGeo/grass * https://svn.osgeo.org/grass/grass-addons/tools/grass-ci/grass-ci.sh * **addons** creation update * on Prague server * creation of https://grass.osgeo.org/addons/grass7/modules.xml * **trac Wiki**: * rewrite https://trac.osgeo.org/grass/wiki/HowToSVN to git usage * WIP: https://trac.osgeo.org/grass/wiki/HowToGit * **grasswiki**: update related pages (search for SVN) * compile and install * **Transifex** update * https://www.transifex.com/grass-gis/grass7/ * https://www.transifex.com/grass-gis/settings/integrations/ * DONE Authorization at: https://github.com/organizations/OSGeo/settings/installations/1004901 * TODO: * New localization files from our GitHub repository will be automatically added to our Transifex project, and * Fully translated content from our Transifex project will be automatically pushed back to our GitHub repository (via commit or Pull Request, depending on which you select). TODO * DONE **docs/infrastructure.txt** update * DONE **CMS**: update download, bug and development related pages * download pages * bug reports * development * DONE **openhub** update * https://www.openhub.net/p/grass_gis * DONE https://www.openhub.net/p/grass_gis/enlistments * https://www.openhub.net/p/grass_gis_addons * DONE https://www.openhub.net/p/grass_gis_addons/enlistments * **IRC** notification via Travis * apparently already ok * DONE **grass-commit** emails * DONE mailman * DONE **cronjobs** (weekly snapshots + manuals) on grass.osgeo.org * DONE switch svn update -> git fetch in cronjob files === Setup of Gitlab mirror === * Import project from !GitHub to !GitLab: https://docs.gitlab.com/ee/user/project/import/github.html * Open problem: how to mirror !GitHub issues? === Setup of OSGeo gitea mirror === * https://git.osgeo.org/gitea * DONE https://git.osgeo.org/gitea/grass_gis/grass == Future plans not being part of initial migration == === Backporting bot === * Usage of github backporting application * https://github.com/apps/backporting (already used by GDAL, PROJ, QGIS, ...) === Zeonodo based DOIs === * DOI support through zenodo by connecting repos on !GitHub with Zenodo: * https://guides.github.com/activities/citable-code/ * According to the Zenodo helpdesk, two options exist to connect the GRASS codebase to the Zenodo archive: 1. REST-API (of Zenodo) or manual upload into Zenodo 2. !GitHub Integration * It is possible to start the process with !GitHub Integration and then (for whatever reason) fall back to the REST-API/manual upload. * It is NOT possible to start with the REST-API/manual upload and to switch to !GitHub Integration later. * Zenodo helpdesk on !GitHub Integration: ''If you want to use our !GitHub integration, then you must move the source code to !GitHub and activate the repository in Zenodo (see the !GitHub guide). Afterwards, you make a new release in GitHub for each of your releases (see also the !GitHub guide). You have to make the releases in the order you want them to appear in Zenodo. If you have tags push to !GitHub, then you can upgrade a tag to a release in the !GitHub interface.'' * Zenodo helpdesk on DOI versioning scheme: ''Zenodo keeps the version number and date in a metadata field that you can change as you see fit even after publishing. By default Zenodo orders the releases in the order we receive them (i.e. by date). This is however only for display purposes, and it is essentially the same way !GitHub orders their releases page. In the metadata however, we do not care about ordering (because it's very hard to model the ordering correctly). We simply have a concept DOI that links to all the the specific version DOIs via !HasVersion/IsVersionOf relationship.'' * Zenodo helpdesk on mishaps: ''...in case you make a mistake, we do have the possibility to reorder the releases manually. Naturally we would like to avoid this, however just rest assured that we can fix it if you make a mistake.'' * WIP: https://grasswiki.osgeo.org/wiki/GitHub-Zenodo_linkage === CI/CD and QA === * Travis-CI: https://travis-ci.org/GRASS-GIS/grass-ci * TODO: check https://travis-ci.org/OSGeo/grass * Scanning of defects, real-time: https://scan.coverity.com/projects/grass * Azure pipelines: https://azure.microsoft.com/en-us/services/devops/pipelines/ === Next release === * Release of GRASS GIS 8.0.0 (or earlier) from !GitHub * See https://github.com/OSGeo/grass/releases == See also == * Remarks on SVN-trac to GitHub migration: https://courses.neteler.org/remarks-on-svn-trac-to-github-migration/