Opened 8 months ago

Closed 6 months ago

Last modified 6 months ago

#2410 closed task (fixed)

artifact manager to replace download.osgeo.org/webdav/geotools

Reported by: jive Owned by: sac@…
Priority: normal Milestone: Sysadmin Contract 2020-I
Component: Systems Admin Keywords:
Cc:

Description

Planet has informed us that they would like to retire repo.boundlessgeo.com (an artifactory instance) used by the GeoServer? project. The GeoServer? PSC would like to ask OSGeo if it is possible to host an artifact manager on OSGeo hardware?

As discussed in 2019 (see https://trac.osgeo.org/osgeo/ticket/2261) this is a good oppertunity to setup an artifact manager to replace http://download.osgeo.org/webdav/geotools/

There are two good options:

Change History (21)

comment:1 Changed 8 months ago by jive

This is a somewhat time sensitive activity as we would like to rescue the contents of repo.boundlessgeo.com (mostly prior release of GeoServer? and GeoWebCache?).

comment:2 Changed 8 months ago by jive

As noted in email discussion these repository managers can be configured to manage maven, docker, node, etc... and may be of interest to a sider group of osgeo projects.

comment:3 Changed 8 months ago by wildintellect

Possible: Yes How: we need some members to step up to set up a container dedicated to this purpose and hopefully have some expertise or at least time to vet the options.

comment:4 Changed 8 months ago by robe

jive -- as a clean replacement sounds like might be better to go with jfrog since that is what repo.boundlessgeo.com is using.

What does sonatype do additional that jfrog doesn't that would be a possible need?

I'd be willing to take a stab at installing it -- you have details of how we can pull over the artifacts from boundless?

comment:5 Changed 8 months ago by jive

Answering your question *What does sonatype do additional that jfrog doesn't that would be a possible need?*

Frog artifactory:

  • https://jfrog.com/open-source/
  • personal experience: easier administration screens to manage
  • Only supports java projects (Maven, Gradle and Ivy)
  • Used by repo.boundlessgeo.com (after a poor experience with cloud hosting)

Nexus:

  • https://www.sonatype.com/download-oss-sonatype
  • personal experience: more complicated administration screens
  • This is the clear choice for OSGeo as it offers capabilities for more than just java projects (docker images, node packages, ruby, python, go, etc...)
  • Used by eclipse foundation (with their strong emphasis on control),

I have personally used both as an administrator (not installing but managing).

Links:

comment:6 Changed 8 months ago by jive

I think to copy the contents Tom Ingold did:

  1. Set up repo as a cache of old repo
  2. Have a script that accessed all the content via the new URL
  3. Turn of the "cache" and keep the content

comment:7 Changed 7 months ago by jive

So what happens next?

comment:8 Changed 7 months ago by robe

jive,

I'm planning to setup osgeo3 this week and thinking I would set up a container for nexus and jfrog.

I'm thinking maybe it would be easier if I setup jfrog first that way you have a clean migration from boundless? and then we can do nexus and slowly port over?

I don't think it's necessary for all projects to use the same thing so if you feel more comfortable with artifactory I'd say you should use that.

comment:9 Changed 7 months ago by robe

Milestone: Sysadmin Contract 2020-I

comment:10 Changed 6 months ago by robe

Jody - I setup nexus using the docker container from here https://hub.docker.com/r/sonatype/nexus3/ - I sent you the login details

https://nexus.osgeo.org

It's running in a dedicated LXD OS container so we could put other dockery stuff in there you need.

It's running on the new osgeo3 lxd host I set up today.

comment:11 Changed 6 months ago by robe

nexus supports LDAP so I configured it to use OSGeo LDAP and set jive, wildintellect, strk as an admins.

There is a setting in the LDAP config to use LDAP groups for role management. But I wasn't sure if we wanted to do that for the first go round.

comment:12 Changed 6 months ago by robe

based on popular demand I have added

https://repo.osgeo.org as an alias for https://nexus.osgeo.org

we can get rid of nexus but that's what the container is called and the config I have at moment and some other names were proposed like docker.osgeo.org, apt.osgeo.org etc.

comment:13 Changed 6 months ago by jive

Thanks for the hard work, I am collecting notes on how to transfer content.

  1. webdav

This script would like to be run on osgeo machine hosting webdav folder.

This replies on direct file system access and should be fairly quick.

Once content is migrated we can redirect http://download.osgeo.org/webdav/geotools/ --> repo.osgeo.org/release

  1. repo.boundlessgeo.com to

This maven repository provisioner can be run from anywhere, and just uses maven API.

I expect this will be quite bandwidth intensive, as such I would prefer to run this script on osgeo hardware.

  1. when complete

Update the builds for osgeo projects and communicate the change to downstream applications

comment:14 Changed 6 months ago by robe

jive are you all set here? Last we discussed you had root access to nexus container and download container and can install anything you need on the nexus container.

I also want to do an upgrade of nexus install to deal with vulnerability that was announced a couple of days ago. I'm not sure you have any groovy scripts so may be a non-issue.

comment:15 Changed 6 months ago by robe

I did the upgrade on nexus.

comment:16 Changed 6 months ago by jive

Tom Ingold was able to transfer content from ​https://repo.boundlessgeo.com/artifactory/release to Geoserver-releases (no idea why that ended up with a capital but it does not matter so much as it mostly an internal detail). The maven repository provisioner did not work, but he was able to figure some scripts out (wget, find, xargs and curl).

Initial testing found that some things were missed (apache beanutils) so some testing is needed.

comment:17 Changed 6 months ago by jive

I have written up a wiki page https://wiki.osgeo.org/wiki/SAC:Repo describing how projects can join the repository.

We have our initial builds working from GeoTools?, I think this issue can be closed!

comment:18 Changed 6 months ago by jive

aside: I was able to complete migrating contents from webdav folder.

Should we make plans to shutoff that webdav folder, or redirect to https://repo.osgeo.org/repository/release/ ?

comment:19 Changed 6 months ago by robe

Resolution: fixed
Status: newclosed

comment:20 in reply to:  18 Changed 6 months ago by robe

Replying to jive:

aside: I was able to complete migrating contents from webdav folder.

Should we make plans to shutoff that webdav folder, or redirect to https://repo.osgeo.org/repository/release/ ?

Yes we should -- you going to be at next SAC meeting? https://wiki.osgeo.org/wiki/SAC_Meeting-2020-04 maybe we can have a working meeting

comment:21 Changed 6 months ago by strk

Would users have a benefit from redirect ? If so please go for it !

Note: See TracTickets for help on using tickets.