Jenkins build server to replace build.geoserver.org

[jive]

Planet has informed us that they would like to retire build.geoserver.org (currently hosted on AWS).

The GeoServer? PSC would like to ask explore options with SAC:

• Is it possible to setup Jenkins on an OSGeo virtual machine(s)
• Some care needs to be taken with both resource use and security
• Or is it recommended to take over AWS hosting (will ask for an estimate of monthly costs)

We do have the jenkins configuration and so on saved in private OSGeo gitlab repo presently. The server is setup to create and throwaway workers.

[jive]

Long term it would be good to have windows and macOS jenkins servers (to assist open source projects in creating signed installers). We should plan for this needed even if we do not implement right now.

[jive]

Discussion: It may be appropriate to use Microsoft Azure pipelines with offer linux, windows and macOS builds. With enough automation that could be used to sign an installer.

[wildintellect]

Can someone chime in how this might relate to our previous and ongoing build infrastructure? strk or robe?

We should make a list of options to pick between. There seems to be many ways to go, and something on a cloud provider may scale better. I would like to rename the ticket to need Build, or CI service to highlight it need not be Jenkins (which I've had issues securing in the past).

Jody can you provide a timeline on which this needs to happen.

[strk]

The only CI service currently offered by OSGeo is ​http://dronie.osgeo.org.

It is a "Drone CI" server and an handful of "agents" either donated by devs/users or hosted in OSGeo infrastructure. See ​https://wiki.osgeo.org/wiki/Dronie

It is currently used by (to my knowledge):

• PostGIS
• GEOS
• librttopo

The server integrates with Gitea, automatically building what's pushed to a gitea repository (git.osgeo.org/gitea) and cannot be connected to other sources (each server connects to a single source).

Windows runners are supported, although we don't currently run any: ​https://readme.drone.io/runner/docker/overview/

Jenkins is something PostGIS/GEOS also do use, but outside of OSGeo infrastructure. It may be good to provide it as a shared facility for different users. I would note that Jenkins is also known to possibly integrate with Gitea.

[robe]

I have been thinking about this a bit. Sadly I don't think a shared Jenkins would work too well or at least not in my skill set. We could setup a dedicate VM/Container for Jenkins.

PostGIS does run it's main jenkins in LXD/LXC container but it doesn't use throw away slaves.

jive, can we discuss on OSGeo SAC irc the needs of GeoServer. Depending on your bandwidth needs Amazon or some other Cloud provider might be your best fit and off course the funds would be paid by OSGeo.

[jive]

GeoCat has volunteered a build server also, going to discuss in tomorrow's GeoServer meeting.

[jive]

A shared build server was used previously for geotools, geowebcache, geoserver, geoscript, geogig, etc...

[jive]

I am ready to work on this if a Nexus instance can be made available.

[robe]

jive are we talking about nexus here or jenkins or both?

[robe]

jive it just occurred to me that I can take an image of your geotools jenkins server like I did with the old osgeo servers when I migrated them. That might save some work -- if you can give me access or let me know someone to talk to to orchestrate this, that might be the easiest course of action.

I also have osgeo3 all setup with lxd installed, so we can use that.

[jive]

The new jenkins server is now available at build.geoserver.org ...

But we are having some kind of problem accessing repo.osgeo.org with consistent performance and wanted to ask if there is any network throttling or limiting in place on repo.osgeo.org? A maven build and a denial of service attack look very similar with bursty traffic patterns.

Our builds are failing after a sustained load on repo.osgeo.org with connection timeouts:

[ERROR] Failed to execute goal on project geoserver: 
Could not resolve dependencies for project org.geoserver:geoserver:pom:2.18-SNAPSHOT: 
Could not transfer artifact org.geotools:gt-opengis:jar:24-20200503.120242-25 from/to osgeo-snapshots (https://repo.osgeo.org/repository/snapshot/): 
Connect to repo.osgeo.org:443 [repo.osgeo.org/140.211.15.6] failed: Connection timed out (Connection timed out) -> [Help 1]

We have:

• Observed this from build.geoserver.org
• Set up a second build on a different cloud provider to verify the problem
• however building as individuals around the world does not show this problem
• builds of other downstream projects (geomea) is are not having a problem

Next steps:

• we are going to test a build that does not use repo.osgeo.org in order to double check this is an osgeo networking issue

[jive]

I have also looked at the stats from monitor.osgeo.org and from repo.osgeo.org and cannot see any problems (but I do not know what I am looking for).

Chat for this activity is on ​https://gitter.im/geotools/geotools

[jive]

Okay so we ended up where:

a) build job was stuck b) nexus was not doing anything

And then it got unstuck!

Here’s the chunk of log around where it was downloading the metadata:

[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ geotools ---
[INFO] Downloading from nexus: https://repo.osgeo.org/repository/geotools-snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from nexus: https://repo.osgeo.org/repository/geotools-snapshots/org/geotools/geotools/24-SNAPSHOT/maven-metadata.xml (593 B at 0 B/s)
[INFO] Uploading to nexus: https://repo.osgeo.org/repository/geotools-snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom
[INFO] Uploaded to nexus: https://repo.osgeo.org/repository/geotools-snapshots/org/geotools/geotools/24-SNAPSHOT/geotools-24-20200505.181440-26.pom (86 kB at 40 kB/s)

(593 B at 0 B/s) really looks like some sort of throttling to me.

[jive]

Seeing inconsistent timeouts on the geotools doc deploys:

+ scp -oStrictHostKeyChecking=no index.zip geotools@docs.geotools.org:/home/geotools/stable
ssh: connect to host docs.geotools.org port 22: Connection timed out
lost connection

This is a second osgeo server, The interesting part is that they sometimes work fine (this failure did happen while geotools-master was downloading maven-metadata, so might be related)

[jive]

Update, we are getting wildly different build times with some very long delays in europe. A wide range of folks are running builds in different parts of the world to try and figure out of this is a jenkins issue or a nexus issue.

Examples:

• usa: 07:26 min
• amsterdam: 52:47 min
• amsterdam: 06:57 min

As shown above "something" has happened and performance has now improved, and we can see that improvement on the jenkins build server jobs which are presently running at a steady clip.

[jive]

Thanks to GeoCat for hosting!