Opened 2 years ago

Closed 2 years ago

#2411 closed task (fixed)

Jenkins build server to replace

Reported by: jive Owned by: sac@…
Priority: normal Milestone:
Component: Systems Admin Keywords:


Planet has informed us that they would like to retire (currently hosted on AWS).

The GeoServer? PSC would like to ask explore options with SAC:

  • Is it possible to setup Jenkins on an OSGeo virtual machine(s)
  • Some care needs to be taken with both resource use and security
  • Or is it recommended to take over AWS hosting (will ask for an estimate of monthly costs)

We do have the jenkins configuration and so on saved in private OSGeo gitlab repo presently. The server is setup to create and throwaway workers.

Change History (16)

comment:1 by jive, 2 years ago

Long term it would be good to have windows and macOS jenkins servers (to assist open source projects in creating signed installers). We should plan for this needed even if we do not implement right now.

comment:2 by jive, 2 years ago

Discussion: It may be appropriate to use Microsoft Azure pipelines with offer linux, windows and macOS builds. With enough automation that could be used to sign an installer.

comment:3 by wildintellect, 2 years ago

Can someone chime in how this might relate to our previous and ongoing build infrastructure? strk or robe?

We should make a list of options to pick between. There seems to be many ways to go, and something on a cloud provider may scale better. I would like to rename the ticket to need Build, or CI service to highlight it need not be Jenkins (which I've had issues securing in the past).

Jody can you provide a timeline on which this needs to happen.

comment:4 by strk, 2 years ago

The only CI service currently offered by OSGeo is

It is a "Drone CI" server and an handful of "agents" either donated by devs/users or hosted in OSGeo infrastructure. See

It is currently used by (to my knowledge):

  • PostGIS
  • GEOS
  • librttopo

The server integrates with Gitea, automatically building what's pushed to a gitea repository ( and cannot be connected to other sources (each server connects to a single source).

Windows runners are supported, although we don't currently run any:

Jenkins is something PostGIS/GEOS also do use, but outside of OSGeo infrastructure. It may be good to provide it as a shared facility for different users. I would note that Jenkins is also known to possibly integrate with Gitea.

comment:5 by robe, 2 years ago

I have been thinking about this a bit. Sadly I don't think a shared Jenkins would work too well or at least not in my skill set. We could setup a dedicate VM/Container for Jenkins.

PostGIS does run it's main jenkins in LXD/LXC container but it doesn't use throw away slaves.

jive, can we discuss on OSGeo SAC irc the needs of GeoServer. Depending on your bandwidth needs Amazon or some other Cloud provider might be your best fit and off course the funds would be paid by OSGeo.

comment:6 by jive, 2 years ago

GeoCat has volunteered a build server also, going to discuss in tomorrow's GeoServer meeting.

comment:7 by jive, 2 years ago

A shared build server was used previously for geotools, geowebcache, geoserver, geoscript, geogig, etc...

comment:8 by jive, 2 years ago

I am ready to work on this if a Nexus instance can be made available.

comment:9 by robe, 2 years ago

jive are we talking about nexus here or jenkins or both?

comment:10 by robe, 2 years ago

jive it just occurred to me that I can take an image of your geotools jenkins server like I did with the old osgeo servers when I migrated them. That might save some work -- if you can give me access or let me know someone to talk to to orchestrate this, that might be the easiest course of action.

I also have osgeo3 all setup with lxd installed, so we can use that.

comment:11 by jive, 2 years ago

The new jenkins server is now available at ...

But we are having some kind of problem accessing with consistent performance and wanted to ask if there is any network throttling or limiting in place on A maven build and a denial of service attack look very similar with bursty traffic patterns.

Our builds are failing after a sustained load on with connection timeouts:

[ERROR] Failed to execute goal on project geoserver: 
Could not resolve dependencies for project org.geoserver:geoserver:pom:2.18-SNAPSHOT: 
Could not transfer artifact org.geotools:gt-opengis:jar:24-20200503.120242-25 from/to osgeo-snapshots ( 
Connect to [] failed: Connection timed out (Connection timed out) -> [Help 1]

We have:

  • Observed this from
  • Set up a second build on a different cloud provider to verify the problem
  • however building as individuals around the world does not show this problem
  • builds of other downstream projects (geomea) is are not having a problem

Next steps:

  • we are going to test a build that does not use in order to double check this is an osgeo networking issue
Last edited 2 years ago by jive (previous) (diff)

comment:12 by jive, 2 years ago

I have also looked at the stats from and from and cannot see any problems (but I do not know what I am looking for).

Chat for this activity is on

comment:13 by jive, 2 years ago

Okay so we ended up where:

a) build job was stuck b) nexus was not doing anything

And then it got unstuck!

Here’s the chunk of log around where it was downloading the metadata:

[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ geotools ---
[INFO] Downloading from nexus:
[INFO] Downloaded from nexus: (593 B at 0 B/s)
[INFO] Uploading to nexus:
[INFO] Uploaded to nexus: (86 kB at 40 kB/s)

(593 B at 0 B/s) really looks like some sort of throttling to me.

comment:14 by jive, 2 years ago

Seeing inconsistent timeouts on the geotools doc deploys:

+ scp -oStrictHostKeyChecking=no
ssh: connect to host port 22: Connection timed out
lost connection

This is a second osgeo server, The interesting part is that they sometimes work fine (this failure did happen while geotools-master was downloading maven-metadata, so might be related)

comment:15 by jive, 2 years ago

Update, we are getting wildly different build times with some very long delays in europe. A wide range of folks are running builds in different parts of the world to try and figure out of this is a jenkins issue or a nexus issue.


  • usa: 07:26 min
  • amsterdam: 52:47 min
  • amsterdam: 06:57 min

As shown above "something" has happened and performance has now improved, and we can see that improvement on the jenkins build server jobs which are presently running at a steady clip.

comment:16 by jive, 2 years ago

Resolution: fixed
Status: newclosed

Thanks to GeoCat for hosting!

Note: See TracTickets for help on using tickets.