Move download.osgeo.org to OSGEO7

[robe]

This is to be done as part of SAC 2019 contract to resetup download.osgeo.org as an LXD container on OSGeo7

[strk]

Please use a Milestone for this

[robe]

strk sorry last time I didn't have rights to create milestones so didn't think I could. I guess I have those rights now, so created a new Milestone.

[robe]

IN meeting now. Jodygarnett said geotools relies on webdav, so we'll need that on the new box.

I do have a new download countainer running Debian 9 (I couldn't get ldap to work properly on Debian 10 which is the only reason I didn't go with that).

I put macoosgeo (Homebrew bottle.download.osgeo.org) on that already since they were in a rush and I didn't want to put them on the old one and have them change things twice.

[robe]

We had discussions about this at the SAC meeting today with

jodygarnett, markusN, jef, dbb expressing their needs and concerns.

Key takeaways from the meeting about new download

1) Make sure to maintain timestamps and permissions as part of the sync

2) Maintain sudos

3) The size of osgeo4w + qgis (both users of the NFS) together take up about 100GB, so we could move these back in the folder and dispense with the NFS

4) Geotools currently relies on WebDav for pushing artifacts so will need that

OR - an interesting idea Jody Garnett brought up is they would like to eventually switch to using something like artifactory / nexus

​https://jfrog.com/open-source/

and this might be something that would be useful for other projects.

With that said - might be good to keep download.osgeo.org simple with just the nginx I have installed and no WebDav and setup a separate container for artifactory use that geotools can use and other projects can eventually take advantage of.

jodygarnett proposed a domain name of repos.osgeo.org

[robe]

did the final resync and changed the dns. So download.osgeo.org and upload.osgeo.org are officially on osgeo7 now.

[jive]

Looks like we missed something, builds fail with:

Failed to read artifact descriptor for org.geotools.data:refdataset-1.0:jar:1.0.7: Could not transfer artifact org.geotools.data:refdataset-1.0:pom:1.0.7 from/to osgeo (​http://download.osgeo.org/webdav/geotools/): Not authorized -> [Help 1]

Indeed navigating to ​http://download.osgeo.org/webdav/geotools/ is:

a) redirected to ​https://download.osgeo.org/webdav/geotools/ b) asked for authorization

[jef]

Moved the basic auth into limit_except. Please verify that it works.

[robe]

jive -- did you not get any of my emails? I sent to your gmail account.

I told you I could not implement webdav/ldap so I had set up an htaccess account for geotools. It does not use the ldap one as I don't know the password for that one.

Anyway if you can write to me at my lr at pcorp dot us account we can sort this out.

jef,

I don't seem to be able to get to ​https://download.osgeo.org/webdav/geotools/ or ​https://download.osgeo.org/geotools/ anymore. I had set up so webdav uses webdav and geotools is just for download

[robe]

ah never mind ​https://dwonload.osgeo.org/geotools still seems to work.

[jive]

Replying to robe:

ah never mind ​https://dwonload.osgeo.org/geotools still seems to work.

I am not sure what you mean? If I go to ​https://download.osgeo.org/geotools it results in a 404 not found.

Please note that the build systems use the following:

 <repositories>
  <repository>
    <id>osgeo</id>
    <name>Open Source Geospatial Foundation Repository</name>
    <url>http://download.osgeo.org/webdav/geotools/</url>
    <!-- release repository used by geotools (and third-party dependencies) -->
  </repository>
</repositories>

And there are hundreds of them around the world. If we wish to change (even from http --> https) we should coordinate.

So there are two issues:

a) making sure the contents are available at ​http://download.osgeo.org/webdav/geotools/

b) sorting up some way for build.geoserver.org and others with an OSGeo userid to deploy new content

[jive]

We are committed to the public URL ​http://download.osgeo.org/webdav/geotools/ for consumption only.

The build servers (or deploying) can be a different endpoint and that is easy to change in one spot.

Here is the GeoTools maven script describing distribution management for deploying (the credentials are kept in a settings.xml file on the build server or on the developers machine):

  <distributionManagement>
    <repository>
      <uniqueVersion>false</uniqueVersion>
      <id>osgeo</id>
      <name>Open Source Geospatial Foundation - Maven 2 repo</name>
      <url>dav:http://download.osgeo.org/webdav/geotools/</url>
    </repository>
    <snapshotRepository>
        <id>boundless</id>
        <uniqueVersion>false</uniqueVersion>
        <name>Boundless Snapshot Repository</name>
        <url>https://repo.boundlessgeo.com/snapshot/</url>
    </snapshotRepository>
  </distributionManagement>

[jive]

Thanks for the help, end of the day:

• http get access via ​http://download.osgeo.org/webdav/geotools/ (public endpoint remains unchanged)
• web dev access via ​http://download.osgeo.org/upload/geotools/ (private endpoint used by deploy process)

The pull request to fix geotools deploy ​https://github.com/geotools/geotools/pull/2385