geotools maven repository missing?

[jive]

The geotools maven repository URL is 404!

• ​http://download.osgeo.org/webdav/geotools/

[jive]

Experimenting shows the content is available here:

• ​http://download.osgeo.org/webdav//geotools/

[jive]

I think this was working on June 28th or June 29th?

Many builds will not notice until they build with a "-U" update dependencies options at which point they will get broken.

[robe]

I got rid of the rewrite rule which I think jef had added and that seems to have fixed it.

Maybe jef can look into this again.

[jive]

I think this is working, not really sure if the build tools are smart enough to handle http --> https redirect.

Going to mark as closed and reopen if their is any further feedback.

[jive]

It was not smart enough to follow https redirect. Can we have both http and https at the same time?

This will give us time to change our documentation and get the word out.

[robe]

Jody,

I think the https redirect is happening on the browser. All of download.osgeo.org is redirecting and I don't see any redirect in configs to cause that. When I use wget, I was still getting a 302 move just for webdav (but it wasn't redirecting to https just another http) and the others weren't redirecting.

I did still have in place a left over REWRITE rule for the regular non https webdav which might have been what you were experiencing.

​https://git.osgeo.org/gitea/sac/download-apache-config/commit/b2c5ddce447dd33bbc75486aea53c5a4e23caf7d

I've taken that out, and also enabled webdav permissions (those were remarked out on the https and not on the http) and they seem to control who can do webdav posts

​https://git.osgeo.org/gitea/sac/download-apache-config/commit/da3a29bf64983a0218986b155738ecaa6b95fb60

Can you see if you can still do webdav posts and also download.

[jef]

What ever is using this is constantly requesting non-existing stuff, which get logged in download_error_log:

# head -1 /var/log/apache2/download_error_log
[Thu Jul 05 09:39:01 2018] [error] [client X.X.X.X] File does not exist: /osgeo/download/webdav/geotools/org/springframework/spring-core/2.0.5
# wc -l /var/log/apache2/download_error_log
197851 /var/log/apache2/download_error_log
# fgrep /osgeo/download/webdav/ /var/log/apache2/download_error_log | wc -l
194565

ie. 197851 total errors in less than 24h - 194565 webdav requests.

# b=; while sleep 1; do a=$(wc -l </var/log/apache2/download_error_log); [ -n "$b" ] && echo $a $(( a-b )); b=$a; done
203651 15
203663 12
203687 24
203704 17
203716 12
203727 11
203735 8
203747 12
203757 10
203761 4
203769 8
203788 19
203809 21
203816 7

more than ~10 per second… Any idea why that happens? These requests also get logged in access.log:

# fgrep webdav /var/log/apache2/download_access_log|wc -l; grep "webdav.* 404 " /var/log/apache2/download_access_log | wc -l
636603
593517

so most of the requests are 404 from webdav. But this includes the time while the rule was in place - I can't check the previous log as I just re-enabled rotation and download_access_log.1 is 22 GB (twenty two gigabytes since Feb 11th).

That's also why I addressed this at all - / was full.

[jive]

Replying to robe:

Jody,

I think the https redirect is happening on the browser. All of download.osgeo.org is redirecting and I don't see any redirect in configs to cause that. When I use wget, I was still getting a 302 move just for webdav (but it wasn't redirecting to https just another http) and the others weren't redirecting.

I was not using a browser, but using the repository from the maven build tool.

Changing maven pom.xml repository definition ​https://download.osgeo.org/webdav/geotools/ worked, while using ​http://download.osgeo.org/webdav/geotools/ did not.

I did still have in place a left over REWRITE rule for the regular non https webdav which might have been what you were experiencing.

​https://git.osgeo.org/gitea/sac/download-apache-config/commit/b2c5ddce447dd33bbc75486aea53c5a4e23caf7d

I've taken that out, and also enabled webdav permissions (those were remarked out on the https and not on the http) and they seem to control who can do webdav posts

​https://git.osgeo.org/gitea/sac/download-apache-config/commit/da3a29bf64983a0218986b155738ecaa6b95fb60

Can you see if you can still do webdav posts and also download.

1. The use of maven build tool is now working with the "http" repository reference.

2. The use of web dev with both "http" and "https" is now working

[jive]

Replying to jef:

What ever is using this is constantly requesting non-existing stuff, which get logged in download_error_log:

# head -1 /var/log/apache2/download_error_log
[Thu Jul 05 09:39:01 2018] [error] [client X.X.X.X] File does not exist: /osgeo/download/webdav/geotools/org/springframework/spring-core/2.0.5
# wc -l /var/log/apache2/download_error_log
197851 /var/log/apache2/download_error_log
# fgrep /osgeo/download/webdav/ /var/log/apache2/download_error_log | wc -l
194565

ie. 197851 total errors in less than 24h - 194565 webdav requests.

The tool used is ​https://maven.apache.org is actually working as expected:

1. For each project a number of repositories is configured.

2. Each dependency is represented by a "pom" (project object model) and "jars", "javadocs", "source", etc... as required. The "pom" file describes the dependency and what it needs in turn.

3. Maven uses this to build up a tree like a package manager and downloads all the components needed as part of the build process.

4. For each dependency it asks each repository if they have it, and then downloads from one of them.

     Downloading from osgeo: https://download.osgeo.org/webdav/geotools/org/codehaus/plexus/plexus-io/2.0.3/plexus-io-2.0.3.pom
     Downloading from boundless: https://repo.boundlessgeo.com/snapshot/org/codehaus/plexus/plexus-io/2.0.3/plexus-io-2.0.3.pom
     Downloading from sonatype-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/codehaus/plexus/plexus-io/2.0.3/plexus-io-2.0.3.pom
     Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.3/plexus-io-2.0.3.pom
     Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.3/plexus-io-2.0.3.pom (1.7 kB at 54 kB/s)

5. Because our repository is popular it is used by not only the geotools build, but by all the projects that make use of geotools world wide. Each of those projects is going to talk to us .. all the time.

There are two ways out:

a) Use a maven repository application (rather than web dev) for hosting artifacts. Artififactory is used by boundless, Nexus is used by LocationTech

b) Deploy on to "maven central" rather than out own repository.

The part of the maven infrastructure we are not making use of is called "maven central", where each group responsible for a project can upload their jars, which are then mirrored around the world.

My understanding is we are unable to unable to upload our jars because we do not meet maven central policies ... our software depends on non open source software ... Java Advanced Imaging.

[robe]

I'm not sure we want to consider changing how its setup right now.

However since we are planning to move download.osgeo.org to OSGeo7, then we should add this as part of our on-going OSGeo7 spec

​https://wiki.osgeo.org/wiki/Osgeo7

[jive]

With respect to OSGeo7, I have spoken with James Hughes and he will see if he can help GeoTools release to maven central. If we can do this then a lot of pressure on our server will be lifted.