repo.osgeo.org 500 internal server error on deploy

[jive]

In the last three days the automated builds have all started failing with:

[ERROR] Failed to execute goal org.apache.maven.plugins:
  maven-deploy-plugin:3.1.1:deploy (default-deploy) on project geoserver:
  Failed to deploy artifacts:
  Could not transfer artifact org.geoserver:geoserver:pom:2.26-20240715.131524-134
  from/to nexus (https://repo.osgeo.org/repository/geoserver-snapshots/):
  status code: 500, reason phrase: Internal Server Error (500) -> [Help 1]

This is not unique to geoserver: geotools, mapfish-print-v2 and others are all failing.

What happened 3 days ago? Some credential time out ...

[jive]

Seems to be a deplicate of ​https://trac.osgeo.org/osgeo/ticket/3219 which is now closed.

[jive]

Testing locally fails also, with the same error, so it is not specific to the build server.

[jive]

testing with a local deploy

[jive]

testing with a local deploy, and using the nexus "Marker to insert into log" I have produced the attached nexus-deploy-log.txt showing the problem...

2024-07-15 18:30:22,728+0000 WARN  [qtp145527698-407114]
  *UNKNOWN org.sonatype.nexus.siesta.internal.WebappExceptionMapper -
  (ID a3211e4b-2913-4e05-b2d1-200a66f93d9a) Response:
  [404] (no entity/body); mapped from: javax.ws.rs.WebApplicationException:
  Path not found

[robe]

@jive,

I haven't changed anything in repo.osgeo.org since the last patch update which was way more than 3 days ago. I can try maybe rebooting the server to see if that helps.

[robe]

ah okay I'm seeing something on the console it is saying we reached total capacity of components -- 100,000 allowed and we have 100,469 for the OSS version of nexus. So that sounds like it's probably the culprit. Just points to this page - ​https://www.sonatype.com/products/sonatype-nexus-repo/oss-upgrade

[robe]

I wonder if they changed policy too cause that page lists LDAP as only on the paid version, but we are using LDAP.

[robe]

@jive I had a talk with sales rep and sent you the details via mail of their correspondence

[jive]

I have run cleanup task and got the number of components down to 94,639... However a local deploy still results internal sever error above.

[robe]

Replying to jive:

I have run cleanup task and got the number of components down to 94,639... However a local deploy still results internal sever error above.

I'm not sure how to test this maybe you can walk me thru this sometime. I could try a restart to rule out its something checked during start up. I doubt it but you never know.

[robe]

I just rebooted repo.osgeo.org can you check again to see if that made any difference.

[jive]

The what is new article shown when signed in talks about usage limits which is probably where some of our problems are.

Reference Architectures and Usage Alerts

Is your Sonatype Nexus Repository deployment appropriate for your scale? If not, your deployment’s stability and performance could be at risk.

Sonatype is now providing a set of Nexus Repository Reference Architectures to help administrators implement a deployment model where all instances have sufficient resources. We have designed each reference architecture against the anticipated total load for a given profile based on Sonatype testing data and our experiences supporting enterprises at scale.

Improper use of an embedded database is one common issue among Nexus Repository deployments. To help you determine if this is happening, deployments using an embedded database will now see usage alerts when usage approaches or exceeds 20,000 requests per day or 100,000 components.

Details about usage alerts and metrics are available in our Usage Metrics documentation . And, of course, be sure to check out our Nexus Repository Reference Architectures .

Configure Maven and Docker Cleanup Policies to Retain Recent Versions

Nexus Repository Pro customers using a PostgreSQL database can now take advantage of more powerful and flexible cleanup policies for Maven and Docker formats. You now have the option to configure your cleanup policies to retain a certain number of most recent artifact versions regardless of if they meet other cleanup criteria.

Learn more about how to use this feature in the cleanup policies help documentation.

It links to two articles to try tomorrow:

• ​https://help.sonatype.com/en/sonatype-nexus-repository-reference-architectures.html
• ​https://help.sonatype.com/en/cleanup-policies.html

[jive]

aside: The feature about retraining a number of recent artifacts is the reason we use tasks and not the cleanup policies provided in the free edition.

[robe]

FYI I think I might have found the culprit. Seems to be nginx is out of space. I'm working on cleaning that up now and will let you know when it's ready for you to try again.

[robe]

Give it a try again.

[jive]

I am able to deploy! Thanks ... so in the end it was nginx being out of space?

I expect the 100000 component limit is to generate sales leads and is not a hard limitation. That said it may be negotiating for "pro" and using a postgresql backend rather than an embeded database.

[robe]

I think this is all set. Clearing disk space on osgeo3-nginx fixed the issue.