Issue with Maven OSGEO (repo.osgeo.org) - no remote packages available

[robe]

Email thread I got about this

---

The archetype metadata file in the root is empty. Our Artifactory instance sees no remote packages on the suggested url. Also Maven Central days there is only 1 java package in the repo.

---

It seems that the OSGeo Nexus Repository is broken. Our Artifactory instance is no longer correctly mirroring the OSGeo Maven Release Repo. I couldn’t find any proper group on the mailing list to send a message to, so I’ve looked at who changed the repository documentation, and it was you.

If you are not the correct person to tell this to, please accept my sincerest apology, and could you be so kind as to point me to the correct person to help me. Thanks in advance! Kind Regards, Thijs Cramer

---

[jive]

I have not had any reports of trouble from my automated build boxes or email lists.

Can you confirm that *the suggested url* is: ​https://repo.osgeo.org/repository/release/ ?

Doing a geoserver build with an empty local repository is functioning (making use of the above repository to fetch geotools dependencies).

[jive]

It looks like there may be a "maven index" that we should publish once a day, used by other clients "nexus instances, or m2eclipse"

I am going to schedule a task to produce this once a day and we can ask if it fixes the problem. This is the first I have heard of this index as it is not something our previous webdav folder offered.

Reference:

• ​https://support.sonatype.com/hc/en-us/articles/213465468-What-do-all-of-the-search-index-related-scheduled-tasks-do-and-when-should-I-schedule-them-

[jive]

I have setup tasks to create an index for the release and snapshot repositories; please let me know if that resolves the problem.

[jive]

Why would maven central know about our repo? we have not requested to be mirrored yet ...

[thijscramer]

I can confirm our configured URL is set to ​https://repo.osgeo.org/repository/release/. Using the "test" function Artifactory gives us a 404. We can ignore the test result and just add the repository, but it stays empty on the Artifactory side. It just doesn't see any remote packages. This is one of many maven(2) repositories we have configured, and we have no issues with any other repo's (like Maven Central for example). Which uses the exact same configuration, just another URL.

There's not much we can configure for a remote repo: ​https://www.jfrog.com/confluence/display/JFROG/Remote+Repositories

[thijscramer]

If we use the url ​https://repo.osgeo.org/service/rest/repository/browse/release/ we get a good status back from Artifactory when testing. But the remote repo is filled up with only directory's. There are no artifacts shown.

[robe]

When I browse it I do see real files --

​https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/1.9.2/

e.g. there is a jar in that folder.

Does not come thru?

jive - did everything replicate over from boundless?

[meedel]

It is correct that you can list the artifacts within a browser. But in our binary resource manager Artifactory I only get the directory listing and there are no artifacts shown! I have to use the url ​https://repo.osgeo.org/service/rest/repository/browse/release/ in Artifactory to see the directory listing. If I use the suggested url from the documentation ​https://repo.osgeo.org/repository/release/ I get a 404 in the test from Artifactory and the only directory what is shown is java3d/vecmath/1.5.2/ without artifacts.

[jive]

Only the things we asked for replicated from boundless; they had lots of their own open source projects (non osgeo) which we did not take.

I am a bit confused about this thread, while I believe that artifactory has some secret api to communicate maven index - we have setup a task to refresh and publish this api. But have no way of testing if it is working or not ...

The browse links, and searching in the user interface, are fine for checking the contents visually. The actual maven build tool is finding everything it needs just fine. It is this "repository index" api which a downstream nexus is trying to make use of ....

meedel do you have any documentation on what needs to be done? I have set up two tasks as shown in the attachments. They both seem to be running ...

[jive]

Replying to thijscramer:

If we use the url ​https://repo.osgeo.org/service/rest/repository/browse/release/ we get a good status back from Artifactory when testing. But the remote repo is filled up with only directory's. There are no artifacts shown.

You are using a URL for browsing visually, what you want is the url we have documented: ​https://repo.osgeo.org/repository/release/

[jive]

Reading a bit about indexer files here: ​https://strongbox.github.io/developer-guide/maven-indexer.html Apparently this is a packed lucense index, cannot find the api URL to hit in order to check if this is working.

[meedel]

Replying to jive:

Reading a bit about indexer files here: ​https://strongbox.github.io/developer-guide/maven-indexer.html Apparently this is a packed lucense index, cannot find the api URL to hit in order to check if this is working.

Thanks Jive for your search and support. I have replaced the url to the documented url. ( stil get a 404 when testing ) The index files are shown, but no artifacts or directories. I have also made an issue for jfrog ​https://www.jfrog.com/jira/browse/RTFACT-21963

[jive]

I do not know if indexer files for a group repository are a thing (it shows it took zero seconds for the last run so I am thinking not).

I have just changed the task to make the indexer files for *all* repositories, I hope that helps, and running it now.

[jive]

If you get a chance to learn of a specific URL that we can use to test if indexer is working that would be great.

Reading that documentation page they give examples of:

• Hosted repositories: have: Local: strongbox-vault/storages/${storageId}/${repositoryId}/local/.index
• Proxy repositories have: Remote: strongbox-vault/storages/${storageId}/${repositoryId}/remote/.index
• Group repositories have: Local: strongbox-vault/storages/${storageId}/${repositoryId}/local/.index

But that is for a specific product"strongbox", I do not know what the URLs are for nexus.

Sonyatype has an example program here: ​https://blog.sonatype.com/2009/06/nexus-indexer-api-part-2/ which may be useful also.

[jive]

Okay the job did something this time, it ran for4m4s!

Can you check now and see if things have improved?

[jive]

I am hoping with the change this is now resolved for you.

[jive]

Still no feedback, I am only seeing a few warnings in the logs:

2020-05-05 18:09:44,875+0000 WARN  [qtp623246820-513087]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Bad request. Reason: Repository path must have another '/' after initial '/'
2020-05-05 18:09:49,653+0000 WARN  [qtp623246820-513091]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Bad request. Reason: Repository path must have another '/' after initial '/'

[thijscramer]

Sorry for the late response, 4th and 5th are National Holidays in NL. I'll check the server first thing tomorrow morning.

[thijscramer]

Our Artifactory instance is still reporting 404's on the test, and listing nothing in the remote repository.

[meedel]

Replying to jive:

If you get a chance to learn of a specific URL that we can use to test if indexer is working that would be great.

Jive, Our Artifactory also services the maven central remote repository ​https://repo1.maven.org/maven2. And this one is working fine. Would that help?

[meedel]

Jive, If I try to download the two index files: nexus-maven-repository-index.gz or nexus-maven-repository-index.properties I get an error: "status" : 404,

"message" : "Resource has expired"

The two files are created 06-05-20 05:33:30 ( our local time )

These are the messages I got from the Artifactory logging from this morning:

020-05-06 05:33:27,521 [art-exec-75] [INFO ] (o.a.r.HttpRepo :470) - osgeo-remote downloading ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz 5.04 MB 2020-05-06 05:33:29,940 [art-exec-75] [INFO ] (o.a.r.HttpRepo :483) - osgeo-remote downloaded ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz 5.04 MB at 2,134.51 KB/sec 2020-05-06 05:33:30,108 [art-exec-75] [INFO ] (o.a.r.HttpRepo :470) - osgeo-remote downloading ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties 180 bytes 2020-05-06 05:33:30,108 [art-exec-75] [INFO ] (o.a.r.HttpRepo :483) - osgeo-remote downloaded ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties 180 bytes at 545.51 KB/sec 2020-05-06 05:33:30,294 [art-exec-75] [INFO ] (o.a.m.i.MavenIndexManager:268) - Successfully saved index file 'osgeo-remote-cache:.index/nexus-maven-repository-index.gz' and index info 'osgeo-remote-cache:.index/nexus-maven-repository-index.properties'.

[robe]

@meedel,

I'm still a bit confused. The logs from artifactory don't show a 404 and give the file size.

When are you getting the 404 error?

Also the maven central link, what would be the equivalent for us

? this? which does give a 404? I just tried and was able to download

​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties

and ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz

just fine

​https://repo.osgeo.org/maven2

[meedel]

Replying to robe:

@meedel,

I'm still a bit confused. The logs from artifactory don't show a 404 and give the file size.

When are you getting the 404 error?

Also the maven central link, what would be the equivalent for us

? this? which does give a 404? I just tried and was able to download

​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties

and ​https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz

just fine

​https://repo.osgeo.org/maven2

Jive, sorry not to be clear enough for you. Artifactory itself is able to download the two index files, but when I want to download these files from Artifactory I get the 404. I mentioned the ​https://repo1.maven.org/maven2 repository to see if there are differences in the makeup of the index files! Maybe there is a difference in the index files which Artifactory can't read. I am not able to check these files because they are binary files.

[meedel]

Jive, I have an answer from support from Jfrog. ( Artifactory ) They say the following:

Hello Bert,

It looks like Nexus 3.x doesn't offer an HTML Directory structure through the URL the Artifacts need to be resolved from. As you've encountered, you can still resolve Artifacts but you won't be able to use the Artifact browser.

If you would like to partially use the Artifact browser, use the following setup:

1. Remote repository pointing to ​https://repo.osgeo.org/repository/release/
2. Remote repository pointing to ​https://repo.osgeo.org/service/rest/repository/browse/release/
3. Virtual repository containing both remote repositories.

This will allow you to view the directory structures of the Artifacts before resolving through your build. You will not be able to download via the UI as the files do not exist in the /service/ context. For example, if you navigate to ​https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/, the folders maintain the /service/rest/ context, but the files switch to the /repository/release/ context.

To resolve the artifacts in a 2 step way is not an option for us.

[robe]

@meedel - is this related to this thread

​https://community.sonatype.com/t/maven-nexus-v3-remote-repository-integration-with-artifactory/2149

[meedel]

Replying to robe:

@meedel - is this related to this thread

​https://community.sonatype.com/t/maven-nexus-v3-remote-repository-integration-with-artifactory/2149

Yes we have the same experiance. Did the remarks of Jfrog support give you any answer?

[jive]

So if I understand ​2149:

Known issue with Artifactory integration written for Nexus 2, we use Nexus 3 leading to the following workaround:

To integrate repo.osgeo.org as a remote repository into artifactory:

1. Use the nexus URL: ​https://repo.osgeo.org/repository/releases/

2. Note that testing the URL before saving it will give a 404 warning

3. After running your maven build artifactory will be able to fetch the the artifacts

This problem occurs as the artifactory test expects to be able to browse to the Nexus URL location. However in Nexus 3 repository URLs are machine readable, with a separate URL for browsing.

If the above is correct I will add it to our wiki and resolve this issue.

[jive]

I have gone ahead and updated the wiki here: ​https://wiki.osgeo.org/wiki/SAC:Repo#How_to_integrate_with_Artifactory_as_a_remote_repository

[robe]

Is this still an issue or can we close this out?

[meedel]

Hello Jive, We stil have issues with it, but you can close this one. The problem is within Artifactory.

The answer I got from Jfrog support was:

It looks like Nexus 3.x doesn't offer an HTML Directory structure through the URL the Artifacts need to be resolved from. As you've encountered, you can still resolve Artifacts but you won't be able to use the Artifact browser.

If you would like to partially use the Artifact browser, use the following setup:

1. Remote repository pointing to ​https://repo.osgeo.org/repository/release/
2. Remote repository pointing to ​https://repo.osgeo.org/service/rest/repository/browse/release/
3. Virtual repository containing both remote repositories.

This will allow you to view the directory structures of the Artifacts before resolving through your build. You will not be able to download via the UI as the files do not exist in the /service/ context. For example, if you navigate to ​https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/, the folders maintain the /service/rest/ context, but the files switch to the /repository/release/ context.

Best regards, John Wright JFrog Support

We have tested the 2 step way to get the right information but that's not working for us. So we use the native repo url in our software.

Thanks for your support