#2445 closed task (wontfix)

Issue with Maven OSGEO (repo.osgeo.org) - no remote packages available

Reported by: robe Owned by: jive
Priority: normal Milestone: Unplanned
Component: Systems Admin Keywords:
Cc:

Description

Email thread I got about this


The archetype metadata file in the root is empty. Our Artifactory instance sees no remote packages on the suggested url. Also Maven Central days there is only 1 java package in the repo.


It seems that the OSGeo Nexus Repository is broken. Our Artifactory instance is no longer correctly mirroring the OSGeo Maven Release Repo. I couldn’t find any proper group on the mailing list to send a message to, so I’ve looked at who changed the repository documentation, and it was you.

If you are not the correct person to tell this to, please accept my sincerest apology, and could you be so kind as to point me to the correct person to help me. Thanks in advance! Kind Regards, Thijs Cramer


Attachments (2)

index-snapshot.png (65.8 KB) - added by jive 17 months ago.
index-release.png (63.4 KB) - added by jive 17 months ago.

Download all attachments as: .zip

Change History (33)

comment:1 Changed 17 months ago by jive

I have not had any reports of trouble from my automated build boxes or email lists.

Can you confirm that *the suggested url* is: https://repo.osgeo.org/repository/release/ ?

Doing a geoserver build with an empty local repository is functioning (making use of the above repository to fetch geotools dependencies).

comment:2 Changed 17 months ago by jive

It looks like there may be a "maven index" that we should publish once a day, used by other clients "nexus instances, or m2eclipse"

I am going to schedule a task to produce this once a day and we can ask if it fixes the problem. This is the first I have heard of this index as it is not something our previous webdav folder offered.

Reference:

comment:3 Changed 17 months ago by jive

I have setup tasks to create an index for the release and snapshot repositories; please let me know if that resolves the problem.

comment:4 Changed 17 months ago by jive

Why would maven central know about our repo? we have not requested to be mirrored yet ...

comment:5 Changed 17 months ago by thijscramer

I can confirm our configured URL is set to https://repo.osgeo.org/repository/release/. Using the "test" function Artifactory gives us a 404. We can ignore the test result and just add the repository, but it stays empty on the Artifactory side. It just doesn't see any remote packages. This is one of many maven(2) repositories we have configured, and we have no issues with any other repo's (like Maven Central for example). Which uses the exact same configuration, just another URL.

There's not much we can configure for a remote repo: https://www.jfrog.com/confluence/display/JFROG/Remote+Repositories

comment:6 Changed 17 months ago by thijscramer

If we use the url https://repo.osgeo.org/service/rest/repository/browse/release/ we get a good status back from Artifactory when testing. But the remote repo is filled up with only directory's. There are no artifacts shown.

comment:7 Changed 17 months ago by robe

When I browse it I do see real files --

https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/1.9.2/

e.g. there is a jar in that folder.

Does not come thru?

jive - did everything replicate over from boundless?

comment:8 Changed 17 months ago by meedel

It is correct that you can list the artifacts within a browser. But in our binary resource manager Artifactory I only get the directory listing and there are no artifacts shown! I have to use the url https://repo.osgeo.org/service/rest/repository/browse/release/ in Artifactory to see the directory listing. If I use the suggested url from the documentation https://repo.osgeo.org/repository/release/ I get a 404 in the test from Artifactory and the only directory what is shown is java3d/vecmath/1.5.2/ also without artifacts.

Version 1, edited 17 months ago by meedel (previous) (next) (diff)

comment:9 Changed 17 months ago by jive

Only the things we asked for replicated from boundless; they had lots of their own open source projects (non osgeo) which we did not take.

I am a bit confused about this thread, while I believe that artifactory has some secret api to communicate maven index - we have setup a task to refresh and publish this api. But have no way of testing if it is working or not ...

The browse links, and searching in the user interface, are fine for checking the contents visually. The actual maven build tool is finding everything it needs just fine. It is this "repository index" api which a downstream nexus is trying to make use of ....

meedel do you have any documentation on what needs to be done? I have set up two tasks as shown in the attachments. They both seem to be running ...

Changed 17 months ago by jive

Attachment: index-snapshot.png added

Changed 17 months ago by jive

Attachment: index-release.png added

comment:10 in reply to:  6 Changed 17 months ago by jive

Replying to thijscramer:

If we use the url https://repo.osgeo.org/service/rest/repository/browse/release/ we get a good status back from Artifactory when testing. But the remote repo is filled up with only directory's. There are no artifacts shown.

You are using a URL for browsing visually, what you want is the url we have documented: https://repo.osgeo.org/repository/release/

comment:11 Changed 17 months ago by jive

Reading a bit about indexer files here: https://strongbox.github.io/developer-guide/maven-indexer.html Apparently this is a packed lucense index, cannot find the api URL to hit in order to check if this is working.

comment:12 in reply to:  11 Changed 17 months ago by meedel

Replying to jive:

Reading a bit about indexer files here: https://strongbox.github.io/developer-guide/maven-indexer.html Apparently this is a packed lucense index, cannot find the api URL to hit in order to check if this is working.

Thanks Jive for your search and support. I have replaced the url to the documented url. ( stil get a 404 when testing ) The index files are shown, but no artifacts or directories. I have also made an issue for jfrog https://www.jfrog.com/jira/browse/RTFACT-21963

Last edited 17 months ago by meedel (previous) (diff)

comment:13 Changed 17 months ago by jive

I do not know if indexer files for a group repository are a thing (it shows it took zero seconds for the last run so I am thinking not).

I have just changed the task to make the indexer files for *all* repositories, I hope that helps, and running it now.

comment:14 Changed 17 months ago by jive

If you get a chance to learn of a specific URL that we can use to test if indexer is working that would be great.

Reading that documentation page they give examples of:

  • Hosted repositories: have: Local: strongbox-vault/storages/${storageId}/${repositoryId}/local/.index
  • Proxy repositories have: Remote: strongbox-vault/storages/${storageId}/${repositoryId}/remote/.index
  • Group repositories have: Local: strongbox-vault/storages/${storageId}/${repositoryId}/local/.index

But that is for a specific product"strongbox", I do not know what the URLs are for nexus.

Sonyatype has an example program here: https://blog.sonatype.com/2009/06/nexus-indexer-api-part-2/ which may be useful also.

comment:15 Changed 17 months ago by jive

Okay the job did something this time, it ran for4m4s!

Can you check now and see if things have improved?

comment:16 Changed 17 months ago by jive

Owner: changed from sac@… to jive
Status: newassigned

I am hoping with the change this is now resolved for you.

comment:17 Changed 17 months ago by jive

Still no feedback, I am only seeing a few warnings in the logs:

2020-05-05 18:09:44,875+0000 WARN  [qtp623246820-513087]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Bad request. Reason: Repository path must have another '/' after initial '/'
2020-05-05 18:09:49,653+0000 WARN  [qtp623246820-513091]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Bad request. Reason: Repository path must have another '/' after initial '/'

comment:18 Changed 17 months ago by thijscramer

Sorry for the late response, 4th and 5th are National Holidays in NL. I'll check the server first thing tomorrow morning.

comment:19 Changed 17 months ago by thijscramer

Our Artifactory instance is still reporting 404's on the test, and listing nothing in the remote repository.

comment:20 in reply to:  14 Changed 17 months ago by meedel

Replying to jive:

If you get a chance to learn of a specific URL that we can use to test if indexer is working that would be great.

Jive, Our Artifactory also services the maven central remote repository https://repo1.maven.org/maven2. And this one is working fine. Would that help?

comment:21 Changed 17 months ago by meedel

Jive, If I try to download the two index files: nexus-maven-repository-index.gz or nexus-maven-repository-index.properties I get an error: "status" : 404,

"message" : "Resource has expired"

The two files are created 06-05-20 05:33:30 ( our local time )

These are the messages I got from the Artifactory logging from this morning:

020-05-06 05:33:27,521 [art-exec-75] [INFO ] (o.a.r.HttpRepo? :470) - osgeo-remote downloading https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz 5.04 MB 2020-05-06 05:33:29,940 [art-exec-75] [INFO ] (o.a.r.HttpRepo? :483) - osgeo-remote downloaded https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz 5.04 MB at 2,134.51 KB/sec 2020-05-06 05:33:30,108 [art-exec-75] [INFO ] (o.a.r.HttpRepo? :470) - osgeo-remote downloading https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties 180 bytes 2020-05-06 05:33:30,108 [art-exec-75] [INFO ] (o.a.r.HttpRepo? :483) - osgeo-remote downloaded https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties 180 bytes at 545.51 KB/sec 2020-05-06 05:33:30,294 [art-exec-75] [INFO ] (o.a.m.i.MavenIndexManager:268) - Successfully saved index file 'osgeo-remote-cache:.index/nexus-maven-repository-index.gz' and index info 'osgeo-remote-cache:.index/nexus-maven-repository-index.properties'.

comment:22 Changed 17 months ago by robe

@meedel,

I'm still a bit confused. The logs from artifactory don't show a 404 and give the file size.

When are you getting the 404 error?

Also the maven central link, what would be the equivalent for us

? this? which does give a 404? I just tried and was able to download

https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties

and https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz

just fine

https://repo.osgeo.org/maven2

comment:23 in reply to:  22 Changed 17 months ago by meedel

Replying to robe:

@meedel,

I'm still a bit confused. The logs from artifactory don't show a 404 and give the file size.

When are you getting the 404 error?

Also the maven central link, what would be the equivalent for us

? this? which does give a 404? I just tried and was able to download

https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.properties

and https://repo.osgeo.org/repository/release/.index/nexus-maven-repository-index.gz

just fine

https://repo.osgeo.org/maven2

Jive, sorry not to be clear enough for you. Artifactory itself is able to download the two index files, but when I want to download these files from Artifactory I get the 404. I mentioned the https://repo1.maven.org/maven2 repository to see if there are differences in the makeup of the index files! Maybe there is a difference in the index files which Artifactory can't read. I am not able to check these files because they are binary files.

comment:24 Changed 17 months ago by meedel

Jive, I have an answer from support from Jfrog. ( Artifactory ) They say the following:

It looks like Nexus 3.x doesn't offer an HTML Directory structure through the URL the Artifacts need to be resolved from. As you've encountered, you can still resolve Artifacts but you won't be able to use the Artifact browser.

If you would like to partially use the Artifact browser, use the following setup:

  1. Remote repository pointing to https://repo.osgeo.org/repository/release/
  2. Remote repository pointing to https://repo.osgeo.org/service/rest/repository/browse/release/
  3. Virtual repository containing both remote repositories.

This will allow you to view the directory structures of the Artifacts before resolving through your build. You will not be able to download via the UI as the files do not exist in the /service/ context. For example, if you navigate to https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/, the folders maintain the /service/rest/ context, but the files switch to the /repository/release/ context.

To resolve the artifacts in a 2 step way is not an option for us.

Last edited 17 months ago by meedel (previous) (diff)

comment:26 in reply to:  25 Changed 16 months ago by meedel

Replying to robe:

@meedel - is this related to this thread

https://community.sonatype.com/t/maven-nexus-v3-remote-repository-integration-with-artifactory/2149

Yes we have the same experiance. Did the remarks of Jfrog support give you any answer?

comment:27 Changed 16 months ago by jive

So if I understand 2149:

Known issue with Artifactory integration written for Nexus 2, we use Nexus 3 leading to the following workaround:

To integrate repo.osgeo.org as a remote repository into artifactory:

  1. Use the nexus URL: https://repo.osgeo.org/repository/releases/
  1. Note that testing the URL before saving it will give a 404 warning
  1. After running your maven build artifactory will be able to fetch the the artifacts

This problem occurs as the artifactory test expects to be able to browse to the Nexus URL location. However in Nexus 3 repository URLs are machine readable, with a separate URL for browsing.

If the above is correct I will add it to our wiki and resolve this issue.

comment:29 Changed 15 months ago by robe

Is this still an issue or can we close this out?

comment:30 Changed 15 months ago by meedel

Hello Jive, We stil have issues with it, but you can close this one. The problem is within Artifactory.

The answer I got from Jfrog support was:

It looks like Nexus 3.x doesn't offer an HTML Directory structure through the URL the Artifacts need to be resolved from. As you've encountered, you can still resolve Artifacts but you won't be able to use the Artifact browser.

If you would like to partially use the Artifact browser, use the following setup:

  1. Remote repository pointing to https://repo.osgeo.org/repository/release/
  2. Remote repository pointing to https://repo.osgeo.org/service/rest/repository/browse/release/
  3. Virtual repository containing both remote repositories.

This will allow you to view the directory structures of the Artifacts before resolving through your build. You will not be able to download via the UI as the files do not exist in the /service/ context. For example, if you navigate to https://repo.osgeo.org/service/rest/repository/browse/release/commons-beanutils/commons-beanutils/, the folders maintain the /service/rest/ context, but the files switch to the /repository/release/ context.

Best regards, John Wright JFrog Support

We have tested the 2 step way to get the right information but that's not working for us. So we use the native repo url in our software.

Thanks for your support

comment:31 Changed 15 months ago by robe

Resolution: wontfix
Status: assignedclosed
Note: See TracTickets for help on using tickets.