svn http 502 bad gateway errors on MapGuide/FDO repos

[jng]

I've been experiencing http 502 bad gateway errors on the MapGuide/FDO svn repos.

Some examples:

1. Using the ​TortoiseSVN windows client, trying to bring up the log of any directory of a svn checkout within brings up the bad gateway error (see attached image)

2. I've been experimenting with some git mirroring/migration scripts and the MapGuide repo fails with bad gateway when I try to run ​svn2git

Example svn2git command:

svn2git https://svn.osgeo.org/mapguide --authors ~/svn_authors.txt --metadata --no-minimize-url

It fails trying to pull out commit r1 with

RA layer request failed: Unexpected HTTP status 502 'Bad Gateway' on '/mapguide/!svn/rvr/29/trunk/MgDev/UnitTest/TestData/FeatureService/FeatureServiceTest.dump' at /usr/share/perl5/Git/SVN/Ra.pm line 312

For 1. I can somewhat workaround this by using the SVN command-line client, where the bad gateway errors aren't happening, though I much prefer to do merges through the TortoiseSVN GUI, so this problem hampers by productivity somewhat.

For 2. this is a more pressing problem as I would one day like to see MapGuide/FDO formally be migrated to git, but if I can't pull out the full history of these repos because of bad gateway errors, that idea of git mirroring/migration with full history is dead in the water.

I theorize this started happening when http://svn.osgeo.org was taken down and everything redirected to https://svn.osgeo.org. My git mirroring scripts worked then and I don't recall encountering bad gateway errors with TortoiseSVN.

[jng]

TortoiseSVN bad gateway error

[robe]

I'll check this out later today.

[robe]

Okay I just tried with my Windows Tortoise SVN

doing checkout of this ​https://svn.osgeo.org/mapguide/

and was able and was going well pulled a ton of file but then I did finally get a Bad Gateway: Error: Unexpected HTTP status 502 'Bad Gateway' on

'/mapguide/!svn/rvr/8981/trunk/MgDev/UnitTest/TestData/FeatureService/FeatureServiceTest.dump'

Doing ShowLog from Windows explorer tortoise svn brings up the show log fine though:

I see last revision 9720 - jng - August 18th, 2020

This might have to do with some measures strk put in a while back as I think our svn was being hit a lot and suffering dos attacks. I'll check with him to see if he remembers what he changed.

[strk]

There was an apache module, I think mod_evasive or similar. Are we still using Apache ?

[robe]

Yes we still use apache for trac and svn though the apache traffic proxies thru nginx. So don't discount nginx at fault.

[jng]

Just to add. The gateway error on TortoiseSVN log problem happens on the FDO repo.

It doesn't happen for me on the MapGuide one, that gateway error happens in a way described in example 2.

[robe]

hmm I get the same behavior with FDO and MapGuide using TortoiseSVN.

e.g. I can checkout and it goes for a while until it gives a bad gateway.

I can always see the log using the TortoiseSVN show log button. I'll try to look thru this mod_evasive module strk mentioned. I'm guessing it has to be some denial of service protection thing getting in the way and perhaps windows GUI is just more aggressive than the commandline and thus is more likely to trigger it.

[robe]

move open times to new contract

[robe]

oops moved to wrong milestone last run

[jng]

It seems it is no longer possible to do a clean SVN checkout of MapGuide.

It will inevitably fail with the 502 bad gateway on /mapguide/!svn/rvr/8981/trunk/MgDev/UnitTest/TestData/FeatureService/FeatureServiceTest.dump and no matter how many retries of svn cleanup and svn update to resume it, it will never finish the incomplete working copy.

For FDO, I still get 502 bad gateway on fresh checkout, but I can retry svn cleanup and svn update to repair the incomplete working copy to completion.

[robe]

jng sorry unfortunately I haven't figured out why this is happening. I did confirm I have the same issue too trying to do the pulls.

Not sure if it's the size of the pull or what.

Any chance you'd be amenable to moving to git. Seems most other OSGeo users have moved on from svn. We can at very least do a mirror to see if it's an issue with the server the way we have timeout configured or SVN itself.

Both git.osgeo.org and SVN are housed on tracsvn container.

[robe]

Disregard my comment about git. I forgot your last bit about trying to mirror to git.

I did find some misconfigurations in the nginx proxy which I fixed but that didn't fix the issue.

I also tried to pull using the svn commandline that comes with Visual SVN Server and it crapped out at the same spot

svn: E175002: Unexpected HTTP status 502 'Bad Gateway' on '/mapguide/!svn/rvr/8981/trunk/MgDev/UnitTest/TestData/FeatureService/FeatureServiceTest.dump'

And svn update on fdo commandline fails on:

svn: E175002: Unexpected HTTP status 502 'Bad Gateway' on '/fdo/!svn/rvr/7016/branches/3.9/Thirdparty/boost/boost/typeof/vector200.hpp'

I'm going to do an local checkout to see if I have same issue on server to rule out corruption and then look at nginx config to see if I'm missing a size set.

The vector200.hpp is 2.4 MB when I look thru the TortoiseSVN repo browser. The mapguide when I got thru repo browser just crashes on me (that might be my machine though)

[robe]

Sorry took so long to figure this out.

The nginx logs were showing the error so the issue was in nginx.

​https://ma.ttias.be/nginx-proxy-upstream-sent-big-header-reading-response-header-upstream/

I changed just the svn.osgeo.org conf for now but plan to change the main one included in all the sites once I confirm this will not cause any issues.

Please give another try and close this out if this fixed your issue.

[robe]

Committed at ​https://git.osgeo.org/gitea/sac/osgeo7/commit/1ae14bd40390441cf542ec53626bfe0945423847

[jng]

I can confirm that SVN checkouts of MapGuide/FDO/CS-Map now run to completion.

I still get sproadic failures when attempting to use svn2git on these repos, but I can re-run svn2git again afterwards and it continues the sync to completion.

I can also confirm that I no longer get 502 bad gateway errors when viewing SVN log history through TortoiseSVN.

So for all intents and purposes, I think this issue is resolved. Thank you very much for solving this problem.

[robe]

Milestone renamed