SVN buildbot failures

[robe]

Been getting errors on both Debbie and Winnie when connecting to SVN for past couple of runs.

Caused by: org.tmatesoft.svn.core.SVNCancelException: svn: E200015: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
svn: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:60)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.findStartingProperties(DAVUtil.java:133)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getBaselineProperties(DAVUtil.java:242)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getBaselineInfo(DAVUtil.java:200)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.logImpl(DAVRepository.java:1021)
	at org.tmatesoft.svn.core.io.SVNRepository.log(SVNRepository.java:1038)
	at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:181)
	at org.tmatesoft.svn.core.internal.wc2.remote.SvnRemoteLog.run(SvnRemoteLog.java:35)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:968)
	at org.tmatesoft.svn.core.wc.SVNLogClient.doLog(SVNLogClient.java:873)
	at hudson.scm.SubversionChangeLogBuilder.buildModule(SubversionChangeLogBuilder.java:187)
	... 12 more
Caused by: org.tmatesoft.svn.core.SVNCancelException: svn: E200015: ISVNAuthentication provider did not provide credentials; HTTP authorization cancelled.
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:723)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:398)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:386)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:863)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doPropfind(DAVConnection.java:151)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getProperties(DAVUtil.java:70)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getResourceProperties(DAVUtil.java:76)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getStartingProperties(DAVUtil.java:100)
	at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.findStartingProperties(DAVUtil.java:122)

problem I think is intermittent

[strk]

They might have been cought by recent anti-DOS and anti-SPAM measures. I'll take a look. Can you give me some timestamps about when that is happening ?

[strk]

This is Debbie (times are in PDT timezone)

May 12 06:48:06 git mod_evasive[903]: Blacklisting address 45.58.41.142: possible DoS attack.

Is debbie using git.osgeo.org to fetch code ?

I cannot find a match for Winnie, does she fetch code via some proxy ?

[strk]

I can confirm that Debbie did make more than 30 requests within a second at

[12/May/2016:06:17:01 -0700]

All the requests were SVN related, something like a full checkout of GDAL SVN (all branches). Beside relaxing the rule, could you tweak the script to be lighter ? Like, do you need _all branches_ ?

[strk]

See ​osgeo:#1678

[strk]

Sorry, I just realized it takes 50 requests per second, not 30, to block. Or 3 requests per second to the _same_ URL. In this case, though, I do see an svn checkout operation can issue multiple PROPFIND requests to the _same_ URI (would sound like an SVN client bug to me).

[strk]

Ok these are more than 3 requests for /postgis/!svn/vcc/default in the exact second of the DOS prevention trigger (REPORT, PROPFIND, REPORT, PROPFIND):

svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "REPORT /postgis/!svn/vcc/default HTTP/1.1" 200 444
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis/trunk HTTP/1.1" 207 651
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis/!svn/vcc/default HTTP/1.1" 207 552
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis/!svn/bc/14897/trunk HTTP/1.1" 207 514
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis/trunk HTTP/1.1" 207 651
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "REPORT /postgis/!svn/vcc/default HTTP/1.1" 200 444
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis HTTP/1.1" 207 637
svn-access.log:45.58.41.142 - - [12/May/2016:06:48:06 -0700] "PROPFIND /postgis/!svn/vcc/default HTTP/1.1" 401 647

[robe]

hmm should I upgrade SVN? or might have something to do with matrix. it's annoying because it ceates a separate repo for each matrix sub job which I don't even use. Though those are sequential so I wouldn't think it would trigger.

[strk]

I don't know if updating SVN would help, but I'd like to find out which exact client command triggers the requests flood, and see if it can be made less intrusive.

[strk]

I believe the rules on the svn server are more relaxed now, feel free to reopen if this is still a problem

[robe]

Milestone renamed