Downloads.osgeo Upgrade/Replacement

[wildintellect]

I was hoping to have an upgrade for Download.osgeo by the next release of the LiveDVD(August), looks like the current issues with the Blades has made this a more urgent matter. Below are some notes on ideas we might want to implement in the replacement service.

1. Finish ticket:336 - Antivirus scan new uploads (unclear if it was finished)
2. For non-SAC members with access use rssh to limit accounts to SFTP only. This can be used to prevent shell access, port forwarding and all sorts of other ways hacker might try use a compromised account.
3. Consider implementing a mirroring system, with something like ​http://www.mirrorbrain.org/ or ​https://fedorahosted.org/mirrormanager/
   • These applications keep an eye on the mirrors and can be configured so md5 etc are hosted on the main site but files can be selective mirrored on other servers. They also scan for status and verify that files match the main mirror.
   • The url to download is still the same, the manager automatically routes the download with a redirect to geographically closer mirrors, less impacted mirrors, etc.
   • Mirrors do not have to be complete mirrors they can be partial, and TOS rules can limit bandwidth on mirrors with limitations.
   • simgislab has offered a mirror(Ukraine), I may be able to offer a mirror (US-West) and a mirror in Asia and Australia(Lisasoft?) would immensely increase international download speeds.
   • This may also solve ticket:515 as it will route requests to multiple mirrors
4. A Newer OS than the current is needed, something that allows for larger than 2 GB file downloads(Possible Apache issue). Consider XFS filesystem or EXT4 for increased speed.

Please add additional ideas to the ticket so we can discuss them when we discuss the resolution of the current Blade hosted services.

[warmerdam]

ticket #336 was not completed. I experimented with clamav but did not get anything automated running.

Use of rssh to limit most access to sftp might be prudent if we are particularly concerned about a lot of people having login access, though it is not clear if that has been a serious problem.

I'm not sure why the proposed mirroring approach is better than the simple rsync based mirroring we already had in place, and can be setup fairly easily?

I'm presuming John will install something newish OS wise when he scrubs the blade.

I'm honestly still not so keen on livedvd's sloshing around on the download site, but I imagine I'm just going to have to live with it.

There has been some discussion with Chris about whether download.osgeo.org ought to live at OSU OSL with telascience hosting a live backup/mirror. I'm not sure how that decision will be resolved.

[wildintellect]

I should clarify our previous conversation on the topic. MirrorBrain does not do the mirroring, that would still be left to rsync. What it does do is allow us to seamlessly allow for many mirrors. Basically we don't have to choose OSL or Telescience, it could be both plus 2-3 more donated mirrors around the world but to the end user it's still only 1 url.

This lets us increase our download capacity, load balance between mirrors, selectively redirect things like the Live DVD downloads to specific mirrors, ensure integrity of mirrors, and keep logs of all downloads traffic even if they get redirected to a mirror. ​More details

[hamish]

Replying to warmerdam:

I'm honestly still not so keen on livedvd's sloshing around on the download site, but I imagine I'm just going to have to live with it.

fwiw I notice on the Server Status wiki page that xblade-15 is listed for ISO downloads...? that does not solve the reported (AFAIU) ">2gb files from apache2 needing a 64bit OS" problem, but as I just saw that I thought I'd mention it as another option for the future.

for my part I don't really care where it goes as long as we have a semi-official, semi-restricted, and mostly-static (for years) place to stash stuff, including things like large binary sample datasets & custom installer packages which do not belong in the build script Subversion. (I'm talking about max 100mb files used in the build, not the final ISOs & overhead of backing up the multi-gigabyte files)

it is perhaps nice to be able to chmod any md5sum listing on the download server to be go-w, but in the past I've mostly had to fix it the other way when the uploader did not make it g+w so it required more effort to maintain things...

regards, Hamish

[wildintellect]

1. Still todo
2. not deemed critical at this time
3. mirrorbrain being tested by wildintellect on a non-osgeo server in conjunction with the Live DVD downloads, download server is one of the mirrors.
4. solved with fresh vm