revive geo-torrents?

[hamish]

Hi,

since ER Mapper is no more their geo torrents.org site has gone away leaving a void in the large geodata dataset distribution world.

Due to an apache bug we can't serve the un-split live-dvd ISO in one piece. (>2gb)

A bright spark in the liveDVD mailing list has figured out how to use openbittorrent.com to act as the tracker,

​http://lists.osgeo.org/pipermail/live-demo/2009-September/000569.html

has uploaded the .torrent to download.osgeo.org, and is now distributing the latest liveDVD .iso build with that. Only problem is that it still relies on his home computer as the primary seed.

if download.osgeo.org were running a bittorrent client, OSGeo's fat pipes & uptime could take over that seeding responsibility. There's also the option that other impatient types could start downloading the ISO before it finished uploading (which takes about 4 hours), but it may be preferable to have the osgeo server only allow torrent data to flow outwards.

I'm really not sure what goes into setting this up, before doing any research on the matter I thought I'd check if the geodata crowd would want on board, what their needs were, and if the SAC had any reservations or ideas about such a thing.

?

cheers, Hamish

[wildintellect]

I'm on the SAC mailing list now so we can properly discuss this. Sure long term we need to upgrade Apache to get rid of this 2GB. In the mean time and possibly a long term helpful idea I suggest we start seeding torrents from an osgeo server or 2.

I've narrowed down the software options to ​transmission or ​rtorrent

Both would work well on a headless server and can handle auto torrenting from a watched folder. Meaning the rules could be that if it's in the torrent folder it will be seeded, remove it from said folder to stop seeding. Not sure if symlinks work on that.

The other big decision would be what to cap the upload rate at. Note: this could actually be used to help upload too for some of our associates in far off places, the occasionally have to restart big uploads, but if they seeded instead md5 would be checked automatically and others could help pull chunks to closer machines.

See ​http://wiki.osgeo.org/wiki/GISVM_Build#Build_the_Live_DVD_ISO_image_build for some open free trackers that we're using.

Side notes: rtorrent does included a tracker, but we might all consider hosting a mirror of the OpenTracker project - they provide hardware if we have space(or something like that) transmission has an optional web management interface

[hamish]

Replying to wildintellect:

Both would work well on a headless server and can handle auto torrenting from a watched folder. Meaning the rules could be that if it's in the torrent folder it will be seeded, remove it from said folder to stop seeding. Not sure if symlinks work on that.

mmm, it is more work to admin but I more favor an explicit approach to distribution. makes it harder to mess up really really badly.

The other big decision would be what to cap the upload rate at.

why have a cap at all? AFAIK bandwidth is "not really a problem" for the SDSC.

Side notes: rtorrent does included a tracker, but we might all consider hosting a mirror of the OpenTracker project - they provide hardware if we have space(or something like that)

[we can (and do) use the opentracker already which is grand; the less stuff we have to admin the better]

as for hosting a mirror of it... um, I'm not really sure what else is tracked by that site (AFAICT from their website neither do/can they) but the U.S. in general and California in particular are perhaps not the legal jurisdictions you want to test that in. The state is pretty much bankrupt and Hollywood does bring a few tax dollars I imagine. That, and our hosts might get grumpy about all the OT traffic.

transmission has an optional web management interface

is that much of a plus over command line reports? I guess it would be a bit like sitestats apache log viewers to show you what's going on? If so, could be an interesting feature. But if it's just a webform crutch for people who are afraid of the command line I don't see much gain.

obviously we don't want this abused by 3rd parties in any way, so the more ways we can find to lock it down the better :)

Hamish

[wildintellect]

The bandwidth cap is to prevent the torrents from making the server unusable. Torrents have been know to essentially prevent ssh, and take up all the system resources if you let them. Seems reasonable to also ensure that other downloads from the server continue as is.

The web GUI as I understand it will let you see the health, seeders, leachers and speed of the torrents much in the way most torrent GUIs on the desktop do. I would say firewall the port it's on and require it to be fowarded over ssh if you want to see it. Might be easier to use than the cli for reviewing status...not sure though since I haven't used either.

I'm not concerned about legal issues, OSGeo does not host or seed anything of questionable nature.

[wildintellect]

Did some more research on the option, short story rtorrent looks harder to use. Basically now just need SAC to decide which and best install approach.

Transmission
Pro

• Can create torrents
• Web interface can restrict requests by IP address, default is 127.0.0.1
• CLI takes args and displays results, does not run by default but will stay running till stopped
• Better Docs

Con

• Requires RPC daemon for watch folder feature, this enables it's web interface, so need to make sure to firewall the port.
• CLI and Daemon are 2 separate apps
• Needs a startup script written if you want it auto on boot up
• ​Maybe theres a package for FC4 Not sure what version

rTorrent
Pro

• Inlcudes a Tracker (If we want one)
• Can do Superseeding (Jump start torrents quickly)

Con

• NCurses interface, not clear how to exit and keep torrents running
• Version with Super Seeding not available as ​rpm

[hamish]

Perhaps the Ibiblio people would be interested in hosting a Geo Channel on their new torrent site.

Hamish

[robe]

moot