Opened 10 years ago

Closed 10 years ago

#435 closed task (fixed)

Benchmarking data download server

Reported by: Jeff McKenna Owned by: sac@…
Priority: normal Milestone:
Component: Systems Admin Keywords:
Cc: aaime, dmorissette, warmerdam, aboudreault

Description

As discussed in the last FOSS4G benchmarking meeting, we need a place to host the benchmarking data download. It's currently on a USACE server but connection speed is very slow...and in the long run we think OSGeo should host the data (and benchmarking test package) for the public to download. The data is ~32 GB in size.

Change History (9)

comment:1 Changed 10 years ago by Jeff McKenna

Can we set this up soon? (I'd like to point ESRI and ERDAS to this) Eventually this will be how the public can access our Web Server benchmarking files.

-jeff

comment:2 Changed 10 years ago by Jeff McKenna

Cc: dmorissette added

Daniel, I wonder if maptools.org could host the benchmarking data? (the data is about 32GB) Can we handle this extra bandwidth demand?

(on IRC FrankW stated that this could not be hosted on telascience servers)

comment:3 Changed 10 years ago by aaime

Mind the biggest part of the data is uncompressed. The shapefile and the big geotiffs (which are uncompressed) should go down a lot if they are tar.gz-ed (the ecw will not go down one bit instead).

One tgz per package would be a nice to redistribute the lot.

comment:4 Changed 10 years ago by warmerdam

Cc: warmerdam added

For the raster data I would suggest we only distribute the ecw, and provide scripts to translate that to bigtiff and tiles and add overviews. I can prepare those scripts at some point.

comment:5 Changed 10 years ago by aaime

Frank, you read my mind. I was actually looking for those scripts to replicate the tiffs on my box hoping to find them somewhere on goliath, but I had no such luck

comment:6 Changed 10 years ago by dmorissette

Cc: aboudreault added

Jeff, yes, we could host it on maptools.org as long as we don't get hundreds of downloads of it per month. However I would not put in the usual "dl" directory to avoid also wasting space on the master server which is replicated to the live server. I'd place the files directly in a directory on the live server bypassing the normal replication mechanism. Let's look into that with Alan on Monday.

comment:7 Changed 10 years ago by Jeff McKenna

Alan, let me know when you can discuss using maptools.org to host the benchmarking package.

All: I generated a TIGER 2008 dataset for GeoServer? and MapServer testing, that I will upload to this new server (it contains both as-is county files as well as processed merged files). I'm looking at TIGER 2008 styles now.

comment:8 Changed 10 years ago by Jeff McKenna

Alan, the files have transferred to your machine. Please upload the 4 zip files to maptools.org

raster.zip (.4 GB) vector-data-original.zip (.5 GB) vector-data-tiger08-tx-merged.zip (1.1 GB) vector-date-tiger08-tx-counties.zip (1.1 GB)

comment:9 Changed 10 years ago by Jeff McKenna

Resolution: fixed
Status: newclosed

Alan has set this up on maptools.org (notes at http://wiki.osgeo.org/wiki/Benchmarking_2009#Download).

marking as fixed.

Note: See TracTickets for help on using tickets.