Add "Big Data" to common datasets

[schpidi]

There was some ​discussion about adding "Big Data" to the common datasets. This ticket is created to track the status for version 7.0.

Known tasks are:

• Evaluate common datasets particularly with respect to storage size
• Define Big Data to be added. Suggestion: A hyperspectral dataset of ~60MB (are there any free AVIRIS, Hyperion, CHRIS/PROBA, MODIS datasets for North Carolina where we have a full dataset?)
• Add an overview for "Big Data" similar to the one for ​Natural Earth

[rasdaman]

Four dimensional datasets such as climate data with x,y,h,t would also be nice to include (Alan)

[pebau]

Right, Natural Earth seems to have only 2D; for the full world of raster data (1D through 4D), see the rasdaman demo (​http://live.osgeo.org/en/quickstart/rasdaman_quickstart.html).

[kalxas]

Hi Peter and Alan. There are some new data in the disk ​http://download.osgeo.org/livedvd/data/netcdf/ Can those be used by rasdaman or should we look for hyperspectral? Thoughts?

Angelos

[pebau]

Hi Angelos, they could be used in principle, but they are boring AFAICS: 3 time slices, that's not a time series. And, indeed, hyperspectral can do more interesting stuff (eg, NDVI). We have something on stock for various situations in life, see again the rasdaman demo mentioned above. There's 1D through 4D ("Variety" aspect in Big Data!) which certainly could be used by other tools as well. BTW, following our "Big Data" discussion last year which ended up in "sorry, no space left on device", maybe we also can utilize the door through which these data now have passed successfully ;-) Also, let me renew our offer to sponsor USBs if we can get some decent space for the rasdaman demos in turn - last year there was a tech showstopper, some systems allegedly were not be able to boot from 8 GB sticks, maybe this is obsolete meantime?

-Peter

[kalxas]

Hi Peter.

I would suggest to re-open this issue in the next weekly meeting (Thursday 20:30 UTC) on IRC and see what can be done.

Cheers, Angelos

[kalxas]

Ticket retargeted after milestone closed

[kalxas]

Ticket retargeted after milestone closed

[kalxas]

Now we know that the 4GB size limit is a hard one for the live system due to USB file system and squashfs limitations