raster/test/regress/loader/TiledAuto takes ~20 seconds

[Algunenano]

I don't know what raster/test/regress/loader/TiledAuto regress test is doing, but it takes 18-20 seconds out of every run.

As it's executed twice (normal + upgrade), a total of 40 seconds are used just by that single test. In my machine, that's 15-20% of the total time.

It should be either simplified, replaced or removed (depending on how important it is).

[robe]

ah indeed gitlab ci shows this

 ./raster/test/regress/loader/Tiled10x10Copy .... ok in 387 ms
 ./raster/test/regress/loader/Tiled8x8 .... ok in 397 ms
 ./raster/test/regress/loader/TiledAuto .... ok in 14968 ms
 ./raster/test/regress/loader/TiledAutoSkipNoData .... ok in 2181 ms

A little lower than yours— my machine is broken at the moment can't run any raster load tests.

It's too bad we don't have timings on the 3.0 branch to see if it's a new issue or has always taken that long.

Looking at the timestamps on Debbie's 3.0 logs, I guess it's an issue in 3.0 too

From Debbie's last 3.0 run

13:57:34  loader/Tiled8x8 .... ok 
13:57:34  loader/TiledAuto .... ok 
13:57:48  loader/TiledAutoSkipNoData .... ok 
13:57:51  clean .. ok

Still not as sucky as your timings but close.

[robe]

These tests do not exist in PostGIS 2.5. I'm guessing they were introduced by Darafei in 3.0 when he changed raster to by default skip no data, and then introduced the -k switch to allow people to have backward compatibility.

  -k  Skip NODATA value checks for each raster band.

It is surprising though such a small raster could take that much effort.. We can probably take out the -C (constraint adding check) since I think the point of this exercise is that the TiledAuto generates a lot more tiles than the TiledAutoSkipNoData.

Unfortunately my loading is broken (thanks strk). So I can't test out the theory of taking out the -C check in the TiledAuto.opts file.

[strk]

Unfortunately my loading is broken (thanks strk).

Did you file a ticket ? Reference ?

[strk]

For the record: on my machine the test takes 7 seconds with -C and 6.8 seconds without -C. I'm using GDAL 3.1.0, released 2020/05/03

[strk]

The test raster is really too big to me:

testraster2.tif TIFF 3577x6128 3577x6128+0+0 64-bit Grayscale Gray 44459B 0.000u 0:00.000

Compare with the other test tif files:

testraster.tif TIFF 90x50 90x50+0+0 8-bit sRGB 13674B 0.000u 0:00.000

Projected.tif TIFF 90x50 90x50+0+0 8-bit sRGB 13894B 0.000u 0:00.000

Komzpa: do we need such a big raster ? Or can you reduce it ? (ref: #4413)

[robe]

Did we do something about this test? My local output is showing

  Postgis 3.1.0alpha3 - (3.1.0alpha2-164-gd2d5c9f98) - 2020-11-19 02:47:08
  scripts 3.1.0alpha3 3.1.0alpha2-164-gd2d5c9f98
  raster scripts 3.1.0alpha3 3.1.0alpha2-164-gd2d5c9f98
  GEOS: 3.9.0-CAPI-1.14.0
  PROJ: 7.2.0
  SFCGAL: 1.3.8
  GDAL: GDAL 3.2.0, released 2020/10/26

 ./raster/test/regress/loader/TiledAuto .... ok in 8618 ms
 ./raster/test/regress/loader/TiledAutoSkipNoData .... ok in 1382 ms

But I can't remember if I ever looked locally but my timings are half.

[strk]

Nothing done, we still want to reduce that raster

[stefanpetrea]

I ran the tests before and after resizing the .tif file locally just to check how much faster the tests would run:

gdalwarp -ts 200 0 ./raster/test/regress/loader/testraster2.tif ./raster/test/regress/loader/testraster3.tif
mv ./raster/test/regress/loader/testraster{3,2}.tif

Before:

 ./loader/TiledAuto .... ok in 4872 ms
 ./loader/TiledAutoSkipNoData .... ok in 1547 ms

After:

 ./loader/TiledAuto .... ok in 158 ms
 ./loader/TiledAutoSkipNoData .... ok in 115 ms

Setup used:

PostgreSQL 11.11 (Debian 11.11-0+deb10u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
  Postgis 3.2.0dev - (3.1.0rc1-180-g4e9b22630) - 2021-05-05 14:10:15
  scripts 3.2.0dev 3.1.0rc1-180-g4e9b22630
  raster scripts 3.2.0dev 3.1.0rc1-180-g4e9b22630
  GEOS: 3.7.1-CAPI-1.11.1 27a5e771
  PROJ: Rel. 5.2.0, September 15th, 2018
  GDAL: GDAL 2.4.0, released 2018/12/14

[robe]

Thanks - can you add your resized to a pull request and I'll commit.

[stefanpetrea]

Okay, ​PR 613 is available

[Regina Obe <lr@…>]

In 5ab22a7/git:

resized testraster2.tif and updated tests
Closes #4783
previous size: 3577x6128
current size: 200x343