Opened 12 years ago

Closed 12 years ago

#4365 closed defect (fixed)

/vsicurl does not work with ftp url

Reported by: etourigny Owned by: warmerdam
Priority: normal Milestone: 1.9.0
Component: default Version: unspecified
Severity: normal Keywords: vsicurl ftp
Cc: Even Rouault

Description (last modified by etourigny)

The /vsicurl/ftp: mechanism does not work (in my machine, ubuntu 10.04, gdal-1.9.0) with the url used in autotest http_4 (in gdalhttp.py). 

  TEST: http_4 ... ERROR 4: `/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif' not recognised as a supported file format.

The same file can be accessed through http: but not ftp: , and the file is accessible through 

 $ gdalinfo --debug on /vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 82943228, time = 2009/10/01 00:00:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 80196041, time = 2009/10/08 00:00:00
VSICURL: Downloading 0-16383 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
ERROR 4: `/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif' not recognised as a supported file format.

$ gdalinfo --debug on /vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 0, time = 2009/10/01 13:21:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 0, time = 2009/10/08 16:10:00
VSICURL: Downloading 0-16383 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: GetFileSize(http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)=82943228  response_code=206
VSICURL: Downloading 16384-32767 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Downloading 32768-65535 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
GDAL: GDALOpen(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0xf3b450) succeeds as GTiff.
Driver: GTiff/GeoTIFF
 gdalinfo --debug on /vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 82943228, time = 2009/10/01 00:00:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 80196041, time = 2009/10/08 00:00:00
VSICURL: Downloading 0-16383 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
VSICURL: Downloading 16384-32767 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
VSICURL: Downloading 32768-65535 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
GDAL: GDALOpen(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x18dfc80) succeeds as GTiff.
Driver: GTiff/GeoTIFF
GDAL: GDALDefaultOverviews::OverviewScan()
Files: /vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
Size is 12178, 13435
Coordinate System is `'
Metadata:
  TIFFTAG_DATETIME=2009:10:01 08:12:22
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=Adobe Photoshop CS2 Windows
  TIFFTAG_XRESOLUTION=300
  TIFFTAG_YRESOLUTION=300
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,13435.0)
Upper Right (12178.0,    0.0)
Lower Right (12178.0,13435.0)
Center      ( 6089.0, 6717.5)
Band 1 Block=12178x7 Type=Byte, ColorInterp=Red
Band 2 Block=12178x7 Type=Byte, ColorInterp=Green
Band 3 Block=12178x7 Type=Byte, ColorInterp=Blue
GDAL: GDALClose(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x18dfc80)

$ gdalinfo --debug on /vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 0, time = 2009/10/01 13:21:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 0, time = 2009/10/08 16:10:00
VSICURL: Downloading 0-16383 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
VSICURL: GetFileSize(http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)=82943228  response_code=206
VSICURL: Downloading 16384-32767 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
VSICURL: Downloading 32768-65535 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
GDAL: GDALOpen(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x198e450) succeeds as GTiff.
Driver: GTiff/GeoTIFF
GDAL: GDALDefaultOverviews::OverviewScan()
Files: /vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
Size is 12178, 13435
Coordinate System is `'
Metadata:
  TIFFTAG_DATETIME=2009:10:01 08:12:22
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=Adobe Photoshop CS2 Windows
  TIFFTAG_XRESOLUTION=300
  TIFFTAG_YRESOLUTION=300
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,13435.0)
Upper Right (12178.0,    0.0)
Lower Right (12178.0,13435.0)
Center      ( 6089.0, 6717.5)
Band 1 Block=12178x7 Type=Byte, ColorInterp=Red
Band 2 Block=12178x7 Type=Byte, ColorInterp=Green
Band 3 Block=12178x7 Type=Byte, ColorInterp=Blue
GDAL: GDALClose(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x198e450)

Change History (6)

comment:1 by Even Rouault, 12 years ago

Resolution: → fixed
Status: new → closed

r23462 /trunk/autotest/gdrivers/gdalhttp.py: http_4 test : just skip it in case of failure (#4365)

comment:2 by etourigny, 12 years ago

Would it not be worth investigating the cause of this problem?

$ curl --version curl 7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.5 libidn/1.18 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

comment:3 by Even Rouault, 12 years ago

The main reason for me to give up is that I don't manage to reproduce the issue on any of my test environments... You're free to investigate of course ;-)

By the way, how come do you have curl 7.21 on Ubuntu 10.04 ? Mine has 7.19.7.

in reply to:  3 comment:4 by etourigny, 12 years ago

Resolution: fixed
Status: closed → reopened

Replying to rouault:

The main reason for me to give up is that I don't manage to reproduce the issue on any of my test environments... You're free to investigate of course ;-)

Although I don't know enough about libcurl, I would be tempted to leave this bug open (as it still is unresolved). I don't know which component to put it in though. If I have the time (some day) I'll look into it.

By the way, how come do you have curl 7.21 on Ubuntu 10.04 ? Mine has 7.19.7.

My mistake, I use Ubuntu 11.04 (probably switch to latest Mint during the holidays)

comment:5 by etourigny, 12 years ago

Cc: Even Rouault added
Description: modified (diff)

I get the response code 225 (Data connection open, no transfer in progress.), in function VSICurlHandle::DownloadRegion, line 659: curl_easy_getinfo(hCurlHandle, CURLINFO_HTTP_CODE, &response_code);

Adding test for that response code seems to work, although the transfer is longer than http (more latency I guess).

in ftp mode: VSICURL: Got reponse_code=225 content_type=[(null)]

in http mode: VSICURL: Got reponse_code=206 content_type=[image/tiff]

Even, can you test this in your system, and see what response code you get?

code: 

    long response_code = 0;
    curl_easy_getinfo(hCurlHandle, CURLINFO_HTTP_CODE, &response_code);

    char *content_type = 0;
    curl_easy_getinfo(hCurlHandle, CURLINFO_CONTENT_TYPE, &content_type);

+    if (ENABLE_DEBUG)
+        CPLDebug("VSICURL", "Got reponse_code=%ld content_type=[%s]", response_code, content_type);

+    if ((response_code != 200 && response_code != 206 &&
+        response_code != 226 && response_code != 426 && response_code != 225) || sWriteFuncHeaderData.bError)
-    if ((response_code != 200 && response_code != 206 &&
-        response_code != 226 && response_code != 426) || sWriteFuncHeaderData.bError)
    {

output after fix: 

$ gdalinfo --debug on /vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 82943228, time = 2009/10/01 00:00:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 80196041, time = 2009/10/08 00:00:00
VSICURL: Downloading 0-16383 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
VSICURL: Downloading 16384-32767 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
VSICURL: Downloading 32768-65535 (ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=225 content_type=[(null)]
GDAL: GDALOpen(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x18dfc80) succeeds as GTiff.
Driver: GTiff/GeoTIFF
GDAL: GDALDefaultOverviews::OverviewScan()
Files: /vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
Size is 12178, 13435
Coordinate System is `'
Metadata:
  TIFFTAG_DATETIME=2009:10:01 08:12:22
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=Adobe Photoshop CS2 Windows
  TIFFTAG_XRESOLUTION=300
  TIFFTAG_YRESOLUTION=300
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,13435.0)
Upper Right (12178.0,    0.0)
Lower Right (12178.0,13435.0)
Center      ( 6089.0, 6717.5)
Band 1 Block=12178x7 Type=Byte, ColorInterp=Red
Band 2 Block=12178x7 Type=Byte, ColorInterp=Green
Band 3 Block=12178x7 Type=Byte, ColorInterp=Blue
GDAL: GDALClose(/vsicurl/ftp://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x18dfc80)

 $ gdalinfo --debug on /vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
VSICURL: GetFileList(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif)
VSICURL: File[0] = MCR2010_01.tif, is_dir = 0, size = 0, time = 2009/10/01 13:21:00
VSICURL: File[1] = MCR2010_tif.zip, is_dir = 0, size = 0, time = 2009/10/08 16:10:00
VSICURL: Downloading 0-16383 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
VSICURL: GetFileSize(http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)=82943228  response_code=206
VSICURL: Downloading 16384-32767 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
VSICURL: Downloading 32768-65535 (http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif)...
VSICURL: Got reponse_code=206 content_type=[image/tiff]
GDAL: GDALOpen(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x198e450) succeeds as GTiff.
Driver: GTiff/GeoTIFF
GDAL: GDALDefaultOverviews::OverviewScan()
Files: /vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif
Size is 12178, 13435
Coordinate System is `'
Metadata:
  TIFFTAG_DATETIME=2009:10:01 08:12:22
  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
  TIFFTAG_SOFTWARE=Adobe Photoshop CS2 Windows
  TIFFTAG_XRESOLUTION=300
  TIFFTAG_YRESOLUTION=300
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,13435.0)
Upper Right (12178.0,    0.0)
Lower Right (12178.0,13435.0)
Center      ( 6089.0, 6717.5)
Band 1 Block=12178x7 Type=Byte, ColorInterp=Red
Band 2 Block=12178x7 Type=Byte, ColorInterp=Green
Band 3 Block=12178x7 Type=Byte, ColorInterp=Blue
GDAL: GDALClose(/vsicurl/http://ftp2.cits.rncan.gc.ca/pub/cantopo/250k_tif/MCR2010_01.tif, this=0x198e450)

comment:6 by Even Rouault, 12 years ago

Milestone: → 1.9.0
Resolution: → fixed
Status: reopened → closed

Thanks for investigating this ! (I got response_code=426.)

r23464 /trunk/ (autotest/gdrivers/gdalhttp.py gdal/port/cpl_vsil_curl.cpp): /vsicurl/ : accept 225 as a valid response code for FTP downloads (#4365)

Note: See TracTickets for help on using tickets.