Opened 6 years ago

Closed 6 years ago

#4164 closed defect (fixed)

vsicurl directory parsing slow with large directories

Reported by: warmerdam Owned by: warmerdam
Priority: normal Milestone: 1.9.0
Component: default Version: unspecified
Severity: normal Keywords: curl
Cc:

Description

Parsing large directory listings via the vsicurl dirver seems to be painfully slow.

eg.

http://hyperquad.telascience.org/naipsource/Texas/20100801/naip10_1m_2597_03_2_20100424.tif

The problem seems to be ordern(n-squared) behavior in the directory parsing. ie. use of strstr() to find the next of something which can be expensive in a long document if it is not found for each entry.

I will dig into this a bit.

Change History (2)

comment:1 Changed 6 years ago by warmerdam

I have correct the worst offending part of the code in trunk (r22728). There is still an n-squared problem with adding the filenames to the filename list but I'm addressing that a bit distinctly since my approach will be more complicated than is suitable to back port into 1.8.

comment:2 Changed 6 years ago by warmerdam

Keywords: curl added
Milestone: 1.9.0
Resolution: fixed
Status: newclosed

I also patched things to use a more efficient allocation strategy for the string list of filenames (r22731). Due to use of CPLStringList this is not suitable to back port.

Note: See TracTickets for help on using tickets.