Opened 12 years ago

Closed 12 years ago

#4164 closed defect (fixed)

vsicurl directory parsing slow with large directories

Reported by: warmerdam Owned by: warmerdam
Priority: normal Milestone: 1.9.0
Component: default Version: unspecified
Severity: normal Keywords: curl


Parsing large directory listings via the vsicurl dirver seems to be painfully slow.


The problem seems to be ordern(n-squared) behavior in the directory parsing. ie. use of strstr() to find the next of something which can be expensive in a long document if it is not found for each entry.

I will dig into this a bit.

Change History (2)

comment:1 by warmerdam, 12 years ago

I have correct the worst offending part of the code in trunk (r22728). There is still an n-squared problem with adding the filenames to the filename list but I'm addressing that a bit distinctly since my approach will be more complicated than is suitable to back port into 1.8.

comment:2 by warmerdam, 12 years ago

Keywords: curl added
Milestone: 1.9.0
Resolution: fixed
Status: newclosed

I also patched things to use a more efficient allocation strategy for the string list of filenames (r22731). Due to use of CPLStringList this is not suitable to back port.

Note: See TracTickets for help on using tickets.