Speed up ReadDirRecursive implementation

[cdestigter]

We discovered that gdal.ReadDirRecursive is listing every directory and then calling stat() on each one to determine if it's a directory, even on normal local filesystems. This makes performance quite slow for large hierarchies.

My understanding is that this shouldn't normally be necessary, since the normal readdir functions yield dirent structs which disambiguate files and directories. I'd kind of assume that most remote filesystems would also yield that information in whatever their equivalents of readdir are

Python 3.5 added os.scandir() to the stdlib, which massively speeds up os.walk(), so ​PEP 471 might be a good description of the improvements possible here.

[Even Rouault]

That would require a new virtual method in the file system handlers to not only return the file name, but also their nature (+ size + last modified date)

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.