gdal_GRASS.dll issue on windows Vista/Seven

[lutra]

Hi,

lately I have made quite a lot (>20) fresh osgeo4w installations on windows platforms (xp,vista and seven) and I'm pretty sure I can confirm the problem.

I have been told that have been a little controversy about this matter, you'll forgive me if I completely misunderstand what is going on (also because I'm no osgeo4w/gdal guru), so don't hesitate to close this ticket. In this case if you can give me (us) an explanation/hint I will appreciate.

All started when testing under windows a plugin for qgis called "gdal tools" a nice interface for many gdal utilities: the plugin works fine under linux but under windows has problems apparently because of issues with the osgeo4w gdal utilities.

This should be confirmed by the fact that the same tools under fwtools work fine.

A few details:

*) under windows XP all the tools that are in the form of .exe files work fine (both from the command line and from the plugin)

*) the same tools under Vista/Seven? return errors (both from the command line and from the plugin) like

"gdalinfo.exe has stopped working"

or

"gdalbuildvrt.exe has stopped working"

eventually they produce the correct output

this doesn't happens with fwtools binaries.

Under all windows versions there are also issues with the tools in the form of python scripts

*) If during the osgeo4w install it is added *just* the gdal16/gdal16-python libraries/packages, then all the gdal utilities that are wrote in python fail to work. The error is always like (both from the command line and from the plugin):

'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "c:/osgeo4w/apps/gdal-16/bin/gdal_merge.py", line 31, in <module>
    import gdal
ImportError: No module named gdal

*) I then tested also to install with osgeo4w the gdal/gdal-python libraries/packages (I know, probably not a good idea, but was just a test). This way the gdal merge utility then seems to work (as also rgb2pct), while all others still fail, example:

gdal.Polygonize() not available.  You are likely using "old gen"
bindings or an older version of the next gen bindings.

Traceback (most recent call last): 
File "C:\osgeo4w\apps\gdal-16\bin\gdal_proximity.py", line 170, in 
gdal.ComputeProximity( srcband, dstband, options, 
AttributeError: 'module' object has no attribute 'ComputeProximity'

[brushtyler]

Replying to lutra:

*) under windows XP all the tools that are in the form of .exe files work fine (both from the command line and from the plugin)
*) the same tools under Vista/Seven? return errors (both from the command line and from the plugin) like "gdalinfo.exe has stopped working" or "gdalbuildvrt.exe has stopped working" eventually they produce the correct output

All the executables (.exe) from gdal16 crash in Vista/Seven?, instead the same tools from gdal lib work fine.

[jef]

Hi,

Replying to lutra:

*) If during the osgeo4w install it is added *just* the gdal16/gdal16-python libraries/packages, then all the gdal utilities that are wrote in python fail to work. The error is always like (both from the command line and from the plugin):

GDAL 1.5 is currently the default GDAL in OSGeo4W and once installed works out of the box.

QGIS however uses GDAL 1.6 (to allow attribute additions to shape files). So you probably only installed QGIS and got GDAL 1.6 with it - and no GDAL 1.5.

If you want to use GDAL 1.6 from the command line you need to run gdal16.bat manually first. Running GDAL 1.5 binaries after that is probably not a good idea either.

If you start QGIS using the desktop/start menu link gdal16.bat is run automatically. So everything should be fine from within QGIS.

[lutra]

Hi Jurgen,

QGIS however uses GDAL 1.6 (to allow attribute additions to shape files). So you probably only installed QGIS and got GDAL 1.6 with it - and no GDAL 1.5.

yes, it is the case.

If you want to use GDAL 1.6 from the command line you need to run gdal16.bat manually first. Running GDAL 1.5 binaries after that is probably not a good idea either. If you start QGIS using the desktop/start menu link gdal16.bat is run automatically. So everything should be fine from within QGIS.

Sorry, just tested on a fresh qgis/osgeo4w install under windows Seven (just gdal16) and it doesn't work. I also launched manually gdal16.bat but apparently with no effects. Both from the command line and the plugin the message is always like

Traceback (most recent call last):
  File "c:/osgeo4w/apps/gdal-16/bin/gdal_merge.py", line 31, in <module>
    import gdal
ImportError: No module named gdal

[brushtyler]

Replying to jef:

If you want to use GDAL 1.6 from the command line you need to run gdal16.bat manually first. Running GDAL 1.5 binaries after that is probably not a good idea either.
If you start QGIS using the desktop/start menu link gdal16.bat is run automatically. So everything should be fine from within QGIS.

I use qgis-dev but no gdal16 .exe tools seems work fine. I tried to call the gdal16.bat too, but nothing.

I'm enough sure this is no a dlls conflict. If you remember time ago I wrote in ML about this problem.

I've tried installing only gdal16 (no gdal lib): each .exe tool started its execution, but it crashed before exit (from terminal too).

So I think these are 2 different problems, I don't know if are related.
1) gdal16 executables crashes
2) gdal python tools require to work both gdal-python and gdal16-python (??).

[jef]

Replying to lutra:

> Traceback (most recent call last):
>   File "c:/osgeo4w/apps/gdal-16/bin/gdal_merge.py", line 31, in <module>
>     import gdal
> ImportError: No module named gdal

Just to make sure: you are aware that you need to call gdal_merge from the "shell" from which you started gdal16.bat?

But from what brushtyler said this is probably a Vista/W7 issue. Could you run [ depends] on %OSGEO4W_ROOT%/apps/gdal-16/pymod/osgeo/_gdal.pyd?

[lutra]

Just to make sure: you are aware that you need to call gdal_merge from the "shell" from which you started gdal16.bat?

100% positive

[lutra]

So I think these are 2 different problems, I don't know if are related.
1) gdal16 executables crashes

confirmed Vista/Seven?. Under XP they works.

2) gdal python tools require to work both gdal-python and gdal16-python (??).

it seems so, but for what I have seen only merge/rgb2pct/pct2rgb works after installing gdal-python. The other tools fail, but at least the

ImportError: No module named gdal

message is gone

[mwtoews]

Replying to jef:

Could you run [ depends] on %OSGEO4W_ROOT%/apps/gdal-16/pymod/osgeo/_gdal.pyd?

I did this on both an XP system, with only a few warnings, and with Win7 x64 with errors. My result is a 3.5MB file -- too big to upload on this Trac system. (I can upload or email this file on request: append @gmail.com to my username).

Here are my exact steps from an OSGeo4W Shell on the Win7 x64 system:

C:\>gdal16

C:\>SET GDAL_DATA=C:\OSGeo4W\apps\gdal-16\share\gdal

C:\>SET GDAL_DRIVER_PATH=C:\OSGeo4W\apps\gdal-16\bin\gdalplugins

C:\>SET PYTHONPATH=C:\OSGeo4W\apps\gdal-16\pymod;C:\Program Files (x86)\ArcGIS\b
in

C:\>PATH=C:\OSGeo4W\apps\gdal-16\bin;C:\OSGeo4W\bin;C:\Windows\system32;C:\Windo
ws;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Pytho
n25;C:\Python25\Scripts;C:\Program Files (x86)\PostgreSQL\8.3\bin;C:\Program Fil
es (x86)\QuickTime\QTSystem\;C:\Program Files\TortoiseSVN\bin;C:\Users\mobile20\
AppData\Local\Start++\LNKs;C:\Users\mobile20\AppData\Local\Start++\CMDs;C:\OSGeo
4W\apps\msys\bin

C:\>depends OSGeo4W\apps\gdal-16\pymod\osgeo\_gdal.pyd

The Dependency Walker log details:

Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module.
Error: Modules with different CPU types were found.
Warning: At least one delay-load dependency module was not found.
Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.

[lutra]

But from what brushtyler said this is probably a Vista/W7 issue. Could you run [ depends] on %OSGEO4W_ROOT%/apps/gdal-16/pymod/osgeo/_gdal.pyd?

attached a screenshot. I don't know the tool and I'm not finding a way to save a log.

[jef]

Replying to lutra:

But from what brushtyler said this is probably a Vista/W7 issue. Could you run [ depends] on %OSGEO4W_ROOT%/apps/gdal-16/pymod/osgeo/_gdal.pyd?

attached a screenshot. I don't know the tool and I'm not finding a way to save a log.

Um, File/Save? As...? But it looks like you didn't start depends from the shell where you ran gdal16.bat and therefore GDAL16.DLL and co are not found.

[jef]

attached a text file from depends from a box where gdal16-python apparently works

[lutra]

Replying to jef:

Replying to lutra:

But it looks like you didn't start depends from the shell where you ran gdal16.bat and therefore GDAL16.DLL and co are not found.

Hi, I have run depends from the same shell where gdal16.bat was run. The result is the same. See attached file.

Cheers

[brushtyler]

output of depends

[jef]

Replying to lutra:

Hi, I have run depends from the same shell where gdal16.bat was run. The result is the same. See attached file.

Ah ok, you're not running from the OSGeo4W shell. If you use the "standard" command prompt you need to run o4w_env.bat to setup the environment for OSGeo4W. Without it %OSGEO4W_ROOT% is unset and gdal16.bat won't set proper paths.

[brushtyler]

Attached the output of depends.

By running only the gdal16.bat the result is wrong because there are some missing paths in PATH. I edited the qgis-dev.bat, replaced the call to qgis with cmd.exe and then ran the depends.exe

Look at IEFRAME.DLL and SHWAPI.DLL

[lutra]

Ah ok, you're not running from the OSGeo4W shell. If you use the "standard" command prompt you need to run o4w_env.bat to setup the environment for OSGeo4W. Without it %OSGEO4W_ROOT% is unset and gdal16.bat won't set proper paths.

new log attached.

The gdal python commands issued from the osego4w shell are working this way. I see no changes when issuing the same one from the gdal tools plugin (qgis launched from the very same osgeo4w shell)

Traceback (most recent call last): 
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_proximity.py", line 35, in 
import gdal 
ImportError: No module named gdal

[lutra]

sorry, log file too big and handling files in a VM is a pain. Depends screenshot attached.

[mwtoews]

Oh, of course, save as type txt. See attached/compressed file for the text output from Dependency Walker for my WinXP system, which only has a few warnings and works, and my office Win7 x64 system, which has the error described in this ticket.

Again, if you want the compressed dwi files just email me, since they are too large to attach here.

[mwtoews]

Dependency Walker TXT Result of XP system without issue and Win7x64 with errror

[brushtyler]

I tried gdal_merge.bat from osgeo4w shell, and it works!

I looked at code, "import gdal" is not related to gdal_merge.bat. It's called by python code to retrieve the extent. :(

Now it's explicit becouse without gdal lib the merge tools return this error. But how can I import the gdal16 lib in python?
This is almost resolved:

2) gdal python tools require to work both gdal-python and gdal16-python (??).

Now remain this, finding the issue will be more difficult here:

1) gdal16 executables crash

This is sure not related with gdaltools-plugin, I re-tried by osgeo4w shell (to be very very sure) and it crashes.

[lutra]

Now it's explicit becouse without gdal lib the merge tools return this error. But how can I import the gdal16 lib in python?

another important question, once you'll find a way to call gdal16, it will be enough or users will have in any case to run gdal16.bat from the osgeo4w shell before?

Now remain this, finding the issue will be more difficult here:

1) gdal16 executables crash

This is sure not related with gdaltools-plugin, I re-tried by osgeo4w shell (to be very very sure) and it crashes.

For what it's worth, I made a *dirt* experiment that gave good results: I copied fwtools gdal utilities executables into the osgeo4w gdal16 binary folder (overwriting the files): they works fine under Seven with both the command line and from the gdal tools plugin, no crashes at all.

[jef]

Replying to brushtyler:

But how can I import the gdal16 lib in python?

You're not supposed to. You run python from a OSGeo4W shell after running gdal16.bat or prepare a batchfile you can run anywhere (like %OSGEO4W_ROOT%\bin\qgis-dev.bat).

OSGeo4W should not interfere with the system wide settings, so it doesn't set global PATHS or installs stuff to global places (like system32). But that means that you have to use the prepared environment or if you don't like that, take into account that you need to create the environment yourself.

Other stuff that is installed system wide might however interfere with OSGeo4W (mainly DLLs that are installed in system32 that override stuff in %OSGEO4W_ROOT%\bin).

[gislab]

For what it worth, I confirm under Win7 64bit: 1. gdal utilites crash after successfully completing, it takes more time than usual 2. Overwriting gdal16\bin with FWTools dlls (tried with this set libtiff_fw, geotiff_fw, geos_fw, gdal_fw, NCScnet_fw, NCSEcw_fw, NCSUtil_fw, lti_dsk и gdalinfo.exe) makes gdalinfo work without glitches.

Hope this helps, I'm ready to provide additional info if needed.

[brushtyler]

Replying to jef:

Replying to brushtyler:

But how can I import the gdal16 lib in python?

You're not supposed to. You run python from a OSGeo4W shell after running gdal16.bat or prepare a batchfile you can run anywhere (like %OSGEO4W_ROOT%\bin\qgis-dev.bat).

No, I meant import gdal16 in python plugins into qgis.
In this moment I run qgis-dev.bat then the gdal16.bat was yet called. But gdaltools-plugin fails to import gdal16-python tools by calling "import gdal", because it tried to import the inexistent gdal lib!
About gdal in qgis, in linux gdal1.6 *overwrite* gdal1.5, then why the osgeo4w on win doesn't have the same behaviour?
If I install gdal16 lib, I suppose and want (if supported) it will be used!

I think this is an osgeo4w issue, do you?

[jef]

Replying to brushtyler:

About gdal in qgis, in linux gdal1.6 *overwrite* gdal1.5, then why the osgeo4w on win doesn't have the same behaviour?
If I install gdal16 lib, I suppose and want (if supported) it will be used!

You can have both GDALs. In OSGeo4W and under Linux. Which one is used is controlled by environment variables (PATH on Windows, LD_LIBRARY_PATH on Linux).

gdal16.bat sets up the environment so that the GDAL 1.6 binaries and libraries are used and also sets PYTHONPATH so that the right GDAL bindings are found.

I think this is an osgeo4w issue, do you?

Probably. But if that is only happening on Windows 7/Vista, we must be missing something special there, but generally it should work - and unfortunately also works here with Win7 64Bit.

[lutra]

Probably. But if that is only happening on Windows 7/Vista, we must be missing something special there, but generally it should work - and unfortunately also works here with Win7 64Bit.

Hi Jurgen, until now I haven't find a single Vista/Seven? machine where the osgeo4w gdal .exe utilities do work without crashing. In all the pc where I installed osgeo4w were typical user machines, nothing special installed related to development & co. This should be have some meaning... that -I guess- in general the osgeo4w gdal utilities doesn't work (under Vista/Seven?).

It should be easy during the hackfest make the necessary comparison from your machine and another Vista/Seven? installation. I'll miss the hackfest but I'm sure Giuseppe can have a look to it with you.

Cheers

[jef]

Two separate issues:

1. Looks like the GDAL 1.6 GRASS plugin causes that problem. Maybe because GDAL 1.6 was rebuild and the GRASS plugin was not. Try to rename or remove %SOGEO4W_ROOT%\apps\gdal-16\bin\gdalplugins\gdal_GRASS.dll to verify (QGIS trunk doesn't need it anymore).

2. The QGIS GRASS plugin (from the nightly build) mixed up PATH and PYTHONPATH, ie. when the GRASS plugin was adding the GRASS python binding path to PATH and use that for PYTHONPATH - that way it wouldn't find GDAL 1.6 bindings, use the GDAL 1.5 bindings or simply fail (fixed in [ QGIS r13068], the next nightly build should have it).

[jef]

Replying to jef:

1. Looks like the GDAL 1.6 GRASS plugin causes that problem. Maybe because GDAL 1.6 was rebuild and the GRASS plugin was not. Try to rename or remove %SOGEO4W_ROOT%\apps\gdal-16\bin\gdalplugins\gdal_GRASS.dll to verify (QGIS trunk doesn't need it anymore).

I've rebuild the GDAL GRASS plugin. Please test if an upgrade using osgeo4w-setup -t helps.

[lutra]

I've rebuild the GDAL GRASS plugin. Please test if an upgrade using osgeo4w-setup -t helps.

works fine for me! great!

[gislab]

I've just updated qgis-dev and gdal16-grass 1.0.6-7.

Unfortunately, I still have the same problem, but removing gdal_GRASS.dll helped. Size of the updated dll is 72 192.

[lutra]

New description:

on windows Vista/Seven? the gdal_GRASS.dll causes gdal .exe utilities to not work correctly. Deleting the file manually is a workaround on qgis-dev where the GRASS plugin has a new provider for rasters so it probably won't work on qgis 1.4.

[lutra]

I received a report claiming that this problem now shows also under XP.

[lutra]

I believe that with all the changes in QGIS in the last 12 months this ticket/problems con be considered fixed/obsolete. It's ok to close?

[jef]

the gdal-grass package is obsolete. QGIS doesn't use it anymore.