python 2.7.4 dll conflicts

[maphew]

the recent python 2.7.4-1 package (#297) broke GRASS and QGIS for some, see ​http://lists.osgeo.org/pipermail/osgeo4w-dev/2013-April/002200.html (sqlite, #305)

​http://lists.osgeo.org/pipermail/osgeo4w-dev/2013-April/002208.html (duplicate python27)

[maphew]

the sqlite problem has been fixed by removing sqlite3.dll from python-core package and making it require pkg-sqlite (which had been done for previous release, but I overlooked for this one).

The issue arising from duplicate yet conflicting python27.dll's in PATH is, strictly speaking, not our problem unless the dupe is from within the o4w system (the ones reported aren't). I do have an idea to help people though and am interested in feedback on the soundness of it:

If we replace etc\ini\python.bat with the below, on each invocation of osgeo4w shell it will search PATH for the presence of duplicate PythonXX.dll's and issue a warning if found.

SET PYTHONHOME=%OSGEO4W_ROOT%\apps\Python27
@call :check_path
SET PATH=%PATH%;%OSGEO4W_ROOT%\apps\Python27\Scripts
goto :eof

:check_path
    @echo off
    setlocal
    set _DLL=%PYTHONHOME:~-8,8%.dll
    for %%G in ("%path:;=" "%") do (
        if exist %%G\%_DLL% call :conflicted %%G %_DLL%
        )
    if exist %SYSTEMROOT%\SysWOW64\%_DLL% call :conflicted %SYSTEMROOT%\SysWOW64 %_DLL%
    if "%_delay%"=="yes" ping -n 10 localhost > nul
    endlocal    
    goto :eof
    
:conflicted
    echo.   
    echo.   **** Possibly conflicting %2 found in
    echo.   **** %1
    echo.   Please see http://trac.osgeo.org/osgeo4w/wiki/pkg-python/xxxxx
    echo.
    set _delay=yes
    goto :eof

sample result:

B:\>SET PYTHONHOME=B:\o4w\\apps\Python27

   **** Possibly conflicting Python27.dll found in
   **** "C:\Program Files\TortoiseHg\"
   Please see http://trac.osgeo.org/osgeo4w/wiki/pkg-python/xxxxx


   **** Possibly conflicting Python27.dll found in
   **** C:\Windows\SysWOW64
   Please see http://trac.osgeo.org/osgeo4w/wiki/pkg-python/xxxxx


B:\>

Why I'm a bit leary of this: a) a very simple 2 line batch file anyone can understand is a dozen times longer and leverages some arcane CMD processing ==> harder to maintain, b) if this becomes standard practice for every DLL issue we end up with a lot of special case code ==> harder to maintain. However in spite of this, it does seem to work and it would save time troubleshooting, especially for those whom the ini directory and shell setup is not usually even given a passing glance. Your thoughts?

[hellik]

Replying to maphew:

the recent python 2.7.4-1 package (#297) broke GRASS and QGIS for some, see ​http://lists.osgeo.org/pipermail/osgeo4w-dev/2013-April/002200.html (sqlite, #305)

AFAICT these are two different issues

(1) (sqlite, #305):

it's a dll conflict within osgeo4w

(2) ​http://lists.osgeo.org/pipermail/osgeo4w-dev/2013-April/002200.html :

after some tests it's a dll conflict between osgeo4w and an python 2.7.4 installed by arcgis

Helmut

[hellik]

Replying to maphew:

[...] it does seem to work and it would save time troubleshooting,

save time troubleshooting would be nice

Helmut

[jef]

for qgis this is fixed by moving the qgis.exe to the main bin (the directory of the executable that load a DLL takes precedence over system32 and PATH).b

[saultdon]

Replying to jef:

for qgis this is fixed by moving the qgis.exe to the main bin (the directory of the executable that load a DLL takes precedence over system32 and PATH).b

But then this stops qgis from being launched at the terminal because the .bat file will not be called first - it defaults to call the .exe first which results in a missing dll error.

You can still launch it either typing qgis.bat specifically or using the desktop and start menu qgis shortcuts which call qgis.bat directly.

You need to incorporate this fix (http://trac.osgeo.org/osgeo4w/ticket/358?replyto=12#comment:7) in order to place .exe and .bat files in the same folder and launch them reliably.

Simply placing a single line...

set PATHEXT=.BAT;.COM;.EXE;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.RB;.RBW

...in the OSGeo4W\OSGeo4W.bat file is enough. If you're interested...

[maphew]

I'm ok with changing the executable extension order, +1 from me, if what follows isn't acceptable:

I'd really like to find a reliable way to strip the Windows system directory from the DLL search path, but after much reading and searching it does look as if this isn't possible. The only other alternative I've found* is to put move the .dll next to the .exe, in apps\qgis\bin, as opposed to the current solution of putting the .exe with the .dll in bin.

• ​http://stackoverflow.com/questions/16245757/python-not-search-windows-system32-for-pythonxx-dll

If we created hardlinks it needn't waste disk space, though only for NTFS. Here's how it might be implemented in etc\postinstall\qgis.bat:

:: for NTFS
mklink  /h apps\qgis\bin\python27.dll  bin\python27.dll

:: for FAT
if not exist apps\qgis\bin\python27.dll ^
copy bin\python27.dll apps\qgis\bin\python27.dll

[jef]

Replying to maphew:

I'd really like to find a reliable way to strip the Windows system directory from the DLL search path, but after much reading and searching it does look as if this isn't possible. The only other alternative I've found* is to put move the .dll next to the .exe, in apps\qgis\bin, as opposed to the current solution of putting the .exe with the .dll in bin.

That goes for all DLLs - not just for the python DLL. Every DLL in bin (or PATH) can potentially be masked by a DLL in system32 (this also happenen with the openssl DLLs). So we'd need to copy all DLLs to whereever .exes live (apps/qgis, apps/qgis-dev, grass...)

Moving the qgis.exe into bin is a solution for that - and typing qgis.bat instead of qgis isn't such a huge burden to justify any significant work put into this - most people probably use the shortcuts anyway.

[jef]

Replying to jef:

Moving the qgis.exe into bin is a solution for that - and typing qgis.bat instead of qgis isn't such a huge burden to justify any significant work put into this - most people probably use the shortcuts anyway.

The qgis.exe was renamed to qgis-bin.exe - so the clash between qgis.bat and qgis.exe is gone.