Opened 3 months ago

Closed 3 months ago

#814 closed defect (fixed)

stdlib venv's broken on new Python 3.9.18

Reported by: akominlsfi Owned by: osgeo4w-dev@…
Priority: critical Component: Package
Version: Keywords: python, stdlib, venv, dll
Cc:

Description

After Python update to 3.9.18, stdlib venv module usage is broken.

This may be due to venv .exes being the same exact exe as the original, copied in this line? Not sure if the compilation even outputs these venvlauncher exes, which I assume would be needed in that directory instead of the original exes?

Is using the bundled Python standalone without any PATH/env manipulation even a supported use case? This can be patched manually for our workflow by grabbing previous 3.9 version launcher exes and dropping those into the python lib venv directory, since then the created venvs use the launchers, and correctly resolve the source exes. Version mismatch between the launcher and source seems to be an actually valid use case (for not needing venv recreation after the Python itself is updated).

Change History (2)

comment:1 by akominlsfi, 3 months ago

For context our current plugin development install workflow on Windows looks this this (related to previous issue about stdlib dll patching):

  • Install QGIS via network installer (each version or ltr/non-ltr branch to separate root directory)
  • Patch the installation stdlib & site-packages
    • For apps/python39/python.exe to run without adding /bin to PATH:
      • Copy zlib.dll from /bin to /apps/python39
    • For venvs without --system-site-packages
      • Allow stdlib to work by copying sqlite/libcrypto/libssl dlls from /bin to /apps/python39/dlls, since .pths in system paths are not sourced
      • EDIT: It seems also liblzma.dll is needed
    • For venvs with --system-site-packages
      • Allow site-packages modules to access dlls in /bin by adding /apps/python39/bin-dlls.pth file with os.add_dll_directory('path-to-/bin')
      • Allow site-packages PyQt5 to access its dlls in /apps/Qt5/bin by adding /apps/python39/qt-dlls.pth file with os.add_dll_directory('path-to-/apps/qt5/bin')

For the installed branch of QGIS Python to work directly as well (referring the last issue conversation, this is only applicable when there is only one ltr/non-ltr branch in same root directory):

  • Patch the site-packages to include the installed branch of QGIS
    • Add a qgis.pth file with os.add_dll_directory('path-to-/apps/<qgis or qgis-ltr>/bin, and plain sys.path addition for /apps/<qgis or qgis-ltr>/python

With this workflow we never need to add anything to PATH or environment (since we might have multiple installations on the same machine), and venvs work automatically when using /apps/python39/python.exe directly.

Last edited 3 months ago by akominlsfi (previous) (diff)
Note: See TracTickets for help on using tickets.