Modules missing in OSGeo4w installation since since QGIS 3.18 update

[jelstner]

Following this advice ​https://lists.osgeo.org/pipermail/qgis-user/2021-February/048137.html I did a complete reinstall of my OSGeo4w environment for the QGIS 3.18 update.

Since then I experience some issues that might be caused by something missing in the new packaging:

1. At least one core python module is missing; doctests (​https://docs.python.org/3/library/doctest.html). I found that out when sphinx.ext.autodoc failed to start. The problem was solved when I put a fresh Python install (manually downloaded from python.org) over the C:\OSGeo4W64\apps\Python39 directory. Not the gentle way. but does its job.

2. My PyCharm and my Sphinx installation can't read the QGIS .pyd files anymore. Python in QGIS itself still works, but PyCharm doesn't build skeletons for the QGIS API and Sphinx fails to document my plugin. They use the same paths as QGIS, I already double-checked that.

The error message in the PyCharm console suggests to me that PyCharm even finds the _core.pyd file, but fails to read it:

Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
from qgis._core import QgsFeature
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.2.5\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: DLL load failed while importing _core: Das angegebene Modul wurde nicht gefunden.

The Sphinx warning looks like this:

WARNING: autodoc: failed to import module 'config' from module 'RailTopoGIS'; the following exception was raised:
DLL load failed while importing _core: Das angegebene Modul wurde nicht gefunden.

I don't know whether both points are related to each other. If you think it's better, I can split it up into two issues.

[jef]

doctest.py is in python3-test.

[jelstner]

Okay, but Sphinx doesn't list this as package requirement during installation (pip). Is this a fault on Sphinx side?

[jef]

Fixed in python3-core-3.9.0-8

[jelstner]

What has been fixed, the missing package dependencies that affected sphinx?

The other error with .pyd files still persists for me, after the update.

[jef]

doctest.py was moved to python3-core.

[jelstner]

Regarding the remaining issue with QGIS API: I also posted it into QGIS (as I in the meantime assumed it could be directly QGIS-related), but there someone made the test with a plain QGIS installation and there it worked: ​https://github.com/qgis/QGIS/issues/42485#issuecomment-807996393

So it seems to be a packaging issue.

[jef]

Replying to jelstner:

Regarding the remaining issue with QGIS API: I also posted it into QGIS (as I in the meantime assumed it could be directly QGIS-related), but there someone made the test with a plain QGIS installation and there it worked: ​https://github.com/qgis/QGIS/issues/42485#issuecomment-807996393

So it seems to be a packaging issue.

The plain installation is a OSGeo4W installation (even if it's disguised in standalone installer - they use the same binaries)

[andreaerdna]

jef, jelstner wants to say that while importing from qgis.core fails using python-qgis in the OSGeo4W testing shell:

C:\OSGeo4W64_new\bin>python-qgis.bat
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from qgis.core import QgsFeature
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\OSGEO4~2\apps\qgis\python\qgis\__init__.py", line 70, in <module>
    from qgis.PyQt import QtCore
  File "C:\OSGEO4~2\apps\qgis\python\qgis\PyQt\QtCore.py", line 24, in <module>
    from PyQt5.QtCore import *
ImportError: DLL load failed while importing QtCore: The specified module could not be found.
>>>

instead there are no errors using the OSGeo4W regular (non testing) shell:

F:\QGIS\OSGeo4W_3.18.1\qgis>python-qgis.bat
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from qgis.core import QgsFeature
>>>

[jef]

Fixed rebuilds underway…

[andreaerdna]

Thanks. It seems to me the issue is fixed now in OSGeo4W testing with qgis-3.18.1-3 and qgis-common-3.18.1-3 packages.

F:\QGIS\SourceDir\qgis>python-qgis.bat
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from qgis.core import QgsFeature
>>>