MSI and EXE installers: different Python locale settings leads to issues with the Python console script editor

[andreaerdna]

With the old OSGeo4W repository EXE installer (QGIS 3.18.1):

import sys
bool(sys.flags.utf8_mode)
True

import locale
locale.getpreferredencoding(False)
'UTF-8'

while with the new OSGeo4W repository MSI installer (QGIS 3.18.1):

import sys
bool(sys.flags.utf8_mode)
False

import locale
locale.getpreferredencoding(False)
'cp1252'

With the new MSI installer, this leads to issues with the Python console script editor if the script contains non cp1252 encoded characters, while there is no issue with the old EXE installer, as reported on GitHub issue tracker ​https://github.com/qgis/QGIS/issues/43834

How to Reproduce

Enter

print('áéŐŰÍ')

in the Python script editor window and run the script.

The script will execute without issue using QGIS from the old repository EXE installer, while it will generate the following error using QGIS from the new repository MSI installer:

Traceback (most recent call last):
  File "F:\QGIS\SOURCE~1\qgis\apps\Python39\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "F:\QGIS\SOURCE~1\qgis\apps\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 12: character maps to <undefined>

[PedroNGV]

@andreaerdna

This seems the issue I've reported here: ​https://github.com/qgis/QGIS/issues/41601

[andreaerdna]

Yes, most likely this is the same underlying reason that causes also the issue you reported, too.

Probably also the issue fixed by ​https://github.com/qgis/QGIS/pull/43249 was due to the same reason.

[jef]

Please try adding set PYTHONUTF8=1 to etc\ini\python3.bat and re-run qgis.bat --postinstall from the OSGeo4W shell.

[nicogodet]

I add set PYTHONUTF8=1, re-run qgis.bat --postinstall but nope.

I need to run qgis.bat instead of qgis-bin.exe to have PYTHONUTF8 in os.environ

>>> os.environ["PYTHONUTF8"]
'1'

>>> exec(open('C:/Users/Nicolas Godet/Desktop/Sans titre 1.py'.encode('utf-8')).read())
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python39\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "C:\OSGeo4W\apps\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 12: character maps to <undefined>

[andreaerdna]

It seems Python used in QGIS is not affected by the PYTHONUTF8 env var, while the Python used in OSGeo4W console is affected:

OSGeo4W old repositroy, OSGeo4W console:

F:\QGIS\OSGeo4W_3.18.1\qgis>python-qgis
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "F:\QGIS\OSGEO4~1.1\qgis\apps\Python37\lib\os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYTHONUTF8'
>>> import sys
>>> sys.flags.utf8_mode
0
>>> import locale
>>> locale.getpreferredencoding(False)
'cp1252'

F:\QGIS\OSGeo4W_3.18.1\qgis>set PYTHONUTF8=1

F:\QGIS\OSGeo4W_3.18.1\qgis>python-qgis
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
'1'
>>> import sys
>>> sys.flags.utf8_mode
1
>>> import locale
>>> locale.getpreferredencoding(False)
'UTF-8'

F:\QGIS\OSGeo4W_3.18.1\qgis>set PYTHONUTF8=0

F:\QGIS\OSGeo4W_3.18.1\qgis>python-qgis
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
'0'
>>> import sys
>>> sys.flags.utf8_mode
0
>>> import locale
>>> locale.getpreferredencoding(False)
'cp1252'

OSGeo4W old repositroy, QGIS Python console:

sys.version
'3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]'
os.environ["PYTHONUTF8"]
Traceback (most recent call last):
  File "F:\QGIS\OSGEO4~1.1\qgis\apps\Python37\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "F:\QGIS\OSGEO4~1.1\qgis\apps\Python37\lib\os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYTHONUTF8'
sys.flags.utf8_mode
-1
import locale
locale.getpreferredencoding()
'UTF-8'

sys.version
'3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]'
os.environ["PYTHONUTF8"]
'1'
sys.flags.utf8_mode
-1
import locale
locale.getpreferredencoding()
'UTF-8'

sys.version
'3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]'
os.environ["PYTHONUTF8"]
'0'
sys.flags.utf8_mode
-1
import locale
locale.getpreferredencoding()
'UTF-8'

[andreaerdna]

OSGeo4W new repositroy, OSGeo4W console:

F:\QGIS\SourceDir\qgis>python-qgis
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "F:\QGIS\SOURCE~1\qgis\apps\Python39\lib\os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYTHONUTF8'
>>> import sys
>>> sys.flags.utf8_mode
0
>>> import locale
>>> locale.getpreferredencoding(False)
'cp1252'

F:\QGIS\SourceDir\qgis>set PYTHONUTF8=1

F:\QGIS\SourceDir\qgis>python-qgis
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
'1'
>>> import sys
>>> sys.flags.utf8_mode
1
>>> import locale
>>> locale.getpreferredencoding(False)
'UTF-8'

F:\QGIS\SourceDir\qgis>set PYTHONUTF8=0

F:\QGIS\SourceDir\qgis>python-qgis
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ['PYTHONUTF8']
'0'
>>> import sys
>>> sys.flags.utf8_mode
0
>>> import locale
>>> locale.getpreferredencoding(False)
'cp1252'

OSGeo4W new repositroy, QGIS Python console:

sys.version
'3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]'
os.environ['PYTHONUTF8']
Traceback (most recent call last):
  File "F:\QGIS\SOURCE~1\qgis\apps\Python39\lib\code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "F:\QGIS\SOURCE~1\qgis\apps\Python39\lib\os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYTHONUTF8'
sys.flags.utf8_mode
0
import locale
locale.getpreferredencoding()
'cp1252'

sys.version
'3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]'
os.environ['PYTHONUTF8']
'1'
sys.flags.utf8_mode
0
import locale
locale.getpreferredencoding()
'cp1252'

sys.version
'3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]'
os.environ['PYTHONUTF8']
'0'
sys.flags.utf8_mode
0
import locale
locale.getpreferredencoding()
'cp1252'

[jef]

3.18.1? 3.9.0? Why not 3.20.0 and 3.9.5?

[PedroNGV]

Replying to jef:

3.18.1? 3.9.0? Why not 3.20.0 and 3.9.5?

I get the same with QGIS 3.20.0 and Python 3.9.5:

sys.version
'3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)]'
os.environ['PYTHONUTF8']
'1'
sys.flags.utf8_mode
0
import locale
locale.getpreferredencoding()
'cp1252'

[andreaerdna]

Replying to jef:

3.18.1? 3.9.0? Why not 3.20.0 and 3.9.5?

QGIS 3.20.0 is not available in old OSGeo4W repository, so I can't compare the different behavior between new and old repository using the same QGIS version.

It seems to me the issue is repository dependent, and probably affect also QGIS 3.16, besides 3.18 and 3.20, in the new repository.