Opened 11 months ago

Closed 3 months ago

#3781 closed defect (fixed)

v.in.geonames - fails in winGRASS7.7 (python3)

Reported by: hellik Owned by: grass-dev@…
Priority: normal Milestone: 7.8.1
Component: Python Version: git-releasebranch78
Keywords: python3, py3 Cc:
CPU: x86-64 Platform: MSWindows

Description

tested with

System Info                                                                     
GRASS version: 7.7.svn                                                          
GRASS SVN revision: r74150M                                                     
Build date: 2019-03-05                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 3.7.0                                                                   
wxPython: 4.0.3                                                                 
Platform: Windows-10-10.0.17763-SP0 (OSGeo4W) 

v.in.geonames fails with

v.in.geonames input=D:\data\private\presentation\lernwerkstatt\AT\AT.txt output=at_out
Traceback (most recent call last):
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 150, in <module>
    main()
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 67, in main
    num_places = sum(1 for _ in f)
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 67, in <genexpr>
    num_places = sum(1 for _ in f)
  File "C:\OSGEO4~1\apps\Python37\lib\encodings\cp1252.py",
line 23, in decode
    return
codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d
in position 3815: character maps to <undefined>

Change History (8)

comment:1 in reply to:  description Changed 11 months ago by hellik

Replying to hellik:

tested with

System Info                                                                     
GRASS version: 7.7.svn                                                          
GRASS SVN revision: r74150M                                                     
Build date: 2019-03-05                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 3.7.0                                                                   
wxPython: 4.0.3                                                                 
Platform: Windows-10-10.0.17763-SP0 (OSGeo4W) 

v.in.geonames fails with

v.in.geonames input=D:\data\private\presentation\lernwerkstatt\AT\AT.txt output=at_out
Traceback (most recent call last):
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 150, in <module>
    main()
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 67, in main
    num_places = sum(1 for _ in f)
  File
"C:\OSGEO4~1\apps\grass\grass77/scripts/v.in.geonames.py",
line 67, in <genexpr>
    num_places = sum(1 for _ in f)
  File "C:\OSGEO4~1\apps\Python37\lib\encodings\cp1252.py",
line 23, in decode
    return
codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d
in position 3815: character maps to <undefined>

same data set works with

GRASS version: 7.6.0                                                            
GRASS SVN revision: r73958                                                      
Build date: 2019-01-17                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.0                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.0                                                                     
SQLite: 3.26.0                                                                  
Python: 2.7.14                                                                  
wxPython: 2.8.12.1                                                              
Platform: Windows-10-10.0.17763 (OSGeo4W)
v.in.geonames input=D:\data\private\presentation\lernwerkstatt\AT\AT.txt output=at2
Converting 51995 place names...
Scanning input for column types...
Number of columns: 19
Number of rows: 51995
WARNING: Column number 13 <admin3code> defined as string has only integer values
WARNING: Column number 14 <admin4code> defined as string has only integer values
Importing points...
Populating table...
Building topology for vector map <at2@data>...
Registering primitives...
(Wed Mar 06 22:25:18 2019) Command finished (9 sec)                            

comment:2 Changed 8 months ago by annakrat

In 74481:

v.in.geonames: read as utf8, see #3781

comment:3 Changed 5 months ago by neteler

Version: svn-trunkgit-releasebranch78

Tested on Linux:

wget http://download.geonames.org/export/dump/AT.zip
unzip AT.zip

v.in.geonames input=AT.txt output=at_out
Converting 52003 place names...
Scanning input for column types...
Number of columns: 19
Number of rows: 52003
WARNING: Column number 13 <admin3code> defined as string has only integer
         values
WARNING: Column number 14 <admin4code> defined as string has only integer
         values
Importing points...
 100%
Populating table...
Building topology for vector map <at_out@user1>...
Registering primitives...

Still an issue?

comment:4 Changed 5 months ago by annakrat

The issue was on Windows, so someone should confirm the fix worked.

comment:5 Changed 5 months ago by neteler

Milestone: 7.8.07.8.1

Ticket retargeted after milestone closed

comment:6 Changed 3 months ago by neteler

Ping Helli, still an issue?

comment:7 Changed 3 months ago by hellik

tested with

GRASS Version: 7.8.1dev                                                         
Code revision: d1c4ad132                                                        
Build date: 2019-10-22                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.1                                                                     
PROJ: 5.2.0                                                                     
GEOS: 3.8.0                                                                     
SQLite: 3.29.0                                                                  
Python: 3.7.0                                                                   
wxPython: 4.0.3                                                                 
Platform: Windows-10-10.0.18362-SP0 (OSGeo4W) 
v.in.geonames --verbose input=D:\temp\geonames\AT\AT.txt output=at_out          
Converting 51999 place names...
Using native format
Scanne die Eingabe zur Ermittelung der Spaltentypen...
Maximale Länge der Inputzeile: 2204
Number of columns: 19
Number of rows: 51999
Column: 1  type: integer
Column: 2  type: string length: 79
Column: 3  type: string length: 79
Column: 4  type: string length: 2089
Column: 5  type: double
Column: 6  type: double
Column: 7  type: string length: 1
Column: 8  type: string length: 5
Column: 9  type: string length: 2
Column: 10  type: string length: 8
Column: 11  type: string length: 2
Column: 12  type: string length: 2
Column: 13  type: integer
Column: 14  type: integer
Column: 15  type: integer
Column: 16  type: integer
Column: 17  type: integer
Column: 18  type: string length: 13
Column: 19  type: string length: 10
WARNING: Spalte Nummer 13 <admin3code> ist als string definiert und hat nur Integer-Werte.
WARNING: Spalte Nummer 14 <admin4code> ist als string definiert und hat nur Integer-Werte.
Importiere Punkte...
Fülle Tabelle...
Erstelle Topologie für die Vektorkarte <at_out@data>...
Registriere Primitive...
GRASS_INFO_PROGRESS: 10000
GRASS_INFO_PROGRESS: 20000
GRASS_INFO_PROGRESS: 30000
GRASS_INFO_PROGRESS: 40000
GRASS_INFO_PROGRESS: 50000
51999 primitives registered
51999 vertices registered
Die Topologie wurde erstellt.
Anzahl der Knoten: 0
Anzahl der Primitive: 51999
Anzahl der Punkte: 51999
Anzahl der Linien: 0
Anzahl der Grenzen: 0
Anzahl der Zentroide: 0
Anzahl der Flächen: 0
Anzahl der Inseln: 0
(Wed Oct 23 23:07:58 2019) Befehl ausgeführt (12 Sek)                           

seems to work now, closing ticket

comment:8 Changed 3 months ago by hellik

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.