data catalog does not show all raster layers, while showing some non-existing

[pvanbosgeo]

In various of my mapsets,

• raster layers are not showing in the Data catalog, or
• some raster layers are showing, others are not, or
• some rasters are not showing, some rasters that show do actually not exist (see example of this case in attached screenshot)

In all cases, vector layer are showing. In some mapsets, all raster layers are showing as well. I can't find really a pattern, nor anything out of the ordinary in the folders of the database.

[pvanbosgeo]

layers shown in data catalog, compared to the layers actually present in the mapset (here the cats folder is shown as example)

[pvanbosgeo]

This example shows the layers in the data catalog and in the 'add selected layers in layer tree' menu.. clearly showing that the data catalog is not showing most layers

[pvanbosgeo]

Currently, no data is showing at all in the data catalogue

[annakrat]

Some time ago I rewrote loading the layers and it was working perfectly for me (Ubuntu). Any error? Does it work for others?

[martinl]

Replying to pvanbosgeo:

Currently, no data is showing at all in the data catalogue

Confirmed, there is something wrong, the condition source:grass/trunk/gui/wxpython/datacatalog/tree.py#L160​ seems to be never fulfilled (loc_count == len(locations)).

[martinl]

Replying to martinl:

Confirmed, there is something wrong, the condition source:grass/trunk/gui/wxpython/datacatalog/tree.py#L160​ seems to be never fulfilled (loc_count == len(locations)).

In my case I have 36 locations, the loop hangs when scanning 32nd location. 33rd is never scanned.

[annakrat]

Replying to martinl:

Replying to martinl:

Confirmed, there is something wrong, the condition source:grass/trunk/gui/wxpython/datacatalog/tree.py#L160​ seems to be never fulfilled (loc_count == len(locations)).

In my case I have 36 locations, the loop hangs when scanning 32nd location. 33rd is never scanned.

I can't replicate it, I tried it on different computers and operating systems. There might be something special about your locations. I am afraid you have to investigate further.

[pvanbosgeo]

Unlike martinl I have only few locations in my GRASS databases (3 in one, 8 in another), with between 2 and 20 mapsets per location. The data catalogue shows empty in both databases.

On the other hand, when creating a new GRASS database, the data catalogue shows the locations/map sets. There are no error messages or warnings, so I am not sure how to further investigate this?

[veroandreo]

I also don't see anything at all in Data catalog. I tried creating a new location and mapset in the GUI, but nothing appears. I have 9 locations with 3-10 mapsets each in my grassdata.

I'm using grass71 r67485 under fedora 22.

[pvanbosgeo]

Replying to veroandreo:

I also don't see anything at all in Data catalog. I tried creating a new location and mapset in the GUI, but nothing appears. I have 9 locations with 3-10 mapsets each in my grassdata.

And if you create a whole new GRASS database?

I'm using grass71 r67485 under fedora 22.

[veroandreo]

Yes, sure!

I created a new grassdata and pasted the North Carolina sample Location inside it. Now, Data catalog shows it perfectly. After that, I created a new Location in this new grassdata but I could only see it in the Data catalog after restarting GUI.

[annakrat]

Replying to veroandreo:

grassdata but I could only see it in the Data catalog after restarting GUI.

Yes, we need a reload button

Martin, any chance you could try to debug it? I can't do anything about it right now, unless someone can send me a grass database which is not visible in the catalog.

[martinl]

Replying to annakrat:

Martin, any chance you could try to debug it? I can't do anything about it right now, unless someone can send me a grass database which is not visible in the catalog.

hopefully tomorrow...

[martinl]

It seems that it hangs on source:grass/trunk/gui/wxpython/datacatalog/tree.py#L169​ when reading locations with more maps. Setting timeout to 0.1 (r67531) helped on my PC. Testing welcome.

[martinl]

Does r67531 fixes originally reported issue?

[annakrat]

Replying to martinl:

It seems that it hangs on source:grass/trunk/gui/wxpython/datacatalog/tree.py#L169​ when reading locations with more maps. Setting timeout to 0.1 (r67531) helped on my PC. Testing welcome.

I am not sure if this solves the problem. I guess it won't show then the correct results? I am wondering why the process hangs, is it just because there is a lot of maps or because g.list or g.mapsets have some other problem? How many maps do you have in the biggest location, so that I can try to reproduce it?

[martinl]

Replying to annakrat:

Replying to martinl: I am not sure if this solves the problem. I guess it won't show then the correct results? I am wondering why the process hangs, is it just because there is a lot of maps or because g.list or g.mapsets have some other problem? How many maps do you have in the biggest location, so that I can try to reproduce it?

it hanged on nc_spm_08_grass7 location with 6142 maps (modis included). The location is 19th from 34 locations.

[martinl]

Replying to annakrat:

I am not sure if this solves the problem. I guess it won't show then the correct results? I am

I checked nc_spm_08_grass7 location in datacatalog and the result seems to be OK.

[veroandreo]

Replying to martinl:

Replying to annakrat:

I am not sure if this solves the problem. I guess it won't show then the correct results? I am

I checked nc_spm_08_grass7 location in datacatalog and the result seems to be OK.

Works for me, too. I can see all locations, mapsets and their content. It seems slow though with the mapset that has more than 15000 raster maps, but maybe that's fine :)

[annakrat]

Replying to martinl:

Replying to annakrat:

Replying to martinl: I am not sure if this solves the problem. I guess it won't show then the correct results? I am wondering why the process hangs, is it just because there is a lot of maps or because g.list or g.mapsets have some other problem? How many maps do you have in the biggest location, so that I can try to reproduce it?

it hanged on nc_spm_08_grass7 location with 6142 maps (modis included). The location is 19th from 34 locations.

I can now reproduce it, I will look at it later today. Thanks

[annakrat]

It should be now fixed in r67532, there was a deadlock because queue.get must be before joining the process, there is an example in python documentation.

Regarding the original issue, the new implementation (of loading maps) which isolates the environment fixes it. So I suggest to close it after you confirm it works.

[annakrat]

Replying to veroandreo:

Replying to martinl:

Replying to annakrat:

I am not sure if this solves the problem. I guess it won't show then the correct results? I am

I checked nc_spm_08_grass7 location in datacatalog and the result seems to be OK.

Works for me, too. I can see all locations, mapsets and their content. It seems slow though with the mapset that has more than 15000 raster maps, but maybe that's fine :)

I was testing with 12000 maps and 35 locations on a computer with 8 cores and SSD disk and it took less then 1 second to display the tree. It's parallelized on the level of locations, so if you have all maps in one location, it won't help you much. And depends on your computer specifications. What exactly is slow for you?

[pvanbosgeo]

Replying to annakrat:

It should be now fixed in r67532, there was a deadlock because queue.get must be before joining the process, there is an example in python documentation.

Regarding the original issue, the new implementation (of loading maps) which isolates the environment fixes it. So I suggest to close it after you confirm it works.

I can confirm the issue has been fixed. Great work, thanks a lot! I am closing the ticket.

[neteler]

Milestone renamed