MapServer seems to restrict organising layers to two levels: One can have regular map layers right at the top-most level, or one can specify groups and assign the layers to them. Assigning groups to groups, that is, specifying nested groups in a tree is not supported. This limitation is reflected in the file MapServer/php/LoadMap.php where a line of comment mentiones this limitation.
I have played with that piece of code a bit, and it seems that the limitation can easily be overcome. In the function getGroupObject(..), the layer that is passed to the function can be queried for its parent. If the layer corresponds to the group, it will know about it parent group, and this information can be stored in group->parent as well as in group->parentUniqueId.
(I think that there is a bug where the function is called, a few lines above, because the layer that is passed to the function may or may not be the group layer in question. The function queries for group information such as METADATA "groupLegendLabel" which returns empty results if the layer is not the group layer of interest. I'll post a separate ticket on this.)
After having changed the code in this way, the groups are indeed nested in the legend, they behave as they should when checked and unchecked, and the maps are drawn correctly accounting for the visibility status of the groups and layers.
