OK, several problems now confirmed.
1) Creating jpeg compressed overviews on tiff with 1.6-esri does produce the expected error. Presumably the .ovr file is not created as big tiff when compression is used since it is hard to predict how large the file will get.
2) Creating .rrd files on a large file where each band overview is less than 2GB does not result in production of a spill file - at least for my test case. And the resulting >2GB file ends up having "2GB barrier wrap" issues so the imagery is corrupt - possibly the cause of apparent greyscale imagery?
Working on resolutions.