It appears that the smoothing needs to use an updated mask with the filled data. This happens easily when a dataset is update in place, using GDALFlushRasterCache() on the mask band for the data. If the mask is external, though, the values are reverted back to the original no data mask. I *think* the original intention was to use the updated mask, which worked if filling in place. The assert would make more sense then, as the filtered data would have to have a new value, not no data.
To resolve this, before starting the smoothing, I replace the supplied mask band with the filled target mask band, recomputed. This works with the test data on this ticket, the data on #4539, and I tested all three mask options, default, nomask, and a mask file. So #4539 can be closed if this works,
patch attached.
