logrotate on osgeo3-nginx not working causing disk runout

[robe]

So osgeo3-nginx ran out of disk space, which I think caused all the issues reported about repo.osgeo.org namely #3222, #3219 and some issues mentioned on osgeo-sac channel about docker giving random 503 errors during image pull. As well as my matrix disconnecting (which I had blamed on being connected to VPNs that randomly disconnect me from stuff).

Anyway what happened is the logrotate service is supposed to rotate the nginx and other logs. Instead the nginx logs kept on growing, the nexus one had balllooned to 850GB or some crazy number.

[robe]

To fix for now I enabled nesting on the container. Which is how I fixed the grass one on osgeo9.

I suspect something in apparmor changes is to blame here as I had annoying issues dist upgrading like peertube, and just completely killed apparmor in that case.

Haven't tried doing that here.

I'm going to close this out for now.

Also note that because of the disk issue, I did have to copy over nginx cause the original container I couldn't increase disk size / snapshot or anything when it's run out of allocated space.

lxc cp osgeo3-nginx --instance-only osgeo3-nginx-2
lxc exec osgeo3-nginx -- shutdown now
lxc start osgeo3-nginx-2
lxc config edit osgeo3-nginx-2 security.nesting true

This is currently running and taking a while

lxc mv osgeo3-nginx osgeo3-nginx-old

once that is done, I'll rename osgeo3-nginx-2 back to its original name and close this ticket out