woodie builds fail due to exceeded quota

[strk]

Especially when we have many steps, where each step creates a container, it's easy to end up out of space.

This pull request for woodpecker-ci should help with this problem: ​https://github.com/woodpecker-ci/woodpecker/pull/2681

Meanwhile our best chance is to docker system prune as often as possible.

[robe]

I thought I did do a daily docker system prune on woodie-server and all the woodie-client

It only runs daily though.

If I'm already doing that, I can up the space available on them. I think I had made it a modest 150G or so.

[strk]

How about hourly instead of daily ?

[robe]

We could do that, but ultimately I think the issue is docker system prune is not pruning everything. Even after doing

docker system prune --all --force

There still seem to be large numbers of orphan /var/lib/docker/vfs left around amounting to about 60 GB still left. I ended up just reinitializing all of them with the base image or snapshot from osgeo8 woodie-client/reset

This is safe to do with the clients since I think the logs are kept on woodie-server, so nothing is lost.

I do plan to try podman to see if it does a better job of cleaning up when you ask it to clean up.

The reset image ends up creating a container with 1.4 GB in use (which includes the woodie client running, but no images beside the woodpecker one)