tracsvn been down since for 4 hrs out of disk space

[robe]

Looks like trac went down again as it ran out of disk space.

I'm assuming it's the same issue as before with gitea cache going to check.

I increased it's disk quota to buy some time. Quota was 900GB before and is now at 1.2TB

[robe]

Garbage collecting cleared up 100GB but even with 100GB the server was only taking up about 300GB, so probably mostly resulting from big snapshots of the server which are retained for 30 days.

[robe]

On closer inspection I think this might be related to our slow network in backup. I'm going to reboot osgeo4 in a bit to see if it helps.

I saw 4 tracsvn backups in progress and just killed them. Which means it was probably holding 4 copies of tracsvn (so like 150GB for each). What puzzles me the the disk space is not reclaimed, I see some temp snapshots in zfs, but they don't add up the the 800GB some odd that is claimed to be used, and the instance itself is only taking up 200GB. I'm sure some is being held in limbo and is in use but not counting towards snapshot space, but still eating up the alloted space. I plan to reboot osgeo7 this weekend too to try to clear up some of this.

[robe]

I'm suspecting this issue will be resolved after I reboot osgeo7. There is some dead zfs space resulting from failed backups to osgeo4 and I think they are locked. At very least I should be able to delete these orphan snapshots once I reboot.

[robe]

Moving my prior still open items to the next proposed Milestone

[strk]

Sounds like long done