Opened 15 months ago
Last modified 7 months ago
#2928 new task
tracsvn been down since for 4 hrs out of disk space
| Reported by: | robe | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | Sysadmin Contract 2024-I |
| Component: | SysAdmin | Keywords: | |
| Cc: |
Description
Looks like trac went down again as it ran out of disk space.
I'm assuming it's the same issue as before with gitea cache going to check.
I increased it's disk quota to buy some time. Quota was 900GB before and is now at 1.2TB
Change History (4)
comment:1 by , 15 months ago
comment:2 by , 15 months ago
On closer inspection I think this might be related to our slow network in backup. I'm going to reboot osgeo4 in a bit to see if it helps.
I saw 4 tracsvn backups in progress and just killed them. Which means it was probably holding 4 copies of tracsvn (so like 150GB for each). What puzzles me the the disk space is not reclaimed, I see some temp snapshots in zfs, but they don't add up the the 800GB some odd that is claimed to be used, and the instance itself is only taking up 200GB. I'm sure some is being held in limbo and is in use but not counting towards snapshot space, but still eating up the alloted space. I plan to reboot osgeo7 this weekend too to try to clear up some of this.
comment:3 by , 14 months ago
I'm suspecting this issue will be resolved after I reboot osgeo7. There is some dead zfs space resulting from failed backups to osgeo4 and I think they are locked. At very least I should be able to delete these orphan snapshots once I reboot.
comment:4 by , 7 months ago
| Milestone: | Sysadmin Contract 2023-I → Sysadmin Contract 2024-I |
|---|
Moving my prior still open items to the next proposed Milestone
Garbage collecting cleared up 100GB but even with 100GB the server was only taking up about 300GB, so probably mostly resulting from big snapshots of the server which are retained for 30 days.