Opened 13 months ago

Last modified 5 months ago

#2928 new task

tracsvn been down since for 4 hrs out of disk space

Reported by: robe Owned by: sac@…
Priority: normal Milestone: Sysadmin Contract 2024-I
Component: SysAdmin Keywords:
Cc:

Description

Looks like trac went down again as it ran out of disk space.

I'm assuming it's the same issue as before with gitea cache going to check.

I increased it's disk quota to buy some time. Quota was 900GB before and is now at 1.2TB

Change History (4)

comment:1 by robe, 13 months ago

Garbage collecting cleared up 100GB but even with 100GB the server was only taking up about 300GB, so probably mostly resulting from big snapshots of the server which are retained for 30 days.

comment:2 by robe, 13 months ago

On closer inspection I think this might be related to our slow network in backup. I'm going to reboot osgeo4 in a bit to see if it helps.

I saw 4 tracsvn backups in progress and just killed them. Which means it was probably holding 4 copies of tracsvn (so like 150GB for each). What puzzles me the the disk space is not reclaimed, I see some temp snapshots in zfs, but they don't add up the the 800GB some odd that is claimed to be used, and the instance itself is only taking up 200GB. I'm sure some is being held in limbo and is in use but not counting towards snapshot space, but still eating up the alloted space. I plan to reboot osgeo7 this weekend too to try to clear up some of this.

comment:3 by robe, 12 months ago

I'm suspecting this issue will be resolved after I reboot osgeo7. There is some dead zfs space resulting from failed backups to osgeo4 and I think they are locked. At very least I should be able to delete these orphan snapshots once I reboot.

comment:4 by robe, 5 months ago

Milestone: Sysadmin Contract 2023-ISysadmin Contract 2024-I

Moving my prior still open items to the next proposed Milestone

Note: See TracTickets for help on using tickets.