Opened 2 years ago

Closed 2 years ago

#2853 closed task (fixed)

tracsvn ran out of disk space

Reported by: robe Owned by: sac@…
Priority: normal Milestone: Sysadmin Contract 2022-II
Component: SysAdmin Keywords:
Cc:

Description

Tracsvn ran out of disk space again. I deleted a snapshot to clear space.

I'm going to check if there is something eating up space.

So this broke both gitea and trac since when server runs out of disk space, PostgreSQL goes into panick mode and shuts itself down to prevent data corruption. Both use PostgreSQL

Change History (1)

comment:1 by robe, 2 years ago

Resolution: → fixed
Status: new → closed

I didn't see anything out of the ordinary. gitea's git folder is taking up 66GB and /var/www/svn takes up about 25GB. Those are the biggest eaters of disk space. Followed by /var/log which is about 2.5GB

So I assume it's just churn of data. I looked at the snapshots which the 500GB I had allocated was partly used for. I deleted a couple more snapshots and upped the capacity from 400G to 700GB. using 

lxc config edit tracsvn

So now there is 241G available.

Command to check snapshot usage is below for future reference 

 zfs list -t snapshot | grep tracsvn

The snapshot sizes are averaging anywhere from 2GB - 19 GB and we have a retention of 30 days. as shown below 

osgeo7/containers/tracsvn@snapshot-snap937                                                                            2.19G      -     97.6G  -
osgeo7/containers/tracsvn@snapshot-snap938                                                                            1.45G      -     98.2G  -
osgeo7/containers/tracsvn@snapshot-snap939                                                                            2.71G      -      104G  -
osgeo7/containers/tracsvn@snapshot-snap940                                                                            5.63G      -      108G  -
osgeo7/containers/tracsvn@snapshot-snap941                                                                             386M      -     99.8G  -
osgeo7/containers/tracsvn@snapshot-snap942                                                                            2.98G      -      104G  -
osgeo7/containers/tracsvn@snapshot-snap943                                                                            9.61G      -      115G  -
osgeo7/containers/tracsvn@snapshot-snap944                                                                            6.83G      -      114G  -
osgeo7/containers/tracsvn@snapshot-snap945                                                                            8.88G      -      113G  -
osgeo7/containers/tracsvn@snapshot-snap946                                                                            14.0G      -      121G  -
osgeo7/containers/tracsvn@snapshot-snap947                                                                            10.0G      -      119G  -
osgeo7/containers/tracsvn@snapshot-snap948                                                                            7.70G      -      115G  -
osgeo7/containers/tracsvn@snapshot-snap949                                                                            10.7G      -      119G  -
osgeo7/containers/tracsvn@snapshot-snap950                                                                            76.1M      -      130G  -
osgeo7/containers/tracsvn@snapshot-snap951                                                                             143M      -      134G  -
osgeo7/containers/tracsvn@snapshot-snap952                                                                            1.76G      -      121G  -
osgeo7/containers/tracsvn@snapshot-snap953                                                                            10.5G      -      124G  -
osgeo7/containers/tracsvn@snapshot-snap954                                                                            6.52G      -      123G  -
osgeo7/containers/tracsvn@snapshot-snap955                                                                            16.3G      -      131G  -
osgeo7/containers/tracsvn@snapshot-snap956                                                                            10.3G      -      117G  -
osgeo7/containers/tracsvn@snapshot-snap957                                                                            24.2G      -      121G  -
osgeo7/containers/tracsvn@snapshot-snap960                                                                            17.5G      -      114G  -
osgeo7/containers/tracsvn@snapshot-snap961                                                                            6.64G      -      111G  -
osgeo7/containers/tracsvn@snapshot-snap962                                                                            7.10G      -      119G  -
osgeo7/containers/tracsvn@snapshot-snap963                                                                            6.42G      -      114G  -
osgeo7/containers/tracsvn@snapshot-snap964                                                                            8.96G      -      113G  -
osgeo7/containers/tracsvn@snapshot-snap965                                                                            6.49G      -      111G  -
osgeo7/containers/tracsvn@snapshot-snap966                                                                            9.31G      -      114G  -
osgeo7/containers/tracsvn@snapshot-snap967                                                                            85.9M      -      109G  -
osgeo7/containers/tracsvn@snapshot-for-osgeo4                                                                         1.21G      -      115G  -

So worsed case 30*20 = 600Gb for snapshot and actual container is using 110GB. So we'd need 600 + 110GB = 710GB to handle worsed case scenario.

Note: See TracTickets for help on using tickets.