Opened 5 months ago

Last modified 5 months ago

#3131 assigned task

Getting 500 errors on osgeo7

Reported by: robe Owned by: robe
Priority: normal Milestone: Sysadmin Contract 2024-I
Component: SysAdmin Keywords:
Cc:

Description

Just noticed it. Might be happening on trac as well.

Change History (2)

comment:1 by robe, 5 months ago

Component: SysAdmin/GiteaSysAdmin
Summary: Getting 500 errors on giteaGetting 500 errors on osgeo7

Actually this is more than trac. I think it's osgeo7 since my alerts are going off on other sites on osgeo7

comment:2 by robe, 5 months ago

It was happening today during our nextcloud meeting. The error logs showing too many open files.

I upped the max open files limits

By adding these lines to /etc/security/limits.conf as sugested in

https://serverfault.com/questions/1003364/nginx-500-24-too-many-open-files

www-data    soft    nofile   20000
www-data       hard    nofile  40000

Then adding at the top of /etc/nginx/nginx.conf

worker_rlimit_nofile 16384; 
systemctl nginx restart
ps -faux #using the root proc for nginx fill in below
cat /proc/<nginx-pid>/limits

And verified the settings took before the open files soft limit was 1024 after

Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            16384                16384                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       513936               513936               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

I still need to verify if the /etc/security/limits.conf was really necessary as I don't think it should have been. It seemed without that putting in the worker_rlimit_nofile 16384; did not take.

I'll keep this open and monitor.

Note: See TracTickets for help on using tickets.