Getting random 500s on osgeo7 hosts

[robe]

Just noticed in past 5 minutes

​https://trac.osgeo.org/osgeo ​https://git.osgeo.org/gitea ​https://photoprism.osgeo.org ​https://download.osgeo.org

[robe]

Seems to be fine now. I rebooted nginx after doing some patch updates.

The prometheus nginx exporter was failing because of apparmor.

I haven't figured out what settings to do with apparmor and what benefit it has for us exactly so I disabled it but seems to have started back on it's own, so doesn't stay disabled

systemctl disable apparmor

I suspect if I reboot the nginx-prometheus will stop working again. Not a huge deal.

[robe]

Still seems to be an issue but just on osgeo7 sites.

[robe]

Okay I think there is some extra traffic happening cause the /var/log/nginx/error.log was showing errors

768 worker_connections not enough

So it was going up and down.

I've upped that 1024, but 768 was kinda low, so surprised we haven't run into issues with this before. I'm going to check our other nginx.

I also just deleted that rewrite file in /etc/nginx/sites-enabled as I suspect those have all been moved to osgeo9 and they are mostly redirects for .osgeo.org sites that redirect to www.osgeo.org

I'm going to keep this open for now until I have reviewed the situation more.

[robe]

oops this could be related to my removal of spatialreference.org in #3101 I'm see a lot of errors in the error logs of sites trying to hit www.spatialreference.org which maybe a cache issue cause

www.spatialreference.org when I ping does go to cdn-185-199-108-153.github.com

[robe]

This seems to have dissipated after increasing max worker_processes and haven't seen these 500s in the past 6 hrs.