#2934 closed task (fixed)
osgeo.org domains not reachable
| Reported by: | robe | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | Sysadmin Contract 2023-I |
| Component: | SysAdmin | Keywords: | |
| Cc: |
Description
osgeo7 went down and appeared to be down for about 15-20 minutes, though osgeo7 and all the instances seemed fine.
All came back fine.
I'm suspecting it was a momentary issue with osgeo7 nginx, as trying to run
systemctl status nginx
took longer than usual
Change History (10)
comment:1 by , 14 months ago
comment:2 by , 14 months ago
Put in a support ticket with PAIRS. They confirmed its a DNS attack against the osgeo.org domain, so that is why our other domains housed on PAIRs were not affected unless they had a CNAME to osgeo.org. details here
https://lists.osgeo.org/pipermail/sac/2023-May/015247.html
At the moment things seem to be back to normal, but I give it a day or so before confirm it is all set.
comment:3 by , 14 months ago
No real change yet. They are still patching. I'll call again to ask if they can move osgeo.org to separate NS servers. As DLange observed the other domains we have are not on NS1 and NS2 pairdomains, but oare on like NS3-NS8.
Then again foss4g is on NS1 and NS2 and haven't had trouble hitting them, so if it is a DOS on particular set of domains that might not help.
comment:4 by , 14 months ago
| Summary: | osgeo7 systems went down → osgeo.org domains not reachable |
|---|
I called PAIRs again asking them to move our osgeo.org domain from NS1, NS2 to NS3 - NS8 since the other domains we have on those domain servers are not having issues.
They have that ticketed now but person on staff unfortunately is not allowed to make the change since he's on the hosting instead of the DNS side.
comment:5 by , 14 months ago
Received an update from PAIRs note below
I wanted to follow up again. While the initial attack was mitigated, we are still seeing many requests per second on what appears to be random sub-domains of osgeo.org. The software to mitigate attacks has been installed on all of our name server, and our system admins are monitoring them. Switching to different ns*.pairnic.com name servers would not help. Since osgeo.org is being targeted, the attack will shift to whatever name servers osgeo.org uses.
comment:6 by , 14 months ago
At several peoples requests, I've changed our DNS over to cloudflare. Will monitor how this goes and close this out. I need to change the email address on cloudflare, but I changed it too many times so am blocked at the moment from changing it.
Once I change, I'll put the passwords and email account in passwordstore.
comment:7 by , 14 months ago
A lot of things have improved! repo.osgeo.org is back and builds are green :)
One server that is not back yet is docs.geoserver.org, and geo-docs.geoserver.org (used for uploading content).
I am not sure if this is related to the DNS trouble; of if this specific instance is not yet happy.
comment:8 by , 14 months ago
geo-docs.geoserver.org and docs.geoserver.org should be fixed now.
I think it's because I still had web.osgeo3.osgeo.org still in proxy mode. I've removed that so it's DNS only now.
comment:9 by , 14 months ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
Things still seem to be working. I'v committed the account info to the password store.
@strk and @jef, please confirm you can read them.
I thought it was just osgeo7 but got failures on osgeo3 and osgeo9 as well. Given that I can ssh into the hosts and they don't seem to be taxed, I'm guessing it's either an issue on my end, or something up with the PAIRS dns.