download.osgeo.org SSL certificate expired

[Bas Couwenberg]

The download.osgeo.org SSL certificate expired today and was not automatically renewed as you'd expect for Let's Encrypt certificates:

$ echo QUIT | openssl s_client -connect download.osgeo.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = download-cache.osgeo.org
verify error:num=10:certificate has expired
notAfter=Jun 24 01:39:47 2022 GMT
verify return:1
depth=0 CN = download-cache.osgeo.org
notAfter=Jun 24 01:39:47 2022 GMT
verify return:1
---
Certificate chain
 0 s:CN = download-cache.osgeo.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 26 01:39:48 2022 GMT; NotAfter: Jun 24 01:39:47 2022 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFSzCCBDOgAwIBAgISBJ1tI8X3ITREuxOXHNdm6YvxMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjAzMjYwMTM5NDhaFw0yMjA2MjQwMTM5NDdaMCMxITAfBgNVBAMT
GGRvd25sb2FkLWNhY2hlLm9zZ2VvLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL2u0YrDTtrvsvxwhSxI0bx9pLMKjMpC+9wxNtIH1gAOeMaIdq/y
QnQUDhC7OicP1hf9l9WRDKdVe/OkdHLGqCu9jvNzn+TtNGjQzsJL8jJ8eWAzMGlc
t8WyT4SW2tdQC0dUeSxnesfqqQZTff8lxfvd62WwzXC7xh+XWimdLhy4OL4AJKiO
6AVmifAvSiYgdlgawvO0uMMm8+kv8o1yNTXzAqJbiGfeqH7zTBlfeHg3g5NLf1h6
RzboSE6xdho32Ve26l7CaiBdYpRqacQLRd46NX2RcmSSrSaKT6u53u0FLIuW6Ww3
LL6qpHd6nonXeOvCwxJii3tYa44IfqWJtW8CAwEAAaOCAmgwggJkMA4GA1UdDwEB
/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/
BAIwADAdBgNVHQ4EFgQUgtHQkj2Z7nv/P3RL2YgaZ9IgELUwHwYDVR0jBBgwFoAU
FC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzAB
hhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5p
LmxlbmNyLm9yZy8wNwYDVR0RBDAwLoIYZG93bmxvYWQtY2FjaGUub3NnZW8ub3Jn
ghJkb3dubG9hZC5vc2dlby5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYB
BAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5v
cmcwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdgApeb7wnjk5IfBWc59jpXflvld9
nGAK+PlNXSZcJV3HhAAAAX/EGBwSAAAEAwBHMEUCIEiWn7bykjYwp4UGEOJee/IS
xAnL7aSRHj+06tra/4cPAiEAnZuoTb7ha8Nqmy9F8aTgZuC2uVwSR0h2sVgwOW52
/8QAdwBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZEwAAAX/EGBxMAAAE
AwBIMEYCIQCyXg9GCefoSoqal2jEBfXqSDFWDeukDEl8TmNliD0MYwIhAKvNQcNx
gy2CyZQQGgB64N4SmM5FhuEVc+awmqnA4YhKMA0GCSqGSIb3DQEBCwUAA4IBAQBT
cegcMdc5zSVWTIA/2EUNsFvA9J2FZhOZwCC6HNGZjMzDfYmp9pynrdj2X3evPBKT
xqfq4GGs2SxlTdFCmwJnrnZkmY20kEB7SN4wqhU6Y35TTgrARq/fYxhRDH50CDqX
gXPn9zZUNKkbp4oCXEocFUoHRdZ71ktTwzX429KKZfcs1LOnlfMX2Ek/6szizU7m
35cLmh4hqGCVIwUp9/2BRHDp8WAIijmBSvva0d7jQNeaSEtFFEeGWQwKet4r7mmU
EJSntyEQG7a6u6cppoPYsl5fqgxsWigHx7MFb0oz+9Zilrd/8tffG0LnjvsDpx+4
TVjEXDG8qJQl7nt/FShR
-----END CERTIFICATE-----
subject=CN = download-cache.osgeo.org
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4615 bytes and written 400 bytes
Verification error: certificate has expired
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)
---
DONE

[jef]

See also ​OSGeo4W #751

[robe]

This should be fixed now, but I'm leaving this open until I verify my changes and commit them to ansible.

[Bas Couwenberg]

The certificate validates again:

$ echo QUIT | openssl s_client -connect download.osgeo.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = download-cache.osgeo.org
verify return:1
---
Certificate chain
 0 s:CN = download-cache.osgeo.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 24 15:12:35 2022 GMT; NotAfter: Sep 22 15:12:34 2022 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFSTCCBDGgAwIBAgISA0IAjfwb60R4qyvr6M2l/wm9MA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA2MjQxNTEyMzVaFw0yMjA5MjIxNTEyMzRaMCMxITAfBgNVBAMT
GGRvd25sb2FkLWNhY2hlLm9zZ2VvLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAKVb0FTEv+5U3JN9pAUpUxWGscdwY2rI/eSa3klAmsXTc/o9xRNT
QG1OPhYbPIaM939jqlHcVc9g7XEv5cCWtZj7roi0lWN2oCjyY4qBiVN4PMkmJrki
H5V/wEVmBYJTvvPulKP2sA95z+GUZPMrI91Dj00Vq1T4iUwpzkm5x/fhmj+8qu2w
OQBf0fCdlY4jKbYTkf8MezF54xxLWOGtnwCM10YLaOLISd9G4nYM6Nvlg3JsKEU+
tSYAetNzEIAPAnA6XvicQCXoXn0uhxkE9Ho+v+6UcN2ThRiPO1W+ETnMzJwOfPJR
ewGT8vAG2uvr6B5CXxJLBz/uZp1+nLGFxDkCAwEAAaOCAmYwggJiMA4GA1UdDwEB
/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/
BAIwADAdBgNVHQ4EFgQUybLRBDCuHihmw1YhmmVwxRYk864wHwYDVR0jBBgwFoAU
FC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzAB
hhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5p
LmxlbmNyLm9yZy8wNwYDVR0RBDAwLoIYZG93bmxvYWQtY2FjaGUub3NnZW8ub3Jn
ghJkb3dubG9hZC5vc2dlby5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYB
BAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5v
cmcwggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdQBByMqx3yJGShDGoToJQodeTjGL
GwPr60vHaPCQYpYG9gAAAYGWfJV6AAAEAwBGMEQCIF9OYWNKqe4W/znZDb7vmHoQ
9/haoTzfPSegmWm6t60PAiB8LkXyvjMFnT7oxJQADFTM7VxswdevXiQUBM+t97gh
TQB2AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr8vxw/m1HAAABgZZ8lZEAAAQD
AEcwRQIhAK4FP1sr2Qy0aDMrlpMDBOAGVJ+zVvjq4RO9RTRu1M51AiAa8IfMrABD
nPyrt9FI0AbkyS+MGThvBLQ/uP/FDR638jANBgkqhkiG9w0BAQsFAAOCAQEAA58v
4P/SNI/D1iT+iTJ5zIdNK3tk5/8LkP2+gidpyo41cMF1OfIm3DUHnZIHmc7QIllC
YT8N00JkPwJt8jQNBUVPWlbIJUH11IkjjJ3qTcqiyF3nXSqMmTFhBz6MPHd71fA0
isk2k8oYDpl9PwA/uBR+A/x0oAsRF65dC8XuwijBF5EppO1qmjEQbT+qXlrtqgDH
RBk0L6WctptkvcOhZg+ex9pzd2e8Gvc/Q1aXf8HF16BmOj7AyBQxc7oPQTjYVlR7
pztC6+26O2YLs7+5AhsZbsHkHX+WgF/lRdI6y10dckmmtqg73XKPvG2nPqIzMrOb
03JEOu/wTp/yo2wJ3Q==
-----END CERTIFICATE-----
subject=CN = download-cache.osgeo.org
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4613 bytes and written 400 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

What caused the autorenewal to fail?

[robe]

I'm still figuring out the best way to set this up. It has to do with the round-robin not being able to fetch from remote server. I thought I had it working, but still having issue, so keeping this open until I resolve.

[Bas Couwenberg]

You could use NFS for the letsencrypt directory to make the certificate available on multiple hosts.

If the problem is that not all hosts in the round-robin are available when the autorenewal tests the availability of the hostnames, the dns-01 challenge may be an option, but it seems that PairNIC doesn't have an API to manage DNS nor does it seem to support RFC 2136 Dynamic Updates.

[robe]

I think it was a number of things going on here.

First was my misunderstanding of where certbot was writting the challenge files so had my challenge nginx config set wrong.

and second I think on one of the servers, the renewal config was being overwritten back to using default nginx instead of webroot.

Anyway I did dry run renew and all servers are passing now.

certbot renew --dry-run

I'll check again in 2 months to make sure it renews and the configs haven't been reverted by the process.

[Bas Couwenberg]

Consider monitoring the certificates with something like ​check_ssl_cert, I use that for my certbot setup which tends to fail when IPv6 is not working correctly.

[robe]

Thanks for the suggestion. I'll take a look at implementing that on our end.

[strk]

Regina: we noticed, with cvvergara, that the change you committed in ansible-deployment referencing this ticket ( ​https://git.osgeo.org/gitea/sac/ansible-deployment/commit/bc2f8566bb3fae86ccb82de8c75c5c5ea866934c ) changed the @acme2 location IP of download-cache from 32 to 30 but on the osgeo9 nginx still had ip 32. I've helped Vicky making the ansible file match the nginx container file ( ​https://git.osgeo.org/gitea/sac/ansible-deployment/commit/fc3b14ea47603a5f998b1d739ffc0aac651d2fa6 ) but given the state of affair it would be good to have your confirmation about correctness of those IPs.