Opened 2 years ago

Closed 8 months ago

#2459 closed task (fixed)

new dedicated VM for demo.mapserver.org

Reported by: Jeff McKenna Owned by: sac@…
Priority: normal Milestone: Sysadmin Contract 2021-II
Component: Systems Admin Keywords:
Cc: sdlime

Description (last modified by Jeff McKenna)

Hi Regina,

The many demo.mapserver.org services running off of the old adhoc server have finally out-grown the architecture there. Recent GDAL/PROJ and even MapServer demand a more updated compiler (adhoc runs Debian Wheezy and gcc 4.7.2). I've pushed that old server to its limits, but it's finally time to upgrade.

I've always been very good to record any changes and maintenance at https://wiki.osgeo.org/wiki/MapServer_at_AdhocVM

I request on behalf of the MapServer project for a new dedicated VM to host all of the heavily used demo.mapserver.org services. Some more notes:

  • this would be similar I guess to pycsw's recent VM (#2452)
  • it will honestly take me much effort to move all of the services to this new VM, so I propose that we keep both the adhoc and the new VM up during, and I will record my progress on a new wiki page (linked of course from https://wiki.osgeo.org/wiki/SAC_Service_Status )
  • if you ask my preference, I am much friendly to Ubuntu 18.04 , but can work in any environment that you provide me.
  • proposed VM name could be 'mapserver'
  • my LDAP account is: jmckenna
  • my SSH key is installed on osgeo7

Let me know what you think. And thank you for this.

(marking ticket as 'major' as with the recent MS 7.6.0 release done now, I'd like to get moving on upgrading all services, and especially since our 'msautotest' scripts will be testing now against old Adhoc MS 7.4.4 for now)

mercy buckets! :)

-jeff

Change History (42)

comment:1 by Jeff McKenna, 2 years ago

Description: modified (diff)

comment:2 by Jeff McKenna, 2 years ago

Description: modified (diff)

comment:3 by Jeff McKenna, 2 years ago

Description: modified (diff)

comment:4 by Jeff McKenna, 2 years ago

Note this is related to ticket #2384 (upgrade old-adhoc from Wheezy). Please let me know what is the best way to move forward on this.

comment:5 by robe, 2 years ago

Sounds like a good plan. Jeff -- I'll set up a debian 10 server, give you admin rights to it and you can install what you need on it and let me know if you need me to do anything. I unfortunately don't have a Ubuntu 18.04 with ldap authentication image in place, but I think debian / ubuntu are much the same as far as installing stuff.

comment:6 by Jeff McKenna, 2 years ago

thanks, yes I figured that also, after my many edits here ha.

comment:7 by robe, 2 years ago

Milestone: UnplannedSysadmin Contract 2020-I

Jeff I set up a dedicated called "mapserver" on osgeo7 and gave you sudo rights. You can add others as you see fit.

You can log in similar to how you log into old-adhoc, but mapserver as detailed here https://wiki.osgeo.org/wiki/SAC_Service_Status#Accessing_osgeo7_containers_via_ssh

The container I set up similar to pycw has the following specs, ability to log in with OSGeo LDAP


Debian 10 Docker 19.03.8 4 GB ram provisioned 200 GB disk (this includes backup space as well so you will see less) 4 CPU Prometheus Node Exporter (still need to register this so visible on monitor.osgeo.org)


We can see later after metrics have collected and you start using it if these limits are sufficient. Let me know if you need help installing anything. I didn't install apache or anything as I wasn't sure your preferences. Also didn't install PostgreSQL but can help with that if you need help installing those.

comment:8 by robe, 2 years ago

forgot to ask - should I setup something like staging.demo.mapserver.org so you can start testing before move? If so let me know what port on the server to connect to. Doesn't need to be port 80.

comment:9 by Jeff McKenna, 2 years ago

great idea for staging, yes please. how about port 8081 ?

comment:10 by robe, 2 years ago

Okay done - http://staging.demo.mapserver.org

should start showing a site instead of bad gateway when you are done

comment:11 by Jeff McKenna, 2 years ago

thanks Regina, I'm connected. (at first I was trying to ProxyJump from my home Windows machine through download onto this new instance, but then realized that is forbidden) It of course works fine as you said, connecting to download first and then connecting to the new instance. thanks again.

(I guess I was trying to just jump, so I didn't have to copy my private key to a cloud server)

Anyway out of my efforts I've now updated to the Windows OpenSSH 8.1-beta release ha, which in the end wasn't needed because jumping through download is forbidden ha. Oh well, I'm sure my new Windows OpenSSH expertise will come in handy later ha! :)

Short summary: I'm off and running on the new server. thanks again for this.

comment:12 by robe, 2 years ago

Hmm you shouldn't need to copy your private key. I'm running msys2 and using that so maybe it's different.

Not sure other modes of connecting via windows. That shouldn't be forbidden.

My .ssh/config file looks like this:

Host download.osgeo.org
    IdentityFile "/path/to/private.key"

Host osgeo7-*
	ProxyCommand ssh robe@download.osgeo.org -W $(sed -e "s/^osgeo7-//;s/$/.lxd/" <<< "%h"):%p
    IdentityFile "/path/to/private.key"

I do notice I need both entries though.

Then I do

ssh robe@osgeo7-mapserver

That did prompt me for my LDAP password since I didn't have that installed on mapserver container. But typing in my password works fine since I don't have password access blocked on the internal servers.

Last edited 2 years ago by robe (previous) (diff)

comment:13 by Jeff McKenna, 2 years ago

I totally agree that I shouldn't have to copy private key remotely.

However all throughout my testing I get this error (same setup as yours) :

  debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
  channel 0: open failed: administratively prohibited: open failed
  stdio forwarding failed
  kex_exchange_identification: Connection closed by remote host

StackExchange says the error is because of a missing setting on the setup of the Container (I have no access to that part).

comment:14 by Jeff McKenna, 2 years ago

It's actually ok though, as I tried so many different methods, and kept hitting that wall (container setting). I'm ok now to just move foward with private key on server. (2 days battling this ha)

comment:15 by Jeff McKenna, 2 years ago

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    IdentitiesOnly  yes
    #ServerAliveInterval 240
    #AllowTcpForwarding yes
    #PermitOpen any    
    
Host osgeo7-mapserver
    HostName     osgeo7-mapserver.lxd
    #Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jump nc %h %p
Last edited 2 years ago by Jeff McKenna (previous) (diff)

comment:16 by Jeff McKenna, 2 years ago

ssh -v osgeo7-mapserver

(my local and remote keys are accepted, as below) :

OpenSSH_for_Windows_8.1p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\Jeff/.ssh/config
debug1: C:\\Users\\Jeff/.ssh/config line 12: Applying options for osgeo7-mapserver
debug1: Setting implicit ProxyCommand from ProxyJump: "C:\\WINDOWS\\System32\\OpenSSH\\ssh.exe"  -v -W "[%h]:%p" jump
debug1: Executing proxy command: exec "C:\\WINDOWS\\System32\\OpenSSH\\ssh.exe"  -v -W "[osgeo7-mapserver.lxd]:22" jump
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt type -1
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
OpenSSH_for_Windows_8.1p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\Jeff/.ssh/config
debug1: C:\\Users\\Jeff/.ssh/config line 1: Applying options for jump
debug1: Connecting to jump [140.211.15.30] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt type -1
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to download.osgeo.org:22 as 'jmckenna'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:9Rj8e6GTNUeah218p0NaUqh143OD/90r2+MPpv90yeQ
debug1: Host 'download.osgeo.org' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\Jeff/.ssh/known_hosts:7
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt
debug1: Authentication succeeded (publickey).
Authenticated to download.osgeo.org ([140.211.15.30]:22).
debug1: channel_connect_stdio_fwd osgeo7-mapserver.lxd:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
Last edited 2 years ago by Jeff McKenna (previous) (diff)

comment:17 by Jeff McKenna, 2 years ago

By the way, from reading the many forums, for Windows users this is the recommended SSH tool now (I've learned a lot ha): OpenSSH for Windows: https://github.com/PowerShell/Win32-OpenSSH/releases The latest beta allows direct ProxyJump.

comment:18 by Jeff McKenna, 2 years ago

PS. I think next version of MS4W will include these SSH tools, so very nice! (OpenSSH for Windows)

comment:19 by Jeff McKenna, 2 years ago

My guess is that the remote server has AllowTcpForwarding or PermitOpen disabled, but I haven't been able to prove that.

Yikes :)

Last edited 2 years ago by Jeff McKenna (previous) (diff)

comment:20 by Jeff McKenna, 2 years ago

SOLVED!!!! Oh my lord, it was the hostname (notice that I specified the IP address instead of container name). YIKES! *solved* Phew! deserves a long-weekend beer on my patio now ha.

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    IdentitiesOnly  yes
    #ServerAliveInterval 240
    #AllowTcpForwarding yes
    #PermitOpen any  

Host osgeo7-mapserver
    #HostName     osgeo7-mapserver.lxd
    HostName      140.211.15.30
    #Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jump nc %h %p

comment:21 by Jeff McKenna, 2 years ago

(funny how I could look at that verbose debug output for 2 days, but pasting it here into Trac let me finally notice the one important line:

debug1: getpeername failed: Bad file descriptor

...so specificing the IP address fixed it. phew. humbled again :)

Thanks for listening to me rant ha!

comment:22 by Jeff McKenna, 2 years ago

That actually didn't solve it. Anyway as you said there are other ways to connect. Sorry for all this noise. I'm happy to connect period ha. Enjoy your sunday:)

comment:23 by robe, 2 years ago

Is your HostName like below. Wasn't clear if that is what you changed it to or not. That is what it should be.

HostName     osgeo7-mapserver.lxd

seems like you had that earlier on. Also keep in mind that each host has a separate jump. I created one for download.osgeo.org to be consistent with the osgeo3 and osgeo4.

So osgeo3 = hop.osgeo3.osgeo.org

osgeo4 = hop.osgeo4.osgeo.org

osgeo7 = download.osgeo.org with alias hop.osgeo7.osgeo.org

Can you do below or is ProxyCommand not supported on your version of OpenSSH

Host hop.osgeo7.osgeo.org
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

Host osgeo7-mapserver
    ProxyCommand ssh jckenna@hop.osgeo7.osgeo.org -W %h:%p
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

and then

ssh jmkenna@osgeo7-mapserver

Note the .lxd with or without is optional since they are on the same network anyway so the .lxd is assumed.

Last edited 2 years ago by robe (previous) (diff)

comment:24 by Jeff McKenna, 2 years ago

Yes sorry, I was testing with ProxyCommand 2 days ago, the working syntax is above but commented out, for 'OpenSSH for Windows' (the new Windows native SSH client).

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump

The error is the same:

channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host

comment:25 by Jeff McKenna, 2 years ago

I think I will keep going in circles ha. I am happy that I am able to connect now period (unfortunately not through the new Windows native SSH client). That's definitely ok. Thanks for helping.

comment:26 by robe, 2 years ago

Hah I do have openssh windows installed. Must have gotten dragged in by VS Studio or VS Studio Code.

Seems ProxyCommand the way we have documented doesn't seem to work with all the sed stuff, bummer. I'll stick with mys2 thank you :)

It just occurred to me you should be using mapserver for the host, The osgeo7-mapserver only works if you are using the hack that strips off osgeo7- from it whcih stupid windows ssh seems incapable of using.

The real hostname is just mapserver.

This worked for me with windows open ssh, off course using my id_rsa and name not yours :)

Give the below a try

Host jump
    HostName     hop.osgeo7.osgeo.org
    Port         22
    User         jmkenna
    IdentityFile "C:\Users\JMckenna\.ssh\id_rsa.openssl-decrypt"

Host mapserver
    HostName  mapserver
    ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe  -W %h:%p jump
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

And then you should be able to do

ssh jmckenna@mapserver
Last edited 2 years ago by robe (previous) (diff)

comment:27 by Jeff McKenna, 2 years ago

Yes all Windows users now have the native SSH included. Nice trick eh?

Not surprised it was the hostname, I was trying so many options for it, the IP etc ha (it was the error 'getpeername failed" that was telling me that). Here is my working settings, hopefully this helps someone else someday down the road...

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    
Host osgeo7-mapserver
    HostName     mapserver
    User         jmckenna
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump

Thanks!

PS. I don't use unix emulators for anything, I find I learn much more this way, the hard way ha.

Wishing you a nice holiday Monday from Canada! Thanks for your help Regina.

comment:28 by robe, 2 years ago

I'm closing out since the server has already been created and Jeff can log into it and I also have the nginx set up for.

Jeff - put in another ticket if you need anything else.

comment:29 by robe, 20 months ago

I guess I did never close this out -- Jeff are you done setting up the new server. Can we please move demo.mapserver.org to it now?

After restarting old-adhoc, it appears the demo.mapserver.org site did not come back.

It is going thru

 http://old-adhoc.lxd:8080/

The /geowebcache is using

http://old-adhoc.lxd:8081/geocache

As far as I can tell it is working

https://staging.demo.mapserver.org/tutorial/example1-1.html

comment:30 by Jeff McKenna, 20 months ago

I left the ticket open as I did move all demo services to the new server, all except one service, which I am half-way through tackling. I've manually restarted old-adhoc Apache now, all good (which I've been doing for decades on old-adhoc each time after reboot, my bots would have notified me of your reboot in an hour, and then I jump into action ha, this is the usual process). Will keep you posted here once I solve the last MapServer service on staging, sorry for the delay. thanks for the nudge :)

Last edited 20 months ago by Jeff McKenna (previous) (diff)

comment:31 by robe, 19 months ago

Milestone: Sysadmin Contract 2020-ISysadmin Contract 2020-II

move open times to new contract

comment:32 by robe, 19 months ago

Milestone: Sysadmin Contract 2020-IISysadmin Contract 2020-III

oops moved to wrong milestone last run

comment:33 by robe, 18 months ago

Milestone: Sysadmin Contract 2020-IIISysadmin Contract 2021-I

Milestone renamed

comment:34 by robe, 16 months ago

Priority: majornormal

This is still waiting on jmkenna and sdlime. Much of the work is done - the container exists.

comment:35 by robe, 15 months ago

Milestone: Sysadmin Contract 2021-ISysadmin Contract 2021-II

Move these to next contract milestone

comment:36 by Jeff McKenna, 13 months ago

@robe you'll be happy to know that I think it's finally time to make the switch. I worked on this again today and yesterday. The last service has been setup on staging, so I think we should now:

  • point demo.mapserver.org to the 'mapserver' container on osgeo7
  • I assume that will be port 80 (it is now 8081, as we discussed before). Let me know and I can change that in the virtualhost
  • point the former demo.mapserver.org on the 'old-adhoc' container to something like old.demo.mapserver.org (just in case we find a few other files we need)
  • leave old.demo.mapserver.org running until we confirm that all files are removed from old-adhoc

Does that sound like a good plan?

Thanks. -jeff

comment:37 by robe, 8 months ago

jeff not sure how I missed this -- I'm so sorry. I'll go ahead and finish this off.

comment:38 by robe, 8 months ago

Resolution: fixed
Status: newclosed

Okay done

https://demo.mapserver.org -> mapserver:8081 (https (port 443) and http (port 80) https://old.demo.mapserver.org -> old-adhoc:8080 (https port 443 and http (port 80) 0

I'll close this ticket out. Could I interest you in moving mapserver.org as well? I think it's on osgeo6. If so please create a new ticket for that.

I also removed staging.demo.mapserver.org since that is now demo.mapserver.org

Last edited 8 months ago by robe (previous) (diff)

comment:40 by Jeff McKenna, 8 months ago

thanks @robe ! Works great. I agree, mapserver.org should be on this same container, will create a ticket. Thanks again!

comment:41 by Jeff McKenna, 8 months ago

Resolution: fixed
Status: closedreopened

Reopening. There seems to be a problem with HTTPS & port redirection. Clients such as QGIS call services on this demo, and they hit this url (notice the ":80" port, which oddly appears in the QGIS logs)

Even though I specify this request (notice no port), which in the browser returns a valid map image:

I'm not sure why the server redirects to the 80 port and causes this error.

(to be clear: I am not specifying the port in QGIS, but I notice that in the QGIS logs, that the request is sent to a port 80 and then fails with an HTTPS error)

(also to be clear: this does not occur with the old demo.mapserver.org so there must be something different in the HTTPS/nginx settings?) Not sure.

Last edited 8 months ago by Jeff McKenna (previous) (diff)

comment:42 by Jeff McKenna, 8 months ago

Resolution: fixed
Status: reopenedclosed

Oops my fault! Missed an important metadata change. Fixed. Sorry for the noise.

Note: See TracTickets for help on using tickets.