Opened 5 months ago

Last modified 4 months ago

#2459 new task

new dedicated VM for demo.mapserver.org

Reported by: Jeff McKenna Owned by: sac@…
Priority: major Milestone: Sysadmin Contract 2020-I
Component: Systems Admin Keywords:
Cc: sdlime

Description (last modified by Jeff McKenna)

Hi Regina,

The many demo.mapserver.org services running off of the old adhoc server have finally out-grown the architecture there. Recent GDAL/PROJ and even MapServer demand a more updated compiler (adhoc runs Debian Wheezy and gcc 4.7.2). I've pushed that old server to its limits, but it's finally time to upgrade.

I've always been very good to record any changes and maintenance at https://wiki.osgeo.org/wiki/MapServer_at_AdhocVM

I request on behalf of the MapServer project for a new dedicated VM to host all of the heavily used demo.mapserver.org services. Some more notes:

  • this would be similar I guess to pycsw's recent VM (#2452)
  • it will honestly take me much effort to move all of the services to this new VM, so I propose that we keep both the adhoc and the new VM up during, and I will record my progress on a new wiki page (linked of course from https://wiki.osgeo.org/wiki/SAC_Service_Status )
  • if you ask my preference, I am much friendly to Ubuntu 18.04 , but can work in any environment that you provide me.
  • proposed VM name could be 'mapserver'
  • my LDAP account is: jmckenna
  • my SSH key is installed on osgeo7

Let me know what you think. And thank you for this.

(marking ticket as 'major' as with the recent MS 7.6.0 release done now, I'd like to get moving on upgrading all services, and especially since our 'msautotest' scripts will be testing now against old Adhoc MS 7.4.4 for now)

mercy buckets! :)

-jeff

Change History (28)

comment:1 Changed 5 months ago by Jeff McKenna

Description: modified (diff)

comment:2 Changed 5 months ago by Jeff McKenna

Description: modified (diff)

comment:3 Changed 5 months ago by Jeff McKenna

Description: modified (diff)

comment:4 Changed 5 months ago by Jeff McKenna

Note this is related to ticket #2384 (upgrade old-adhoc from Wheezy). Please let me know what is the best way to move forward on this.

comment:5 Changed 5 months ago by robe

Sounds like a good plan. Jeff -- I'll set up a debian 10 server, give you admin rights to it and you can install what you need on it and let me know if you need me to do anything. I unfortunately don't have a Ubuntu 18.04 with ldap authentication image in place, but I think debian / ubuntu are much the same as far as installing stuff.

comment:6 Changed 5 months ago by Jeff McKenna

thanks, yes I figured that also, after my many edits here ha.

comment:7 Changed 5 months ago by robe

Milestone: UnplannedSysadmin Contract 2020-I

Jeff I set up a dedicated called "mapserver" on osgeo7 and gave you sudo rights. You can add others as you see fit.

You can log in similar to how you log into old-adhoc, but mapserver as detailed here https://wiki.osgeo.org/wiki/SAC_Service_Status#Accessing_osgeo7_containers_via_ssh

The container I set up similar to pycw has the following specs, ability to log in with OSGeo LDAP


Debian 10 Docker 19.03.8 4 GB ram provisioned 200 GB disk (this includes backup space as well so you will see less) 4 CPU Prometheus Node Exporter (still need to register this so visible on monitor.osgeo.org)


We can see later after metrics have collected and you start using it if these limits are sufficient. Let me know if you need help installing anything. I didn't install apache or anything as I wasn't sure your preferences. Also didn't install PostgreSQL but can help with that if you need help installing those.

comment:8 Changed 5 months ago by robe

forgot to ask - should I setup something like staging.demo.mapserver.org so you can start testing before move? If so let me know what port on the server to connect to. Doesn't need to be port 80.

comment:9 Changed 5 months ago by Jeff McKenna

great idea for staging, yes please. how about port 8081 ?

comment:10 Changed 5 months ago by robe

Okay done - http://staging.demo.mapserver.org

should start showing a site instead of bad gateway when you are done

comment:11 Changed 5 months ago by Jeff McKenna

thanks Regina, I'm connected. (at first I was trying to ProxyJump from my home Windows machine through download onto this new instance, but then realized that is forbidden) It of course works fine as you said, connecting to download first and then connecting to the new instance. thanks again.

(I guess I was trying to just jump, so I didn't have to copy my private key to a cloud server)

Anyway out of my efforts I've now updated to the Windows OpenSSH 8.1-beta release ha, which in the end wasn't needed because jumping through download is forbidden ha. Oh well, I'm sure my new Windows OpenSSH expertise will come in handy later ha! :)

Short summary: I'm off and running on the new server. thanks again for this.

comment:12 Changed 5 months ago by robe

Hmm you shouldn't need to copy your private key. I'm running msys2 and using that so maybe it's different.

Not sure other modes of connecting via windows. That shouldn't be forbidden.

My conf file looks like this:

Host download.osgeo.org
    IdentityFile "/path/to/private.key"

Host osgeo7-*
	ProxyCommand ssh robe@download.osgeo.org -W $(sed -e "s/^osgeo7-//;s/$/.lxd/" <<< "%h"):%p
    IdentityFile "/path/to/private.key"

I do notice I need both entries though.

Then I do

ssh robe@osgeo7-mapserver

That did prompt me for my LDAP password since I didn't have that installed on mapserver container. But typing in my password works fine since I don't have password access blocked on the internal servers.

Version 0, edited 5 months ago by robe (next)

comment:13 Changed 5 months ago by Jeff McKenna

I totally agree that I shouldn't have to copy private key remotely.

However all throughout my testing I get this error (same setup as yours) :

  debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
  channel 0: open failed: administratively prohibited: open failed
  stdio forwarding failed
  kex_exchange_identification: Connection closed by remote host

StackExchange? says the error is because of a missing setting on the setup of the Container (I have no access to that part).

comment:14 Changed 5 months ago by Jeff McKenna

It's actually ok though, as I tried so many different methods, and kept hitting that wall (container setting). I'm ok now to just move foward with private key on server. (2 days battling this ha)

comment:15 Changed 5 months ago by Jeff McKenna

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    IdentitiesOnly  yes
    #ServerAliveInterval 240
    #AllowTcpForwarding yes
    #PermitOpen any    
    
Host osgeo7-mapserver
    HostName     osgeo7-mapserver.lxd
    #Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jump nc %h %p
Last edited 5 months ago by Jeff McKenna (previous) (diff)

comment:16 Changed 5 months ago by Jeff McKenna

ssh -v osgeo7-mapserver

(my local and remote keys are accepted, as below) :

OpenSSH_for_Windows_8.1p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\Jeff/.ssh/config
debug1: C:\\Users\\Jeff/.ssh/config line 12: Applying options for osgeo7-mapserver
debug1: Setting implicit ProxyCommand from ProxyJump: "C:\\WINDOWS\\System32\\OpenSSH\\ssh.exe"  -v -W "[%h]:%p" jump
debug1: Executing proxy command: exec "C:\\WINDOWS\\System32\\OpenSSH\\ssh.exe"  -v -W "[osgeo7-mapserver.lxd]:22" jump
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt type -1
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
OpenSSH_for_Windows_8.1p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\Jeff/.ssh/config
debug1: C:\\Users\\Jeff/.ssh/config line 1: Applying options for jump
debug1: Connecting to jump [140.211.15.30] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt type -1
debug1: identity file C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to download.osgeo.org:22 as 'jmckenna'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:9Rj8e6GTNUeah218p0NaUqh143OD/90r2+MPpv90yeQ
debug1: Host 'download.osgeo.org' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\Jeff/.ssh/known_hosts:7
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\Jeff\\.ssh\\id_rsa.openssl-decrypt
debug1: Authentication succeeded (publickey).
Authenticated to download.osgeo.org ([140.211.15.30]:22).
debug1: channel_connect_stdio_fwd osgeo7-mapserver.lxd:22
debug1: channel 0: new [stdio-forward]
debug1: getpeername failed: Bad file descriptor
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
Last edited 5 months ago by Jeff McKenna (previous) (diff)

comment:17 Changed 5 months ago by Jeff McKenna

By the way, from reading the many forums, for Windows users this is the recommended SSH tool now (I've learned a lot ha): OpenSSH for Windows: https://github.com/PowerShell/Win32-OpenSSH/releases The latest beta allows direct ProxyJump.

comment:18 Changed 5 months ago by Jeff McKenna

PS. I think next version of MS4W will include these SSH tools, so very nice! (OpenSSH for Windows)

comment:19 Changed 5 months ago by Jeff McKenna

My guess is that the remote server has AllowTcpForwarding or PermitOpen disabled, but I haven't been able to prove that.

Yikes :)

Last edited 5 months ago by Jeff McKenna (previous) (diff)

comment:20 Changed 5 months ago by Jeff McKenna

SOLVED!!!! Oh my lord, it was the hostname (notice that I specified the IP address instead of container name). YIKES! *solved* Phew! deserves a long-weekend beer on my patio now ha.

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    IdentitiesOnly  yes
    #ServerAliveInterval 240
    #AllowTcpForwarding yes
    #PermitOpen any  

Host osgeo7-mapserver
    #HostName     osgeo7-mapserver.lxd
    HostName      140.211.15.30
    #Port         22
    User         jmckenna
    #IdentityFile C:\Users\Jeff\.ssh\id_rsa
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jump nc %h %p

comment:21 Changed 5 months ago by Jeff McKenna

(funny how I could look at that verbose debug output for 2 days, but pasting it here into Trac let me finally notice the one important line:

debug1: getpeername failed: Bad file descriptor

...so specificing the IP address fixed it. phew. humbled again :)

Thanks for listening to me rant ha!

comment:22 Changed 5 months ago by Jeff McKenna

That actually didn't solve it. Anyway as you said there are other ways to connect. Sorry for all this noise. I'm happy to connect period ha. Enjoy your sunday:)

comment:23 Changed 5 months ago by robe

Is your HostName? like below. Wasn't clear if that is what you changed it to or not. That is what it should be.

HostName     osgeo7-mapserver.lxd

seems like you had that earlier on. Also keep in mind that each host has a separate jump. I created one for download.osgeo.org to be consistent with the osgeo3 and osgeo4.

So osgeo3 = hop.osgeo3.osgeo.org

osgeo4 = hop.osgeo4.osgeo.org

osgeo7 = download.osgeo.org with alias hop.osgeo7.osgeo.org

Can you do below or is ProxyCommand? not supported on your version of OpenSSH

Host hop.osgeo7.osgeo.org
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

Host osgeo7-mapserver
    ProxyCommand ssh jckenna@hop.osgeo7.osgeo.org -W %h:%p
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

and then

ssh jmkenna@osgeo7-mapserver

Note the .lxd with or without is optional since they are on the same network anyway so the .lxd is assumed.

Last edited 5 months ago by robe (previous) (diff)

comment:24 Changed 5 months ago by Jeff McKenna

Yes sorry, I was testing with ProxyCommand 2 days ago, the working syntax is above but commented out, for 'OpenSSH for Windows' (the new Windows native SSH client).

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump

The error is the same:

channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host

comment:25 Changed 5 months ago by Jeff McKenna

I think I will keep going in circles ha. I am happy that I am able to connect now period (unfortunately not through the new Windows native SSH client). That's definitely ok. Thanks for helping.

comment:26 Changed 5 months ago by robe

Hah I do have openssh windows installed. Must have gotten dragged in by VS Studio or VS Studio Code.

Seems ProxyCommand? the way we have documented doesn't seem to work with all the sed stuff, bummer. I'll stick with mys2 thank you :)

It just occurred to me you should be using mapserver for the host, The osgeo7-mapserver only works if you are using the hack that strips off osgeo7- from it whcih stupid windows ssh seems incapable of using.

The real hostname is just mapserver.

This worked for me with windows open ssh, off course using my id_rsa and name not yours :)

Give the below a try

Host jump
    HostName     hop.osgeo7.osgeo.org
    Port         22
    User         jmkenna
    IdentityFile "C:\Users\JMckenna\.ssh\id_rsa.openssl-decrypt"

Host mapserver
    HostName  mapserver
    ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe  -W %h:%p jump
    IdentityFile "C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt"

And then you should be able to do

ssh jmckenna@mapserver
Last edited 5 months ago by robe (previous) (diff)

comment:27 Changed 5 months ago by Jeff McKenna

Yes all Windows users now have the native SSH included. Nice trick eh?

Not surprised it was the hostname, I was trying so many options for it, the IP etc ha (it was the error 'getpeername failed" that was telling me that). Here is my working settings, hopefully this helps someone else someday down the road...

Host jump
    HostName     download.osgeo.org
    Port         22
    User         jmckenna
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    
Host osgeo7-mapserver
    HostName     mapserver
    User         jmckenna
    IdentityFile C:\Users\Jeff\.ssh\id_rsa.openssl-decrypt
    #ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p jump
    ProxyJump jump

Thanks!

PS. I don't use unix emulators for anything, I find I learn much more this way, the hard way ha.

Wishing you a nice holiday Monday from Canada! Thanks for your help Regina.

comment:28 Changed 4 months ago by robe

I'm closing out since the server has already been created and Jeff can log into it and I also have the nginx set up for.

Jeff - put in another ticket if you need anything else.

Note: See TracTickets for help on using tickets.