new VM for demo.pygeoapi.io

[pvgenuchten]

​​http://demo.pycsw.org is running off a rented server at hezner, provided by psc members.

We (psc of pygeoapi) would like a new VM so we can setup an autodeploy via git, which will lessen the presure budget of psc members

The vm will have latest and stable demo of pygeoapi, documentation and website (mkdocs).

[robe]

What operating system do you want on it?

I currently have images for debian 10, debian 11, and Ubuntu 20.04 configured with OSGeo LDAP.

Also let me know if you want to run docker in it. I have to put in some provisions to allow that. You can decide later on that if you want.

[just]

Yes, need Docker, but we need to be able to provision the entire VM with GitOps/Ansible. That requires (key-based) root-access. The site btw is ​https://demo.pygeoapi.io. The current Ansible code for provisioning is in: ​https://github.com/geopython/demo.pygeoapi.io/tree/master/ansible. Basically the only install needed is Python 3. The entire VM will be bootstrapped/installed/maintained with Ansible.

[robe]

just

Can you try logging in: Setting up your ssh as detailed in -

​https://wiki.osgeo.org/wiki/SAC_Service_Status#sshing_into_osgeo3_containers

ssh just@osgeo3-demo-pygeoapi

I gave your just account sudo rights so you should be able to add any additional accounts you need and also install docker with your ansible script.

So if that works I guess we can setup a staging domain for testing.

Does something like demo.pygeoapi.osgeo.org work okay for you to test out the setup before cut over? I just need to proxy your 443/80 via our nginx proxy for that.

[just]

Thanks @robe ! Yes, I can login on the VM, also directly using the SSH Proxy config and with key. Sudo works. btw the domain name of the demo server is demo.pygeoapi.io (not .org). I've changed in the Wiki.

Now the challenge is to get Ansible working both from my/a local system and later from within the GitHub CI/CD (GitHub Workflows). Usually Ansible with (key) root access is used as it will provision system daemons etc. This should be fully automated without any password-prompts etc. We will setup a dedicated admin user. Think there are two challenges:

• how a CI/CD process can access the VM directly over the hop (needs SSH Proxy Config)
• how Ansible (within the CI/CD processes) can make direct sudo-changes without password prompts

And yes, could be first tested with a staging server (domain).

Within the project we need to decide if we first move to full GitOps (GH Workflows) or migrate with the existing Ansible setup and commit webhooks. Also the Docker build is currently migrated from DockerHub to GitHub.

[robe]

Okay a couple of thoughts on this:

1) how a CI/CD process can access the VM directly over the hop (needs SSH Proxy Config)

If it's an issue I could open up a different port on the server to allow direct ssh to the container. Can github actions use an alternative port(not 22)?

e.g would be able to do something like below (where 41022 is the port I'd open up)

ssh -p 41022 ansible@demo.pygeoapi.io

I don't see any huge security hole with this so am fine with that change as long as you turn off password authentication on your container with

in /etc/ssh/sshd_config 

change to

PasswordAuthentication no
 

But you need to make sure before you do that - to install your public keys in your container under your account otherwise you won't be able to log in.

Also note your proxy account doesn't need to be the same as the account you log into the container with. But whatever account you use for ansible the public key has to be also registered in your ldap profile (or we could setup a different profile - e.g. we have one for geoserver I think with some email alias like pygeoapi@… that points to your email).

You can also have more than one ldap public key on your profile.

2) How Ansible (within the CI/CD processes) can make direct sudo-changes without password prompts what comes to mind

If you create an ansible account with sudo rights

Then assuming you have the proxy worked out, ansible would log in as something like:

ssh ansible@osgeo3-demo-pygeoapi

To allow ansible ability to sudo without password prompt, just change the visudo accordingly

visudo

Then add a line at end:

ansible ALL=(ALL) NOPASSWD: ALL

again I don't think there is a huge risk to this since if ansible gets compromised, would only affect your VM not the others.

[robe]

@just I setup a staging domain - ​https://demo.pygeoapi.osgeo.org that points to your VM port 80.

Right now it just shows 502 Bad Gateway. Once you have setup everything it should show your demo site.

Once you feel confident in your setup, you can then CNAME your demo.pygeoapi.osgeo.org to web.osgeo3.osgeo.org (and I can then get a letsencrypt cert for that and turn off the osgeo.org one)

[just]

@robe thanks again! Some answers:

1) yes Ansible can use alternative ports, configured per-target-host

2) this is in fact what our Ansible playbook ​https://github.com/geopython/demo.pygeoapi.io/blob/master/ansible/bootstrap.yml does: creating an admin user with those sudo-rights. That is why root user access is needed. But we can overcome by making those changes manually, i.e. create the Ansible user as you indicated. Ansible actions are idempotent, so no harm if user already exists.

domains) our stack has Traefik at the frontend. Traefik automatically creates/updates letsencrypt certificates. We also apply SSL-settings to obtain SSL A+ status. Are you proposing to use the domain demo.pygeoapi.osgeo.org for the production demo? We rather keep the existing domain: demo.pygeoapi.io (.io not .org). This has been communicated all over in the past 2.5 years (slides, FOSS4Gs etc).

[robe]

No the site would be demo.pygeoapi.io.

I was proposing we use demo.pygeoapi.osgeo.org for testing to work out any kinks and then once you feel comfortable with the setup, it will become demo.pygeoapi.io.

But you bring up another point. Right now we have all the web sites proxy thru an nginx server which does the work of getting the letsencypt cert for all the sites regardless which server they are on. Would that work for you or you really need the Traefik to manage the letsencrypt? I think that would require that we assign a dedicated IP to demo.pygeoapi.io or somehow pass the cert request to the demo.pygeoapi.io server.

[robe]

Replying to just:

@robe thanks again! Some answers:

1) yes Ansible can use alternative ports, configured per-target-host

2) this is in fact what our Ansible playbook ​https://github.com/geopython/demo.pygeoapi.io/blob/master/ansible/bootstrap.yml does: creating an admin user with those sudo-rights. That is why root user access is needed. But we can overcome by making those changes manually, i.e. create the Ansible user as you indicated. Ansible actions are idempotent, so no harm if user already exists.

BTW if you did want ansible to do everything, no reason you can't log in as

ssh root@osgeo3-demo-pygeoapi

as long as you have the key registered under your osgeo ldap account or we do 1) of having an alternative-port you can do it. Would still mean though you have to do some manual work of adding the public key to the root account.

[robe]

@just yah sorry for the confusion earlier, I had a typo in domain earlier with CNAME, so the plan is:

1. Test with demo.pygeoapi.osgeo.org -- which is already setup to proxy thru to the demo-pygeoapi vm.

2. If all looks good we remove the demo.pygeoapi.osgeo.org and you setup in your dns CNAME demo.pygeoapi.io -> web.osgeo3.osgeo.org

If you need to manage letsencrypt entirely via ansible, then let me know. I think the easiest in that case would be to allocate a dedicated ip for the osgeo3-demo-pygeoapi so I would need to get extra SAC approval for that. I think we have a couple spare ips but something I'd like to avoid. In that case I'd just point 80/443 ports directly to the vm dedicated ip and you would do an A instead of CName in your DNS.

3. Also let me know if it's easier for you to go thru a non-standard port than having to do the proxy jump. That is easy enough to do. As mentioned I'm not too concerned as long as you block off password access (and only allow key access) in your vm. That means you need to put in your ssh public key since the VMs are not set up to pull ldap stored public keys.

[just]

Ok, and thanks again @robe! We are currently debating among the PSC (on Gitter) the order of actions w.r.t. pygeoapi Docker and CI/CD for the demo site. This will be surely not decided before Christmas. I let you know here, Happy Christmas!

[strk]

Ticket retargeted after milestone closed

[robe]

@just checking on how things are going. If you still plan to use this server or have decided otherwise.

[just]

No decision yet. Will need to talk within the pygeoapi PSC. Hold on!

[robe]

pushing to next milestone since my contract funds have been used.

[robe]

@just is this something you are still interested in. At this point, I think I should just rebuild it from scratch using a new VM template since 16 months have already passed.

If I don't hear anything in a couple of days, I'll shut down the server.

[robe]

Moving my prior still open items to the next proposed Milestone

[robe]

I have not heard back on this so closing as a won't fix. Feel free to reopen if you have renewed interest