Opened 4 weeks ago

Closed 2 weeks ago

#3286 closed task (fixed)

gitea git commit hooks failing to trac upgrade of trac.db.api.DatabaseManager

Reported by: robe Owned by: strk
Priority: normal Milestone: Sysadmin Contract 2024-I
Component: SysAdmin/Trac Keywords:
Cc:

Description

We've been having this issue on trac since recent upgrades that gitea commit hooks to trac are failing with

Error: Unable to check for upgrade of trac.db.api.DatabaseManager: TimeoutError: Unable to get database connection within 0 seconds.

making the calls directly via trac-admin works fine.

Change History (9)

comment:1 by lnicola, 4 weeks ago

What if you run trac-admin as git?

comment:2 by strk, 4 weeks ago

I've just tried and succeeded (sudo -u git trac-admin /var/www/trac/env/postgis/ changeset added git bdecd88fe3f506b2b1f64de3f6cf25dc8f102f66 resulting in https://trac.osgeo.org/postgis/ticket/5782#comment:41

comment:3 by robe, 3 weeks ago

Yah bizarred direct works. I'm really tempted to rebuild tracsvn from scratch -- maybe we can do that next year and just call it tracgit.

tracsvn has been around for 20 years and gone thru continuous upgrades - e.g. my first real experience with it was when it was a ganetti vm and I migrated it to lxd and then upgraded it from stretch. It probably was upgraded even early before me.

comment:4 by strk, 3 weeks ago

For the record it took me 3 reloads to see this ticket, first attempts all gave me the "cannot get db connection within 0 seconds" error.

I wanted to say that manually running trac-admin usually works for me, I've continued using this workaround and it seemed to always work, while the git log seems to always fail. It would probably be wise to file a ticket upstream to trac itself to see if they have any hint about it.

comment:6 by robe, 2 weeks ago

Upstream walked me thru adding some logging in the python code around debugging db and also to turn on db logging.

The logs now show this as the culprit

psycopg2.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: fe_sendauth: no password supplied

Why it doesn't trigger that when tracadmin is called directly using the same accounts gitea is using is a bit puzzling.

comment:7 by robe, 2 weeks ago

I confirmed that is the issue cause when I set to trust for local connection in /etc/postgresql/16/main/pg_hba.conf it worked fine. I checked and the git account does have a .pgpass entry for trac account for all databases and it can connect to all databases. But I guess for some reason maybe psyqcopg2 can't access the file or it's running under another account when it does tracadmin call.

comment:8 by robe, 2 weeks ago

For now I just put in the trac password in the /var/www/trac/env/postgis/trac.ini file and that seems to work okay. I also turned postgres logging back to what it was since the postgresql.log had ballooned to over 90GB.

comment:9 by robe, 2 weeks ago

Milestone: Sysadmin Contract 2024-I
Resolution: fixed
Status: newclosed

Okay issue resolved. Troubleshooted with lnicola and strk by adding env logging to the postgis git hook to see what environment variables it was using.

added to the top of /home/git/gitea-repositories/postgis/postgis.git/hooks/post-receive.d/osgeo

env > /home/git/env_from_git

Turns out that gitea overrides the HOME variable to:

HOME=/home/git/gitea/data/home

So it was expecting the .pgpass file to be there instead of /home/git/.pgpass. Copying the .pgpass there fixed the issue.

Note: See TracTickets for help on using tickets.