Opened 5 years ago

Closed 3 years ago

#1693 closed task (fixed)

Slow DNS lookups on tracsvn

Reported by: strk Owned by: sac@…
Priority: normal Milestone:
Component: DNS Keywords: tracsvn, dns
Cc: martin

Description

I've just updated the Gogs implementation to cache DNS lookups because otherwise it would take up to 14 seconds to render the user-explore page, due to having to lookup the domain of each user's email in the page (~20 per page).

Now, with the cached lookups the time to render the page goes down from 14 second to 5 _milli_ seconds.

This tells me there's an issue with the DNS resolution. I don't know which other services do use DNS lookups, but if any production service does it too (beside Gogs which is only experimental) we'd better install a local caching server, like dnsmasq (or, as I've been suggested, "unbound").

Change History (19)

comment:1 Changed 3 years ago by TemptorSent

Component: Systems AdminDNS

comment:2 Changed 3 years ago by TemptorSent

A local caching DNS server with DNSSEC capability is mandatory for proper operation. As mentioned in the ticket, either dnsmasq -- http://www.thekelleys.org.uk/dnsmasq/doc.html or Unbound would be a good option -- https://www.unbound.net/index.html and both should be available to install using the package manager.

comment:3 Changed 3 years ago by TemptorSent

To explain further -- A recursive caching name server is used to look up and locally cache DNS information from the appropriate authoritative name server, which it discovers more or less in the following manner:

  • starting at the root name server, it sends a query for the target record; the root nameserver replies based on the TLD (top level domain - osgeo.org, for example) with the address of the name servers for that domain
  • it then queries each successive name server recursively until it find an authoritative record for the target
  • each record has an associated TTL or "Time To Live", which determines how long it is allowed to be cached for

DNSSEC uses cryptographic techniques to ensure that only records actually originating from an authorized authoritative nameserver are accepted -- without this, a technique called DNS cache poisoning can be used to insert records linking valid names to malicious addresses. Ensuring that only DNSSEC authenticated records are cached when available will prevent a large class of DNS related exploits.

comment:4 Changed 3 years ago by wildintellect

I'm in favor of trying unbound, I have seen it in use around my work. Easy to remove if it doesn't work. Question is, can we get it from repos on such an old box?

comment:5 Changed 3 years ago by strk

It would be also helpful to understand which services are doing dns lookups, in case those lookups can be disabled. For example I know for sure that Gitea is doing DNS lookups for finding user avatars, and I also know Gitea must be compiled in a certain way for it to be using the system resolver

comment:6 Changed 3 years ago by TemptorSent

Pretty much every service that logs incoming connections will be hitting DNS (unless you explicitly turn it of, leaving just IPs), as does the mail server (mandatory), and any thing else that wants to know the name, ip address, mail server, dkim keys, or anything else that DNS provides. Disabling nslookups in very specific cases where many cache-misses occur on a regular basis, but otherwise it just makes sense to use a local fast cache. We don't necessarily need to use the system resolver from gitea, just point it at a the ip of the local caching server to use for lookups.

comment:7 Changed 3 years ago by strk

Now available as packages in current TracSVN machine:

unbound 1.4.22-1~bpo70+1 (Installed-Size: 1585)
dnsmasq 2.62-3+deb7u4 (Installed-Size: 120)

I'd go the smaller route unless Unbound does anything that dnsmasq is not doing..

comment:8 Changed 3 years ago by strk

Here are instructions to setup dnsmasq on Debian: https://wiki.debian.org/HowTo/dnsmasq

comment:9 Changed 3 years ago by martin

According to my experience, setting up dnsmasq as DNS cache on nowadays Debian systems requires nothing else but installing the package.

Please let me know if it doesn't.

comment:10 Changed 3 years ago by strk

I've installed dnsmasq but /etc/resolv.conf is not automatically updated to point to localhost, which makes me think it is NOT enough to install the package.

I've manually added 127.0.0.1 as first entry i /etc/resolv.conf

Not sure how to test/compare performances now though (nor how to convince Gitea to use 127.0.0.1 to lookup names)

comment:11 Changed 3 years ago by martin

Ah ok, thanks for responding, maybe I'm using more recent versions of the respective package.

The installer of the dnsmasq package I know puts the local host as the one and only nameserver in /etc/resolv.conf and writes the forwarder into a file somewhere in /var/run/dnsmasq/ or the like. If the dnsmasq process proves to be robust, let's try that.

comment:12 Changed 3 years ago by strk

I did not follow the debian guide completely, so for example I don't know if dnsmasq is serving every host rather than just localhost, can you check that part Martin ?

comment:13 Changed 3 years ago by martin

Is everybody using the system resolver or do some local services rely on custom DNS configuration ?

comment:14 Changed 3 years ago by strk

I'm not sure what Gitea (focus of this ticket) is using, don't know how to tell. No idea about any other service.

comment:15 Changed 3 years ago by martin

I'd recommend to make the DNS cache listen on localhost only and configure resolv.conf to connect localhost first.
If we're unsure, which services rely on connecting the cache, we might simply enable the configuration that way and see if it breaks anything. Ok ?

comment:16 Changed 3 years ago by strk

Fine by me

comment:17 Changed 3 years ago by strk

Just make sure to create a dnsmasq configuration file (and please put under revision control) because the trick of having non-localhost addresses in /etc/resolv.conf is supported by dnsmasq as a poor man's configuration of it (dnsmasq will use the non-localhost addresses as upstreams)

comment:18 Changed 3 years ago by martin

Dnsmasq is now set to listen on the loopback interface only - let's have a look who might be complaining.

comment:19 Changed 3 years ago by strk

Resolution: fixed
Status: newclosed

Assuming fixed

Note: See TracTickets for help on using tickets.