Here are some DNS-OARC development highlights from the past months. Previous blog posts are available on our Medium blog.
New DSC Grafana example dashboards!
Finished creating the new example dashboards for DSC data in Grafana! Have a look at the links below which shows DSC data coming from Check My DNS.
The “Statistics” dashboard contains some simple stats on packets, queries, responses and response codes (RCODE), and their relation as a difference graph such as “queries vs responses”.
Then there’s a “Query Failure Rate” graph which shows in percent how many responses seen with an error related to how many queries seen. Note, there is no relationship between query and response here, but it can still show some interesting stuff! What I’ve seen is that sometimes it goes way beyond 100%, and it looks like someone is sending bursts of faked/spoofed failed responses. Possibly some cache poisoning attempts or something else.
Last on the dashboard we have two graphs showing queries with DO and RD bit set and their ratio of all queries seen.
The second dashboard “Locations and Queries” uses the new Geomap plugin to display where queries are coming from based on country code lookups using MaxMind databases along with other stats based on the query.
Now the “RCODE per IP/Net (top20)” graph is something I’m a bit proud of! This was one of the graphs I just couldn’t recreate a few years ago when I started using Grafana for DSC data. It’s because you need to turn the data on its side, and this is now possible thanks to Grafana’s new transformations. Similar things can be done with InfluxDB’s new Flux query language, but I haven’t had enough time to get into that yet.
I hope these example graphs are helpful for people! And if you want to share your own graphs/dashboards I’ve created a repository for that.
OARC 40 and DSC+Grafana session
The day before OARC 40 in Atlanta I held a knowledge exchange session around DSC, Grafana and the new example dashboards. We had about 10 people joining that and while I had hoped to do half demo half discussions it ended up with me talking for the whole 2 hours because there is just so much to go over before doing even the simplest graph.
With that experience fresh in mind, I’m happy to announce a refresh of the “DSC+Grafana crash course”(!) that I did little over a year ago. It will be open for anyone to join, but discounted for OARC members (so join!), and can be done as a small group of individuals or separate just for you and your team. If your interested in this then please contact me.
pcap_dispatch() issue
An important fix was included in the release of dsc v2.13.2 and the helper library pcap-thread v4.0.1. Something had apparently changed in libpcap, or underlying packet capturing, which could make pcap_dispatch() capture packets indefinitely under high load / attacks.
The man-page says that pcap_dispatch() will “only one bufferful of packets is read at a time”. However, in December I received a report from Klaus Darilion (NIC.AT) that the dsc process ran amok, consumed all memory and produced an XML containing several hours of data even though the interval was set to 1 minute.
Thanks to the help from Klaus I was able to track down the issue to pcap_dispatch() and deploy a fix that checks the timestamp in the PCAP header object against the timeout for capturing. If the callback finds itself running over it will now pcap_breakloop() from processing packets and go ahead to write out the data.
dnscap v2.1.1 — anonymizing fixes
This new release includes a couple of fixes and I would like to highlight one of them as it might be important to you if you use dnscap for anonymizing PCAPs.
Duane Wessels (Verisign) reported that all the anonymizing plugins were not anonymizing the client addresses if the client was sending using the server’s port!
This has been fixed so now both the sending and receiving IP addresses are anonymized if both are using the server port (command line option, default 53).
Please see full release notes for additional fixes.
dnsperf v2.11.0 — latency histogram
Petr Špaček (ISC) has been busy adding new features in DNSPerf to track and display latency which he also showed results on in his presentation “Detecting latency spikes in DNS server implementation(s)” during OARC 40.
This uses a 64-bit histogram data structure created by Tony Finch (ISC) and can be enabled with -O latency-histogram.
The release includes more new features and a few fixes, for full details see release notes.
DNS Hackathon 2023
I will be running a hackathon together with Johanna Eriksson (Netnod) and Vesna Manojlovic (RIPE NCC) during the weekend before RIPE 86 in Rotterdam! We also have a great team of volunteers on our Program Committee to help guide and organize participants and projects. For full information about the hackathon please see https://www.netnod.se/join-the-dns-hackathon-2023.
Space is limited so please sign up as soon as possible if you wish to participate!
Cheers,
Jerry
