Many of you may not realize it but being a member of OARC , and subscribing to some of our mailing lists actually gives you a tremendous amount of real time access, not just to data sets or services but to the community. And while having insight to what our peers think about particular issues is nice, its also useful when trying to impact immediate change, like for example the Firefox ANY Queries event.
The thing about DNS ANY Queries is that they’ve been a bit of a pain for a while now, as a vector for amplification attacks. Dyn’s blog explains more about how ANY queries work in amplification attacks and how to mitigate those attacks. But ANY queries do have two non-malicous functions:
- debugging , diagnostics, and/or troubleshooting/
- the “qmail” mail routing software, (and yes we know many of you may argue that this uses ANY queries in a way which is obsolescent and should be disabled)
So what happened? Long story short, Mozilla released Firefox 36.0 on February 26th. By the 27th there was a 5% spike in ANY Queries, the issue was raised on the dns operations mailing list, and a bug was filed with Mozilla. And on March 5th, after community members and OARC personnel had reached out to their own trusted contacts within Mozilla, the change was reverted.
From the discussion that ensued on the OARC dns-operations list, we can conclude that Mozilla introduced a new feature to improve Firefox DNS resolver query pre-fetching performance. But it seems like maybe Mozilla made some invalid assumptions. That being said when you think about it, it’s kind of astounding that Mozilla not only acknowledge that there was a problem but also resolved it, in about a week.
One of the tools we use at OARC for analyzing events like this is DSC. For those of you not in the know, DSC stands for Domain Statistics Collector, it looks at all in coming and out going traffic on individual Root Servers, aggregates it, and then summarizes that traffic for us. Admittedly the DSC is a bit like a vintage family heirloom for OARC, it’s precious to us, but could use a good dusting off, so let us know if you’ve got an idea of how you’d like to see it evolve.
Anyway if you look at the graph above, you can see how DSC used data provided by RIPE NCC (K- Root) and Cogent (C-Root)to show the impact of the Firefox release query load at the Root over time. And even more techno-babble, can be seen in Cloudflare’s perspective .
This event, understandably threw everyone into a bit of a tailspin for a moment, but from OARCs point of view it was really rewarding to see critical information being shared via the mailing list we run, and to see DSC provide us with some metrics for what was happening. It also reminded us that these tools and services are only as useful as the community is engaged. So please, if there’s something on your mind, find a way to reach out to us personally. If you’re hesitant to participate in mailing lists, don’t feel comfortable sharing data, or even if you’ve got a great idea for how we can improve one of our tools, we want to know, and more importantly we want to help!