RIPE 92 side room.
11am: DNS
MORITZ MULLER: Hello, welcome to the DNS working group session here at RIPE 92 and welcome to the people online, great to see you all here, we are the co‑chairs of the DNS working group and today we have again a very full agenda, even with one time permitting presentation by Maynard which will be a short announcement and we have to see whether we can fit it, we have a couple of announcements.
There's, of course, the RIPE PC election voting will close today at 5pm, remember to vote all. Talking about voting, remember also to rate the talks. We appreciate your feedback and also the speakers appreciate the feedback as well.
And then we have one very important announcement, we have found a new co‑chair for the RIPE working group, Mark Feneral from from the French registry.
(APPLAUSE.)
Other than that, we are also have ‑‑ because he will replace me by the way as a co‑chair so this is my last working group session as a co‑chair.
Other than that, we also have a lively discussion about the co‑chair selection process in the DNS working group on the mailing list, if you have opinions, feel free to share them and my fellow co‑chairs will have to figure out what to do with them. That's not my problem any more.
And yeah, I think this is it, right. So I think we can start with our
ULRICH WISSER: Almost, almost, we have to to say thank you to Moritz.
(APPLAUSE.)
And thank you for doing a wonderful job and putting your time into the RIPE community.
MORITZ MULLER: Thank you. With that, Ondrej.
ONDREJ SURY: OK, hi from Ondrej from IFC, this will be a little bit interactive to if you can scan your GR code with your phone, it will, or go to wooclap.com and type in DNSRIPE92. I will give you a brief moment.
OK. So this talk is about delegations and mostly about how many DNS queries does it take to resolve a name. So here's the first question. So teamsMicrosoft.com, how many DNS queries does it cold‑cache lookup take?
SPEAKER: Which version of BIND!
ONDREJ SURY: I will show you. OK, seems like the middle of the field is winning. OK. Here's the answer. OK. So I will show you why. So that's, it was a surprising number for me too and yeah, DNS is magic and I am still surprised it works every day. So what I am going to show you here, all numbers come from BIND 9 so its implementation specific, we will add one more complexity at a time, there's delegation types, CNAME chains, I will show you the DNSSEC doesn't really cost much and the PTR and PTR recurs are important in this community.
So I am sure that you know this, so resolution in few seconds, every hop it's TTL, and cache hides most of it until it doesn't. You know, this picture like this is very naive for how many queries it takes to resolve something. You would think three is the number. So the real world case. It's this. Again example.com. And that's because it doesn't use ‑‑ it does use cloud name servers it adds complexity.
Quick recap of the delegation taxonomy, like we have like in domain, NSname list in the zone it itself and the glue is required, CRA cost, zero, if it's in bailiwick or sibling glue, it lives in the parent of the zone, it's the glue can be used so there's some extra cost or none, and if it's out of the bailiwick, the name service lives in a completely unrelate zone, glue is not allowed and extra cost is large.
And we will be talking about it mainly difference between the line one and three in this talk.
So the in domain glue, strict glue, which is mandatory, this is very cheap, so for example which can media.Org is delegated to zero and something else, blew must be in the parent, this is the best case and a couple of top 20 domain does this. And there is a reason for this.
In bailiwick‑sibling glue, this is RIPE.net and all the name servers are in the.er net zone in the parent and the parent can shift zone because it's authoritative for the sibling zones and the resolver can use these, but it's complicated.
Before there was a CVE in this area, after the CVE it's cost more because we can't use the, at least in BIND, we can't use the discipline glue, out of the bailiwick examples, examples, Org,.com, glue if ever present is untrusted and should be thrown away and resolver must resolve the NSnames from the scratch.
So just like recap, so you go example.com, you look up.Org or Org delegation, it goes to... and you go back to the root, look up .com, look up cloudflare.com and you get IP working name servers it's like three plus three multiplied by number out of the bailiwick parents per name.
So again it's for you on the phones.
OK. Kind of surprising. I didn't get just four lines. So I guess some people will have to do some homework but basically the top four lines is the correct answer.
So what is spread around the different domains cost you. If you have nothing in the cache, we are resolving CNAME at the moment and as you can see the DNSSEC doesn't add much and the difference between 920 and 921 is the child centric versus parent centric delegations, that's why it dropped down. As you can see, we have been improving in this area the implementation, but still just a single name, BIND 9.16 or BIND 9.18 is over a hundred queries alone. This is measured by running a name the instance and counting the local lines how many out going queries has been actually issued. So it can cost something, depending if you upgrade to the latest versions, from 16 to almost 200 cold‑cache queries just for a single name.
Teams.Microsoft is a CNAME and it's a CNAME chain. So there's like five, four CNAMEs and the last one is the A record, and each CNAME a fresh resolution and as you can see, it also switches the TLDs, so each target zone again might be out of the bailiwick and each NSset might send multiple TLDs and each of the domain can have a similar situation as well as I will show you in the, on some next slides. So if you count the whole resolution chain, you come to the number I was showing you at the beginning. So with BIND 9.16, it's 247 queries. Now it gets better. We have the parent‑centric delegations, it gets better with 9.20 versus 9.18 but still it's a lot. And that's just single name to resolve when the cache is cold or the records expire from the cache.
So more fun with PTR. Reverse DNS is everywhere, it's mail server, SSH, traceroute, whatever, and they regularly have out of the bailiwick delegations at every level and separate infrastructure, separate organisations, it might be broken, missing glue, and TTLs that nobody looks at.
IPv6 each can be delegated, delegation level, each can be different out of bounds delegation level.
So this is one of the trace for one of the IPv4 addresses and we will need more slides to show you this. So when you start, you start at arpa with Qminimisation where this is where it makes a difference because you are asking for arpa, not .n‑arpa so there are two steps. But still we are in this streaked sibling, streaked, that's fine, then we go to RIPE and this mostly fine because it still is .net but it's .arpa so you know, it's out of the bailiwick. But at least they are all in the node net but not exactly.
Then .net we go to root servers again this is bailiwick, strict glue, sibling glue, go to RIPE.net and now there's like TLD.Org so we went from.arpa to .org so if only.
So for PTR reverse records, the number of the varies is huge and that's for resolver that does every reverse lookup, you issue ping, you have a mouse service, you do reverse lookups. So these are some names, I have non‑mised the IPv4 address, we go almost to 400 and that was somebody complaining that BIND doesn't do its basic job and resolve names because with the cold‑cache it's 400 out going queries so it's not a basic resolution really.
So again, there's a great tool from Verisign labs, transitionaltrust.Verisignlabs.com, you can test it on your domains. Again this is very huge graph. I will show you just parts of it. So RIPE.net goes to.net, it's still fine, OK. Go TLD services .net goes to .com. Come with me, we have .net, .com to LACNIC.net goes to .org, OK. TLD .org goes to Brassil and Chile, now we have six TLDs and that's just for a single name and Chile has a bonus Canada, OK, and we are not yet done.
We also have Sweden. So we started with.arpa and half have now spanned the globe and visited many countries including Sweden.
Six TLDs. So. Question for you, how ‑‑ there is no right answer, obviously ‑‑ how would you describe your zones today?
I see Google people are very quick.
SPEAKER: Which one of them!
ONDREJ SURY: So many people with in domain delegations, I am pleasantly surprised.
OK. So back to the implementation, there are two different approaches and BIND comes from the left side, that it tried to resolve everything because the idea was to fill the cache and have the full cache and then you are basically done. So for every delegation level, store all the names, resolve both A and quote A, but it explodes under attack. The other side would be how naive for the v resolve bare minimum and for every delegation level you pick one name, one IP address per family, but it removes resiliency because the server might be slow or dead so everything gets slower and you know, we need DNS to be fast to be serving ads to people.
So neither of these are viable and something in between has to be implemented. So there are various limits, how many name servers are resolved at various per delegation level, limited number of IP addresses and use sibling glues as much as possible, that helps a lot.
So, what went sideways, I already mentioned the CVE, there was a CVE that showed that you can use sibling glue to poison the name. So we had two remove this from from BIND. And as you can see the BIND 9.20.13 was mostly parent centric, it used whatever was available.
And then the CVE came and we bumped from 7 to 37 or from 37 to 122, so these are like the top ten something names. And still we improved that a little bit with the latest version, like remove some of the queries but it's still quite a large number.
So what went well, when we switched to parent centric delegation model, most of the domains that alphabet or Meta has is just seven queries which is really nice. So if you don't know how to do that, look at these companies, pick a top ten list and you will see that in domain delegations are like fast and can be done.
So what you can do as a domain operator. Prefer in domain delegations if you can, if you must use managed DNS provider, pick one or two at most. And not for each across different four TLDs and across like and each of them has different TLDs. I know that people like to play stupid DNS tricks with load balances and stuff, but it adds complexity to DNS delegations, you might not realise it's there. The number at the beginning was really surprising for everyone in this room.
And keep the PTR delegations boring, you remove work from the DNS. For managed DNS providers, again just do in domain delegations for the client name servers, it will help all of your clients. And for resolver operators, also little you can do except deploy the latest version of the resolvers becausement implementations improve and get better. Aggressive NSEC might help a little bit but only a little bit for sign zones and the sign ... so, to take away from this, this is something I hope you take home. The single look up is not just a few queries, it's tenses or even hundreds of queries when the cache is cold. Or expired.
Ought of the bailiwick name servers and CNAME chains like multiplicate and PTR is the worst case, nobody talks about it because it's just like hidden and there's the caching only hides the cost.
So do this and you can get the free again with DNSSEC it would be six because you need, you know, DSand DNS key for every level.
So questions. And meanwhile you can look up your domain and just post the number.
MORITZ MULLER: Thank you Ondrej, we have time for two quick questions? .
SPEAKER: Joe ... from CloudFlare, this is all very fun, I like, thank you for doing all this, but this is just on an empty cache and most caches are not empty most of the time, after a cache has been started, 30 seconds later it's very not empty and probably all the queries needed for teams definitely in the cache at that point, and all subsequent queries are quick, so I wonder whether your advice is the right general advice, sounds like good DNS specific advice, good cold‑cache advice if but if the steady state is not an empty cache and other risks people are mitigating against like use multiple registries because everything depends on your set of credentials or one registry operator, maybe the overall picture for most people is different.
ONDREJ SURY: I expected this question. So if you have a domain that lives in specific TLD and that TLD goes down, it doesn't matter how many providers you have. So if, you know, if something goes down then you don't get to the delegations at all, and it doesn't matter that you are in .org or .info .com, and I don't know, .IT. So that's why my recommendation was just use, don't use many providers, just use one or two and use some that are well managed so that's my answer.
And it's not cold‑cache, it's stuff under attack as well. And the attacks happen. So you can force resolvers to clean the cache, you know, sub‑domain attacks to fill the cache with stuff and it gets complicated very quickly.
SPEAKER: ... CZ‑NIC, thank you for this research, and for the... memebers; however, I think that local route is becoming more of an industry standard already and I can it would be be changing the numbers sell out so could you maybe perhaps redo the same research with local root enabled because the point is that I think that with the local root it no longer like cares if those names are in different TLDs or if they are in the same TLD. And I think this was like one of main points of your presentation. But I don't really see in this case, yeah, of course there is the complexity of the further like labels in the names and stuff that will be very nice to see but for the TLD specifically, local routes obviously and that's my point.
ONDREJ SURY: I can do new numbers, I also have a like branch where the number comes from BIND as and EDE, extended DNS error, like some random number.
SPEAKER: Thank you.
SPEAKER: Thank you,
JIM REID: Jim Reid, unaffiliated guy just hanging ‑‑ just wondered in off the street.
ONDREJ SURY: From Glasgow, I know
JIM REID: Not from this city. This is interesting work and it's a good idea to publicise this, but I wonder what we can do to progress is it, is it worth trying to write it up as a document? Because I think there's an issue here about the Goldilocks number of the, not too many, not too few, where is that boundary? And it might not be a fixed answer to that but I think it would be useful to give guidance about what sort of things to consider how to give yourself enough robustness in terms of your delegation, at the same time not overwhelming the resolving process with having to make too many queries, I don't know the limit but it would be good to get guidance and write it down.
ONDREJ SURY: Sure I wrote a blog post on this 15 years ago but nobody knows about it, don't look it up because it's horrible broken English.
JIM REID: Who needs blogs, come on, it's something the working group could pick up and publish as a RIPE document.
ONDREJ SURY: I am happy to just put the presentation into more words.
JIM REID: OK, Ondrej, thanks.
MORITZ MULLER: Thank you Ondrej.
Next we have Elizabeth Boswell from the University of the Glasgow about Black Holes and prisoners and understanding AS112 deployment characteristics.
ELIZABETH BOSWELL: Hello everybody, this is my second talk on AS112, this week, I am sure many of you heard the talk at DNS Org already and this is about the other side, who runs the AS112 network.
So the issue that I'm talking about here are DNS junk queries. So not all queries that are sent to the DNS have a meaningful response, for example the query that would correspond to what domain name resolves to 192.168 doesn't have a meaningful because, because it's a private IP dress address, it can't correspond to one single domain name in the public DNS so this kind of query is essentially meaningful noise that's sent to the DNS and wastes resources, however these kinds of queries are often sent by misconfigured software and the AS112 network was created. It's anycast DNS network that captures these junk queries and diverts them away from the arpa servers, as we just hear, these queries will still take up some resources because of the delegation process but at least this way, those ones can be cached and hopefully prevent further queries from wasting everyone's time. So AS112 the authoritative name server for reverse DNS queries for private and linked local IPv4 addresses and home.ARRPA and service.arpa the interesting thing this is a volunteer run network, anyone can just add a site to AS112 by setting up a DNS server and announcing the AS112 to Anycast prefixes over BGP and a portion of the world's junk queries will be routed to that operator. The project is loosely coordinated by dns.org which maintains a list of operators with some information on the website AS112.net.
So as a result of this volunteer run model, we don't really know how many AS112 sets there are, who runs them, where they are located and how does it compare to any other Anycast networks. However knowing this is important because the queries that are sent to AS112 can contain some sensitive information, for example they can contain host names or they can be server queries and these queries can be received by anyone, we want to know where these queries are actually going.
Secondly, AS112 protects important parts of the DNS, it knees to be resilient to handle this load and finally it's an interesting uniquely coordinated DNS deployment, it's interesting to see how it compares to any other Anycast networks. All AS112 sites are required to correspond to these TXT queries for hosting ... with information about the location and operator of the site that received the query. So for example if I sent this query from my office I can see that my network AS 102 queries is going to the site run by RIPE in Amsterdam, from sending such queries from a large number of vantage points, we can discover a large number of AS 102 sites in operation, we sent these host name queries from 12,000 RIPE Atlas probes and 34,000 open recursive resolvers which were found through the census internet scan and we discovered 469 AS112 sites, this is significantly more than the 72 and 65 sites found in previous measurements performed ten years ago. 40 of the 97 operators are not on the official list of operators and this includes three of the top ten operators.
Based on the number of vantage points that query them and as you can see, CloudFlare is the main operator, that runs the, contacted the most but many other organisations which are present at RIPE right now also contribute to this project.
So CloudFlare is a major operator, it runs almost half of the sites we found and it was created by about 40% of the probes and resolvers and it was actually told by people from CloudFlare that weekend that they have more than 216 sites, so this is a lower bound but of course they can they can't tell how much AS112 queries the other sites receive since I wasn't able to discover them using essentially all discoverable open recursive resolvers on the internet. The fact that CloudFlare runs so many sites is a positive development because they obviously greatly increase the number of AS112 sites in operation. But it also creates a resilience issue because CloudFlare could at any point decide to stop contributing to AS112 because this is a volunteer run project and all the queries that go to CloudFlare would go to other sites and these sites may not be equipped to handle the load of these queries.
In terms of geographic coverage, it's pretty good in Europe, North America and Oceania, the coverage is less good in Africa and South America. Some sites have a large geographical reach; for example, the RIPE NCC site in Amsterdam isn't just queried from the Netherlands or from Europe, but it actually receives queries from virtual all over the world, even though those clients would have geographically closer AS112 sites, they should probably query there instead.
This is also shown in how 32% of the probes and resolvers sent queries across borders and as expected, this is especially prevalent in areas where deployment is low, so in this map, the darker the colour, the higher the percentage of queries sent across borders and you can see that the map of Africa is quite dark here because deployment is quite low.
In terms of the network distribution, only about 8.5% of probes and resolvers are in an AS that's directly connected to AS112, so ASes where the operator has set up an AS112 site or somehow provisioned direct connectivity, and only about 5% of probes and resolvers are in such an AS and also query the site they are directly connected to, so things could be improved in this regard because creating more direct connection to AS112 you can somewhat make sure that your queries are going somewhere you want them to go and they weren't just going to any random site, run by someone you might not want to receive your queries.
Secondly we compared AS112 to the DNS route, the DNS route is at the top of the DNS heirarchy and it consists of 13 independent Anycast networks, run by 12 different operators, it's important to note here that 12 of the 13 route server networks also server the.arpa zone, this is both a comparison of two types of Anycast networks, un managed AS112 network and the more carefully managed route networks and it also directly compares AS112 to the.arpa servers AS112 is supposed to protect.
In order to compare the two, we use the CHAOS TXT and for these we can determine which side of each route letter the RIPE Atlas probe contacts, we were able to determine the location of its chosen AS112 site and the chosen site of each of the 13 route letters and then we could directly compare the distances and latencies and in doing this, we found that AS112 has about a 36% lower median distance and 20% lower median response time than the tute, obviously these numbers vary a lot because they differ so much in their size. For example in terms of distance, E and F root have lower distances than AS112. However, for about 22% of RIPE Atlas probes, the distance to its chosen AS112 site was less than or equal to the distance to the closest queried route site of all letters, for those probes, the location of AS112 was better or at least as good as the location of all rot sides of any letter and the distance to AS112 could be decreased because only about 22% of the probes and resolver query the geographically closest AS112 site which is a lower percentage than for route letters, this is partly because of some routing inefficiencies which is when the AS112 operator restricts the BGP announcements to only go to certain networks so some probes can't access a site that's close to them because they don't have that connectivity.
So in conclusion, AS112 is a volunteer run Anycast DNS deployment that responds to junk DNS query, it's widely deployed in a similar or possibly better way than some route deployments with 469 sites run by 97 operators, however, the coverage varies and resilience might be limited, this work was published in the passive and active measurement conference this year, if you wanted to learn more, you could search for that and find out many more details about this. But for now, thank you for listening. And does anyone have any questions?
(APPLAUSE.)
MORITZ MULLER: Thank you Elizabeth.
SPEAKER: Robert, RIPE NCC for the purposes of this question, maintaining Atlas. You probably know that we do very similar things as built in into Atlas about the route zone and some others as well. So I guess if there is an ask, if there's a need, we can do similar things for AS112 as well, why not, and then we produce beautiful graphs and maps and colours and all of that stuff.
ELIZABETH BOSWELL: If you have the resources and people are interested that would be for setting that up.
SPEAKER: I guess if the community would like us to do this, then just send a mail to the mailing list and we will know.
ELIZABETH BOSWELL: We will thank you.
SPEAKER: Hi, Lucas from the... university here, thank you for that amazing talk, I didn't know about the AS112 stuff before at all and I was able to understand it all, it was really nice. I was just wondering did you consider like possible biases in your study for example the coverage of the geographic coverage by Atlas probes, I guess there's less Atlas probes in Africa and Asia, and also you said like the RIPE AS112 route servers had a far reaching geographic distance, if this might also be because of using RIPE Atlas probes for yeah the discovery.
ELIZABETH BOSWELL: We definitely considered that which is why we also used the open recursive resolvers as vantage points, at least in theory more widely distributed, in practice this only had limited success, for example we had a lot of open recursive resolvers in China but they all queried AS112 or some AS112 sites for which I wasn't able to determine the location, I wasn't able to use a lot of data from China for example.
So we did consider that, we tried our best to mitigate the bias of RIPE Atlas locations, in terms of the RIPE side, I don't know if that is much of a factor, it was created by the open recursive resolvers which wouldn't have the bias of RIPE Atlas.
SPEAKER: OK, thank you.
SPEAKER: Lars Liman from NetNode, route server operator, this is fascinating, very interesting, thank you very much. I felt that there is room for improvement on my side and did you use Atlas as your test network here, so that there are test results stored in the Atlas database, of course I can see a value for us to undertake some comparative studies for how routing happens for AS112 and for the route to route to learn how to improve our own routing situation so we can improve, would you in that case be willing to share the measured data that you have because I don't know if measurements you run or whether they are public or the not.
ELIZABETH BOSWELL: Yes, make if you can send me an email, that would be the easiest way to share, I have made all the code for this available because because the queries, the responses contain some personal information like names of operators, I wasn't able to make in a public. But if you send me an email, I can send you the links to the RIPE Atlas measurements which are public, you need to be able to find them, I can definitelily do that.
SPEAKER: Thank you very much, to Robert please do, so that we get more data and can learn more. Thanks.
SPEAKER: I can't see how there would be a bias in RIPE Atlas for that node just because we run both on the AS112 node is just connected to AmSix and I suppose a lot of networks are up there, that's the only thing I can think.
ELIZABETH BOSWELL: Yes, I think that's the main reason, it's just good connectivity so....
MORITZ MULLER: Thank you, anything online? No? OK. Then thank you, Elizabeth again.
(APPLAUSE.)
Next we have Lars Liman talking about DNS Tapir.
LARS LIMAN:, yes, thank you, I am Lars‑Johan Liman, for those who don't know me, there's precious few of you, I think. So I work for NetNode, an infrastructure company based in Sweden and we operate exchange points, time frequency servers and DNS.
We are involved in a project called DNS Tapir which is an attempt to improve on or give a chance to let resolver operators have better information for doing filtering if needed.
When we sent into the proposal for the talk, we wanted to focus on the specific part of the Tapir project called the policy processor. But I was requested to give a better overview of the Tapir project in general, so I will start by doing that.
So, we have a problem statement, DNS usage and DNS is not always used for good things, there are lots of bad and illegal activities like side channels, command and control, steering and what not. And also various types of tracking activities, since cookies have been kind of regulated, people try to use other means to do tracking or what people do on the net which are not regulated, for instance DNS and the trend for all these activities is unfortunately negative.
So we have a couple of areas of concern. Privacy leaks. DNS queries carry sensitive information. They are, as I mentioned, uses as replacement for cookies and that is often used without the consent of the actual user.
And it's used used by malicious software for communication miss direction for directing you to phishing sites and keeping things running, pinging home.
And we believe that these things need to be monitored and reactions need to be formed.
So threat intelligence is something that we are looking for, that's data that can be used to mitigate threats. This is data you can purchase from companies that specialise in providing this but this subscription services are often for known bad domain names, they give me a list of what's bad. And you will receive a list that says block this domain name, blah blah blah. But you never or very seldom receive information on why this is blocked, why does this company have this notion as a bad domain name and do you agree with that assessment.
So, data sources for threat analysis. Basically you can use all data, web traffic, routing information. Data from email filtering, you can use the phase of the moon or even reindeer migrations, maybe not reindeer migrations, but but definitely DNS traffic.
And DNS as a source of threat intelligence is a very good source actually, it's very interesting, it contains very specific details, a good thing for regular resolver operator is they awe event see the bad actors as well. Systems that have been inif he canned with various viruses or trojans send DNS queries and often they send them to the regular resolver which means that we can observe them. And that's a way to find command and control traffic and also to find bad sites that users are being directed to.
But DNS data is just one component in this much larger picture, but this is an important component.
So the data integrity when it comes to to DNS, we see the source IP address, almost always it's at the resolver, at the resolver you can see the source IP address of the actual machine that the user uses. So that makes it very, very integrity sensitive.
You can see the exact domain name that's being queried for and see the record type, the usage and there's also a specific point in time you can record exactly when this query came in and all that is sensitive information.
So, most operators of resolvers deny access to the query data and this is in order to protect their customers. And this is a very worthy cause, this is good. Still it would be very useful to be able to analyse that data, so the TAPIR project starting point is to using a gated ‑‑ this is a word I cannot pronounce, pseudonomised DNS data and we believe it can be analysed and used for cybersecurity monitoring and threat mitigation without privacy issues. And that last thing is extremely important to us.
So, we chose a way forward, which is to collect only select data and to aggregate it so that the various detailed information disappears and with that we can achieve a sufficient level of anonymity or also known as pseudonymity, this isn't full anonymity but it's, we believe, good enough.
So, the next step is to try to convince the people who hold the data which are others resolver operators often large ISPs and telcos, to collaborate with us and actually yield the data but only in the limited data that is integrity kind of cleansed from integrity sensitive information.
So we need the collaboration of these resolver operators but the data is sensitive, with he need to gain their trust and this trust we believe that we can earn by being very transparent about what we do and how we do it and the way to be, to create that transparency, is to use OpenSource software for all components of this.
So turning around by you having your fully OpenSource project, we create transparency, through that we create the trust from the ISPs and we get their collaboration, that's the plan at least.
So the robust DNS project is a large project ‑‑ it's not a large project, it's an overarching project where DNS TAPIR is an implementation of the larger project. We wanted to design an OpenSource platform for robust resolver operation and design on OpenSource system for central analysis.
So this has led us to two sub systems or software, one we refer to as the TAPIR Edge and this is something operated by the resolver operator, this is not in the central system, OpenSource as I mentioned, this software will cooperate with the actual resolver through a DNS tap feed; it will receive the queries which are now still integrity sensitive, but this edge software will aggregate and filter out the sensitive information.
So this is something that we hope to see operated by a multitude of operators and so that we can collect data from multiple sources at the same time.
It also receives feedback from the central analysis function and that feedback comes in the shape of observations, not decisions or instructions but only observations. And that makes it possible for this edge software to make ‑‑ to implement a local policy for what to filter and not to filter.
So this is a feedback loop so to speak where we have many hopefully many edge systems feeding into a central analysis function, which will be then feedback to the edge systems, the observations from the collective of these edges.
The central function we refer to as TAPIR core, this is supposed to be one system that collects as much data as possible from as many edges as possible. And it receives this pseudonomised data and that sudden doe mised in a way that's transparent, everybody can see which data is fed up to this central analysis function and all data is then analysed again with transparent OpenSource methods so people can see how we do this and it feeds these intelligence observations back to the resolvers or the edge systems.
So many edges report to the central analysis function, the core and the core feeds back to the edges and from the core you can, of course, also send alarms or these observations to third party people who might be interested.
So if I wasn't an experienced speaker, this is the picture I would show you and you will immediately realise it's impossible to read, I said to myself I can do better.
So the lower part is the edge and it consists of a number of components, you see the resolver to the left in lavender and then you have the edge minimiser, which is the one that cleanses away all the source IP addresses and it also does aggregation of data and it sends two types of local observations to the upper core and these observations are two types, one is statistics, I have seen this many queries of this type and that many queries of this type and that type, but also events, like oh, this is the first time I see this query. This is important; I will send that immediately. Because that can be something that triggers observation of a new thing in the core system.
The core system has databases where it stores those operations an does regular analysis on it and feeds back configuration to change possibly the minimiser to send other types of data, but also to the policy processor that you see to the left in the lower box.
And we will have a little deeper walk into the policy processor. The policy processor consumes these observations from the TAPIR core and combines that with data from other sources. So the policy processor is the one that kind of makes the decisions at the local resolver, how to actually instruct the local resolver. And it can take data from multiple sources and override observations with allow and deny lists, it can apply local policy for filtering and do background garbage collection of old data when necessary and it transforms this into a minimal feed with filter instructions to the local resolver. So here is where the policy is implemented.
It sounds simple but it can actually get rather complex.
So the policy processor takes observations from the TAPIR core from above, in our case we use MQTT for transporting this information, it can receive response policy zones, which is kind of a standard way to send specially information about special domain names across the DNS system. And that ‑‑ for that it can use regular zone transfers. And it will feed that down to the resolver in the shape of this single RPZ feed so that the resolver knows what to do.
It can also receive instructions in the form of deny lists and these are regular text files and allow lists and these come as directed as cyclical word graphs which is a type of database which is very suitable for quick look‑ups for words like domain names so that the policy processor quickly can put together stuff.
And of course all these things run as streams of input but if you cold boot a policy processor, you need to inform it of current state of affairs, so it needs to have a bootstrap function as well that comes from above. And on top of that, there's a Looking Glass where you can observe the status of things so that you know what's going on and you can fiddle with your system.
So local policy for filtering, examples of what you can do in the policy system is to react to flags, sorry, domain names that are flagged from at least this number of sources, remember that we have many edges, it's five edges agree on an observation, that's probably more interesting than if you receive it only from two. You have different TAPIR observations, if there's a combination of observation that you see that can be interesting or if you have at least some combination of X, Y and Z observations from the analysis system.
And of course you can always add allow and deny lists on that.
So this is currently an area of focus in the project an where we are working.
So the current status of the TAPIR project is that we are past the prototype stage, we have operating stuff in test mode, we have real but limited data from Swedish ISP, the Swedish internet foundation runs an operating core system that you can connect to and participate in. All the software components are available as ready‑to‑install binaries. We have Linux packages for both RPM and Debian systems. And one big Swedish SNMP is already on board and providing test data, real data from but a limited set of resolvers, two more are working on internal stuff to be able to participate, they often have to push stuff through their legal departments and they are not always the quickest.
Implementation status. All Edge components are implemented, all core components are implemented, but that will continue to evolve because analysis will change over time, so you cannot really say that it's like done.
Storage and communication is robust and our model for protecting integrity has been cleared by real lawyers. We actually checked with lawyers before doing that.
So next steps are trying to get more ISPs involved in that and product and solidify it even more. And we are also looking into next round of funding, we are good for the coming year but we need a little more.
So, welcome to the TAPIR and the TAPIR filter. And this is just a log of the administration and so, thank you very much.
(APPLAUSE.)
MORITZ MULLER: We have questions and I think the queue is closed now.
LARS‑JOHAN LIMAN: I will be around, you can talk to me afterwards.
SPEAKER: Are the operators of the TAPIR Edge software vetted or can anyone join the project, information flows back impacting their filters?
LARS‑JOHAN LIMAN: Looking at you, we would not really vetting people at such ‑‑ we are happy to work with people but we have to get access to the system through certificates so everything is encrypted that goes around here and we probably reserve the right to ignore people that we feel are not wanting to help, if I phrase it like that.
SPEAKER: It might also be useful to allow to add known trusted sources in the filter policy so you decide who you want to accept your rules from.
LARS‑JOHAN LIMAN: Yes, absolutely, yes yes.
SPEAKER: Thank you.
JIM REID: Jim Reid, unaffiliated DNS geek from Glasgow.
LARS‑JOHAN LIMAN: Are you new here?
JIM REID: Yeah. This is fascinating work, I think you guys are doing something really interesting here, the point the fact you have been open about the source, of the data you are using to try generate these feeds, that's a big step forward and I hope to see more of this kind of thing but my question for you is perhaps a little bit unfair and maybe premature, the effectiveness of the filter data, how does it stack up against the commercial filtering services already in place,
LARS‑JOHAN LIMAN: We don't know yet. Thank you for, it's a very good question. We don't know yet, it hasn't been tested against these commercial services yet but we do have security companies involved in the project as well because they are interested in using this type of technology for their services, I don't have a better answer for you.
JIM REID: Thank you.
PETER HESSLER: Some years ago I ran a project kind of similar to this, using BGP distributed information about email servers, are they sending spam or sending good emails and a big problem I ran into was the trust worthiness of the sources feeding me the data. And so I want to emphasise the first speaker, or the first comment, make sure you have a vetting system, make sure you have a way to remove bad actors and to keep an eye on the possibility of several seemingly unaffiliated actors behaves in concert to punish something, if they can get teams at microsoft.com blocked, a lot of employees will be happy but microsoft will be unhappy.
LARS‑JOHAN LIMAN: Very good advice and thank you for that. Of course the the allow and deny lists will probably help you in that case, they will contain the big players by default. But it's still very good advice and it's being heard, thank you.
SPEAKER: Ben know... a little bit in line with the question from... is it possible that in the Netherlands also run TAPIR core? Not, so, you gave the presentation TAPIR core is run by internet... some in Sweden. But also possible for reasons, protective DNS in the Netherlands, something like or protected DNS in France, that a country wants to run their own core. How many resolvers do you need to have?
LARS‑JOHAN LIMAN: For the first question, absolutely, the idea for the core is that there should be ‑‑ it's should be possible to operate multiple cores in different locations, or different geographic regions, legal systems, whatever, and we hope to be able to develop the system so that you can have a federation of core systems that talk to each other and feed information across, that part is not newly researched and done yet. Sorry I missed your second question.
SPEAKER: No, that's fine. Thank you.
SPEAKER: Thanks for the presentation because it gives me a better idea what TAPIR actually does and one thing I picked on was it gives a better insight, something is blocked for resolvers do you have a reason to give that back to the client request?
LARS‑JOHAN LIMAN: Ah, that means feeding it back through the actual resolver, to say why something is blocked. No, we have not begun working on that. It's a good idea, I think but it's totally separate research or development that we are not working with, at least not right now. But interesting suggestion. Thank you.
MORITZ MULLER: Thank you.
(APPLAUSE.)
Next up we have Peter Spacek talking about NSEC 3.
PETER SPACEK: I work with all things DNS at this time, I would like to debunk some myths about NSEC three and what it does. Usually the people think that NSEC 3 is the which protects your zone at that time so the bad actors cannot see them but in reality, it's more like this it's a very efficient way to burn a lot of DNS servers without getting much and you can misconfigure it in interesting ways and it gives very nice feeling of false security.
So what is this. I will just skip over the slides with a difference, if you are in this room, you know that what DNS is and NSEC gives you a range of names and between these names, nothing exists, that's what NSEC says and the names are in clear text and NSEC 3 is is the same thing but the name is hashed. So it's not immediately obvious what was the original name was before hashing.
So instead of range of names, you have range of hashes, but again you already know this.
If you want to read this in coherent manner, I strongly recommend reading the RFC on the slide, that's 7129, it is with accessible language explaining what is going on and why so we can understand why the heck is this so complicated.
For today, we look at the NSEC3 parameters, so if you have your zone or the zine and you do use NSEC 3, what can you configure and why.
So here we have three parameters in the NSEC 3 one, it's salt, nonsense, just ignore it, leave it out of the second parameter is hash algorithm, there's only one, not the much to configure and the last parameter is the extra iterations and here emphasis on extra because the standard says iterations, not extra iterations but even if you specify zero, it still hashes once, it's like the value you configure is how many extra loops it has to do before sending it to the client.
Here's an example: If you do NSEC hash, this is a common line tool, you can play with it yourself. If you hash it with zero extra iterations, you get a hash, if you use a hundred extra iterations it hashes 101 times, it takes more time, you get different hash. And here's the catch. Of course, hashing more times is more secure, right? Sure, of course.
But the thing is that if someone decides to attack your zone and extract the names, there are just sent random names, completely random legion rated, they will get huge number of the hashes, store them on disc and then eventually when they are bored, they will start cracking the hashes, possibly using GP U farms because nowadays everybody has a GPU or stuff like that, it's very efficient and off‑line, and emphasis on off‑line, while at the same time your DNS servers have to compute the same calculation on Palestine while the client is waiting, this is not really fair game when the attackers and actual DNS servers.
To demonstrate this, we'll have two different test set ups but they are almost the same. One is with random sub domain attacks, originate random number, send a query to a DNS server and on the authoritative side we changed the number of the NSEC3 in the zone so we can see was impact of the different values.
Second test set up is very similar, but we have a DNS resolver in the middle, so we send a random query to DNS resolver and then it goes and attacks the authorative server, with the same random DNSEC. And for resolver just to spice it up, we use second variant of the attack which is random labels, 127 random labels or if they randomly generated because in NSEC 3, resolvers have more work to do if there are more labels.
The test set up, in case you want to play with it yourself, it's on this slide but it's basically switch off IIL so it's easy to measure.
Here this is the important slide. On the X acquisition, we have the number of NSEC used to sign the zone, so that's what the authoritative operator decides and the Y acquisition has performance in QPS, can you see? On my screen it is a little bit chopped. You can see all of it, good.
So again, on the Y acquisition, we have a lot of performance and relative in a sense that if you configure your zone with zero extra iterations, measure the QPSand go on to like 150 iterations, measure it again and you know, compare the ratio, how much slower it is with additional iterations.
Here you can see for yourself that the shape of the lines is the same for NSD and unbound and for BIND, I am sure it will be same for everything, because the shape is property of the protocol, it's not implementation back that it's so slow, it's the property of NSEC 3 which is the important point here. If you go for 150 iterations because you can, you will effectively burn 90% of your DNS resolver capacity by just doing that.
Enjoy the money on fire.
Now we have just zoomed into the beginning of the chart so we can actually see the nuances at the very beginning and you can see for 25 iterations, you burn roughly two‑thirds of the capacity of your server, if you are usually people like to over provision their servers, having enough capacity to ‑‑ so you don't need to care and if you use 25 iterations, you burn two‑thirds of the capacity, if you go to five iterations you still burn like one third of the capacity, even one iteration makes a difference as you can see.
So it's incredibly costly or in other words, we can say that over the last 40 years, we have optimised the heck out of the DNS servers, so even the very fast... makes a difference.
To protect themselves, the DNS resolvers have limits of how much work they are willing to do. The limits have changed over time from like completely out of the blue 2.050 to more sensible numbers nowadays, but the question now is what happens if the limit is exceeded? And that's very important one.
Because you have to two options basically, either Servfail, but then people complain I cannot get there and what not, and the resolver operator doesn't have anyway to fixing the problem on their side, they mostly decided to go insecure in the number of iterations is over that local specific implementation limit but that has some down sides too. Imagine that you have for TLE, that TLE for some reason is signed with NSEC 3 using 151 iterations, that's over the limit for that specific r implemntation, one of them is signed like with DSrecord, nice, sign, secure delegation; the other one, the unknown name, doesn't have DNS record, it's insecure in NSEC term terminology and the attacker will take these two records, combine them together and send them to the resolver, but now there is a catch.
To actually detect that the name in the answer doesn't match, the resolver would have to do the iterations, but it's over the limit. So the resolver will say that's too much work, I am not going to do it and either self fail and people will cry or go insecure. Then people will not cry but they will have a false sense of the security because well there's a secure delegation,st the zone is signed and everything is fine except it's not.
So that's very good reason why the resolvers cannot go with limit zero because it would turn lots of secured delegations into down gradable mode of operation but at the same time not enforcing the limit of zero costs performance and capacity so it's not good.
So long story short, don't do it. NSEC 3 is for very limited use cases, especially if you like...publishes the zone. It can save some memory but but that's about it. If you have to use NSEC 3, there's RFC published a couple of years back which goes to some length to explain why, how to configure and why, in three bull level points, use iterations zero, extra iterations zero, don't use salt, it doesn't make sense or do what people think and no one knows how to use it effectively and don't use opt out if you really don't have to, if you have enough memory, just go for opt out disable, it's more secure and has better properties. Thank you.
(APPLAUSE.)
MORITZ MULLER: There's time for questions.
SPEAKER: Just a quick remark, don't even think about changing the algorithm, it will not help.
SHANE KERR: Are you going to be reporting in the resolver like DNS new reporting?
PETER SPACEK: I think it has code error reporting, yes. No one asked for it yet.
SHANE KERR: I think it would make sense. It kind of is an error.
PETER SPACEK: Yeah, yeah, it can be.
SPEAKER: NSEC for the win and I say that as a TLD.
SPEAKER: Yeah yeah, definitely.
MORITZ MULLER: Thank you, now we have time for the RIPE NCC updates.
With Florian.
FLORIAN OBSER: Hello, I am Florian from the RIPE NCC, I give this update and it's tradition I am standing between you and lunch. I will try to keep this short. I want to give you an update on the team. We have not done this for quite sometime. Last year we had a bit of a re‑org and we are now four people on the team who work on DNS exclusively, so we are doing all the authoritive things at the NCC and things adjacent to that like the signers.
So we have Anand, Leon, Martin and myself.
I want to give you an update on our hosted DNS instances. We have one quarter AuthDNS host DNS instances an three quarters for K‑root, we would like to get this more or less or par like have more auth dance hosted instances, to increase the resilience there more.
And we made some good process there since RIPE 91, we add the the following AuthDNS hosted instances, I am not going to read out all the names, I will probably mispronounce them, we would need Anand on stage for that, we are looking for hosts, you can employ that link and it LIRs all the requirements that we have.
Another thing that we have been working on is service automation, this is a thing that we have been doing for at least ten years, where if a name server answers a Servfail, it automatically withdraws its server prefix, this is of course a dead person's switch, what we do is if we lose ‑‑ if a server loses contact with the controlling infrastructure in Amsterdam, it will continue serving the zone because we think if there's a problem in a region, like you have natural disaster, war going on, those kind of things, it's still better if we can answer questions and we will do this until the zone expires. But at that point when the zone expires, this system becomes useful and we destroy automatically.
However, we do not do any more things automatically, like if there are other issues on the system like hardware faults, that kind of things, we need to deal with them manually and the reason for that is that we quite worried that if we have more automation, at some point that system will just run rampant and withdraw on all our service prefixes and that would not be good.
However, there's a way around this and the idea is if we have covering prefix in place and we will never withdraw that, we can actually automate more and just remove the service prefix. For K‑root, we already have a cover prefix in place for V4 and V6 since the inception of K‑root as far as I know. For DNS we did have space for v4, next to the service prefix, so we could have a covering prefix there but not for v6 we had no space, we had to renumber V6 on ought DNS, which we did in 2025. It took us all of 2025 to wrap this up and we wrapped it it up in January 2026. We moved slowly there.
With that another step of automation that we now have in place is if on a system we are missing the default route or we no longer have a full BGP view, we will also withdraw the service prefix. Because people do not want to draw bloke queries we receive, we always require a path to the whole of the internet.
And as I said, we will never withdraw the covering prefix and the configuration looks like that, the covering prefixes only announce from a core sides and we have /23 for IPv4 and /32 for V6 on K‑root and /44 for AuthDNS. And production and test prefixes. We also always leave test prefixes in place, so that we can observe BGP propagation.
Another project we have been working on last year is ISO27001, for those who never heard of it, congratulations.
No, it's actually not that bad. So it's an information security standard and what it wants you to do is maintain and improve an information security management system, so write things down. You collect your risks, you come up, you decide what controls to put in place to mitigate those risks and you write them in a business continuity plan, more or less of that.
For DNS it was not particularly complicated for us, gives you a list of controls, you pick and choose, we already had our risk matrix in place so we needed to find which controls apply to our risk matrix, that's one part and the other part is link the controls to show that you implement those controls, link this to existing documentation.
Of course everybody in the room who has done operations knows that kind of not is entirely true, so we also took this as a bit of a spring cleaning get rid of all documentation and really double check that the documentation is up to date and we found a few things that needed to clarify but it wasn't not a huge task.
We of course did have a business continuity plan in place, we needed to formalise it a bit more so it's auditable under ISO 27001 and we had time to run our disaster r scenario testing, multiple ways to do it, you do it on a table top exercise, you role play it, you can do a functional test, which is you take one system out but the whole system as a whole continues working or you do full scale test. So, for example, we have disaster scenario, the head office becomes inaccessible. I propose to do a full scale test for that one, we were not allowed, so we could only do it as a table top exercise. However, we all lived through you the pandemic, we knew that we can happily operate this thing like the DNS infrastructure without a head office so nothing surprising there.
Last month we passed an internal audit and the next step is external audit to get the certification second half of this year.
And the intention is to then use this information to security management system to show compliance with various regulations that DNS falls under, for example NIS2.
And last but not least, we finally got around to publishing a statement on RSSAC001, this is the service expectations of root server operators, there are 17 lists in this document, on what the community expects that root server operators should satisfy, it has things in there like server the unique IANA zone and up to date root zone and confirm to BCP 40 and have a business continuity plan, there's a few more in there, you are an important service, the community expects you to do this properly and yes, that's what we are doing. And the recommendation is that we should actually put out a statement saying we do that and we did that by publishing it as a RIPE 859.
And with that, thank you. And if you have questions, I might have answers.
(APPLAUSE.)
SPEAKER: Dick Knight, NomNet. K‑root did not have its covering prefix from the get go, for more information, see your mail from Randy Bush to NANOG in October 2025 titled "OK, can you see" for description of some of the unintenned consequences of running local nodes.
FLORIAN OBSER: So it was 20 years ago so, close enough! Thank you.
MORITZ MULLER: Thank you. That leaves us time for one more short presentation by Maynard.
MAYNARD KOCH: Thank you very much for giving me the opportunity, I will be really quick and make sure you get your lunch on time. This is a joint work together with the Internet Society, we try to measure resilience of recursive resolvers, the specific part of the project, because reciliency is challenged by multiple incidents, that may lead to outages so resilient domain name system should withstand these outages, attacks and so on, so there are a couple of mitigation techniques that try to help make the internet more reliable, the DNS more reliable, more resilient. So recently ICANN has proposed an initiative to codify the best practices into a set of global norms to improve security, the knowledge sharing norms for DNS and naming security. So the goal of this project is to conduct an independent study of the measurable practices of the framework as well as other important DNS reciliency practices, so we tried to better understand and assess the resilience see, how resilient are the recursive resolvers, what is critical infrastructure here, and our approach is we use external data sources and try to use public sources and we complement them with active measurements to try to get a comprehensive picture as much as possible.
So yeah, very brief into the result, so currently we have about 100,000 resolvers in the database we can measure, but stay tuned for more updates here. Just a little homework, so thinking about it, how would you define and assess reciliency in DNS? And if you would like to contribute data to this project, please talk to us. That's about it, thank you very much. (APPLAUSE.)
MORITZ MULLER: Thank you for the quick update. We still have time for comment, questions, if there are any. No? Then I thank all of you for being here for participating actively. And yes, see you next time.
(APPLAUSE.)
Lunch break.