PING: Recent Episodes
APNIC
PING is a podcast for people who want to look behind the scenes into the workings of the Internet. Each fortnight we will chat with people who have built and are improving the health of the Internet. The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, we talk with Job Snijders about secure Internet routing again, focusing on an approach to preserving the state of RPKI as a longterm historical record for research and analysis. Job was last on PING to discuss the “Erik” protocol.Job has been running an archive of RPKI state for some time, as a volunteer activity but an exercise which was feasible when BGP speakers producing ROA objects was measured in the hundreds to low thousands is significantly more expensive when the population of ROA producers is a more realistic percentage of the around 80,000 AS holders worldwide.Because Public Key Cryptography depends on a regular re-signing, and re-cataloging (as another signed object, the RPKI “Manifest” file) even when there is no substantive change in the state of signed information, a large amount of “churn” can be seen in the data, and even a compressed form of this state in turn incurs a huge overhead in storage of hard-to-compress data. Job reached the limits of his free activity, and started to explore a more compact and useful representation. This has emerged in the IETF standards process as two related activities. The Canonical Cache Representation or CCR, and the aggregation over this for data represented as the RPKI Spool data model for “materialising” the state of RPKI objects.Along with his fellow IETF draft authors Bart Bakker , Tim Bruijnzeels, Theo Buehler and Fedor Vompe Job has managed to define a remarkably compact, highly compressible representation of RPKI validated objects, and the cryptographic payloads. These systems have been designed to leverage well known UNIX and other techniques for data management such as the “tar” format for spooled data, ASN.1 for binary encoding, Merkle Trees to hold compact hash state, and data compression using the zstd encoding.It’s a nice approach to solving the scaling problem.
In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss a traffic behaviour in the DNS which Geoff has noticed in the labs advertising based experimental data capture. Virtually every DNS query Geoff sees, he sees twice (or more). For a cohort of about 150 million unique DNS labels on a given day, The Labs system is collecting 270 million incoming DNS queries. Thats a lot of duplication. What's going on?The advertising data collection depends on issuing unique DNS queries, which in turn generate unique web page serves. This allows measurement of internet-wide behaviour on about 30 million browsers, games, devices every day. Not to say that these names and web URLs are not routinely seen by more than one entity, intermediary systems such as caches and proxies as well as re-visiting old open website tabs on a browser can cause this.The point is that after the "first" fetch, the subsequent fetches can usually be held to be re-presentations of the same experiment and can therefore often be discarded (in the case of the web). But, for the DNS which has always had an element of unreliable transport, and which in turn invites measurement of features like DNSSEC which cause SERVFAIL messages, and demand repeated attempts to "find the DNS name-to-address mapping, re-fetching is itself something under test. How many resolvers lie behind a given users systems? How many kinds of resolver (DNSSEC enabled, or not) does the user depend on? Finding that almost all DNS queries are repeated, and it turns out repeated very quickly invited Geoff to have a look at what's going on "under the covers". There are some patterns behind what's being seen which Geoff explored in the APNIC Labs Blog.
In this episode of PING, we talk with Willem Toorop about a measurement of 6 choices for naming the root servers of the global DNS. This measurement was motivated by RSSAC 028, a 2017 technical analysis for ICANN of the naming scheme used by the root DNS servers.Root name servers are part of the global DNS system, and they provide basic bootstrapping for every other DNS resolver and authoritative name server worldwide. This is their primary role, but they also handle query load relating to the global DNS system all day and every day. There are 13 distinctly named root servers, each one identified by a letter from A to M under the domain root-servers.net. Although there are only 13 distinct "labels" there are in fact thousands of machines providing this service as independently operated clusters behind each of these letters, using the BGP "anycast" method we have discussed on PING before.The special zone, root servers.net was instantiated in 1995. It's delegated as normal under the .net zone, which in turn delegates from the root "dot" zone. If you are bootstrapping a new resolver, to find a root server you appear to need to know about the delegation of .net in order to find root servers.net, to find the given server. This is a circular dependency because you don't know how to find .net until you have asked a root server instance. The circular dependency is resolved by the contents of an initial "priming" response. This response provides the additional "glue" information to seed direct knowledge of how to find each of these named root servers as is, without all the intermediate logic of the circular dependency.This glue is inherently not signed in DNSSEC. It's insecure. The priming response is very carefully curated for both its size and the content but it would be good to give it some content security and increase trust in the message. RSSAC 028 explored the impact of applying DNSSEC over this domain, which in turn would add DNSSEC signatures to the priming response and impact a goal of keeping this a small 512 byte backwards compatible UDP packet.By keeping this response small, it would continue to work on all legacy systems worldwide. Nothing happens quickly in DNS evolution, and this naming question has now been under consideration for almost a decade. Willem and his colleagues at NLNet Labs and SIDN produced two reports reflecting on the candidate naming schemes, application of DNSSEC, and the impact on the bootstrap fetch. There are some surprising outcomes!
In recent PING episodes, APNIC Chief Scientist Geoff Huston has asserted that Internet routing has shifted away from traditional IP packet forwarding. Instead, it is increasingly driven by processes that map names to addresses. It’s no longer just about your IP address or the specific endpoint you think you’re connecting to, it’s about your location and which intermediary services can most effectively handle your request.How does this actually work in practice? What processes determine where your request is served from, and who makes those decisions? In the latest episode of PING, we explore this topic.Of course, IP-level routing hasn’t disappeared. For optimizing content delivery using a ‘closest’ node model, anycast remains a critical technique, using BGP to direct traffic to the nearest available server, and it is widely deployed at scale. However, this approach is increasingly supplemented by name-based mechanisms, particularly those driven by DNS, to more precisely determine where requests should be directed.The way endpoints are identified in an Internet protocol exchange is changing, and this shift has broader implications for the nature of the network. As a result, we are seeing some trends emerge:Control over routing decisions for application content has shifted away from ISPs operating at the BGP layer and towards higher-level logistics functions in the stack. These decisions are now typically made by service providers offering optimized content delivery as a managed service.Provision of this optimization is typically carried out through DNS-based mechanisms. As a result, organizations often delegate their DNS to the same content delivery provider, allowing it to control request routing. This gives the intermediary significant influence over how and where traffic for a domain is directed.Together, these trends are likely to reinforce the growth of ‘walled garden’ vertical markets. Application-specific delivery methods are increasingly positioned as competitive advantages over generic services (such as standard video streaming), meaning that both application choice and DNS-based steering can guide users into more closed, vertically integrated environments.
In this episode of PING we’re hearing about secure Internet Routing and its data distribution problem from Job Snijders who has been on PING before talking about his measurements in BGP and RPKI.We caught up at IETF125 in Shenzhen where Job presented to the SIDROPS working group on a new protocol he’s been designing, called Erik. The Erik protocol was named in honour of Erik Bais who died in May 2024. Erik was a stalwart of the RIPE routing community. He was a chair of the Address policy working group, and active in the Dutch cloud community and the data center association.RPKI, the principal mechanism for determining secure inter domain routing intent (hence SIDR) depends on every relying party (or RP) validating the data collecting all the signed statements from all the publication points, worldwide. This is a time consuming process which inherently serialises behind the sequence of bytes fetched to form a given repository state at a publication point, and how the protocol works out whats changed since the last fetch by this user, and what to send. It’s not very efficient and it’s not scaling as well as we’d like as the amount of data rises, and the number of validators or RPs are fetching the data.Job’s “Erik” protocol is designed to improve significantly on the two mechanisms defined at present, the RSYNC protocol, originally designed in the mid 1990s for filesystem synchronisation, and RRDP, a SIDR specific delta protocol which was designed to improve on rsync, using experience gained from the NRTM mechanism used to copy data in the RIR WHOIS databases.Job has been able to find why RPKI fetch is slow, and design a protocol using the Merkle Tree mechanism which can significantly improve the collection delay, as well as allow for intermediaries such as CDN providers to host services in the cloud.
In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss Network Time Protocol or NTP. NTP is one of the older systems we depend on, designed and implemented by Dave Mills who died in 2024. Dave had been working on time synchronisation from the mid 1970s, and cared deeply about synchronising the emerging ARPAnet and the Internet, with the pre-existing worldwide collaborative framework which regulates our sense of time, and how it relates to the world of science, astronomy and civil society.Geoff has been musing about NTP, moves to secure NTP, and the many dependencies in the modern world on the underlying concepts of a coordinated sense of time. This dependency in the modern world on highly synchronised clocks cannot be overstated, it creeps into every sector of daily life from aircraft and space navigation, to finance systems, and event scheduling of all kinds.But our model of time is fundamentally based on the rotation of the earth, and the length of the second, and unfortunately while we now define the length of the second to astonishingly accurate levels, the rotation of the earth isn't as stable as we'd like. our model of time therefore has to make some adjustmentsTo make matters worse, our model of time has been coded over the years to varying models of a start date known as an "epoch", and how we represent time inside the machines, systems and services isn't one unified model any more. It's all coming a bit un-stuck.
This time on PING I'm talking to Alban Kwan from the Trusted Notifier Network (TNN). I caught up with Alban at the APRICOT/APNIC61 meeting held recently in Jakarta, where Alban was attending the policy and governance sessions with a particular interest in the problem of online abuse mitigation.Alban is interested in bridging the gap between the business and technical communities in this problem space. When it comes to crafting a response, business tends to focus on brand integrity and the real world cost of mitigation when their products and services are abused, used for spam or fraud or suffer inbound attacks against them.The technical community tends to be focused on things like network traffic level views of problems, found in deep packet inspection, running honey nets, looking at firewall logs and in turn, tends to drive security activities in network technology. It's a very real cultural divide.We've discussed the more technical, packet-level distributed threats on previous episodes of ping. We've talked with Adli Wahid from APNIC and with Leslie Daigle from Global Cyber Alliance, discussing honey nets which present as low barrier attack targets and can measure the levels of abuse coming to unprotected websites and online services and help pinpoint where they come from.In addition, Leslie has raised her concern that we need a conversation in wider civil society about the governance issues to address this problem because technology alone can't solve what's clearly a societal problem.TNN is part of that civil society response to online abuse, albeit in a slightly different form, focussed on communication between the involved parties and a concept of a higher trust channel, a better signal-to-noise ratio which reduces real world cost and preserves business reputation.Alban has an interesting analytical model of how to think about the problem and I think it's interesting to look at his engagement with business process and how people build mutuality and trust dealing with abuse problems of any kind.
In this episode of PING, APNIC Chief Scientist Geoff Huston discusses the tortuous history of The CIDR reportClassless Inter-Domain Routing or CIDR, is a mechanism defined in the 90s, to replace the former model of fixed sized networks defined in RFC791 called class-A class-B or class-C (there were actually class-D and class-E but for now we can ignore them) -the "Classless" part means no longer obeying the fixed bit-pattern at the "top" of the address (in the top 3 bits) which defined which class you were in, the classes defining how many addresses were in that block: a Class-A was 17 million, a Class B was 65,000 and a Class C was 256. This worked fine for the early life of the Internet, but under the stresses of exponential growth in the 1990s a new method for allocating addresses was defined, which exploited this "classless" model and allowed people to be given sizes between 17 million and 65,000 or between 65,000 and 256. -Which in turn fixed two problems: access to addresses into the future (through the Regional Internet Registry model of justified need for addressing) and the scaling problems of the routing mechanism.Routing has roots which reach back into the 1950s when a class of methods for describing how to exchange information about paths in a system called "Bellman Ford" was defined. This mechanism came all the way into the future alongside the growth of the Internet and replaced other models of routing which had emerged in networks such as DECnet from Digital Equipment corperation, and we now know as the ubiquitous BGP4 for Border Gateway Protocol, version 4 (a very good name, for the 4th version of something which was modified from the equally well named BGP3, to add in CIDR models of prefixes.The CIDR report grew out of the need to understand who was causing the stress inside BGP, a public commons of everyones routing assertions, where if you did what was entirely rational for you to engineer better routes by announcing more of them, you made every other BGP4 speaker incur a cost. The report helped identify who was the "noisy" BGP speaker, which Autonomous Systems (AS) were responsible and how much more effective could they be, and still achieve their engineering outcome. It was an early version of "nudge" theory, using naming-and-shaming to publicly expose the damage any BGP speaker did to the commons, in a public record.Geoff has been running the CIDR report continuously for over 2 decades, following on from the work of Tony Bates and Phil Smith at Cisco.But, carried into the modern era, after so much discussion of the declining importance of BGP routing on the Internet in a world of "names based" steering for content, how relevant IS the CIDR report?
This episode of PING is an interview with Marc Blanchet from Viagenie in Quebec, Canada. Marc has been active in Internet Procotols and the IETF for decades, most recently focussed on Internet Protocol communications in deep space. Marc presented at the recent APRICOT/APNIC61 meeting held in Jakarta.We've got used to the idea of IP working in Low Earth Orbit, with the rise of Starlink as a high speed service which in many cases out-performs terrestrial services available in rural and remote locations. And, we've had IP services mediated via GeoSynchronous orbit satellites like DirecTV, which are now significantly less popular because of one overriding problem: the very long end-to-end delay. People find the half-second of round-trip time to a GEO satellite unacceptable. But, in space, things get much worse.Marc's work looks at space contexts which go far beyond these orbital "shells" around earth, into the distances to the Moon, to the Asteroid belt, and beyond to other planets. The delay component in these networks isn't just an inconvenience, it has very real implications for rount-trip-time effects on a protocol like TCP, which demands a stream of "ACK" signals to manage the sender and receivers models of bandwidth and delay and retransmission. Marc has been working on how to simulate the effects of these very long delays using earth bound Virtual Hosts and code, using the Linux TUN device, and TC-NETM along with code developed by his team. This allows them to programatically define an experiment in delay, loss, re-ordering terms, which can span hours of packet-in-flight time, and look at how switches and routers, intermediate elements of an end-to-end IP exchange can work. Code patches to these systems to represent delay as a 64 bit quantity now mean it's theoretically possible to test IP out to the edge of the galaxy, if you had a way to keep machines running that long.If we look at what we actually want from IP networks in deep space, TCP isn't the right choice for how to get applications to work: There is a much better choice in QUIC, a more modern session-layer like protocol which can deploy over unreliable transport like UDP, and which integrates transport-layer security and IP address agility into the same model.As Marc discusses in this episode. IP in space is already a reality, with deployment of mobile telephony 4G base stations to the moon, and Chinese researchers experimenting with QUIC. The IETF is actively exploring the protocol options in the TIPTOP working group.
In this episode of PING, APNIC Chief Scientist Geoff Huston discusses an emerging concern about how the IETF interprets the concept of ‘Best Current Practice‘ (BCP).In his previous episode, Geoff explored emerging questions around DNS provisioning over IPv6, including experimental observations on its performance characteristics. Towards the end of the discussion, we turned to how the Internet Engineering Task Force documents standards and protocols through Best Current Practice (BCP) publications.In the traditional view, BCPs are intended to provide clear, implementation-focused guidance that remains neutral across different operational contexts. This includes networks of all scales — from home environments to enterprise, national, and international infrastructure. These systems rely on a shared understanding of protocol behaviour, and BCPs help define the set of practical, interoperable choices available for deployment today.To be designated as a BCP, an IETF RFC undergoes additional review and scrutiny. This is similar to the ‘Standard’ (STD) designation, which elevates an RFC beyond a general publication and gives it greater weight in operational and procurement contexts.While RFCs are not legally binding, they are frequently referenced in national regulations, procurement frameworks, and service specifications. In contrast, Experimental and Informational RFCs explicitly indicate that they are not intended for such use. BCPs, along with Proposed Standards and Standards, carry greater normative weight due to the more rigorous processes required for their approval, making them more suitable as references for real-world deployment and decision-making.An emerging perspective suggests that BCPs may be treated as aspirational as well as definitional — potentially describing behaviours that are not yet widely deployed, or in some cases not deployed at all. This represents a subtle shift from the traditional view of BCPs as documenting established, operational practice.This tension has surfaced in discussions around the DNS/IPv6 draft mentioned in the earlier PING episode. The conversation involves multiple parts of the IETF process, including the DNSOP Working Group, the DNS Directorate (which reviews DNS-related work across Working Groups) and the Internet Engineering Steering Group (IESG), responsible for final document review and status approval before publication as an RFC.These discussions reflect an evolving question: Should BCPs remain a record of established operational consensus, or can they also serve to guide and shape future practice?Advocacy is an important part of the IETF process. But it’s important to distinguish between advocating for a new path in systems and specifying what the current best choice is in the specification of systems behaviour.
This episode of PING features Thomas Alfroy and Thomas Holterbach from the University of Strasbourg, talking about bgproutes.io - A new approach to BGP data collection and analysis.We've featured bgproutes.io on PING before, when we discussed GILL and DFOH with Professor Cristal Pelsser from Louvain University. At that stage, the project was in an early stage and we focussed on the machine learning and approaches to selecting the "Most valuable Vantage Point" or MVP in the data sources available.This time, the two Thomases discuss the operational deployment of the service, and how they have designed the system to provide fast visibility to data in a 3 month window, and an API for selection of prefixes and origin-AS of interest, to show the BGP transactions seen in the wild. They've been designing "dashboards" to show both the data and a sense of what logic determined the inferences made about the data.bgproutes.io has been written to process the newer BGP Monitoring protocol (BMP) which provides visibility of the discrete states of the individual BGP speakers who peer at the BMP collection point. So, considering an IX this means that a single feed can supply 50 or more distinct views of BGP. This has permitted the project to grow to over 300 points of view worldwide.The service is complementary to those from University of Oregon routeviews or the RIPE RIS project, and includes data from these sources along with PCH and CGTFThomas Alfroy presented at the Sydney SIGCOMM meeting where the system was described in the 2024 SIGCOMM ‘best paper’ award-winning research: “The Next Generation of BGP Data Collection Platforms“. Thomas Holterbach is no stranger to the APNIC community having spend time at the IIJ Research Laboratory in Tokyo.
In this episode of PING, APNIC Chief Scientist Geoff Huston discusses running advertising-based experiments and a problem of interest in the modern DNS.DNS fundamentally requires end users, their chosen resolver provider, and the authoritative servers for the names they query to cooperate in a coordinated exchange over IP protocols to answer DNS questions. The specifics of how these queries are encoded and transmitted become complex very quickly, but a particular issue is emerging in how we define, in normative and strongly binding terms, the way the protocol is expected to behave. This will shape future deployment decisions, implementation choices, and operational dependencies.The question centres on the use of IPv6 within the DNS ecosystem as a whole. Can we yet say that IPv6-only DNS can be relied upon in operational practice? And if so, should that position be written into the guidance an RFC may define, were it to be elevated to the status of a Best Current Practice (BCP) or BCP document?Geoff is exploring how to measure this by exploiting a DNS model known as ‘glueless’. In this approach, the additional ‘glue’ records that are typically passed around behind the scenes to keep DNS resolution working are not provided by the authoritative server to the resolver. As a result, the resolver is forced to issue further queries, which can in turn be constrained to use IPv6 only.These queries are conducted without many of the usual measurement artefacts — such as error introduced when users close a browser session prematurely, or when attention drifts away from the web page that triggered an advertisement-based test.Some interesting variances are emerging when you look at this data by geographic region and origin-AS.DNS fundamentally requires all of the end users, their chosen resolver provider and the authoritative servers of the names they ask about, to cooperate in a dance over IP protocols to answer DNS questions. The specifics of how these questions encode and are passed around get complex very quickly but a specific problem is emerging in how we define "normatively", with strong force, the ways this protocol works. This is going to affect future deployment, code, and operational dependencies.The question relates to the use of IPv6, inside the DNS system at large. Can we yet declare that IPv6 only DNS can be used reliably, and should we write it into the operational practices an RFC can define if it's elevated to the status of a Best Current Practice or BCP document?Geoff is exploring measurement of this question, by exploiting a model of DNS which is called "glueless" -the extra "glue" which is typically passed around behind the scenes to make DNS work, is not given by the authoritative server to the resolver, and this forces the DNS resolver to ask more questions, which can be in turn forced to be delivered over IPv6 only. These questions are run with none of the usual concerns about the error rate due to drop off by users closing a browser session, and the problems in measurement seen with end users, whose minds may wander away from the web page triggering the advert.Some interesting variances are emerging when you look at this data by geographic region and origin-AS.
This episode of PING features two members of the Thai academic and research community and was recorded last year at IETF 122 in Bangkok. With a population of more than 70 million, Thailand has around 80 publicly funded universities and a further 70 or more private institutions, and undertakes substantial research in telecommunications and computing. A leading example is the Asian Institute of Technology (AIT), which has run the AINTEC conference across the Asia Pacific. Both of our guests on this episode are associated with AINTEC.First, we hear from Sukumal Kitisin of Kasetsart University, a state-funded institution. She has been working on an Internet measurement programme for the Thai Consumer Council (TCC). The project leverages well-known tools such as Ookla Speedtest and OpenSignal to capture end-user experiences of mobile Internet services in Thailand.These measurements support modelling of broadband speeds and service quality, which the TCC can then use to present consumer evidence to the national telecommunications regulator, the National Broadcasting and Telecommunications Commission (NBTC). While we are often accustomed to discussing measurement using Atlas probes and other ‘insider’ approaches, this work represents a consumer-friendly, simple, and defensible method for capturing the lived experience of real-world users.Secondly, we hear from Adisorn Lertsinsrubtavee, Director of the Internet Education and Research Laboratory (InterLab) at AIT. Adisorn describes a decade-long measurement effort known as HAZEMON, which focuses on low-bandwidth Internet of Things (IoT) devices used to measure suspended particulate matter such as smoke and fumes from diesel and charcoal fires, as well as carbon monoxide and carbon dioxide.Thailand sits at the heart of a South East Asian climate system that is heavily affected by forest fires and persistent smog, with serious health implications. PM2.5 particles can penetrate deep into the lungs and cause lasting respiratory and other health problems as they are absorbed into the body. Monitoring these levels enables the project to model pollutant density, identify significant events such as forest fires, and support on-the-ground responses by firefighting teams and health professionals.The measurwement system uses small, low-energy computing platforms with off-the-shelf particulate sensors, integrated with low-bandwidth radio systems and powered by solar or battery sources where mains electricity is unavailable. These units are deployed across forests and urban areas, with data federated back to AIT for analysis.
In this episode of PING, APNIC Chief Scientist Geoff Huston returns with his annual review of BGP, reflecting on developments across 2025. Geoff has been publishing this year-in-review analysis of BGP dynamics for more than a decade, and this time he has uncovered some genuinely surprising shifts. His 2025 analysis has been published in two parts on the APNIC Blog.Border Gateway Protocol (BGP) is the mechanism by which network operators announce their Internet address space to the rest of the world and, in turn, learn about the addresses announced by others. Operators participating in the global default-free zone receive all publicly announced routes, each expressed as an IP prefix and associated with its originating Autonomous System Number (ASN). Every BGP speaker has a unique ASN, and all routing information is exchanged and interpreted through this fundamental identifier. In effect, the ASN is the basic unit of interdomain routing.BGP also carries path information that describes how routing announcements traverse the network. This data informs routing policy decisions — which paths to prefer, and through which commercial or technical relationships. While the protocol itself is well understood, the system as a whole is anything but simple. When more than 100,000 ASes are continuously exchanging routing information, complexity is unavoidable.Speaking BGP is about telling things and learning things, but it’s also about deciding what to do with what has been learned. This is the work behind a router, and involves holding all the information and performing routing decisions on it, so the ‘size’ of the information shared and learned has a direct impact on the ‘cost’ of operating as a BGP speaker (cost here ultimately means memory and CPU). For most of the Internet’s history, BGP growth has been relentless, forcing operators to continually ask whether their current routing infrastructure can accommodate future growth.All technology adoption has a life cycle, and is often referred to as the ‘technology adoption curve’. New technologies start out expensive and scarce, become cheaper and widely adopted, and eventually reach a point of saturation where growth slows and replacement becomes the dominant driver. For much of its existence, the Internet has remained firmly in the rapid growth phase of this curve, with sustained increases in users, networks, and routing information.Geoff has detected changes in the pace of growth for both IPv4 and IPv6, which suggest the underlying economics behind investment in Internet, and growth in customers has reached it’s saturation point: We are entering a time where BGP growth may not have the same dynamics we’ve been used to, and questions about capital investment in BGP routing and underlying Internet Addressing are not the same.
Welcome back to PING for 2026 and season 6.This time on PING, we have a pair of interviews with students from the National Institute of Technology Karnataka, Surathkal (NITK), recorded last year at IETF 122. This is the second time we've heard from students from NITK. We previously heard from Vanessa Fernandes and Kavya Bhat when they attended IETF 119 in 2024.NITK is a large, technically focused university located on India’s south-western coast in the state of Karnataka. The state is home to major technology hubs, such as Bengaluru and Mangaluru, alongside institutions like NITK, which play a key role in developing technical talent.Against this backdrop, it is unsurprising that NITK students show a strong interest in network technologies and Internet protocol development. Dr Mohit Tahiliani, Associate Professor at NITK, has led a multi-year program involving undergraduate, postgraduate, and postdoctoral researchers to engage with emerging Internet standards. Through this program, participants explore new ideas, contribute code, and take part in IETF hackathons and Working Group activities.This work has been supported in part by the APNIC Foundation.Last time with Vanessa and Kavya, we explored NITK’s multi-year campus IPv6 deployment, which has been underway for some time. That work has included direct engagement with the IETF, with Dr Mohit Tahiliani’s students attending alongside Nalini Eklins, who is involved both in the IPv6 deployment at NITK and in IPv6 standards work within the IETF.Since then, both students have gone on to work in networking roles or to pursue further study, reflecting the longer-term impact of sustained involvement in operational and standards-based Internet engineering.This time, we've got two different projects and NINE students to hear from.The first group is Rati Preethi Subramanian, Shriya Anil, Mahati Kalale, Anuhya Murki and Supradha Bhat, who explored fair queuing disciplines, FQ_Codel, a derivative FQ_Codel++ and a new proposed model, FQ_Pie. They worked with the NS3 network simulator and CCPerf, exploring how these queueing disciplines compare, and discussed their project with me at IETF 122.The second group are Vartika T Rao, Hayyan Arshad, Siddharth Bhat and Bharadwaja Meherrushi Chittapragada, who looked at the YANG data model in the network management space, and more efficient ways to manage data coming out of networking systems using YANG. They wrote a producer-consumer model in Python code, and explored time-series databases using interface packet count collections as an example YANG dataset to explore, in the CBOR encoding.Finally, I spoke with Dr Mohit Tahiliani, who has been leading this project. He is strongly committed to bringing new and younger voices into IETF work, recognizing the value of exposing students to real-world protocol development early in their careers. This experience benefits participants by grounding their learning in practical standards work, while also helping the IETF engage with new contributors who may return to protocol development in the future.This sustained engagement has already had tangible outcomes: The students involved have gone on to roles in the ICT sector or to further academic study, demonstrating the long-term value of this collaborative model.
In the final podcast for 2025, APNIC Chief Scientist Geoff Huston discusses the problem of independent measurement in an Internet which is increasingly “going dark”.Communications has always included a risk of snooping, and a matching component of work to enhance privacy, from the simplest ciphers used in ancient times, techniques of hiding and discovering messages, attempts to prevent and detect intrusion of the mail, to adoption of telegraph codes, the cutting of telegraph wires in wartime (to force messages into radio where they could be listened to) and the development of modern encryption algorithms typically using the public-private keypair model. There has always been a story of “attack” and “response” in how we communicate privately.Aside from matters of state security, banking and finance at large depend on a degree of privacy and now require it under legislation to enable use of creditcard information online. Many other contexts have an assumption of privacy, and use technology to try and preserve it. Fundamentally, individuals in their use of the Internet are entitled to expect a level of privacy where the state permits it.The publication of RFC7258 “Pervasive Monitoring Is an Attack in 2014 formalised a belief that the intrusion of third parties into a communication between two ends demanded a technology response to exclude them, where possible. Protocol designers and Internet Engineers took up the challenge.This position led over time to a marked increase in the adoption of privacy enhancing protocol features. For example, the web moved from HTTP: denoted URLs to HTTPS: where the content is protected by the Transport Layer Security (TLS) encryption protocol, which now overwhelmingly predominates in the at large.However, significant aspects of Internet communications “leak” information to third parties. Between an individual and a web service lies their provider, unknown numbers of intermediate providers, typically a content distribution system hosting the web site in a local copy, all of whom have opportunities to see and understand what is being done, and by whom. In particular the DNS typically exposes the name and address of the site being connected to across all kinds of protocols (not just the web) and exposes it to unknown intermediary systems as the DNS lookup is processed.In response to this, services are emerging which break down the DNS into dissociated queries: what is being looked for, and who is looking for it, and use intermediary services which may know one, but not both: Questions are seen to be asked, but by who is now hidden. If you know who is asking, you don’t know what they are asking for.Combined with newer network protocols like QUIC which imposes a strong end-to-end encryption model which even hides the inter-packet size and timing information (another form of leak which can be used to reconstruct what kind of traffic is flowing) it has become increasingly hard for an independent researcher to see inside the network: It’s going dark.Geoff explores the nature of privacy in the Internet at large, and how APNIC Labs gets round this problem with it’s measurement system.PING will return in January 2026 with another season of episodes. Until then, enjoy this final recording of 2025, and see you online, in the new year.
This time PING features Emile Aben from the RIPE NCC R&D Department. Emile is a Senior Research Engineer, and for over a decade and a half has been looking at Internet Measurement at RIPE in the Atlas system, and in the RIPE RIS BGP data collection.
Emile and a collaborator Romain Fontugne from IIJ Labs in Tokyo have been exploring a model of the influence and effect on global connectivity in BGP for different AS, based on the impact they have on other AS’s transit choices. They call this “AS Hegemony” and Emile has been using it to adjust for sample bias in the data being collected in RIPE RIS and in the Atlas network. This approach to re-balancing the sources helps Emile to understand changes in network topology and routing under rapid shocks like cable cuts and he’s been applying this to the recent spate of cable outages in the Baltic, around Africa, and the power outage on the Iberian Peninsula.
Emile has also been looking at new ways of holding data, and visualising data. His RIPE colleague Ties de Kok has explored use of “Parquet” as a data abstraction tool and this has allowed Emile to perform rapid analysis and experiment in new data visualisations in 3D, rendered in-browser.
Read more about AS Hegemony, and the new data visualisations on the web:
- How RIS Saw the Iberian Power Outage (RIPE90 plenary presentation, May 2025)
- A Deep Dive Into the Baltic Sea Cable Cuts (RIPE Labs, December 2024)
- AS Hegemony: A Robust Metric for AS Centrality (SIGCOMM 2017 poster)
In this episode of PING, APNIC Chief Scientist Geoff Huston explores the complex landscape of undersea cables. They have always had a component of strategic interest, communications and snooping on communications has been a constant since writing was invented, and the act of connecting two independent nation states by a telegraph wire invokes questions of ownership and jurisdiction right from the start.
After the initial physics of running a long distance wire to make an electric circuit was worked out, telegraph services became a vital part of a states economic and information gathering processes. This is why at the beginning of world war 1 and again in world war 2 the submarine cables linking europe out into the world were cut by the British Navy: forcing the communications flows into radio meant it was possible to listen in, and with luck (and some smart people) decode the signals.
Modern day fibre optic communications are no different in this regard. Many incidents of cable cutting have simple explanations, not all paths the subsea cables run through are especially deep and in shallow waters near landfall with lots of fish, trawlers cause a lot of damage. But there is now good reason to believe state actors are also disrupting fiber communications by breaking links, and a strong trend now to direct which sources of equipment (from the physical fibre up to the active routing systems) are used for a landfall into any given economy. This in turn is influencing the flow of capital, and the paths taken by subsea fibre systems, as a result of the competing pressures.
In this episode of PING, Shumon Huque from Salesforce discusses how protocols with extensible flag fields can benefit from regular testing of the values possible in the packet structure. This technique is known as "greasing" and has a strong metaphorical meaning of "greasing the wheels" to ensure future uses aren't blocked by mistaken beliefs about the possible values.
Intermediate systems (so-called "middleboxes") have to try and determine "risky" packetflows, and one of the mechanisms they use is to consider unexpected values in the known packetflows as possibly dangerous. This is an over-simplistic approach, and risks "ossifying" a protocol into the range of values which are actively in use now. Protocols usually include extra potential values for flag-fields, settings, options and the like, and these frequently have a large range of "reserved" values which are held in trust in an IANA registry, for future use. Greasing is a proposed mechanism to test out some of these values, and see what happens "on the wire" for the protocol in question.
Shumon and his co-author and collaborator Mark Andrews from ISC have been applying the greasing model to the DNS, and we talked about it's history in other protocols, and how in practice greasing can be applied on the global internet.
Read more about Shumon, Mark and Roy Arends' greasing activity on the web:
- DNS Grease (IETF draft, in the IETF Datatracker)
- the TLS DNSSEC Chain Extension ( IETF DANE WG, IETF RFC):
- DELEG Testing Report (with Roy Arends, DNSOP WG interim meeting presentation, IETF)
In this episode of PING, APNIC Chief Scientist Geoff Huston discusses a problem which cropped up recently with the location tagging of IP addresses seen in the APNIC Labs measurement system. For compiling national/economic and regional statistics, and to understand the experimental distribution into each market segment, Labs relies on the freely available geolocation databases from maxmind.com, and IPinfo.io -which in turn are constructed from a variety of sources such as BGP data, the RIR compiled resource distribution reports, Whois and RDAP declarations and the self-asserted RFC8805 format resource distribution statements that ISPs self publish.
At best this mechanism is an approximation, and with increasing mobility of IP addresses worldwide it has become harder to be confident in the specific location of an IP address you see in the source of an internet dataflow, not the least because of the increasing use of Virtual Private Networks (VPN) and address cloaking methods such as Apple Private Relay, or Cloudflare Warp (although as Geoff notes, these systems do the best they can to account for the geographic distribution of their users in a coarse grained “privacy preserving” manner).
Geoff was contacted by Ben Roberts of Digital Economy Kenya, a new boardmember of AFRINIC and long-time industry analyst and technical advisor. He’d noticed anomolies with the reporting of Internet statistics from Yemen, which simply could not be squared away with the realities of that segment of the Internet Economy. This in turn has lead Geoff to examine in detail the impact of Starlink on distribution of internet traffic, and make adjustments to his measurement Geolocation practices, which will become visible in the labs statistics as the smoothing functions work through the changes.
Low Earth Orbit (LEO) Space delivery of Internet has had rapid and sometimes surprising effects on the visibility of Internet worldwide. The orbital mechanics mean that virtually the entire surface of the globe is now fully internet enabled, albiet for a price above many in the local economy. This is altering the fundamentals of how we “see” Internet use and helps explain some of the problems which have been building up in the Labs data model.
Read more about Geolocation and Starlink on the APNIC Blog and on the web:
- Geolocation and Starlink (Geoff Huston, APNIC Blog September 2025)
- RFC8805 A Format for Self-Published IP Geolocation Feeds (IETF RFC website)
- The NRO RIR Statistics on delegations with geographic tagging of the delegated entity (NRO Website)
- Maxmind GeoIP resources (maxmind website)
- IPinfo.io (IPinfo website)
- Labs statistics portal (APNIC Labs website)
RSSAC047 - a document from the Root Server System Advisory Committee proposed a set of metrics to measure DNS root servers, and the DNS root server system as a whole. the document was approved in 2020, and ICANN worked on an implementation of the metrics as code, and a deployment into 20 points of measurement distributed worldwide.
ISC and Verisign, two of the root server operators proposed a review of this measurement and retained SIDN Labs (who are part of the Dutch body operating .NL as a CountryCode Top-Level Domain or ccTLD) to look into how well the measurement was performing.
In this episode of PING, Moritz Mullër from SIDN Labs and Duane Wessels from Verisign respectively, discuss this "measurement of the measurement" exercise, what they found out, and what it may mean for the future of metrics at the DNS Root.
It's an interesting "meta conversation" about measuring things which themselves are measurements. We see this all the time in the real world, for example diagnostic imaging machines designed to measure bone density (for osteoporosis checks) require calibration, and when you want to compare a baseline over time that calibration and the specific machine become questions the clinician may want to check, assessing the results. Change machine, you get different sensitivity. So how do you line up the data?
Moritz's investigations show that in some respects, the ICANN implementation of RSSAC047 was incomplete, and didn't tell an entirely accurate story about the state of the DNS Root Server System. Also, there are questions of scale and location which means a re-implementation or future improvement is worth discussing.
Read more about the DNS Root Server System, Moritz's report, and the RSSAC on the APNIC blog and on the web:
- Root-Servers.org website
- Monitoring highly distributed DNS deployments: Challenges and recommendations (APNIC Blog)
- RSSAC047: RSSAC Advisory on Metrics for the DNS Root Servers and the Root Server System (ICANN)
- SIDN Labs
- Verisign Labs
In this episode of PING, APNIC Chief Scientist Geoff Huston shares a story from the recent AusNOG in Melbourne and connects it to measurement work at APNIC Labs, exploring how modern IP flow control manages ‘fair shares’ of the network.
At AusNOG 2025, Geoff attended a talk by Lincoln Dale of Amazon AWS titled “No Packet Left Behind: AWS’s Approach to Building and Operating Reliable Networks”. The presentation examined how AWS scales its data centre networks, highlighting massive investments in high-speed routers and switches to support both global internet services and the vast flows of traffic between servers and other Amazon resources.
What AWS doesn’t do is rely on highly complex protocols like Segment Routing over IPv6 (SRv6), Resource Reservation Protocol (RSVP), or other modern traffic engineering techniques unless absolutely necessary. Instead, they use a radically simplified, on-chip model of data management, pushing as much processing as possible into a single VLSI circuit and minimizing the amount of ‘smart’ work in the network. The question is: How can simplifying the IP stack to this extent actually work?
Geoff has long been sceptical of higher-layer protocols that try to manage bandwidth reservation and shaping. He recalls an earlier attempt by Digital Equipment Corporation (DEC) to signal congestion with Explicit Congestion Notification (ECN), a mechanism that still exists in the protocol stack and now underpins new bandwidth management approaches such as Apple and Comcast’s ‘L4S’.
APNIC Labs has measured how the wider Internet responds to ECN signals using an advertising-based model, and the results suggest this approach struggles outside tightly controlled, ‘walled garden’ networks. He contrasts this with advances in flow control through Google’s BBR, now in its third version, which refines the aggressive, bandwidth-seeking behaviour of TCP window management.
Read more about the story of IP, flow control and the modern Internet on the APNIC Blog, and the AusNOG website (video recordings of Lincon Dale’s talk and others should be released shortly)
- Measuring Explicit Congestion Notification (ECN) (Geoff Huston, APNIC Blog)
- Notes from AusNOG 2025 (Geoff Huston, APNIC Blog)
- The AusNOG 2025 program (AusNOG Website, videos to be released shortly)
In this episode of PING, Adli Wahid, APNIC's Security Specialist discusses the APNIC honeypot network, an investment in over 400 collectors distributed throughout the Asia Pacific, collecting data on who is trying to break into systems online and use them for malware, destributed denial of service, and command-and-control systems in the bad traffic economy.
Adli discusses how APNIC Members can get access to the results of honeynet traffic capture coming from their network address ranges, and originated from their AS in BGP using the DASH system. and explores some work planned for the APNIC Honeynet systems to extend their systems coverage.
As well as publishing reports on APNIC's Blog and presenting at NOG meetings and conferences, Adli has coordinated information sharing from this collector network with a range of international partners such as the Shadow Server Foundation. He continues to offer training and technical assistance in security to the APNIC community and works with the CERT, CSIRT and FIRST community at large.
Read more about Honeypots, bad traffic and systems security on the APNIC Blog and the web:
- Blogs on the honeynet (APNIC Blog)
- Adli's posts on the APNIC Blog
- The APNIC Dashboard for AS Health (DASH) (requires an APNIC member account)
- The Shadow Server Foundation dashboard
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses the economic inevitability of centrality, in the modern Internet. Despite our best intentions, and a lot of long standing belief amongst the IETF technologists, no amount of open standards and end-to-end protocol design prevents large players at all levels of the network (from the physical infrastructure right up to the applications and the data centres which house them) from seeking to acquire smaller competitors, and avoid sharing the space with anyone else.
Some of this is a consequence of the drive for efficiency. A part has been fuelled by the effects of Moore’s law, and the cost of capital investment against the time available to recover the costs. In an unexpected outcome, networking has become (to all intents and purposes) “free” and instead of end-to-end, we now routinely expect to get data through highly localised, replicated sources. The main cost these days is land, electric power and air-conditioning. This causes a tendency to concentration, and networks and protocols play very little part in the decision about who acquires these assets, and operates them.
The network still exists of course, but increasingly data flows over private links, and is not subject to open protocol design imperatives.
A quote from Peter Thiel highlights how the modern Venture Capitalist in our space does not actively seek to operate in a competitive market. As Peter says: “competition is for losers” – It can be hard to avoid the “good” and “bad” labels talking about this, but Geoff is clear he isn’t here to argue what is right or wrong, simply to observe the behaviour and the consequences.
Geoff presented on centrality to the Decentralised Internet Research Group or DINRG at the recent IETF meeting held in Madrid, and as he observes, “distributed” is not the same as “decentralised” -we’ve managed to achieve the first one, but the second eludes us.
Read more about the policy issues of the modern Internet at the apnic labs blog, the DINRG (IETF) and APNIC Blog
- Decentralizing Services? (Geoff Huston, talk to DINRG IETF123 Madrid)
- Centralization topics at the APNIC Blog
- DINRG at the IETF Wiki (IETF web page)
In this episode of PING, Robert Kisteleki from the RIPE NCC discusses the RIPE Atlas system -a network of over 13,000 measurement devices deployed worldwide in homes, exchange points, stub and transit AS, densely connected regions and sparse island states.
Atlas began with a vision of the world at night -a powerful metaphor for where people are, and where technology reaches. Could a measurement system achieve sufficient density to "light up the internet" in a similar manner? Could network measurement be "democratized" to include internet citizens at large?
From it's launch at the RIPE 61 meeting held in Rome Italy. with 500 probes based on a small ucLinux device designed as an ethernet converter, to 5 generations of probe hardware and now a soft probe design which can be installed on linux, and an "anchor" device which not only sends tests but can receive them, Atlas has become core technology for network monitoring, measurement and research.
Rob discusses the history, design, methodology and futures of this system. A wonderful contribution from the RIPE NCC for the community at large.
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses "a day in the life of BGP" -Not an extraordinary day, not a special day, just the 8th of May.
What happens inside the BGP system, from the point of view of AS4608, one ordinary BGP speaker on the edge of the network? What kinds of things are seen, and why are they seen?
Geoff has been measuring BGP for almost it's entire life as the internet routing protocol, but this time looks at the dynamics at a more "micro" level than usual. In particular there are some things about the rate of messages and changes which points to the problems BGP faces. A small number of BGP speakers produce the vast majority of change, and overall the network information BGP speakers have to deal with as a persisting view of the world increases more slowly. Both kinds of message dynamics have to be dealt with.
Can we fix this? Is there even anything worth fixing here, or is BGP just doing fine?
Read more about the dynamics of BGP on the APNIC Blog:
- A Day in the life of BGP (Geoff Huston June 2025 APNIC Blog)
- BGP topics at the APNIC Blog
- Geoff's archive of BGP data at bgp.potaroo.net
In this episode of PING, Doug Madory from Kentik discusses his rundown of the state of play in secure BGP across 2024 and 2025. Kentik has it’s own internal measurements of BGP behaviour and flow data across the surface of the internet, which combined with the Oregon University curated routeviews archive means Doug can analyse both the publicly visible state of BGP from archives, and Kentik’s own view of the dynamics of BGP change, along side other systems like the worldwide RPKI model, and the Internet Routing Registry systems.
Doug has written about this before on the APNIC Blog in May of 2024.
RPKI demands two outcomes, Firstly that the asset holders who control a given range of Internet Address sign an intent regarding who originates it the ROA, and secondly that the BGP speakers worldwide implement validation of the routing they see, known as Route Origin Validation or ROV. ROA signing is easy, and increases very simply if the delegate uses an RIR hosted system to make the signed objects. ROV is not always simple and has to be deployed carefully so has a slower rate of deployment, and more consequence in costs to the BGP speaker. Doug has been tracking both independently, as well as looking at known routing incidents in the default free zone, and therefore the impact on RPKI active networks, and everywhere else.
Read more about RPKI and BGP on the APNIC Blog, the web, and at Doug’s own blogging at Kentik:
- RPKI ROV reaches a Major Milestone (APNIC Blog, May 2024)
- Blog Articles by Doug Madory on the APNIC Blog
- The Oregon Routeviews Project
- Doug Madory’s blog posts at Kentik
- A shorter interview with Doug Madory on AS_SET problems features in an earlier PING episode, recorded at the ISOC Pulse “PIMF” session at APRICOT 2025.
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses the root zone of the DNS, and some emerging concerns in how much it costs to service query load at the root.
In the absence of cacheing, all queries in the DNS (except ones the DNS system you ask is locally authoritative for anyway) have to be sent through the root of the DNS, to find the right nameserver to ask for the specific information. Thanks to cacheing, this system doesn't drown in the load of every worldwide query, all the time, going through the root. But, even taking cacheing into account there is an astronomical amount of query seen at the root, and it has two interesting qualities
Firstly, its growing significantly faster than the normal rate of growth of the Internet. We're basically at small incremental growth overall in new users, but query load at the root increases significantly faster, even after some more unexpected loads have been reduced.
Secondly, almost all of the queries demand the answer "No, that doesn't exist" and the fact most traffic to the root hunts the answer NO means that the nature of distributed DNS cacheing of negative answers isn't addressing the fundamental burden here.
Geoff thinks we may be ignoring some recent developments in proving the contents of a zone, the ZONEMD record which is a DNSSEC signed check on the entire zone contents, and emerging systems to download the root zone, and localise all the queries sent onwards into a copy of the root held in the resolver.
Basically, "can we do better" -And Geoff thinks, we very probably can.
Read more about the economics of the root zone and ZONEMD at the APNIC Blog and on the web:
- The Root of the DNS (Geoff Huston, APNIC Blog March 2025)
- ZoneMD: Message digest for DNS Zones RFC8976 (IETF RFC)
In this episode of PING, We’re talking to Leslie Daigle from the Global Cyber Alliance (GCA) again, discussing GCA’s honeynet project. Leslie spoke with PING back in January 2024, and in this episode we re-visit things. Honeynets (or Honey farms) are deliberately weakly protected systems put online, to see what kinds of bad traffic exist out in the global Internet, where they come from and what kinds of attack they are mounting.
In the intervening period GCA has continued to develop its honeyfarm, building out it’s own systems images, and can now capture more kinds of bad traffic. They have also bedded in the MANRS community, which is now supported by GCA worldwide.
In this episode, Leslie is actually asking more questions than providing answers. If we accept that there is now a persisting problem at scale, what kinds of approaches do we need to take to “get on top” of bad traffic? It used to be we thought of this in terms of technical solutions but increasingly Leslie feels we now need to broaden the conversation and take this into Public policy and governance communities, to understand what kinds of social cost we can bear, and what socially driven objectives we want to drive to. The problem is, this is one of the tasks technologists are often the least equipped to do: Talk to people.
GCA is showcasing the AIDE system, reachable at https://gcaaide.org/ as a way of opening up the conversation with national strategic policy makers, and the wider community. It’s a simple economy & region model summarising the state of honeynet detected bad traffic levels worldwide, and helps to set an agenda with which the individual ISPs and routing-active community can engage, for their locus of control.
Read more about GCA, Honey nets, AIDE on the APNIC blog and the web:
- The invisible War: Why securing Internet Traffic is Everyone’s Responsibility (Leslie Daigle, APNIC Blog May 2025)
- Global Cyber Alliance measurements (Podcast, PING at APNIC Jan 2024)
- Dealing with the undercurrent of unwanted Traffic (Leslie Daigle, APNIC Blog October 2022)
- AIDE: Addressing unwanted Internet Traffic at it’s source (GCA Website)
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, revisits changes underway in how the Domain Name System (DNS) delegates authority over a given zone and how resolvers discover the new authoritative sources. We last explored this in March 2024.
In DNS, the word ‘domain’ refers to a scope of authority. Within a domain, everything is governed by its delegated authority. While that authority may only directly manage its immediate subdomains (children), its control implicitly extends to all subordinate levels (grandchildren and beyond). If a parent domain withdraws delegation from a child, everything beneath that child disappears. Think of it like a Venn diagram of nested circles — being a subdomain means being entirely within the parent’s scope.
The issue lies in how this delegation is handled. It’s by way of nameserver (NS) records. These are both part of the child zone (where they are defined) and the parent zone (which must reference them). This becomes especially tricky with DNSSEC. The parent can’t authoritatively sign the child’s NS records because they are technically owned by the child. But if the child signs them, it breaks the trust chain from the parent.
Another complication is the emergence of third parties to the delegate, who actually operate the machinery of the DNS. We need mechanisms to give them permission to make changes to operational aspects of delegation, but not to hold all the keys a delegate has regarding their domain name.
A new activity has been spun up in the IETF to discuss how to alter this delegation problem by creating a new kind of DNS record, the DELEG record. This is proposed to follow the Service Binding model defined in RFC 9460. Exactly how this works and what it means for the DNS is still up in the air.
DELEG could fundamentally change how authoritative answers are discovered, how DNS messages are transported, and how intermediaries interact with the DNS ecosystem. In the future, significant portions of DNS traffic might flow over new protocols, introducing novel behaviours in the relationships between resolvers and authoritative servers.
Read more about DELEG on the APNIC Blog and the web:
- DNS and the proposed DELEG record (APNIC Blog, February 2024)
- DELEG Working Group Charter (IETF Website)
- Service Binding and Parameter Specification via the DNS (IETF RFC 9460)
In this episode of PING, Professor Cristel Pelsser who holds the chair of critical embedded systems at UCLouvain Discusses her work measuring BGP and in particular the system described in the 2024 SIGCOMM “best paper” award winning research: “The Next Generation of BGP Data Collection Platforms”
Cristel and her collaborators Thomas Alfroy, Thomas Holterbach, Thomas Krenc and K. C. Claffy have built a system they call GILL, available on the web at https://bgproutes.io This work also features a new service called MVP, to help find the “most valuable vantage point” in the BGP collection system for your particular needs. GILL has been designed for scale, and will be capable of encompassing thousands of peerings. it also has an innovative approach to holding BGP data, focussed on the removal of demonstrably redundant information, and therefore significantly higher compression of the data stream compared to e.g. holding MRT files.
The MVP system exploits machine learning methods to aide in the selection of the most advantageous data collection point reflecting a researchers specific needs. Application of ML methods here permits a significant amount of data to be managed and change reflected in the selection of vantage points.
Their system has already been able to support DFOH, an approach to finding forged origin attacks from peering relationships seen online in BGP, as opposed to the peering expected both from location, and declarations of intent inside systems like peeringDB.
Read more about Cristel’s work, and their BGP analysis tools on the web:
- The Next Generation of BGP Data Collection Platforms (Best Paper Award at ACM SIGCOMM 2024)
- bgproutes.io (web portal to GILL, MVP and DFOH systems)
- Measuring Internet Routing from the Most Valuable Points
- A system to Detect Forged-Origin Hijacks (DFOH)
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses the history and emerging future of how Internet protocols get more than the apparent link bandwidth by using multiple links and multiple paths.
Initially, the model was quite simple, capable of handling up to four links of equal cost and delay reasonably well, typically to connect two points together. At the time, the Internet was built on telecommunications services originally designed for voice networks, with cabling laid between exchanges, from exchanges to customers, or across continents. This straightforward technique allowed the Internet to expand along available cable or fibre paths between two points. However, as the system became more complex, new path options emerged, and bandwidth demands grew beyond the capacity of individual or even equal-cost links, increasingly sophisticated methods for managing these connections had to be developed.
An interesting development at the end of this process is the impact of a fully encrypted transport layer on the intervening infrastructure’s ability to manage traffic distribution across multiple links. With encryption obscuring the contents of the dataflow, traditional methods for intelligently splitting traffic become less effective. Randomly distributing data can often worsen performance, as modern techniques rely on protocols like TCP to sustain high-speed flows by avoiding data misordering and packet loss.
This episode of PING explores how Internet protocols boost bandwidth by using multiple links and paths, and how secure transport layers affect this process.
Read more about multipath network protocols on the web:
- IETF Draft on Multipath for QUIC (IETF, April 2025)
- Multipath TCP: Revolutionising connectivity one path at a time (Cloudflare Blog, January 2025)
- RFC 8684 (IETF, 2020)
Last month, during APRICOT 2025 / APNIC 59, the Internet Society hosted its first Pulse Internet Measurement Forum (PIMF). PIMF brings together people interested in Internet measurement from a wide range of perspectives — from technical details to policy, governance, and social issues. The goal is to create a space for open discussion, uniting both technologists and policy experts.
In this second special episode of PING, we continue our break from the usual one-on-one podcast format and present a recap of why the PIMF forum was held, and the last 3 short interviews from the workshop.
First we hear a repeat of Amreesh Phokeer's presentation. Amreesh is from the Internet Society and discusses his role in managing the Pulse activity within ISOC. Alongside Robbie Mitchell, Amreesh helped organize the forum, aiming to foster collaboration between measurement experts and policy professionals.
Next we hear from Beau Gieskens, a Senior Software Engineer from APNIC Information Products. Beau has been working on the DASH system and discusses his PIMF presentation on a re-design to an event-sourcing model which reduced database query load and improved speed and scaling of the service.
We then have Doug Madory from Kentik who presented to PIMF on a quirk in how Internet Routing Registries or IRR are being used, which can cause massive costs in BGP filter configuration and is related to some recent route leaks being seen at large in the default free zone of BGP.
Finally, we hear from Lia Hestina from the RIPE NCC Atlas project. Lia is the community Development officer, and focusses on Asia Pacific and Africa for the Atlas project. Lia discusses the Atlas system and how it underpins measurements worldwide, including ones discussed in the PIMF meeting.
For more insights from PIMF, be sure to check out the PULSE Forum recording on the Internet Society YouTube feed
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses the surprisingly vexed question of how to say ‘no’ in the DNS. This conversation follows a presentation by Shumon Huque at the recent DNS OARC meeting, who will be on PING in a future episode talking about another aspect of the DNS protocol.
You would hope this is a simple, straightforward answer to a question, but as usual with the DNS, there are more complexities under the surface. The DNS must indicate whether the labels in the requested name do not exist, whether the specific record type is missing, or both. Sometimes, it needs to state both pieces of information, while other times, it only needs to state one.
The problem is made worse by the constraints of signing answers with DNSSEC. There needs to be a way to say ‘no’ authoritatively, and minimize the risk of leaking any other information.
NSEC3 records are designed to limit this exposure by making it harder to enumerate an entire zone. Instead of explicitly listing ‘before’ and ‘after’ labels in a signed response denying a label’s existence, NSEC3 uses hashed values to obscure them. In contrast, the simpler NSEC model reveals adjacent labels, allowing an attacker to systematically map out all existing names — a serious risk for domain registries that depend on name confidentiality. This is documented in RFC 7129.
Saying ‘no’ with authority also raises the question of where signing occurs — at the zone’s centre (by the zone holder) or at the edge (by the zone server). These approaches lead to different solutions, each with its own costs and consequences.
In this episode of PING, Geoff explores the differences between a non-standard, vendor-explored solution, and the emergence of a draft standard in how to say ‘no’ properly.
At the APRICOT/APNIC59 meeting held in Petaling Jaya in Malaysia last month, The internet society held it's first PIMF meeting. PIMF, or the Pulse Internet Measurement Forum is a gathering of people interested in Internet measurement in the widest possible sense, from technical information all the way to policy, governance and social questions. ISOC is interested in creating a space for the discussion to take place amongst the community, and bring both technologists and policy specialists into the same room.
This time on PING, instead of the usual one-on-one format of podcast we've got 5 interviews from this meeting, and after the next episode from Geoff Huston at APNIC Labs we'll play a second part, with 3 more of the presenters from this session.
First up we have Amreesh Phokeer from the Internet Society who manages the PULSE activity in ISOC, and along with Robbie Mitchell set up the meeting.
Then we hear from Christoph Visser from IIJ Labs in Tokyo, who presented on his measurements of the "Steam" Game distribution platform used by Valve Software to share games. It's a complex system of application-specific source selection, using multiple Content Distribution Networks (CDN) to scale across the world, and allows Christoph to see into the link quality from a public API. No extra measurements required, for an insight into the gamer community and their experience of the Internet.
The third interview is with Anand Raje, from AIORI-IMN, India’s Indigenous Internet Measurement System. Anand leads a team which has built out a national measurement system using IoT "orchestration" methods to manage probes and anchors, in a virtual-environment which permits them to run multiple independent measurement systems hosted inside their platform.
After this there's an interview with Andre Robachevsky from Global Cyber Alliance (GCA). Andre established the MANRS system, it's platform and nurtured the organisation into being inside ISOC. MANRS has now moved into the care of GCA and Andre moved with it, and discusses how this complements the existing GCA activities.
FInally we have a conversation with Champika Wijayatunga from ICANN on the KINDNS project. This is a programme designed to bring MANRS-like industry best practice to the DNS community at large, including authoritative DNS delegates and the intermediate resolver and client supporting stub resolver operators. Champika is interested in reaching into the community to get KINDNS more widely understood and encourage its adoption with over 2,000 entities having completed the assessment process already.
Next time we'll here from three more participants in the PIMF session: Doug Madory from Kentik, Beau Gieskins from APNIC Information Products, and Lia Hestina, from the RIPE NCC.
- PULSE Forum recording (Internet Society YouTube feed)
In this episode of PING, APNIC’s Chief Scientist, Geoff Huston explores bgp "Zombies" which are routes which should have been removed, but are still there. They're the living dead of routes. How does this happen?
Back in the early 2000s Gert Döring in the RIPE NCC region was collating a state of BGP for IPv6 report, and knew each of the 300 or so IPv6 announcements directly. He understood what should be seen, and what was not being routed. He discovered in this early stage of IPv6 that some routes he knew had been withdrawn in BGP still existed when he looked into the repositories of known routing state. This is some of the first evidence of a failure mode in BGP where withdrawal of information fails to propagate, and some number of BGP speakers do not learn a route has been taken down. They hang on to it.
Because BGP is a protocol which only sends differences to the current routing state as and when they emerge (if you start afresh you get a LOT of differences, because it has to send everything from ground state of nothing. But after that, you're only told when new things come and old things go away) it can go a long time without saying anything about a particular route: if its stable and up, nothing to say, and if it was withdrawn, you don't have it, to tell people it's gone, once you passed that on. So if somehow in the middle of this conversation a BGP speaker misses something is gone, as long as it doesn't have to tell anyone it exists, nobody is going to know it missed the news.
In more recent times, there has been a concern this may be caused by a problem in how BGP sits inside TCP messages and this has even led to an RFC in the IETF process to define a new way to close things out.
Geoff isn't convinced this diagnosis is actually correct or that the remediation proposed is the right one. From a recent NANOG presentation Geoff has been thinking about the problem, and what to do. He has a simpler approach which may work better.
Read more about BGP zombies at the APNIC Blog and the web:
- BGP Zombies at NANOG 93 (Geoff Huston, APNIC Blog February 2025)
- NANOG 93 presentation on BGP Zombies (Iliana Xygkou from Thousand Eyes, NANOG presentation)
- RFC9687 SendHold Timers (IETF RFC)
In this episode, Job Snijders discusses RPKIViews, his long term project to collect the "views" of RPKI state every day, and maintain an archive of BGP route validation states. The project is named to reflect route views, the long-standing archive of BGP state maintained by the University of Oregon, which has been discussed on PING.
Job is based in the Netherlands, and has worked in BGP routing for large international ISPs and content distribution networks as well as being a board member of the RIPE NCC. He is known for his work producing the Open-Source rpki-client RPKI Validator, implemented in C and distributed widely through the OpenBSD project.
RPKI is the Resource PKI, Resource meaning the Internet Number Resources, the IPv4, IPv6 and Autonomous System (AS) numbers which are used to implement routing in the global internet. The PKI provides cryptographic proofs of delegation of these resources and allows the delegates to sign over their intentions originating specific prefixes in BGP, and the relationships between the AS which speak BGP to each other.
Why rpkiviews? Job explains that there's a necessary conversation between people involved in the operational deployment of secure BGP, and the standards development and research community: How many of the worlds BGP routes are being protected? How many places are producing Route Origin Attestations (ROA) which are the primary cryptographic object used to perform Route Origin Validation (ROV) and how many objects are made? Whats the error rate in production, the rate of growth, a myriad of introspective "meta" questions need to be asked in deploying this kind of system at scale, and one of the best tools to use, is an archive of state, updated frequently, and as for route views collected from a diverse range of places worldwide, to understand the dynamics of the system.
Job is using the archive to produce his annual "RPKI Year in review" report, which was published this year on the APNIC Blog (it's posted to operations, research and standards development mailing lists and presented at conferences and meetings normally) and products are being used by the BGPAlerter service developed by Massimo Candela
Read about the rpkiviews archive on the APNIC Blog, and on the web:
- RPKI's 2024 Year in review - (Job Snijders, APNIC Blog January 2025)
- RPKIViews - (the RPKI views Web archive)
In his first episode of PING for 2025, APNIC’s Chief Scientist, Geoff Huston returns to the Domain Name System (DNS) and explores the many faces of name servers behind domains. Up at the root, (the very top of the namespace, where all top-level domains like .gov or .au or .com are defined to exist) there is a well established principle of 13 root nameservers. Does this mean only 13 hosts worldwide service this space? Nothing could be farther from the truth! literally thousands of hosts act as one of those 13 root server labels, in a highly distributed worldwide mesh known as "anycast" which works through BGP routing.
The thing is, exactly how the number of nameservers for any given domain is chosen, and how resolvers (the querying side of the DNS, the things which ask questions of authoritative nameservers) decide which one of those servers to use isn't as well defined as you might think. The packet sizes, the order of data in the packet, how it's encoded is all very well defined, but "which one should I use from now on, to answer this kind of question" is really not well defined at all.
Geoff has been using the Labs measurement system to test behaviour here, and looking at basic numbers for the delegated domains at the root. The number of servers he sees, their diversity, the nature of their deployment technology in routing is quite variable. But even more interestingly, the diversity of "which one gets used" on the resolver side suggests some very old, out of date and over-simplistic methods are still being used almost everywhere, to decide what to do.
Read more about Geoff's research on DNS nameserver selection and diversity on the APNIC Blog:
- DNS nameservers: Service performance and resilience (Geoff Huston, APNIC Blog February 2025)
Welcome back to PING, at the start of 2025. In this episode, Gautam Akiwate, (now with Apple, but at the time of recording with Stanford University) talks about the 2021 Advanced Network Research Prize winning paper, co-authored with Stefan Savage, Geoffrey Voelker and Kimberly Claffy which was titled "Risky BIZness: Risks Derived from Registrar Name Management".
The paper explores a situation which emerged inside the supply chain behind DNS name delegation, in the use of an IETF protocol called Extensible Provisioning Protocol or EPP. EPP is implemented in XML over the SOAP mechanism, and is how registry-registrar communications take place, on behalf of a given domain name holder (the delegate) to record which DNS nameservers have the authority to publish the delegated zone. The problem doesn't lie in the DNS itself, but in the operational practices which emerged in some registrars, to remove dangling dependencies in the systems when domain names were de-registered. In effect they used an EPP feature to rename the dependency, so they could move on with selling the domain name to somebody else.
The problem is that feature created valid names, which could themselves then be purchased. For some number of DNS consumers, those new valid nameservers would then be permitted to serve the domain, and enable attacks on the integrity of the DNS and the web.
Gautam and his co-authors explored a very interesting quirk of the back end systems and in the process helped improve the security of the DNS and identified weaknesses in a long-standing "daily dump" process to provide audit and historical data.
Read more about RISKY BIZness and the supply chain attack on the web:
- The 2021 ANRP paper "Risky BIZness: Risks Derived from Registrar Name Management"
- 2017 Grand Jury indictment of Zhang et al
- 2022 IMC paper "Retroactive Identification of Targeted DNS Infrastructure Hijacking
- The prevalence, persistence, and perils of lame delegations (APNIC blog, 2021)
In the last episode of PING for 2024, APNIC’s Chief Scientist Geoff Huston discusses the shift from existing public-private key cryptography using the RSA and ECC algorithms to the world of ‘Post Quantum Cryptography. These new algorithms are designed to withstand potential attacks from large-scale quantum computers and are capable of implementing Shor’s algorithm, a theoretical approach for using quantum computing to break the cryptographic keys of RSA and ECC.
Standards agencies like NIST are pushing to develop algorithms that are both efficient on modern hardware and resistant to the potential threats posed by Shor’s Algorithm in future quantum computers. This urgency stems from the need to ensure ‘perfect forward secrecy’ for sensitive data — meaning that information encrypted today remains secure and undecipherable even decades into the future.
To date, maintaining security has been achieved by increasing the recommended key length as computing power improved under Moore’s Law, with faster processors and greater parallelism. However, quantum computing operates differently and will be capable of breaking the encryption of current public-private key methods, regardless of the key length.
Public-private keys are not used to encrypt entire messages or datasets. Instead, they encrypt a temporary ‘ephemeral’ key, which is then used by a symmetric algorithm to secure the data. Symmetric key algorithms (where the same key is used for encryption and decryption) are not vulnerable to Shor’s Algorithm. However, if the symmetric key is exchanged using RSA or ECC — common in protocols like TLS and QUIC when parties lack a pre-established way to share keys — quantum computing could render the protection ineffective. A quantum computer could intercept and decrypt the symmetric key, compromising the entire communication.
Geoff raises concerns that while post-quantum cryptography is essential for managing risks in many online activities — especially for protecting highly sensitive or secret data—it might be misapplied to DNSSEC. In DNSSEC, public-private keys are not used to protect secrets but to ensure the accuracy of DNS data in real-time.
If there’s no need to worry about someone decoding these keys 20 years from now, why invest significant effort in adapting DNSSEC for a post-quantum world? Instead, he questions whether simply using longer RSA or ECC keys and rotating key pairs more frequently might be a more practical approach.
Read more about Post-Quantum Cryptography and DNSSEC on the APNIC blog and the web.
- Post-Quantum Cryptography (Geoff Huston, APNIC Blog November 2024)
- [Podcast] Testing Post-Quantum Cryptography DNSSEC (Podcast July 2024)
- A quantum-safe cryptography DNSSEC testbed (Caspar Schutijser, APNIC Blog 2024)
- [Podcast] The SIDN Labs post-quantum DNSSEC testbed (Podcast August 2024)
- Quantum Computing and the DNS (Paul Hoffman, office of the CTO, ICANN April 2024)
PING will return in early 2025This is the last episode of PING for 2024, we hope you’ve enjoyed listening. The first episode of our new series is expected in late January 2025. In the meantime, catch up on all past episodes.
This time on PING, Peter Thomassen from SSE and DEsec.io discusses his analysis of the failure modes of CDS and CDNSKEY records between parent and child in the DNS. These records are used to provide in-band signalling of the DS record, fundamental to the maintenance of a secure path from the trust anchor to the delegation through all the intermediate parent and grandparent domains. Many people use out-of-band methods to update this DS information, but the CDS and the CDNSKEY records are designed to signal this critical information inside the DNS, avoiding many of the pitfalls of passing through a registry-registrar web service.
The problem is, as Peter has discovered, the information across the various nameservers (denoted by the NS record in the DNS) of the child domain can get out of alignment, and the tests a parent zone need to do checking CDS and CDNSKEY information aren't sufficiently specified to wire down this risk.
Peter performed a "meta analysis" inside a far larger cohort of DNS data captured by Florian Steurer and Tobias Fiebig at the Max Planck Institute and discovered a low but persisting error rate, a drift in the critical keying information between a zones NS and the parent. Some of these related to transitional states in the DNS (such as when you move registry or DNS provider) but by no means all, and this has motivated Peter and his co-authors to look at improved recommendations for managing CDS/CDNSKEY data, to minimise the risk of inconsistency, and the consequent loss of secure entry path to a domain name.
Read more about DNSSEC delegation at the APNIC Blog, and the IETF:
- Authenticated bootstrapping of DNSSEC delegations (NIls Wisiol, APNIC Blog March 2022)
- Measurement of CDS/CDNSKEY inconsistencies (IETF119 Presentation, March 2024)
- Generalised DNS NOTIFY (IETF Draft)
In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston discusses the slowdown in worldwide IPv6 uptake. Although within the Asia-Pacific footprint we have some truly remarkable national statistics, such as India which is now over 80% IPv6 enabled by APNIC Labs measurements, And Vietnam which is not far behind on 70% the problem is that worldwide, adjusted for population and considering levels of internet penetration in the developed economies, the pace of uptake overall has not improved and has been essentially linear since 2016. In some economies like the US, a natural peak of around 50% capability was reached in 2017 and since then uptake has been essentially flat: There is no sign of closure to a global deployment in the US, and many other economies.
Geoff takes a high level view of the logisitic supply curve with the early adopters, early and late majority, and laggards, and sees no clear signal that there is a visible endpoint, where a transition to IPv6 will be "done". Instead we're facing a continual dual-stack operation of both IPv4 (increasingly behind Carrier Grade Nats (CGN) deployed inside the ISP) and IPv6.
There are success stories in mobile (such as seen in India) and in broadband with central management of the customer router. But, it seems that with the shift in the criticality of routing and numbering to a more name-based steering mechanism and the continued rise of content distribution networks, the pace of IPv6 uptake worldwide has not followed the pattern we had planned for.
Read more about the IPv6 transition at the APNIC Blog
- The IPv6 Transition (Geoff Huston, APNIC Blog November 2024)
- The Transition to IPv6 are we there yet (Geoff Huston, APNIC Blog May 2022)
In this episode of PING, Vanessa Fernandez and Kavya Bhat, two students from the National Institute of Technology Karnataka (NITK) discuss the student led, multi-year project to deploy IPv6 at their campus. Kavya & Vanessa have just graduated, and are moving into their next stages of work and study in computer sciences and network engineering.
Across 2023 and 2024 they were able to attend IETF118 and IETF119 and present on their project and it’s experiences to the IPv6 working groups and off-Working Group meetings, in part funded by the APNIC ISIF Project and the APNIC Foundation.
This multi-year project is supervised by the NITK Centre for Open-source Software and Hardware (COSH) and has outside review from Dhruv Dhody (ISOC) and Nalini Elkins (Inside Products inc). Former students have also acted as alumni and remain involved in the project as it progresses.
We often focus on IPv6 deployment at scale in the telco sector, or experiences with small deployments in labs, but another side of the IPv6 experience is the large campus network, in scale equivalent to a significant factory or government department deployment but in this case undertaken by volunteer staff, with little or no prior experience of networking technology. Vanessa and Kavya talk about their time on the project, and what they got to present at IETF.
Read more information on the NITK and their IPv6 deployment project on the APNIC Blog, the IETF website and the APNIC Foundation pages:
- Migrating the NITK Surathkal Campus Network to IPv6 (APNIC Foundation)
- How Deploying IPv6 at NITK Led me to IETF (Vanessa Fernandez, APNIC Blog)
- IPv6 Deployment at NITK (IETF118 Presentation)
In his regular monthly spot on PING, APNIC’s Chief Scientist, Geoff Huston, discusses a large pool of IPv4 addresses left in the IANA registry, from the classful allocation days back in the mid 1980s. This block, from 240.0.0.0 to 255.255.255.255 encompasses 268 million hosts, which is a significant chunk of address space: it's equivalent to 16 class-A blocks, each of 16 million hosts. Seems a shame to waste it, how about we get this back into use?
Back in 2007 Geoff Paul and myself submitted An IETF Draft which would have removed these addresses from the "reserved" status in IANA and used to supplement the RFC1918 private use block. We felt at the time this was the best use of these addresses because of their apparent un-routability, in the global internet. Almost all IP network stacks at that time shared a lineage with the BSD network code developed at the University of California, and released in 1983 as BSD4.2. Subsequent versions of this codebase included a 2 or 3 line rule inside the Kernel which checked the top 4 bits of the 32 bit address field, and refused to forward packets which had these 4 bits set. This reflected the IANA status marking this range as reserved. The draft did not achieve consensus.
A more recent proposal has emerged from Seth Schoen, David Täht and John Gilmore in 2021 which continues to be worked on, but rather than assigning to RFC1918 internal non-routable puts the address into global unicast use. The authors believe that the critical filter in devices has now been lifted, and no longer persists at large in the BSD and Linux derived codebases. This echoes use of the address space which has been noted inside the Datacentre.
Geoff has been measuring reachability at large to this address space, using the APNIC Labs measurement system and a prefix in 240.0.0.0/4 temporarily assigned and routed in BGP. The results were not encouraging, and Geoff thinks routability of the range remains a very high burden.
Read more about 240/4 in the APNIC Blog, and the IETF Datatracker website:
- Looking for 240/4 addresses (Geoff Huston, APNIC Blog September 2024)
- Re-delegation of 240/4 from "future use" to "private use" (expired IETF draft, 2008)
- Unicast use of the formerly reserved 240/4 (active IETF draft, 2024)
In this episode of PING, Nowmay Opalinski from the French Institute of Geopolitics at Paris 8 University discusses his work on resilience, or rather the lack of it, confronting the Internet in Pakistan.
As discussed in his blog post, Nowmay and his colleagues at the French Institute of Geopolitics (IFG), University Paris 8, and LUMS University Pakistan used a combination of technical measurement from sources such as RIPE Atlas, in a methodology devised by the GEODE project, combined with interviews in Pakistan, to explore the reasons behind Pakistan’s comparative fragility in the face of seaborne fibre optical cable connectivity. The approach deliberately combines technical and social-science approaches to exploring the problem space, with quantitative data and qualitative interviews.
Located at the head of the Arabian Sea, but with only two points of connectivity into the global Internet, Pakistan has suffered over 22 ‘cuts’ to the service in the last 20 years, However, as Nowmay explores in this episode, there actually are viable fibre connections to India close to Lahore, which are constrained by politics.
Nowmay is completing a PhD at the institute, and is a member of the GEODE project. His paper on this study was presented at the 2024 AINTEC conference held in Sydney, as part of ACM SIGCOMM 2024.
Read more about GEODE, and Nowmay’s work:
- The GEODE project
- Pakistan, a case study in Internet fragility
- The Quest for a Resilient Internet Access in a Constrained Geopolitical Environment (AINTEC 2024 Paper)
In his regular monthly spot on PING, APNIC’s Chief Scientist, Geoff Huston, discusses another use of DNS Extensions: The EDNS0 Client Subnet option (RFC 7871). This feature, though flagged in its RFC as a security concern, can help route traffic based on the source of a DNS query. Without it, relying only on the IP address of the DNS resolver can lead to incorrect geolocation, especially when the resolver is outside your own ISP’s network.
The EDNS Client Subnet (ECS) signal can help by encoding the client’s address through the resolver, improving accuracy in traffic routing. However, this comes at the cost of privacy, raising significant security concerns. This creates tension between two conflicting goals: Improving routing efficiency and protecting user privacy.
Through the APNIC Labs measurement system, Geoff can monitor the prevalence of ECS usage in the wild. He also gains insights into how much end-users rely on their ISP’s DNS resolvers versus opting for public DNS resolver systems that are openly available.
Read more about EDNS0 and UDP on the APNIC Blog and at APNIC Labs:
- Privacy and DNS Client Subnet (Geoff Huston, APNIC Blog July 2024)
- The use of ECS as measured by APNIC Labs
In this episode of PING, Joao Damas from APNIC Labs explores the mechanics of the Labs measurement system. Commencing over a decade ago, with an "actionscript" (better known as flash) mechanism, backed by a static ISC Bind DNS configuration cycling through a namespace, the Labs advertising measurement system now samples over 15 million end users per day, using Javascript and a hand crafted DNS system which can synthesise DNS names on-the-fly and lead users to varying underlying Internet Protocol transport choices, packet sizes, DNS and DNSSEC parameters in general, along with a range of Internet Routing related experiments.
Joao explains how the system works, and the mixture of technologies used to achieve the goals. There's almost no end to the variety of Internet behaviour which the system can measure, as long as it's capable of being teased out of the user in a javascript enabled advert backed by the DNS!
- Measurements from APNIC Labs
- How we measure: RPKI ROA and ROV (2023)
- How we measure: DNSSEC Validation (2023)
- The APNIC Labs IPv6 Measurement system (2013)
In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston re-visits the question of DNS Extensions, in particular the EDNS0 option signalling maximum UDP packet size accepted, and it’s effect in the modern DNS.
Through the APNIC Labs measurement system Geoff has visibility of the success rate for DNS events where EDNS0 signalling triggers DNS “truncation” and the consequent re-query in TCP as well as the impact of UDP fragmentation even inside the agreed limit, as well as the ability to handle the UDP packet sizes proffered in the settings.
Read more about EDNS0 and UDP on the APNIC Blog and at APNIC Labs
- Revisiting DNS and UDP truncation (Geoff Huston, APNIC Blog July 2024)
- DNS TCP Requery failure rate (APNIC Labs)
In this episode of PING, Casper Schutijser and Ralph Koning from SIDN Labs in the Netherlands discuss their post-quantum testbed project. As mentioned in the previous PING episode about Post Quantum Cryptography (PQC) in DNSSEC with Peter Thomassen from SSE and Jason Goertzen from Sandbox AQ it's vital we understand how this technology shift will affect real-world DNS systems in deployment.
The SIDN Labs system has been designed to be a "one stop shop" for DNS operators to test configurations of DNSSEC for their domain management systems, with a complete virtualised environment to run inside. It's fully scriptable so can be modified to suit a number of different situations and potentially include builds of your own critical software components to include with the system under test.
Read more about the testbed and PQC on the APNIC Blog and at SIDN Labs:
- PATAD: The SIDN Labs post-quantum cryptography DNSSEC testbed
- [Podcast] Testing Post Quantum Cryptography DNSSEC
- A quantum-safe cryptography DNSSEC testbed
- How organizations can prepare for post-quantum cryptography
In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston continues his examination of DNSSEC. In the first part of this two-part story, Geoff explored the problem space, with a review of the comparative failure of DNSSEC to be deployed by zone holders, and the lack of validation by the resolvers. This is visible to APNIC labs from carefully crafted DNS zones with validly and invalidly signed DNSSEC states, which are included in the Labs advertising method of user measurement.
This second episode offers some hope for the future. It reviews the changes which could be made to the DNS protocol, or use of existing aspects of DNS, to make DNSSEC safer to deploy. There is considerable benefit to having trust in names, especially as a "service" to Transport Layer Security (TLS) which is now ubiquitous worldwide in the web.
Read more about DNSSEC and TLS on the APNIC Labs website and the APNIC Blog:
- Calling time on DNSSEC (Geoff Huston, APNIC Blog, June 2024)
- 'Keytrap' attacks on DNSSEC (Geoff Huston, APNIC Blog, June 2024)
- DNS topics at RIPE 88 (Geoff Huston, APNIC Blog, June 2024)
- The Tranco list
- DNSSEC validation client usage (APNIC Labs)
- DNSSEC-enabled domains from Cloudflare public DNS (APNIC Labs)
This time on PING, Peter Thomassen from deSEC and Jason Goertzen from Sandbox AQ discuss their research project on post quantum cryptography in DNSSEC, funded by NLNet Labs.
Post Quantum cryptography is a response to the risk that a future quantum computer will be able to implement Shor's Algorithm -a mechanism to uncover the private key in the RSA public-private key cryptographic mechanism, as well as Diffie-Hellman and Elliptic Curve methods. This would render all existing public-private based security useless, because with knowledge of the private key by a third party, the ability to sign uniquely over things is lost: DNSSEC doesn't depend on secrecy of messages but it does depend on RSA and elliptic curve signatures. We'd lose trust in the DNSSEC protections the private key provides.
Post Quantum Cryptography (PQC) addresses this by implementing methods which are not exposed to the weakness that Shor's Algorithm can exploit. But, the cost and complexity of these PQC methods rises.
Peter and Jason have been exploring implementations of some of the NIST candidate post quantum algorithms, deployed into bind9 and PowerDNS code. They've been able to use the Atlas system to test how reliably the signed contents can be seen in the DNS and have confirmed that some aspects of packet size in the DNS, and new algorithms will be a problem in deployment as things stand.
As they note, it's too soon to move this work into IETF DNS standards process but there is a continuing interest in researching the space, with other activity underway from SIDN which we'll also feature on PING.
In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston discusses DNSSEC and it's apparent failure to deploy at scale in the market after 30 years: Both as the state of signed zone uptake (the supply side) and the low levels of verification seen by DNS client users (the consumption side) there is a strong signal DNSSEC isn't making way, compared to the uptake of TLS which is now ubiquitous in connecting to websites. Geoff can see this by measurement of client DNSSEC use in the APNIC Labs measurement system, and from tests of the DNS behind the Tranco top website rankings.
This is both a problem (the market failure of a trust model in the DNS is a pretty big deal!) and an opportunity (what can we do, to make DNSSEC or some replacement viable) which Geoff explores in the first of two parts.
A classic "cliffhanger" conversation about the problem side of things will be followed in due course by a second episode which offers some hope for the future. In the meantime here's the first part, discussing the scale of the problem.
Read more about DNSSEC and TLS on the APNIC Labs website and the APNIC Blog:
- Calling time on DNSSEC (Geoff Huston, APNIC Blog June 2024)
- "Keytrap" attacks on DNSSEC (Geoff Huston, APNIC Blog June 2024)
- DNS topics at RIPE88 (Geoff Huston, APNIC Blog June 2024)
- The Tranco top website Rankings
- DNSSEC validation client usage (APNIC Labs)
- DNSSEC enabled domains from Cloudflare public DNS (APNIC Labs)
This time on PING, Philip Paeps from the FreeBSD Cluster Administrators and Security teams discusses their approach to systems monitoring and measurement. Its eMail.
“Short podcast” you say, but no, there’s a wealth of war-stories and “why” to explore in this episode.
We caught up at the APNIC57/APRICOT meeting held in Bangkok in February of 2024. Philip has a wealth of experience in systems management and security and a long history of participation in the free software movement. So his ongoing of support of email as a fundamental measure of system health isn’t a random decision, it’s based on experience.
Mail may not seem like the obvious go-to for a measurement podcast, but Philip makes a strong case that it’s one of the best tools available for a high-trust measure of how systems are performing, and in the first and second order derivative can indicate aspects of velocity and rate of change of mail flows, indicative of the continuance or change in the underlying systems issues.
Philip has good examples of how Mail from the FreeBSD cluster systems indicates different aspects of systems health. Network delays, disk issues. He’s realistic that there are other tools in the armoury, especially the Nagios and Zabbix systems which are deployed in parallel. But from time to time, the first best indication of trouble emerges from a review of the behaviour of email.
A delightfully simple, and robust approach to systems monitoring can emerge from use of the fundamental tools which are part of your core distribution.
Read more about Philip, FreeBSD, Zabbix and Nagios at their websites:
- FreeBSD Project home page
- The FreeBSD Foundation welcomes donations!
- The FreeBSD Project and Administration
- Philip’s home page
- Zabbix for systems and network monitoring
- Nagios for systems and network monitoring
In his regular monthly spot on PING, APNIC’s Chief Scientist Geoff Huston discusses the question of subnet structure, looking into the APNIC Labs measurement data which collects around 8 million discrete IPv6 addresses per day, worldwide.
Subnets are a concept which "came along for the ride" in the birth of Internet Protocol, and were baked into the address distribution model as the class-A, class-B and class-C subnet models (there are also class-D and class-E addresses we don't talk about much).
The idea of a sub-net is distinct from a routing network, many pre-Internet models of networking had some kind of public-local split, but the idea of more than one level of structure in what is "local" had to emerge when more complex network designs and protocols came into being.
Subnets are the idea of structure inside the addressing plan, and imply logical and often physical separation of hosts, and structural dependency on routing. There can be subnets inside subnets, its "turtles all the way down" in networks.
IP had an ability out-of-the-box to permit subnets to be defined, and when we moved beyond the classful model into classless inter-domain routing or CIDR, the idea of prefix/length models of networks came to life.
But IPv6 is different, and the assumption we are heading to a net-subnet-host model of networks may not be applicable in IPv6, or in the modern world of high speed complex silicon for routing and switching.
Geoff discusses an approach to modelling how network assignments are being used in deployment, which was raised by Nathan Ward in a recent NZNOG meeting. Geoff has been able to look into his huge collection of IPv6 addresses and see what's really going on.
Read more about networks and subnets and address policy on the APNIC Web and blog
- APNIC's current address policy
- RFC4632 Classless Inter-Domain Routing (CIDR) (IETF RFC)
- IPv6 Prefix Lengths (Geoff Huston, blog article)
This time on PING Doug Madory from Kentik discusses his recent measurements of the RPKI system worldwide, and it's visible impact on the stability and security of BGP.
Doug makes significant use of the Oregon RouteViews repository of BGP data, a collection maintained continuously at the University of Oregon for decades. It includes data from back to 1997, originally collected by the NLANR/MOAT project and has archives of BGP Routing Information Base (RIB) dumps taken every two hours from a variety of sources, and made available in both human-readable and machine readable binary formats.
This collection has become the de-facto standard for publicly available BGP state worldwide, along with the RIPE RIS collection. As Doug discusses, research papers which cite Oregon RouteViews data (over 1,000 are known of, but many more exist which have not registered their use of the data) invite serious appraisal because of the reproducibility of the research, and thus the testability of the conclusions drawn. It is a vehicle for higher quality science about the nature of the Internet through BGP.
Doug presented on RPKI and BGP, at the APOPS session held in February at APRICOT/APNIC57 Bangkok, Thailand
Read more about Doug's presentation, his measurements at Kentik, Oregon RouteViews, the state of BGP and RPKI on the Kentik website, and the APNIC Blog:
- RPKI ROV Reaches Major Milestone/ (APNIC Blog May 2024)
- Doug Madory's blog at Kentik
- Digging into the Orange España Hack (APNIC Blog January 2024)
- What can be learned from BGP hijacks targeting cryptocurrency services? (APNIC Blog November 2022)
- The University of Oregon RouteViews project website
- The RIPE Routing Information Service (RIS) website
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses Starlink again, and the ability of modern TCP flow control algorithms to cope with the highly variant loss and delay seen over this satellite network. Geoff has been doing more measurements using starlink terminals in Australia and the USA, at different times of day exploring the system behaviour.
Starlink has broken new ground in Low Earth Orbit internet services. Unlike Geosynchronous satellite services which have a long delay but constant visibility of the satellite in stationary orbit above, Starlink requires the consumer to continuously re-select a new satellite as they move overhead in orbit. In fact, a new satellite has to be picked every 15 seconds. This means there's a high degree of variability in the behaviour of the link, both between signal quality to each satellite, and in the brief interval of loss ocurring at each satellite re-selection window.
Its a miracle TCP can survive, and in fact in the case of the newer BBR protocol thrive, and achieve remarkably high throughput, if the circumstances permit. This is because of the change from a slow start, fast backoff model used in Cubic and Reno to a much more aggressive link bandwidth estimation model, which continuously probes to see if there is more room to play in.
Read more about Satellites, TCP and flow control algorithms on the APNIC Blog and on the IETF website.
- An explainer on Coherent Optical Transcievers (Geoff Huston, APNIC Blog 2024)
- Low Earth Orbit and the Congestion Control Problem (Geoff Huston, APNIC Blog 2023)
- APNIC Labs measurements of Starlink (APNIC Labs)
- Comparing TCP and QUIC (Geoff Huston APNIC Blog 2022)
- Testing LEO and GEO Satellite Services in Australia
- Transport Protocols and the Network
- Congestion Control at IETF 110
This time on PING, Dr Mona Jaber from Queen Mary University of London (QMUL), discusses her work exploring IoT, Digital Twins and Social Science led research in the field of networking and telecommunications.
Dr Jaber is a senior lecturer in QMUL and is the founder and director of the Digital Twins for Sustainable Development Goals (DT4SDG) at QMUL. She was one of the invited Keynote speakers at the recent APRICOT/APNIC57 meeting held in Bangkok, and the podcast explores the three major themes explored in her keynote presentation.
- The role of deployed fibre optic communication systems in measurement for sustainable green goals
- Digital Twin Simulation platforms for exploring the problem space
- Social Sciences led research, an inter-disciplinary approach to formulating and exploring problems which has been applied to Sustainable Development-related research through technical innovation in IoT, AI, and Digital Twins.
The Fibre Optic measurement method is Distributed Acoustic Sensor or DAS:
"DAS reuses underground fibre optic cables as distributed strain sensing where the strain is caused by moving objects above ground. DAS is not affected by weather or light and the fibre optic cables are often readily available, offering a continuous source for sensing along the length of the cable. Unlike video cameras, DAS systems also offer a GDPR-compliant source of data."
The DASMATE Project at theengineer.co.uk
This Episode of PING was recorded live in the venue and is a bit noisy compared to the usual recordings, but it's well worth putting up with the background chatter!
Read more about Dr Jaber's presentation, the DAS system, Digital Twins and Fibre Optic communications:
- Intelligent IoT for sustainable development Goals: Keynote talk at APRICOT/APNIC57
- The recording of Dr Jaber's Keynote talk
- The DASMATE project: Assisting the uptake of Active Travel Tower Hamlets, London
- The DT4SDG group page at QMUL
- Coherent Optical Tranceivers (Geoff Huston, April 2024)
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the European Union's consideration of taking a role in the IETF, as itself. Network engineers, policy makers and scientists from all around the world have participated in IETF but this is the first time an entity like the EU has considered participation as itself in the process of standards development.
What's lead to this outcome? What is driving the concern that the EU as a law setting and treaty body, an inter-governmental trade bloc needs to participate in the IETF process? Is this a mis-understanding of the nature of Internet Standards development or does it reflect a concern that standards are diverging from society's needs? Geoff wrote this up in a recent opinion piece on the APNIC Blog and the podcast is a conversation around the topic.
Read more about digital sovereignty on the APNIC Blog and on the IETF website.
- Digital sovereignty and standards (Geoff Huston, APNIC Blog)
- As the Balance of Security Controls shifts where does responsibility rest? (Kathleen Moriarty, Guest Author on the APNIC Blog)
- Reflections on Ten Years Past the Snowden Revelations (IETF RFC9446)
- Pervasive Monitoring is an Attack (IETF RFC7528)
This time on PING we have Phil Regnauld from DNS Operations Analysis & Resource Center (DNS-OARC) talking about the three distinct faces OARC presents to the community.
Phil came to the OARC presidents role, replacing Keith Mitchell who was the founding president since 2008 through to this year. Phil previously has worked with the Network Startup Resource Centre (NSRC) and with AFNOG, and the Francophone Internet community at large.
DNS OARC has at least 3 distinct faces. It is a community of DNS operators and researchers, who maintain an active ongoing dialogue face to face in workshops and online in the OARC Mattermost community hub. Secondly it is a home, repository and ongoing development environment for DNS related tools such as DNSVIZ (written by Casey Deccio) hosting the AS112 project, and development of the DSC systems amongst many other tools.
Thirdly it is the organiser and host of the Day In The Life or DITL activity, the periodic collection of 48-72 hours of DNS traffic from the DNS root operators, and other significant sources of DNS traffic. Stretching back over 10 years DITL is a huge resource for DNS research, providing insights in the use of DNS and its behaviour on-the-wire.
Read more about DNS OARC and its activities:
- The Domain Name Service Operations, Analysis and Research Center
- The DSC data collection and analysis system
- DNS OARC software tools catalog
- The Day In The Life (DITL) collection
In this episode of PING, APNICs Chief Scientist Geoff Huston discusses a new proposed DNS resource record called DELEG. The record is being designed to aid in managing where a DNS zone is delegated.
Delegation is the primary mechanism used in the DNS to separate responsibility between child and parent for a given domain name. The DELEG RR is designed to address several problems, including a goal of moving to new transports for the name resolution service the DNS provides to all other Internet protocols.
Additionally, Geoff believes it can help with cost and management issues inherent in out-of-band external domain name management through the registry/registrar process, bound in the whois system and in a protocol called Extensible Provisioning Protocol or EPP.
There are big costs here and they include some problems dealing with intermediaries who manage your DNS on your behalf.
Unlike whois, EPP, and registrar functions, DELEG would be an in-band mechanism between the parent zone, any associated registry, and the delegated child zone. It’s a classic disintermediation story about improved efficiency and enables the domain name holder to nominate intermediaries for their services, via an aliasing mechanism that has until now eluded the DNS.
Read more about DELEG on the APNIC Blog and on the IETF website.
- DNS and the proposed DELEG record (APNIC Blog)
- ‘Extensible Delegation for DNS‘ (IETF draft)
- Extensible Provisioning Protocol (EPP) (IETF RFC)
This time on PING we have Amreesh Phokeer from the Internet Society (ISOC) talking about a system they operate called Pulse, available at https://pulse.internetsociety.org/. Pulse’s purpose is to assess the “resiliency” of the Internet in a given locality.
Similar systems we have discussed before on Ping include APNIC’s DASH service, aimed at resource holding APNIC members, and the MANRS project. Both of these take underlying statistics like resource distribution data, or measurements of RPKI uptake or BGP behaviours and present them to the community, and in the case of MANRS there’s a formalised “score” which shows your ranking against current best practices.
The Pulse system measures resilience in four pillars: Infrastructure, Quality, Security and Market Readiness. Some of these are “hard” measures analogous to MANRS and DASH, but Pulse in addition to these kinds of measurements includes “soft” indicators like the economic impacts of design decisions in an economy of interest, the extent of competition, and less formally defined attributes like the amount of resiliency behind BGP transit. This allows the ISOC Pulse system to consider governance-related aspects of the development of Internet, and has a simple scoring model which allows a single health metric analogous to the use of pulse and blood pressure by a physician to assess your condition, but this time applied to the Internet.
Read more about Pulse:
- The https://pulse.internetsociety.org/ website
- The Pulse Blog
- Don’t put all your internet infrastructure in one basket (Robbie Mitchell in the APNIC Blog)
- Internet Resilience on Pulse
- Internet Resilience Index Methodology
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the role of DNS in directing where your applications connect to, and where content comes from. Although this more “steering” traffic than it “routing” in the strict sense of IP packet forwarding, (that’s still the function of the border gateway protocol or BGP) It does in fact represent a kind of routing decision, to select a content source or server logistically “best” or “closest” to you. So in the spirit of “Orange is the new Black” -DNS is the new BGP.
As this change in delivery of content has emerged, the effective control on this kind of routing decision has also become more concentrated, into the hands of the small number of at-scale Content Distribution Networks (CDN) and associated DNS providers worldwide. This is far less than the 80,000 or so BGP speakers with their own AS and represents another trend to be thought about. How we optimise content delivery isn’t decided in common amongst us, its managed by simpler contractual relationships between content owner and intermediaries.
The upside of course remains the improvement in efficiency of fetch for each client, the reduction in delay and loss. But the evolution of the Internet over time and the implications for governance in “steering” decisions is going to be of increasing concern.
Read more about Geoff’s views of Concentration in the Internet, Governance, and Economics on the APNIC Blog and at APNIC Labs:
- DNS is the new BGP
- Internet Governance in 2023
- On Internet Centrality and Fragmentation
- The Internet as a Public Utility
- An Economic Perspective on Internet Centrality
- Looking at Centrality in the DNS
In this episode of PING, Leslie Daigle from the Global Cyber Alliance (GCA) discusses their honeynet project, measuring bad traffic internet-wide. This was originally focussed on IoT devices with the AIDE project but is clearly more generally informative. Leslie also discusses the quad-nine DNS service, GCA’s domain trust work and the MANRS project. Launched in 2014 with support from ISOC, MANRS now has a continuing relationship with GCA and may represent a model for the routing community regarding the ‘bad traffic’ problem which the AIDE project explores.
Leslie has a long history of work in the public interest, as Chief Internet Technology Officer of the Internet Society, and with the IETF. She is currently the chair of the MOPS working group, has co-authored 22 RFCs and was chair of the IAB for five years.
Read more about GCA, AIDE, domain trust and honeynets:
- The Global Cyber Alliance (GCA)
- The AIDE programme at GCA
- Domain Trust at GCA
- Honeynet tagged blog entries at APNIC
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the change in IP packet fragmentation behaviour adopted by IPv6, and the implications of a change in IETF “Normative Language” regarding use of IPv6 in the DNS.
IPv4 arguably succeeds over so many variant underlying links and networks because it’s highly adaptable to fragmentation in the path. IPv6 has a proscriptive requirement that only the end hosts fragment, which limits how intermediate systems can handle IPv6 data in flight. In the DNS, increasing complexity from things like DNSSEC mean the the DNS packet sizes are getting larger and larger, which risks invoking the IPv6 fragmentation behaviour in UDP. This has consequences for the reliability and timeliness of the DNS service.
For this reason, a revision of the IETF normative language (the use of capitalised MUST MAY SHOULD and MUST NOT) directing how IPv6 integrates into the DNS service in deployment has risks. Geoff argues for a “first, do no harm” approach to this kind of IETF document.
Read more about IPv6, Fragmentation, the DNS and Geoff’s measurements on the APNIC Blog and APNIC Labs:
- IPv6, the DNS and Happy Eyeballs
- How we measure DNSSEC Validation
- DNS is the new BGP
- To DNSSEC or Not
In this episode of PING, Sara Dickinson from Sinodun Internet Technologies and Terry Manderson, VP, Information Security and Network Engineering at ICANN discuss the ICANN DNS stats collector system which ICANN commissioned, and Sinodun wrote for them.
This system consists of two parts, a DNS stats compactor framework which captures data in the C-DNS format, a specified set of data in CBOR format, and the DNS stats visualiser which is uses Grafana. The C-DNS format is not a complete packet capture but allows the recreation of all the DNS context of the query and response. It was standardised in 2019, in an RFC authored by Sara, her partner John, Jim Hague, John Bond and Terry.
Unlike DSC, which is a 5 minute sample aggregation system, this system is able to preserve a significantly larger amount of the seen DNS query information and can even be used to re-create an on-the-wire view of the DNS (albiet not 1 to 1 identical to the original IP packetflows)
Read more about the systems, and IMRS online:
- RFC8618 Compacted-DNS (C-DNS): A Format for DNS Packet Capture
- The ICANN github repository for DNS Stats
- ICANN Managed Root Server (IMRS)
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the rise of Low Earth Orbiting (LEO) Satellite based Internet, and the consequences for end-to-end congestion control in TCP and related protocols.
Modern TCP has mostly been tuned for constant delay, low loss paths and performs very well at balancing bandwidth amongst the cooperating users of such a link, achieving maximum use of the resource. But a consequence of the new LEO internet is a high degree of variability in delay, loss and consequently an unstable bandwidth, which means TCP congestion control methods aren’t working quite as well in this kind of Internet.
A problem is, that with the emergence of TCP bandwidth estimation models such as BBR, and the rise of new transports like QUIC (which continue to use the classic TCP model for congestion control), we have a fundamental mismatch in how competing flows try to share the link. Geoff has been exploring this space with some tests from starlink home routers, and models of satellite visibility. His Labs starlink page shows a visualisation of behaviour of the starlink system, and a movie of views of the satellites in orbit.
Read more about TCP, QUIC, LEO and Geoff’s measurements on the APNIC Blog and APNIC Labs:
- APNIC Labs measurements of Starlink. (2023, Geoff Huston)
- Comparing TCP and QUIC (November 2022, Geoff Huston)
- Testing LEO and GEO Satellite Services in Australia (May 2022, Geoff Huston)
- Transport Protocols and the Network (May 2021, Geoff Huston)
- Congestion Control at IETF110 (March 2021, Geoff Huston)
In this episode of PING, Verisign fellow Duane Wessels discusses a late state (version 08) Internet draft he’s working on with two colleagues from Verisign. The draft is on Negative Caching of DNS Resolution Failures and is co-authored by Duane, William Carroll, and Matt Thomas
This episode discusses the behaviour of the DNS system overall in the face of failures to answer. There are already mechanisms to deny the existence of a queried name or a specific resource type. There are also mechanisms to define how long this negative answer should be cached, just as there are cache lifetimes defined for how long to hold valid answers, things that do exist, and have been supplied.
This time, it’s a cache of not being able to answer. The thing asked about? It might exist, or it might not. This cached data isn’t saying if it does exist or not, it’s a caching failure to be able to answer. As the draft states: “… a non-response due to a resolution failure in which the resolver does not receive any useful information regarding the data’s existence.”
Prior DNS specifications did provide guidance on caching in the context of positive responses and negative responses but the only guidance relating to failing to answer was to avoid aggressive re-querying of the nameservers that should be able to answer.
Read more about the draft, and other DNS-related work by Duane on the APNIC Blog:
- The draft Negative Caching of DNS Resolution Failures (2023, Version 08)
- Adding ZONEMD protections to the root zone (2023, APNIC Blog post)
- [Podcast] Adding ZONEMD protections to the root zone (2023, related podcast on PING)
- [Podcast] A look back at notable root zone changes (Duane discusses three significant root zone changes over the last decade)
In this episode of PING, instead of a conversation with APNIC’s Chief Scientist Geoff Huston we’ve got a panel session from APNIC56 he facilitated, where Geoff and six guests got to discuss the 30 year history of APNIC.
With Geoff on the panel were:
- Professor Jun Murai known as the ‘father of the Internet’ in Japan. In 1984, he developed the Japan University UNIX Network (JUNET), the first-ever inter-university network in that nation. In 1988, he founded the Widely Integrated Distributed Environment (WIDE) Project, a Japanese Internet research consortium, for which he continues to serve as a board member. Along with Geoff, Jun was one of the main progenitors of what became APNIC.
- Elise Gerich, a 31 year veteran of Internet networking, is recognised globally for her significant contributions to the Internet. Before retiring, Elise was President of PTI and prior to that, Vice President of IANA at ICANN. Elise served as the Associate Director National Networking at Merit Network in Michigan. While at Merit she was also a Principal Investigator for NSFNET’s T3 Backbone Project and the Routing Arbiter Project and was responsible for much of the early address management Impetus which led to the creation of the RIR system.
- David Conrad Previously the Chief Technology Officer of ICANN, who was involved in the creation of APNIC as its first full-time employee and founding Director-General.
- Akinori Maemura the JPNICChief Policy Officer, and a member of the APNIC EC for 16 years, 13 of which he was Chair of the EC.
- Gaurab Raj Upadhaya Head of WWW Video Delivery Strategy, Prime Video at Amazon. Gaurab has been active in the Internet community for more than a decade and like Akinori served on the APNIC EC for 12 years, 7 of these as Chair of the EC.
- Paul Wilson has more than thirty years’ involvement with the Internet, including 25 years’ experience as the Director General of APNIC.
The Panel discussed the early years of the Internet and the processes which led to the creation of APNIC along with some significant moments in the life of the registry.
In this episode of PING, Stephen Song discusses his work mapping the Internet. This is a long-term project, which he carries out alongside and supported by Mozilla Corporation, and the Association for Progressive Communications (APC).
Stephen has long championed the case for Open Data in telecommunications decision-making and maintains a list of resources for capacity building and development of the Internet with a particular focus on Africa.
The combination of some opaque business practices and the change from end delivery to mediated proxies from the content distribution network model raises questions about where the things users engage with and depend on are, so network infrastructure can be efficiently and openly planned. The latest episode of PING explores the issues inherent in understanding ‘where things are’ in the modern Internet.
Explore Stephen’s resources:
- Many Possibilities website
- Connectivity indexes, maps, and reports (GitHub)
- Open Data map of Content Distribution Networks around the world
- After Fibre
- Village Telco
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the technique APNIC Labs uses to measure end user behaviour in the global internet. This is probably the only worldwide web advert based measurement system in continuous use since 2010.
Originally written in Adobe Flash, the system is now coded in Javascript and HTML5, and continuously samples as many as 25 million users per day, across mobile devices and desktop PCs, Android, iPhone and Chromebook.
The system was first designed to inform the community on the rate of IPv6 deployment. The APNIC Labs measurements now encompass IPv6, RTT, HTTP/3 (Quic) adoption, DNSSEC, use of public DNS resolvers, IPv6 EH support, RPKI validation amongst other measurements.
Data is available at a per-economy, and per-AS (origin-AS) level, both as a web view and as JSON downloads. No end user identifying material is held, or distributed in any way. The measurement program is generously supported by Google, ICANN and APNIC.
Read more about some recent research outcomes from the labs advert on the APNIC Blog:
- Measuring the use of DNSSEC (September 2023, Geoff Huston)
- Measuring NXDOMAIN responses (July 2023, Geoff Huston)
- A Further Update on IPv6 Extension Headers (June 2023, Geoff Huston)
- A second look at QUIC use (September 2022, Geoff Huston)
In june of this year, the Dashboard for AS Health or DASH, a service operated by APNIC saw a leak of approximately 260,000 BGP routes from a vantage point in Singapore, and sent alerts to around 90 subscribers to our routing mis-alignment notification service which is part of DASH.
BGP is the state of announcements made and heard worldwide, calculated by every BGP speaker for themselves and although its globally connected and represents “the same” network, not everyone sees all things, as a result of filtering and configuration differences around the globe. BGP also should align with two external information systems, the older Internet Routing Registry (IRR) system which uses a notation called RPSL to represent routing policy data, including the “route” object, and Resource Public Key Infrastructure or RPKI, which represents the origin-AS (in BGP, who originates a given prefix) in a cryptographically signed objected called a ROA. The BGP prefix and origin (the route) should align with whats in an IRR route object and an RPKI ROA, but sometimes these disagree. Thats what DASH is designed to do: tell you when these three information sources fall out of alignment.
I discussed this incident, and the APNIC Information Product family (DASH, a collaboration with RIPE NCC called NetOX, and the delegation statistics portal called REX) with Rafael Cintra, the product manager of these systems, and with Dave Phelan who works in the APNIC Academy and has a background in Network Routing Operations.
You can find the APNIC Information products here: (note that the DASH service needs a MyAPNIC login to be used)
- https://dash.apnic.net the DASH portal login page (MyAPNIC resource login needed)
- https://netox.apnic.net NetOX the Network Observatory web service
- https://rex.apnic.net Resource Explorer: delegation statistics for the world
And you can read about the Information Products family in these blog articles:
- New Alert Options for DASH
- Routing Status added to DASH
- Suspicious Traffic Alerts added to DASH
- Using DASH to rank economies by suspicious traffic
- How DASH helps monitor Network Health
- Worldwide REX
- Introducing REX a new approach for the internet directory
- Hands-On with APNIC’s NetOX
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the coming future of VLSI with Moores law coming to an end. This was motivated by a key presentation made at the most recent ANRW session at IETF117, San Francisco.
For over 5 decades we have been able to rely on an annual, latterly bi-annual doubling of speed called Moore's Law, and halving of size of the technology inside a microchip: Very Large Scale Integration (VLSI), the basic building block of the modern age being the transistor.
From it's beginnings off the back of the diode, replacing valves but still discrete components, to the modern reality of trillions of logic "gates" on a single chip, everything we have built in recent times which includes a computer, has been built under the model "it can only get cheaper next time round" -But for various reasons explored in this episode, that isn't true any more, and won't be true into the future.
We're going to have to get used to the idea it isn't always faster, smaller, cheaper, and this will have an impact on how we design Networks, including details inside the protocol stack which go to processing complexity forwarding those packets along the path.
A few times, Both Geoff and myself get our prefixes mixed up and may say millimeters for nanometers or even worse on air. We also confused the order of letters in the company Acronym TSMC -The Taiwan Semiconductor Manufacturing Company.
Read more about the end of Moore's law on APNIC Blog and the IETF:
Chipping Away at Moore's Law (August 2023, Geoff Huston) It’s the End of DRAM As We Know It (July 2023, Philip Levis, IETF117 ANRW session)
In this episode of PING Jaap Akkerhuis (NLNet Labs), Ulrich Spiedel (University of Auckland) and Russ White (Juniper) discuss the issues behind Sunspots, ionisation in the athmosphere and it's effects on satellite communications and terrestrial infrastructure based on wires in the air: Power grids and data services.
In two blogs Good day sunshine and Solar Storms and the Internet we've highlighted the potential risks from increases in solar activity such as solar flares and the associated Coronal Mass Ejection or CME.
Spectacular as the effects on earths atmosphere can be, The risk of these events is quite high, if things line up badly for us: It's possible for there to be compounding effects on Satellite systems orbit, their electrical components, their lifetime in orbit (due to repositioning costs burning fuel to cope with the event) as well as effects on land as the suspended wires in power grids and data communications act as antenna, and produce voltage "spikes" to attached equipment at the end, as well as along the path.
However, as explored in this episode of PING the situation is often overblown by the news cycle, and it's more a story about being prepared with resilience in systems exposed to risk, and understanding those risks.
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the eternal tension between content and carriage.
At the RIPE 86 meeting held in Rotterdam in May of this year, Rudolf van der Berg presented a talk titled ‘The EU Gigabit Connectivity Package and How It Will Hurt the Internet’ (video, slides).
Geoff has previously written about the tensions between content and carriage, transit and Content Distribution Networks (CDNs), and the economics of networks but this episode of PING discusses a new twist: Vodafone’s underlying cost and price issues seem at odds with the European operator community seeking to regulate the ‘cost’ side of carrying domestic content.
Read more about the economics of the Internet on the APNIC Blog:
RIPE 86 bites — Gigabits for EU (June 2023, Geoff Huston on this RIPE 86 presentation) On centrality and fragmentation (July 2023, Geoff Huston) The Internet as a public utility (May 2023, Geoff Huston) An economic perspective on Internet centrality (March 2023, Geoff Huston) Sender pays (September 2022, Geoff Huston) Content vs carriage — who pays? (June 2022, Geoff Huston)
In this episode of PING, Verisign fellow Duane Wessels presents the ZONEMD resource record, defined in RFC8976.
The “MD” in ZONEMD stands for “message digest” and this resource record (RR) is a checksum over the state of a zone, including all its records and the zone serial record (“start of authority” or SOA) which includes a serial number.
This means that by fetching an entire zone, either in the DNS or “out of band” from an FTP or Web server or however you receive it, if it has the ZONEMD record you have a way to check that the entire zone, as it should be for that serial, is exactly what you have in-hand.
ZONEMD is going to permit people who copy zones to serve them (locally, or more widely) now have a basis to trust the state of the zone before publishing it.
Duane talks about the long lifetime of this idea with roots back into the 1990s, and the road to RFC8976 taken by the co-authors. A ZONEMD record with an un-testable signature will be placed in the root zone of the DNS in September of this year, and will become testable in December to allow time for the community to understand it’s behaviour.
This podcast is accompanied by a repost of a Verisign blog Duane wrote recently which has just been republished here on the APNIC Blog: Adding ZONEMD protections to the root zone
Read more about DNS, ZONEMD, and other blogs and podcasts by Duane on the APNIC Blog and elsewhere online:
The Root of the DNS revisited(2023, Geoff Huston) Notes from DNS OARC 38 (2022 APNIC Blog post by Geoff Huston) Notes from DNS OARC 35 (2021 APNIC Blog post by Geoff Huston) RFC8976 (2021 RFC D. Wessels, P. Barber – Verisign; M. Weinberg – Amazon; W. Kumari – Google; & W. Hardaker – USC/ISI) [Podcast] A look back at notable root zone changes (Duane Wessels on PING discusses 3 significant root zone changes over the last decade)
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses how Sweden built a national time distribution system and the nature of time in the modern Internet.
At the RIPE86 Meeting held in Rotterdam in May of this year, Karin Ahl, the CEO of Netnod presented a talk titled “How Sweden Built a World-Leading Time Network”
A central problem in time distribution on the Internet is firstly the lack of security inside the Network Time Protocol (NTP), and secondly the sources and reliability of the time information. The first problem is solved by using the newer Network Time Security (NTS) protocol which adds TLS, and the second by investment in reliable and strategically placed time distribution servers, which is the basis of the Swedish national time initiative.
Geoff attended the Netnod presentation and reflects on the complex and murky history of time, and the emergence of worldwide communities that coordinate both civil time (what the time of day is, in the world) and the nature of how time is measured (how a ‘second’ is defined, for example).
Geoff discusses historic and current attempts to standardise time measurements (such as UT1 and UTC) — with their inherent compromises — against Earth’s revolutions and rotations around the Sun. These measurements have become increasingly critical to modern technology, such as GPS.
Read more about NTP, NTS, and the time problem at the APNIC Blog and elsewhere online:
Watch Karin Ahl’s presentation at RIPE86 Rotterdam RIPE 86 bites — what’s the time? (2023 Geoff Huston’s APNIC Blog write-up on the issues) Network Time Security: new NTP authentication mechanism (2021 APNIC Blog by Martin Langer) How do you know what time it is? (2020 APNIC Blog by Patrik Fälström) Putting a stop to Internet Time Shifters (2019 APNIC Blog by Neta Rosen Schiff) Is the Internet Running Late? (2018 APNIC Blog by Geoff Huston) Steve Allan blogs on time (background reading) Tony Finch blogs on time (background reading)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC. At the RIPE86 Meeting held in Rotterdam in May of this year, Karin Ahl, the CEO of Netnod presented a talk titled “How Sweden Built a World-Leading Time Network”
A central problems in time distribution on the Internet is firstly the lack of security inside the Network Time Protocol (NTP) and secondly the sources and reliability of the time information. The first problem is solved by use of the newer Network Time Security (NTS) protocol which adds TLS, and the second by investment in reliable and strategically placed time distribution servers, which is the basis of the Swedish national time initiative.
Geoff saw this presentation and reflects on the complex and murky history of how we “do” time, and the emergence of a worldwide communities which coordinate both civil time (what the time of day is, in the world) and the nature of how we measure time (what is a “second” exactly and how is it defined?)
Decisions made in the 1950s and 1970s to try and normalise the difference between monotonically increasing “UT1” time and the civil time system we know as “UTC” continue to plague the IT world, as civil time drifts (occasionally) by one second forwards (or very occasionally backwards) against UT1 -And with the emergence of more and more technology and especially satellite based systems like GPS, Bai-Dou, Galileo and GLONASS which provide time, the need to finalise the relative status of each time model becomes greater.
Read more about NTP, SNTP and the Time problem at the APNIC Blog and elsewhere online:
Watch Karin Ahl’s presentation at RIPE86 Rotterdam RIPE 86 bites — what’s the time? (2023 Geoff Huston’s APNIC Blog write-up on the issues) Network Time Security: new NTP authentication mechanism (2021 APNIC Blog by Martin Langer) How do you know what time it is? (2020 APNIC Blog by Patrik Fälström) Putting a stop to Internet Time Shifters (2019 APNIC Blog by Neta Rosen Schiff) Is the Internet Running Late? (2018 APNIC Blog by Geoff Huston) Steve Allan blogs on time (background reading) Tony Finch blogs on time (background reading)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Christian Huitema discusses how looking into the IETF data tracker allowed him to assess "how well we are doing" at document production.
As the IETF has grown, and as the process of developing standards has got more complex its understandable it takes a bit longer to produce a viable RFC but some questions have been made about exactly where in process the delays come from. Are we really doing better or worse than we used to? and, why might that be?
Christian took an interesting approach to the problem, using a random sample of 20 documents from 2018 (initially) and a hand method of collating the issues, and then applied the same methodology back into 2008 and 1998. His approach to measurement was rigorous and careful, separating his own opinions from the underlying data to aide reproducibility.
Christian has a long history of network development and research, with experience in industry, and in the french national computing research institute "INRIA" before joining Bell Communications Research, and Microsoft. He worked on OSI systems, X.500 directories, Satellite communications, and latterly the IPv6 stack including the "Tededo" transition technology, the H/D ratio used in determining IPv6 allocations and assignments in the RIR model, and the QUIC transport layer protocol.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, APNIC’s Chief Scientist Geoff Huston discusses the major themes from his recent blog on “Failed Expectations”
In a trip down memory lane, the podcast ranges over the 40 year plus history of how we came to have the current Internet as we know it, and some of the “road not taken” alternates which were under consideration at the time. In this context. “Failed” doesn’t have to mean “failed to work” -it can mean the technology simply wasn’t chosen, or it can be the “failure” to turn off something which was believed to be at best temporary!
In part, the story of IPv6 deployment is part of this mismatch of expectations and reality, because nobody sought the outcome we’re now living through, of a 20 plus year transition from 32 bit addresses to a world of 128 bit addressing. IPv6 was designed with an eye to the needs of addressing at scale, but the emergence of a transfer model, and continued improvement in NAT (and deployment of Carrier-grade NAT or CGN) at scale, worldwide has perpetuated a 32 bit address and routing world. IPv4 Internet is the “little network which could” and refuses to go away quietly.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Verisign Fellow Duane Wessels discusses notable changes in the DNS root zone over the last 13 years.
Duane joined Verisign in the early stages of DNSSEC deployment and has conducted measurements of DNS for many years, in his measurement factory days, and in DNS OARC as well as inside Verisign.
The significant changes to the DNS root zone, and it's implications for the root zone operators are discussed: Deploying DNSSEC, the first DNSSEC KSK key changes, the increase in packet sizes with RSA keylength changes, and the future KSK and ZSK algorithm changes
Read more about DNS and DNSSEC on the APNIC Blog.
Here's some articles from the blog which discuss the issues:
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, APNIC's Chief Scientist Geoff Huston discusses the question of buffers, flow control and 'efficient' use of a network link.
How do we maximise the use of a given network path, without knowing everything about its size along the way? It turns out, the story isn't as simple as "more is better" because sometimes, adding more memory to the system adds delay. Modern TCP's flow control algorithms are being modified to react to delay as well as loss, and become more efficient at occupying the available space. At the same time bit-marks inside the IP packet are modifying how end hosts can react to signals of congestion along the path. Are these two mechanisms in conflict? how do they stack up, and achieve critical mass in deployment?
Read more about TCP and flow control on the APNIC Blog.
Here's some articles from the blog which discuss the issues:
- Comparing TCP and QUIC (Geoff Huston)
- Does TCP keep pace with QUIC? (Konrad Wolsing)
- TCP Congestion Signatures (Srikanth Sundaresan)
- Striking a balance between bufferbloat and TCP queue oscillation (Ulrich Speidel)
- TCP initial window configurations in the wild (Jan Rüth)
- Underload: The future of congestion control (Safiqul Islam)
- Beyond bufferbloat: End-to-end congestion control cannot avoid latency spikes (Bjørn Teigen)
- Congestion Control at IETF 110 (Geoff Huston)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Dr Romain Fontugne, the deputy director of research at IIJ Labs in Tokyo discusses the IIJ "Internet Health Report" and AS Hegemony (or network centrality) in particular.
This is a data model they have been working on for some time (6 years now) which exposes dependencies between ASs in BGP, both directly (as in customer-cone) and indirectly through transitive dependencies. It's a fascinating insight into how BGP dependencies can be seen through the state of the routing table worldwide, and how IIJ are helping BGP speakers understand the dependencies in the transit paths they use. It's also a fertile space for student engagement with google summer of code opportunities.
You can read more about AS Hegemony on the APNIC blog, as well as other posts Romain has made reflecting IIJ's research.
- AS hegemony: measuring AS interdependence (2018)
- BGP zombies (2019)
- The impact of COVID-19 on last-mile latency (2020)
IIJ's Internet Health Report (IHR) is reachable at http://ihr.iijlab.net
In this episode of PING, APNIC's Chief Scientist Geoff Huston discusses Low Earth Orbit (LEO) satellite communications and the amazing effects they are having on internet reach worldwide.
Read more about the issues in LEO and satellite communications on the APNIC Blog.
Here's some recent articles of note:
- Getting hands-on experience with Starlink (Ulrich Speidel)
- Everything, everywhere, all the time (for the internet at least) (George Michaelson)
- The APRICOT Panel discussing satellite broadband in the Asia Pacific region (Geoff Huston, Dan York, Debopam Batterchee, Ulrich Speidel, Mike Puchol)
- How does starlink compare to broadband (Mohamed Kassem in 2022)
- Fact checking Starlink’s performance figures (Martino Trevisan in 2022)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING Rolf Winter discusses his 0 draft in IETF for "reverse traceroute" which is mechanistically the same as normal traceroute, but in this proposal requires some code point assignments. Rolf and his collaborators have been experimenting with eBPF implementations and testing the viability of their proposed codepoints in live tests on the global internet.
In this episode of PING, APNIC's Chief Scientist Geoff Huston discusses DNSSEC and presents a case "for" and "against" deployment, in the context of complexity, fragility, and impact on the DNS process at large. DNSSEC is net beneficial but its by no means automatic to deploy it protecting a zone.
Read more about the issues in DNSSEC deployment on the APNIC Blog:
- to DNSSEC or not? (Geoff Huston)
- IP Fragmentation and the DNS - Mitigation (Carsten Strotmann)
- DNSSEC Validation - Performance Killer (Petr Špaček)
- [Podcast] DNS spoofing is a non-issue if we all do DNSSEC (Geoff Huston)
- Addressing the challenges of modern DNS (Moritz Müller)
And for Geoff's continuing measurement of DNSSEC see https://stats.labs.apnic.net/dnssec for his daily updated measurements of DNSSEC validation.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Andre Geldeblom from APNIC product development discusses how APNIC is measuring user experience, satisfaction and engagement with the "Orbit" system we deployed to provide web services integrated with email.
Andre discusses the different motivations and mechanisms we're using at APNIC to understand "UX" and how this integrates in our planning to deliver "value for money" to the APNIC community.
Read about the different motivations and mechanisms we’re using at APNIC to understand “UX” and how this integrates in our planning to deliver “value for money” to the APNIC community.
- Upcoming changes to MyAPNIC (Andre, October 2022)
- how APNIC is implementing your feedback (Dale, September 2022)
- Improving Accessibility at APNIC (Dale, August 2021)
- APNIC help centre now live (Andre, February 2021)
- Help APNIC create better products and services (Dale, September 2020)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, APNIC's Chief Scientist Geoff Huston discusses the current situation in BGP across IPv4 and IPv6. Historically. we've met the "running out of memory" problem with incremental upgrades but things have now come to a place where "simply adding more memory" may not be the answer, and the dynamics of BGP growth appear to have changed.
Geoff explores the changing surface of the BGP default-free zone, and what it means for routing technology and the ISP
Read more about the APNIC Labs analysis of BGP on the APNIC Blog:
- BGP in 2022 - The Routing Table
- BGP in 2022 - BGP Updates
- IP Addressing through 2022
And for Geoff's continuing measurement of BGP see bgp.potaroo.net for his daily updated history of BGP from 1994.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Ege Cem Kirci from ETH Zurich discusses his IMC paper on users perceptions of internet outages, measured using Google Trends. This was presented at the IMC conference held in NIce, France in October 2022.
Ege and his co-authors have been exploring the relationship of Internet outages, (for example caused by weather events) and the information in google trends, with a mechanism to combine snapshots of data by time period which auto-scale, to a single unified time series. In their IMC talk they presented SIFT, a detection and analysis tool for capturing user-affecting Internet outages. SIFT leverages users' aggregated web search activity to detect outages.
Watch Ege's presentation :
- "Is my Internet down?": Sifting through User-Affecting Outages with Google Trends
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, APNIC's Chief Scientist Geoff Huston explores how APNIC Labs has been able to look inside behaviour in the DNS, to see signs of the "centrality" problem: How much concentration of delivery of service is there, across different market segments of users, and between the supply side (name serving as an authority) and and request side (recursive resolvers)
Read more about the APNIC Labs measurement of DNS:
- Looking at Centrality in the DNS
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Luuk Hendricks and Willem Toorop from NLNet talk about their work to embed telemetry in the linux kernel using eXpress Data Path (XDP)
Read more about XDP in a series of articles published in the APNIC blog:
- Journeying into XDP: Part 0
- Journeying into XDP: Augmenting the DNS
- Journeying into XDP: Fully-fledged DNS service augmentation
- Journeying into XDP: Augmenting the DNS
and their blog covering this episode of Ping:
- Journeying into XDP: XDPerimenting with DNS telemetry
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, APNIC's Chief Scientist Geoff Huston talks through some of the presentations he saw (and gave) at the recent DNS OARC 39 meeting held in Belgrade.
Read more about DNS OARC and the presentations mentioned:
- DNS OARC
- Casey Deccio's presentation on DITL data and vulnerabilities
- Geoff Huston's presentation on resolverless DNS
- David Lawrence's presentation on the registrar API problem
All of the DNS OARC 39 Speakers and their presentations can also be seen online now.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode of PING, Juan Garcia Pardo from ETH Zurich discusses the Cooperative Lightweight Inter-domain Bandwidth-Reservation Infrastructure (COLIBRI) project for the SCION network, last discussed in episode 18 of PING.
Juan explores the motivations for setting up a bandwidth reservation framework, and how it might be used in practice.
Read more about COLIBRI and SCION:
- The SCION network architecture
- The SCION Inter-domain Routing Architecture. From research to deployment (APNIC54 presentation by Nicola Rustignoli)
- [Podcast] Taking a clean slate to designing the Internet of the future (Ping Episode 18, a podcast with Nicola Rustignoli)
- Can we provide SLOs in an open Internet in a scalable manner? (Giacomo Giuliari)
- A new Internet architecture (Nicola Rustignoli)
- Future Internet at terabit speeds: SCION in P4 (Caspar Schutijse)
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
APNIC's Chief Scientist, Geoff Huston, joins PING for his monthly chat, to share his thoughts on the associated trust with routing security and whether it can hold up as a sustainable model.
We'll talk about the history of trust in communication and associated challenges within routing security and the rise and future of Resource Public Key Infrastructure (RPKI) and BGP security.
Read more about PKI and routing security on the APNIC Blog.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
At the recent APNIC 54 meeting, we sat down with local keynote speaker Alexander Ling, Director of Singapore's Quantum Engineering Programme, in his offices at the National University of Singapore’s Centre for Quantum Technologies.
In this episode, Alex will give us a brief 101 on quantum networking — given it may be a relatively new concept for many of you, even though it's been in development for quite some time — and discuss its current and pending use cases in the next 3 to 5 years, including via satellite technology.
Watch Alex's APNIC 54 keynote presentation From HTTP to HTTQ - Working Towards the Quantum Internet. https://youtu.be/x5HwbtgBEEQ?t=4104
Read more about quantum networking on the APNIC Blog.
APNIC's Chief Scientist, Geoff Huston, joins PING for his monthly chat, to warn the service provider industry about pressuring regulators to make content providers pay to use their networks; a situation currently playing out in Korea as we chatted about in Episode 13.
As he outlines in his recent post, Geoff forecasts the consequences of turning our backs on net neutrality which could jeopardize service providers' current hold on the provision market and Internet users’ access to sites like Netflix and YouTube, increase the centrality of the Internet and in some cases compromise the security of certain economies communication infrastructure.
Read more about net neutrality and centralization on the Internet on the APNIC Blog: https://blog.apnic.net/2022/09/09/sender-pays/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode, we’re returning to a topic that we last discussed in Episode 20, with Robin Marx, and unpacking the intricacies and perceived complexities of the QUIC and other similar new protocols.
To help, we’ve invited Larry Peterson and Bruce Davie, who among their extensive list of experience spanning the past thirty years, are co-authors of the renowned ‘Computer Networks: A Systems Approach’ textbook, which is now in its fifth edition and open source.
Check out the Systems Approach blog systemsapproach.substack.com/ Read more about QUIC on the APNIC Blog: blog.apnic.net/tag/quic/ Join the measurement@APNIC mailing list: www.apnic.net/community/participate/mailinglists/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
APNIC's Chief Scientist, Geoff Huston, joins PING for his month chat, to share his thoughts on discussions at the recent meeting of the Decentralized Internet Infrastructure Research Group (DINRG) at IETF 114.
The DINRG investigates open research issues in decentralizing infrastructure services such as trust management, identity management, name resolution, resource/asset ownership management, and resource discovery with a focus on infrastructure services that can benefit from decentralization or that are difficult to realize in local, potentially connectivity-constrained networks.
For Geoff the discussion about whether technology has led to centralization is misplaced — technology was merely and will remain simply a lever that 'The Big' use to remain big. Rather the discussion needs to focus on the economic factors that have historically repeated themselves and allowed the biggest companies of the world to control their markets.
Read more about Centralization in the Internet on the APNIC Blog: https://blog.apnic.net/tag/centralization/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode, we chat with Anant Shah, Principal Architect at Edgecast, about the challenges with measuring video streaming quality of experience. We'll discuss the fundamentals of video streaming, including the protocols, how it is delivered and different vantage points from which it can be measured.
Understanding the challenges associated with delivering and measuring content can provide insight into how we need to improve observability in the Internet as a whole as well as how we can develop the network to deliver low latency content in the future.
Watch Anant's presentation at PAM 2022 https://www.youtube.com/watch?v=R3uAj9OF1Zs
And read more Internet measurement stories on the APNIC Blog https://blog.apnic.net/tag/measurement/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
APNIC's Chief Scientist, Geoff Huston joins us again on the show, this time to discuss three related presentations by Google, ISC and Mozilla that caught his attention during the recent IETF 114 and DNS-OARC 38 meetings on securing the DNS against spoofing.
DNS spoofing involves third parties intercepting and responding to queries for benign or malicious purposes; recent studies show that DNS spoofing has more doubled since 2016.
Google is protecting its DNS service against spoofing using multiple methods including using a combination of DNS cookies, randomizing the choice of name servers, stripping duplicate queries from the outbound queues, performing rate limiting and unilaterally probing for support of Authoritative DNS over TLS (ADoT); it projects that these measures will cover 99% of queries after the various rollouts are complete.
While such results are impressive, Geoff and others argue that the widespread use of DNSSEC could do just as good as a job and with little impact on performance, as per ISC's and Mozilla's findings in their recent studies.
Read more about DNS Spoofing and DNSSEC on the APNIC Blog.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
We’re continuing the theme of our last episode and learning more about the intricacies and perceived complexities of the QUIC protocol from one of its contributors and proponents Robin Marx, a web protocol postdoc researcher at KULeuven in Belgium and soon-to-be technical solutions architect at Akamai.
Having researched HTTP2 performance as part of his PhD, Robin became involved in the development of HTTP3 and QUIC in the IETF while they were being designed. He also created the popular QUIC and HTTP/3 debugging and testing tools qvis and qlog.
In this episode, we discuss how the perceived complexities are unwarranted and how similar perceptions directed at other new protocols can impact innovation, especially from new contributors to forums like the IETF. Robin also shares his experience with contributing to the development of QUIC in the IETF, and the importance of user experience when it comes to measurement.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode, APNIC’s Chief Scientist, Geoff Huston joins us to discuss a new measurement project that he and his APNIC Labs colleague Joao Damas have established to measure the use and performance of the QUIC protocol (RFC 9000).
Although QUIC has only recently been standardized by the IETF (May 2021), it has been around since 2012 with an initial public release included in Chromium version 29, in August 2013. After nine years, how widely is it being used and is it living up to its billing and improving the reliability, security and speed of the Internet?
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
Today’s Internet is a Frankenstein’s monster of parts that have been bolted together. If it was a house, it would be a knock-down job — far easier to start from scratch.
This is easier said than done or so many thought as we’ll come to learn in this episode featuring Nicola Rustignoli a founding engineer at the SCION Association, which is taking a clean-slate approach to overcoming the architectural limitations of today’s Internet, to provide route control, failure isolation, and explicit trust information for end-to-end communication.
Read more about SCION projects on the APNIC Blog and the SCION website.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode, APNIC’s Chief Scientist, Geoff Huston, joins us to discuss IPv6 — a topic he has investigated exhaustively over the last 30 years.
Specifically, we’re talking about the seemingly never-ending transition to IPv6, the hallmarks it shares with previous IP transitions, how Network Address Translation (NAT) has hindered its deployment for the good and bad of the Internet, and whether 100% IPv6 is even the end goal given how far technology has come since IP was first implemented.
You can read more on the APNIC Blog and IPv6@APNIC portal.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In this episode, we talk with Adrian Farrel about a study he and Daniel King conducted on the challenges semantic routing has had on the Internet's routing system.
Now, this is a topic that a lot of people have a poor opinion of, particularly its link to New IP. But what this study hopes to achieve is to look past the preconceptions and discover the broad underlying issues that semantic routing proposals are seeking to address and how to do it in a generic and scalable way that doesn’t break the network layer.
Further to this, it also seeks to shed a light on the standard system and how to introduce new ideas before shooting them down as inappropriate.
You can read more about this study on the APNIC Blog or via the following sources: An Introduction to Semantic Routing https://gate.sc/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-farrel-irtf-introduction-to-semantic-routing%2F&token=cd7e5a-1-1654660769555
A Survey of Semantic Internet Routing Techniques https://gate.sc/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-king-irtf-semantic-routing-survey%2F&token=1b30f9-1-1654660769555
Challenges for the Internet Routing Systems Introduced by Semantic Routing https://gate.sc/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-king-irtf-challenges-in-routing%2F&token=258f71-1-1654660769555
Also, there is a free virtual IETF Routing Area Working Group (RTGWG) interim meeting on 21 June 2022 (15:00 to 17:00 UTC) to discuss semantic routing. https://datatracker.ietf.org/doc/agenda-interim-2022-rtgwg-01-rtgwg-01/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
Geoff Huston joins us again for his monthly chat, this time to consider the contradictory theory of a resolverless DNS. Contradictory because the DNS is by its nature reliant on resolvers.
We'll discuss the historical process of how DNS names are resolved and how content providers, through their open DNS services, have sought to optimise this process. This optimisation has led some, including Geoff, to consider 'server push' combined with DNSSEC, as a viable means to speed up the DNS, reduce privacy leakage and constrain the potential for failures.
You can read more on the APNIC Blog https://blog.apnic.net/2022/05/17/the-path-to-resolverless-dns/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our fourteenth episode, we're taking a closer look at Resource Public Key Infrastructure (RPKI) in Australia and New Zealand with Terry Sweetser.
Terry recently worked on an Internet Society project to measure RPKI adoption in Australia and New Zealand among its government services and critical infrastructure. We've invited him on to discuss the results and research methodology, including the challenges of working with public sources of data.
Watch Terry's presentation on this project at APRICOT 2022 and check out the RPKI@APNIC portal for more information on RPKI as well as useful deployment case studies, how-to posts and links to hands-on APNIC Academy lessons and labs.
In our thirteenth episode, Geoff Huston joins us again for his monthly chat, this time on the historical feudings between carriers and content providers.
The latest instalment in this feud is currently playing out in Korea, where South Korean Internet Service Provider, SK Broadband, is suing Netflix to pay for the costs of increased network traffic and maintenance work because of a surge in viewers since 2018.
For Geoff, this is merely the latest in a long list of disputes dating back to the 1800s that have regularly been decided by the courts and regulators. But this most recent dispute is being watched closely by all sides around the world with Korean regulators taking a different and deliberate road that for some impedes the free market regulation model that has made the Internet into what it is today.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our twelfth episode, we're talking all things Resource Public Key Infrastructure (RPKI) with Job Snijders, Principal Engineer at Fastly.
Job shares his thoughts on the benefits of managing an RPKI publication point, including the experience and insights it can give you. We also touch on a recent IETF proposal he has assisted with that seeks to provide clearer, authenticated proof of the intent by address delegates and the future of routing securing including IRR and BGPsec.
Check out RPKI @ APNIC portal for more information on RPKI as well as useful deployment case studies, how-to posts and links to hands-on APNIC Academy lessons and labs.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our eleventh episode, we are again talking to APNIC's Chief Scientist, Geoff Huston, this time about certificate revocation.
This has been a hotly debated subject for some time, namely whether it still serves a purpose given that many web browsers give you the option to ignore revoked certificates or ignore them for you. We'll discuss this and possible solutions to addressing the revocation processes issues.
You can read more about certificate revocation on the APNIC Blog, including a post from Geoff: What’s going on with certificate revocation?
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our tenth episode, we're talking all things submarine cables with Alan Mauldin, who is a Research Director at TeleGeography.
Synonymous with their submarine cable maps, TeleGeography also offers a range of open-source data surrounding the health of the Internet, as we'll come to learn.
Read about the latest submarine cable research on the APNIC Blog. https://blog.apnic.net/tag/submarine-cable/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our ninth episode, APNIC's Chief Scientist, Geoff Huston, and I chat with Reliance Jio's Senior Executive Vice President, Mr Nagaraj, who oversees the design, architecture and engineer of Jio's IP network as well as operational support.
With more than 420 million mobile users and 260 million cable users, Reliance Jio has grown to become one of the largest telcos in the world in less than a decade. How were they able to do this in the age of IPv4 exhaustion? By deploying IPv6 from the get-go. Mr Nagaraj takes us through the discussions he had in convincing his management and vendors to go IPv6 and the future of the Jio network, including working more closely with content providers to bring content closer to users than before.
You can also read an abbreviated transcript of this discussion on the APNIC Blog: https://blog.apnic.net/2022/03/03/podcast-taking-ipv6-by-the-horns/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our eighth episode, we're talking to Willem Toorop, Luuk Hendricks and Tom Carpay of NLnet Labs, about how eXpress Data Path, or XDP as it’s better known, can be used to improve the performance of DNS resolvers and increase the versatility of name servers, as well as perform low-level measurements on high-speed links.
This is a project that they've been working on for at least two years now and as you'll come to learn has now started to bear fruits and make the networking community stand up and pay attention to the capabilities and potential of XDP.
You can also read about their journey via the APNIC Blog: blog.apnic.net/tag/xdp/
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our seventh episode, we are talking again to APNIC's Chief Scientist, Geoff Huston, regarding all things time.
We wander through the ways that humans have measured time and discuss the challenges of current Internet timekeepers in trying to make sure time is current — a very important element when securing a network — including how to handle leap seconds and how a recent announcement of the world’s first hardware implementation of network time protocol will assist with securing time on the Internet.
You can read more on Network Time Protocol on the APNIC Blog, including a post from Geoff on leap seconds: Is the Internet running late? https://blog.apnic.net/2018/11/28/is-the-internet-running-late/
Also, check out Netnod's white paper on the world’s first hardware implementation of NTS. https://www.netnod.se/netnod-white-paper-on-the-worlds-first-nts-hardware-implementation
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our sixth episode, we are talking to Romain Fontugne a senior researcher from Internet Initiative Japan, whose research interests include detecting network traffic anomalies and visualizing network traffic.
Romain shares some best practices for collecting and using open data for researching the Internet and discusses the need to have an objective in mind when trying to make sense of the data and why you're collecting it.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our fifth episode, we are talking again to APNIC's Chief Scientist, Geoff Huston, regarding a recent presentation he sat in on at DNS OARC 36 about Slack's unfortunate 24-hour outage on 30 September 2021, where 1% of its users (almost 100,000 users) weren't able to access Slack, due to an issue with deploying it's DNSSEC. We'll discuss two themes that Geoff has blogged about and discussed on this show this year: managing and securing the DNS is difficult but necessary, and the need for greater transparency in the industry when discussing things that have gone wrong, so that the community can learn from them to make sure they don’t happen again.
You can read Geoff's account on this presentation here: Notes from DNS-OARC 36 [ https://blog.apnic.net/2021/12/07/notes-from-dns-oarc-36/?utm_source=podcast&utm_medium=organic&utm_campaign=PING&utm_term=E5&utm_content=description]
Also, here is a link to Slack's account of the situation that we refer to in the show: The Case of the Recursive Resolvers [https://slack.engineering/what-happened-during-slacks-dnssec-rollout/]
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our fourth episode, we talk with Comcast's Jason Livingood, regarding a recent post he wrote about how Comcast has reduced its working latency through Active Queue Management by 90%! We'll talk about these results as well as the fundamental changes in the way that researchers are now looking at latency and its effect on overall Internet quality.
You can read the post here: Working latency — the next QoE frontier.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our third episode, we are talking again to APNIC's Chief Scientist, Geoff Huston, regarding a recent post he wrote following IETF 111 where a call for 'running code' was excavated from the past during one working group discussion and made Geoff question what has become of the standards process.
You can read the post here: https://blog.apnic.net/2021/08/20/running-code-at-ietf/?utm_source=podcast&utm_medium=organic&utm_campaign=PING&utm_content=description
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
In our second episode, we will be talking to Ignacio Castro from the Queen Mary University of London regarding a recent study he was involved in characterising the demographics and trends in the IETF across its history and how this has impacted the evolution of the Internet. Using a unique dataset of 2.4 million emails, 8,711 RFCs and 4,512 authors, the group examined the shifts and trends within the Internet standards development process, showing how protocol complexity and time to produce standards has increased.
You can learn more about the project via sodestream.github.io where you'll also find a link to their IMC 2021 paper.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.
For our first episode, we will be talking to Internet Hall of Famer and APNIC’s Chief Scientist, Geoff Huston about Facebook’s 6-hour outage on the 5th of October 2021. The show builds on Geoff's blog post and discusses how the network design of Facebook and other large content distribution networks are unique from the traditional ways that networks connect to the Internet as they seek to improve efficiency. However, these unique designs have compromised resilience, which results in outages. So what can we learn from these outages and are they such a bad thing if they ultimately seek to evolve the way the Internet works.
The views expressed by the featured speakers are their own and do not necessarily reflect the views of APNIC.