Why Route 53 geolocation routing is not accurate?
Background
Route 53 Geolocation routing lets you choose the resources that serve your traffic based on the geographic location of your users, meaning the location that DNS queries originate from.
For example, when you have create two resources in US and EU. You would like to let the users from US to access the resource in US and let the users from EU to access the resource in EU.
You can set the Route 53 record as following
Then we can use third-party tool(RIPE Atlas) and probes in US and TW to check DNS resolver result.
Probes in US
Probes in TW
Problem
In the previous DNS resolve test, I used 10 probes each in the US and TW to resolve the domain by the probe’s DNS resolver. All resolve results from US probes returned “US”. Regarding the resolve results from TW probes, most of them returned “TW” and a few were “JP” or “US”. This tells us that the Route 53 geolocation routing is not always accurate, even when using just a random sample of 10 probes to test. What issues will this bring? It will cause trouble when you want to load balance the traffic and allow users to access resources that are nearby their geolocation.
To find a solution, we need to identify what is the root cause.
In first, let’s find what information we got
I download the detail result from TW probes which got “JP”.
cat tw-geo.txt | jq -c '.[] | select(.resultset[].result.answers[].RDATA[]=="JP")' | jq .
{
"fw": 5080,
"mver": "2.6.2",
"lts": 12,
"resultset": [
{
"time": 1723674576,
"lts": 12,
"subid": 1,
"submax": 2,
"dst_addr": "1.1.1.1",
"dst_port": "53",
"af": 4,
"src_addr": "10.120.85.39",
"proto": "UDP",
"result": {
"rt": 72.641,
"size": 69,
"ID": 63923,
"ANCOUNT": 1,
"QDCOUNT": 1,
"NSCOUNT": 0,
"ARCOUNT": 1,
"answers": [
{
"TYPE": "TXT",
"NAME": "geo.********.io",
"RDATA": [
"JP"
]
}
]
}
},
{
"time": 1723674577,
"lts": 13,
"subid": 2,
"submax": 2,
"dst_addr": "8.8.8.8",
"dst_port": "53",
"af": 4,
"src_addr": "10.120.85.39",
"proto": "UDP",
"result": {
"rt": 66.373,
"size": 72,
"ID": 36447,
"ANCOUNT": 1,
"QDCOUNT": 1,
"NSCOUNT": 0,
"ARCOUNT": 1,
"answers": [
{
"TYPE": "TXT",
"NAME": "geo.********.io",
"RDATA": [
"TW"
]
}
]
}
}
],
"from": "1.162.77.167",
"type": "dns"
}
From the detail, we can see the Google Public DNS(8.8.8.8) return “TW” but the Cloudflare DNS(1.1.1.1) return “JP”. Both of them are well known public DNS resolver and operator by technical company. What make them have this result?
In Second, let's recall how DNS work
In usually, when we send DNS request to public DNS resolver(e.g. 8.8.8.8, 1.1.1.1). They will help to query with Root DNS nameserver, TLS DNS nameserver and until to Route 53 DNS nameserver.
In the step 6, the DNS query will send to theRoute 53 DNS nameserver and the nameserver will resolve the domain according the information in that query and return the value.
In third, let’s check how Route 53 estimate the location of a user.
According to the AWS public document[1], Route 53 will use edns-client-subnet(ECS), which is one of the extensions of EDNS0.
If the DNS resolver does not support ECS, Route 53 will use the source IP address of the DNS resolver to approximate the location of the user.
If the DNS resolver does support ECS, the DNS resolver sends Route 53 a truncated version of the user’s IP address. Route 53 determines the location of the user based on the truncated IP address rather than the source IP address of the DNS resolver.
Why the value return to 8.8.8.8 is different with the value return to 1.1.1.1?
We can see the query happened at 1723674576 (2024–08–14T22:29:36) and 1723674577 (2024–08–14T22:29:37). After I checked the Route 53 Public DNS query logging, I found there are two records aligned with the timestamp.
1.0 2024-08-14T22:29:37Z Z02***************NOI geo.********.io TXT NOERROR UDP HKG54-C1 2404:6800:4008:c01::12d 1.162.77.0/24
1.0 2024-08-14T22:29:36Z Z02***************NOI geo.********.io TXT NOERROR UDP NRT12-C2 162.158.117.81 -
Each log entry includes following values
- Log format version
- Query timestamp
- Hosted zone ID
- Query name
- Query type
- Response code
- L4 protocol
- Route 53 edge location
- Resolver IP Address
- EDNS client subnet(ECS)
But how do we know which query is from Google and which query is from Cloudflare? We can check the resolver IP address by searching in RIR tools. For example, I put both resolver IPs in RIPEstat and search. I can confirm the IPv6 address (2404:6800:4008:c01::12d) is used by Google and the IPv4 address (162.158.117.81) is used by Cloudflare.
We can see the difference between both queries is ECS. The query from Google has brought ECS and Cloudflare has not. After checking the document[2] about Cloudflare DNS (1.1.1.1), it states that 1.1.1.1 does not send the ECS header to authoritative nameservers due to privacy reasons.
Now we know that 1.1.1.1 does not send ECS and Route 53 nameservers use resolver IP to estimate location. However, how do we know if 162.158.117.81 is not in Taiwan or if Route 53 geolocation routing has issues?
To estimate the location of 162.158.117.81, we can use a third-party tool to test the latency. For example, we use multiple probes from different vendors but all in Tokyo to ping the resolver IP (162.158.117.81). We can see all latencies are around 1ms. Then we use multiple probes from different vendors but all in Taiwan to ping the resolver IP (162.158.117.81). We can see the minimal latency is around 30ms. With both latency results, we can conclude this DNS resolver uses the IP located in Tokyo, JP to query Route 53 DNS nameservers.
Now, we can confirm that the reason the DNS resolver (1.1.1.1) returned “JP” is because Cloudflare used the IP located in Japan to query the Route 53 nameserver without ECS information.
How do we know if the DNS resolver support ECS?
I would suggest you use following command to check if DNS resolver support ECS.
$ dig +short TXT o-o.myaddr.l.google.com @<DNS resolver IP>
For example, if we test with Google Public DNS, Quad9, Cloudflare DNS, and OpenDNS, we can see if the DNS resolver supports ECS. If the dig output contains one line that starts with “edns0-client-subnet,” that means the DNS resolver supports ECS.
// Google Public DNS(8.8.8.8)
$ dig +short TXT o-o.myaddr.l.google.com @8.8.8.8
"edns0-client-subnet 43.207.71.0/24"
"172.253.236.18"
// Quad9 DNS(9.9.9.9)
$ dig +short TXT o-o.myaddr.l.google.com @9.9.9.9
"103.194.167.231"
// Cloudflare DNS(1.1.1.1)
$ dig +short TXT o-o.myaddr.l.google.com @1.1.1.1
"172.68.117.220"
// OpenDNS(208.67.222.220)
$ dig +short TXT o-o.myaddr.l.google.com @208.67.222.220
"2a04:e4c0:20::39"
"edns0-client-subnet 43.207.71.0/24"
Recommendations
To make Route 53 geolocation routing more accurate, I would suggest you try your best to make clients use DNS resolvers like Google Public DNS or OpenDNS. These DNS support ECS and we can estimate the location of client.
However, if you confirm the DNS resolver supports ECS but the Route 53 nameserver returns the wrong value with the wrong judgment, then you might need to do some other troubleshooting. We will have another story to discuss that.
Appendices
