10 Messages
Intermittent access to third party DNS server
This is a repost, as the previous one seems to have been filtered, I'm guessing due to including shell commands to help illustrate the problem and my analysis of it.
Hi, I have AT&T fiber in the Madison, WI area.
I'm having intermittent access issues to external DNS services like dns.adguard-dns.com, either over port 53 UDP or for DNS over HTTPS (DoH).
I've done a little bit of troubleshooting and found that roughly 50% of the IPv4 packets to these systems, which are served by anycast addresses, simply timeout, whereas the IPv6 addresses are significantly delayed without always timing out.
When I test this from non-AT&T networks, e.g., UW-Madison CS or Microsoft networks (where I work) or even local coffee shops, it works fine. I have a friend in the area who has reported a similar experience.
When I contacted the Adguard.com support people, we did some traceroutes and determined the issue was likely in the AT&T ISP network.
Below is some detailed output:
Here's some nmap output:
(commands omiited to avoid being blocked)
Host: 94.140.15.15 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 2a10:50c0::ad1:ff (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 94.140.14.14 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 94.140.14.14 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 2a10:50c0::ad1:ff (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 94.140.15.15 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 94.140.14.14 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 94.140.15.15 (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
Host: 2a10:50c0::ad2:ff (dns.adguard.com) Ports: 53/open|filtered/udp//domain///
As you can see, there's a subset of the 40 requests that appear filtered, instead of just open. Sometimes it's about 40% of them, in this case it was only about 22%.
Here's another analysis:
1. Check the network via fping:
94.140.14.14 : xmt/rcv/%loss = 100/0/100%
94.140.15.15 : xmt/rcv/%loss = 100/100/0%, min/avg/max = 6.25/7.05/9.98
2a10:50c0::ad1:ff : xmt/rcv/%loss = 100/100/0%, min/avg/max = 6.27/8.31/10.3
2a10:50c0::ad2:ff : xmt/rcv/%loss = 100/0/100%
This seems inconclusive - some of the addresses don't respond to ping at all, probably per network policy. Others do 100% of the time with reasonably consistent times. I'd guess from this that the network routes themselves are probably fine.
1. Collect some data on accessing each of the addresses using DNS over HTTPS via curl, with a connect timeout of 5 seconds
(command omitted to avoid being blocked)
2. Analyze the data:
(commands omitted to avoid being blocked)
Attached is a breakdown of the total time it takes for each address over 60 iterations grouped by the nearest second. Anything over 5 is a timeout:
In general, it appears that the IPv4 addresses are bimodal - roughly half of each return in sub-second times, whereas the other half timeout.
However, the IPv6 addresses are somewhat more distributed with a smaller portion returning slowly (>1 second) instead of simply timing out.
{
"url_effective": "https://94.140.14.14/resolve?name=example.com&type=AAAA",
"times": {
"<1": 28,
"<2": 0,
"<3": 0,
"<4": 0,
"<5": 0,
">=5": 32
}
}
{
"url_effective": "https://94.140.15.15/resolve?name=example.com&type=AAAA",
"times": {
"<1": 33,
"<2": 0,
"<3": 0,
"<4": 0,
"<5": 0,
">=5": 27
}
}
{
"url_effective": "https://[2a10:50c0::ad1:ff]/resolve?name=example.com&type=AAAA",
"times": {
"<1": 34,
"<2": 18,
"<3": 2,
"<4": 3,
"<5": 1,
">=5": 2
}
}
{
"url_effective": "https://[2a10:50c0::ad2:ff]/resolve?name=example.com&type=AAAA",
"times": {
"<1": 33,
"<2": 17,
"<3": 1,
"<4": 7,
"<5": 0,
">=5": 2
}
}
Let me know if you need any other info that could help you solve this issues.
Thanks!
bpkroth2
10 Messages
4 months ago
Btw, this one also looks related:
https://forums.att.com/conversations/att-fiber-equipment/no-longer-able-to-use-3rdparty-dns-servers-after-12hour-outage/65dffa69d1337129dfdda619
0
0
bpkroth2
10 Messages
4 months ago
Here was the one I previously tried to post:
https://forums.att.com/conversations/att-fiber-account/intermittent-access-to-third-party-dns-server/65e52ed1d1337129dfddee7c
It'd be great if someone could unblock that account :)
0
0
ATTHelp
Community Support
•
232.9K Messages
4 months ago
Hi @bpkroth2! Thank you for reaching out to us. We understand that you're facing issue using third party DNS servers. We hear you, and happy to assist you.
Please use IP Passthrough and make sure you're configuring IP Passthrough with the steps present in our article.
We recommend that you try optimizing your internet connection, this will help boost your internet speed.
Please let us know the below details to help you better:
Let us know how it goes.
For further assistance we're here to assist you.
Thank you.
Nate, AT&T Community Specialist.
0
0
bpkroth2
10 Messages
4 months ago
Hi, I'm using IP passthru already.
It's less of a speed issue, and more of a packet loss issue, and only to certain anycast hosts, which doesn't occur when I test accessing those hosts from other networks.
1. I have AT&T Fiber 500Mbps plan.
Speed tests via Google (to Chicago) show 283Mbps download and 500Mbps upload. Speed tests via AT&T's official site show 603 and 615 Mbps respectively for download and upload. So something in the egress from AT&T's networks are a bit slow unfortunately. At least one of those went over IPv6.
2/3. I have only a single router device attached to the modem directly, with other home devices attached from there. The issue described above occurs on all of the devices, including the router and even the modem itself. This is part of what makes me think the issue is internal to AT&T's networks.
0
0
ATTHelp
Community Support
•
232.9K Messages
4 months ago
Hi @bpkroth2! Thank you for writing us back! We understand that you're facing issue with packet loss. We hear you, and are happy to assist you.
We recommend that your try resetting your Wi-Fi gateway to original settings.
Heads up: Be sure to jot down your custom settings, like your static IP address if you have one, or your Wi-Fi network name (SSID). You can then change them back after the factory reset.
Options after a gatewat reset:
Find or change your Wi-Fi info:
After resetting your Wi-Fi gateway, please reconfigure IP Passthrough from our article and check.
If issue persists, please let us know.
For further assistance we're here to assist you.
Thank you.
Nate, AT&T Community Specialist
0
0
bpkroth2
10 Messages
4 months ago
Hi, sorry for the delay, I was away for a day.
I've tried resetting the modem, but it didn't help.
Given there are
1. other users with the same problem in my area, and
2. half of the time packets are received to these addresses, and
3. it's only these addresses that seem to be affected, and
4. I don't see the same problem off of the AT&T Fiber ISP network,
I was thinking it was more of an internal routing issue on AT&T's end.
Can you please escalate this? Thanks!
0
0
ATTHelp
Community Support
•
232.9K Messages
4 months ago
Hi @bpkroth2, We understand that you're facing issue using third party DNS servers. We hear you, and happy to assist you.
The Community Forums are a public support option where other users, and AT&T, will try and assist with high level support needs. This means we won’t be able to look into account specific concerns. To get the help you need for your unique issue, please review our Contact Us page - https://www.att.com/support/contact-us/, and choose the best option to reach out to us. You can call, chat, or reach out via social media, and we can review your specific issue and provide you support. If you feel your issue isn’t account specific, and can be answered generally, please let us know, and we’ll be happy to help.
Thank you for contacting AT&T Community and Forums,
Ancy, AT&T Community specialist.
0
0
geastman
1 Message
4 months ago
I'm also in the Madison area on AT&T Fiber and experiencing the same issues mentioned. Other third party DNS servers work fine, but only have problems with adguard, both traditional DNS as well as DNS over TLS configured on the router. If I override DNS locally on an endpoint device (also tried DNS over HTTPS there), I get the same problems while on AT&T, but not elsewhere.
0
bpkroth2
10 Messages
4 months ago
Yeah, definitely seems like an AT&T internal routing issue. Quite frustrating that there's no recourse here. The support desk similarly gives the run around but has no external accountability. Here at least we can see that others are also experiencing the same problem.
0
0
JefferMC
ACE - Expert
•
36K Messages
4 months ago
"anycast" or georouting can lead to issues like this. I'd suggest switching to a service that doesn't use it.
0
0