Recent versions of firmware cause sudden latency increases after being online for a number of days

The first thing I would do is factory reset the gateway and check again.  Which BGW?

Model Number BGW210-700

Software Version 4.26.11

I really don't want to do a factory reset and put all my settings back in.

(about 6 months ago it apparently decided to do a reset itself after an update and dropped my static IP settings, sigh)

Hey @hardaker, we understand you’re experiencing high latency, let's get the help you need & walk you through the right path.

First, we recommend that your view our article that covers High Latency & Ping. It has helpful tips such as what is acceptable per our standards, how to rule out various causes of high latency, and understanding your traceroute!

If this is happening with specific sites or games, we do recommend running a traceroute and sharing the results here. This way we can go over them and see if it's o our end, or the game/sites end.

If you have a 2Wire/Pace gateway, use these steps:

1. Type http://192.168.1.254 to access the gateway.
2. Select Settings > Diagnostics > IP Utilities

If you have an NVG/Motorola model, use thses steps:

1. Type http://192.168.1.254 to access the gateway
2. Select Diagnostics

Let us know if this helps.
Thank you for reaching out on AT&T Community & Forums.
Bruce, AT&T Community Specialist

Does your utilization also increase during these times of high latency?  Perhaps your restart is interrupting some intensive data transfer (that you may not know about or desire).

Nope, the graph above (pings at regular 1m intervals 24hr/day) shows that it happens at all times of the day.  And it's Gb fiber so it's not a bandwidth issue as almost all of the time it's massively underused bandwidth-wise.  And the fact that restarting the box fixes it means it's certainly not related to bandwidth.

Here's 24hrs of ping times (reset happened about 07:00 in the graphs) and 24hrs of bandwidth usage, eg.  They don't align well with ping times showing latency even when bandwidth was low.

Many are the times that rebooting a gateway restored performance because it interrupted usage and the performance was good until that usage resumed.   You could be exhausting the session state table (you can check that in Diagnostics > NAT Table).  Or there could that you have discovered a bug in the firmware; wouldn't be hugely surprised.  I hope that's not it, actually, because who knows if they'll ever fix it.  It's almost impossible to  route feedback to them unless there are a bunch of users showing the exact same symptom.

Are you checking other DNS servers?  Are they all going up together?  I ask because I have one DNS server (Cloudflare) of many I check (Verizon, AT&T, Google) that has doubled or tripled in latency AND is going up even further during peak hours.

I'll check the NAT table (though it's a bit annoying that it's even lists stuff from my static allocation that shouldn't be NATed at all).  I found this table shortly after rebooting.  Good suggestion though, thanks.

But considering this appeared as a new problem in March, I suspect it's a firmware bug.  Looking at my graph of the last *year* of ping data shows almost no issues until March when there is suddenly lots of issues.  The longer graphs are a bit harder to show with detail because of aggregation, but actually my HTTP server monitoring graph much more clearly shows the problem over the longer trend.

Curious, is that Grafana?  If not, what is it?  How/where are you storing your metrics?

Yes, it's grafana.  Metrics are collected stored on a machine behind the router that acts as a firewall between the router and everything else.

(which does bring up a good point -- I should verify the stats with a different cable and port behind the router too.  The reset could just be resetting problems on the downstream side of the router)