Random Spikes of High Ping and Latency

 I currently have AT&T U-Verse and I'm on the AT&T Internet 25 plan. My router is the NVG599 and I am connected to a wireless access point, 2 wireless tv receivers, 2 wired tv receivers, 2 wired Xbox One's, 2 wireless laptops, 3 phones, and a Kindle. 

I like to play lots of games and so does my little brother, hence there being 2 Xbox One's. He still goes to high school while I now go to college, but I still live with him and my parents. Since I'm in college, my school day ends before his does, so in between him not being home and my parents at work, I play games on my Xbox One. Going a few months back, I was playing games on a wireless connection and just recently decided to buy new ethernet cables to make my brother's Xbox and my Xbox wired. I had frequent lagging issues when I played on wireless and when I switched to wired most of those issues went away except for one thing.

That one issue that still remains is my ping randomly skyrocketing. Usually when playing I get around 60-100 ping which is completely understandable since I live in the midwest and live a fair distance from the city, but at least once a day my ping shoots up to 400+ and my download speed goes below 1mbps for at least 5 minutes. I looked through other forums and saw that this issue could be because of the ethernet cable but I tested the connection on my wireless laptop and got the same ping results as I was getting in game on a wired gaming console.

Having a wired connection has really helped solve my lag issues I use to always have, but I still do have lag from time to time. Most of my lag comes from when me and my brother are playing at the same time which again is completely understandable because AT&T doesn't support UPNP and the plan I'm on isn't meant for multiple high speed gaming consoles running at the same time, but the big latency spikes happen when the only thing in my house using the internet in my wired Xbox One. 

Finally I wanted to include that I have called AT&T and had multiple times where someone came to my house to look at my internet. Obviously, since the spikes are random, they never got to see the ping go above 100ms. One time I had my NVG599 router replaced, a couple other times they replaced a line in my backyard, but the most notable one was when someone looked at the neighborhood box and said that it was a mess, which, like I've said before, I don't have a problem with because I live in an older neighborhood and I'm probably the only person in the neighborhood that cares about having good internet so it makes sense that spending thousands of dollars just to make me happy is not a priority. On top of that, I've done some research and I don't think that the box would make my ping spike like it does.

To sum this up, I just want to know if there's anything I can do to stop or potentially lower the severity of these ping spikes or is it something that just happens that I don't have any control over. Besides this I've had no other problems with AT&T and I personally think that the internet I'm getting is exceptional for the price. I've tried calling about this issue before but was told that I would have to sign up for some sort of gaming assistance program that would cost me more which I can't afford so I thought I would come to the forums for help. If anyone has any tips or suggestions they all would be greatly appreciated. Thank you guys very much.