Random disconnects with wired connection

[Fathomless]

Case support information submitted. I might need to submit repro will work on it tonight, if not tomorrow.

[Netduma Fraser]

Thanks Fathomless, really appreciate your help on this issue!

[Fathomless]

Now i am getting repros every day, all i did reconfigured access point(R6700) automatically via the wizard, before it was configured manually i connected, from switch port(4 ports) to switch port on xr500, or maybe XR500 degrades,

Somehow even when USB selected it does capture into Memory, maybe it does not recognize 64GB?... but system tells me that i have 64 GB and it is formatted by the system(i had fat32 formatting before, now my computer cannot access it). So i did not risk to leave capture overnight.

...at this point submitted before, during, after reboot (capturing boot state), after(normal operation).zip

 

going to schedule a call with T2 at 6pm.

[Netduma Fraser]

Awesome thanks Fathomless! Also not sure what happened there with the USB.

 

Please could you also send the files to [email protected] so I can also forward to Netgear to make sure it gets to the people it needs to.

[Fathomless]

done

[rckrz6]

I had a repro yesterday as well, it takes quite a bit of time to repro. This time it happened while I was playing Xbox, and switching it into a hibernation mode a few times a day, there was almost no traffic on the network, but Ethernet still failed. I will try to set debugging today .

 

On positive side, when XR500 works it is pretty good experience, I am taking 1/2 places frequently,  camping less and running with shotgun more, my average K/D went from1.29 to 1.33 in a few days, average K/D was 4 when playing with XR500, I hope it is not a placebo effect

 

Just a random thought, I was wondering if Xbox switching to 10/100 during hibernation might has something to do with it? when it hibernate switch port will have amber light, when xbox is ON light would be white again and probably set to 1000....There was a bug with some cisco switches for a while, when they did not support duplex AUTO selection well, port on the blade configured to Auto duplex would randomly fail, fix was to configure duplex on the switch to 100/1000full. 

 

I called NG. It is impossible to get ahold of the level2 personal, I hope they are all working on this. They suppose to call me back 6pm, I would like to get status on my case and have cross correlation with other similar reports. How many same cases netgear needs to fix this issue, I think they already have debugging info

 

 

This is what my xbox x does in hibernation mode, it usually boots back up ok from hibernation mode with a amber light in sleep, if the light is white for a long period of time while in hibernation mode it comes back to no internet connect and i have to reset the xbox, i have't tired taking out and plugging in the network cable yet.  Im not 100% sure if its an xbox isssue or a router issue but it takes a day or 2 to for it to happen

[Fathomless]

 Yes pattern is consistent if you wake up and switch port light - white and xbox is in hibernation, just reboot the router. All we do now is black box testing, until netgear comes back with acknowledgment of the problem.

 

in this scenario there are a few option to validate:

- remove the cable so it limits switch to stay in one pattern

- install cat3 cable so it will limit switch to stay in 10/100mbps since cable is restrictive.

- install additional switch. it could be 1gbps or 100mpbs(my choice since it is easy to find one, any old router can do as well just disable DHCP and connect everything to switching ports on the router)

 

i found netgear GX105 undamaged switch, connected it after XR500, if problem does not come back tonight there might be a hope for simple mitigation.

 

my current config:

all equipment netgear for simplicity if i need to file another case i will not be asked what driver version my network card on computer has.

 

Comcast(ISP)->CM1000(modem)-> XR500(router)->GS105(switch)->R6700(access point)

 

tonight will play on and off again and tomorrow morning will check for a problem, i will report back around 8am Pacific.

[Netduma Fraser]

I haven't been able to reproduce the Xbox wake from sleep issue. I am not sure on how long it would need to be off in order for me to be able to see the issue. 

 

I did speak with Netgear and they think it could possibly to do with DHCP lease as I think you found Fathom. They'll be going over the logs and will let you know if I need anymore. 

 

For now don't have it in instant on mode and connection should be on everytime you turn it on.

[Fathomless]

would be interested to see what they conclude, if two devices one with static and another with dynamic connected to XR500, they both fail. Device with static IP cannot ping the gateway anymore and device with dynamic cannot reach the DHCP or DHCP request corrupt or maybe DHCP response not reaching the client, in both cases they cannot pass XR500 switch or port. All 4 ports would act the same way.

 

With my current setup where one wire coming to XR500 and another is connected to the switch, issue did not happen yet, will be waiting more... interesting to see that xbox is still in hibernation mode(it has 10/100indicator) and gs105 is showing no signs of port failures. Wired connection from access point to internet is still 160mpbs

 

if issue happen i can get debug logs from both devices XR500 and R6700, if need to do traffic capture i probably need to get managed switch GS105E, so i can do port mirroring and capture the traffic between source and destination, from PC

[Netduma Fraser]

I've forwarded this information onto Netgear along with the captures you sent by email. Will definitely know should I need anything else or if theres another way they'd want to collect information. Would you be available for them to access via TeamViewer if the issue occurs? They are GMT +8.

[Fathomless]

I will be available, after 6pm until 8am,  Sat/Sunday in the morning Pacific time. Not available from May11th to May18th.

Tonight I will add prove to the case why it is not DHCP issue I had boundary condition test done with dynamic, static IPs where I pinned default gateway via wireless and wired connection.

 

When we test switching there will be a few conditions:

 

1. Cables disconnected - probability is that condition impact switching side of the router is low

2. Cable connected time to time when disconnected less power to the switch - at this stage we can verify if overheating is impacting the switch module. probability for failure still low, but if failure happen most likely customer will get disconnect during the game play

3. Devices always connected - that is most difficult scenario

- overheating could be involved

- devices interaction and duplex changes

 

 

I have Xbox one first generation.

[Netduma Fraser]

I don't think they'll be available then unfortunately but I'll pass that on. 

 

Okay do keep us updated, really appreciate your attention to detail!

[Fathomless]

Could you please send me their availability times(local time or pacific would work), and phone number. I can probably find time, to debug it, will take day off or could work from home,  right now I am just waiting if it fails again. If they need to set debugging session I just need heads up maybe 12-24 hours.

[Netduma Fraser]

Great thanks, will let you know!

[rckrz6]

24-48 hours, just happen again i put it in sleep mode last night, came home tonight and it says no network connection, i disconnect the cable from the back of the router and plugged it back in and it doesn't do anything.  I just power reset the router didn't fix it, i switched ports and it didn't fix it..  Only restarting the xbox fixes it so i guess it narrows it down to the xbox

I haven't been able to reproduce the Xbox wake from sleep issue. I am not sure on how long it would need to be off in order for me to be able to see the issue. 

 

I did speak with Netgear and they think it could possibly to do with DHCP lease as I think you found Fathom. They'll be going over the logs and will let you know if I need anymore. 

 

For now don't have it in instant on mode and connection should be on everytime you turn it on.

[Bmann]

I just switched my Xbox over to "instant on" it was in power save mode probably due to an update changing my settings.  I was loosing internet connection from the idle Destiny screen while in orbit but the xbox was still online.  still troubleshooting disconnects issues on the Router itself.

[Fathomless]

I just switched my Xbox over to "instant on" it was in power save mode probably due to an update changing my settings.  I was loosing internet connection from the idle Destiny screen while in orbit but the xbox was still online.  still troubleshooting disconnects issues on the Router itself.

 

I think you suggested another mitigation until Netgear would find a bug with switching, they know how to manufacture switches. i hope they can talk to xbox team, to sort this problem out.

 

...potentially we might have 2 mitigations:

1. Configure Xbox, there was similar problem with x10, which might use similar hardware, strange thing is that Netgear ignored this issue, or did not release the fix yet.

https://community.netgear.com/t5/Nighthawk-WiFi-Routers/X10-R9000/m-p/1295894

 

2. install reliable switch similar to Netgear GS105 to offload switching to business device, still testing XR500 and GS105,  XR500 did not disconnect switch ports yet, and GS105 handles corks of XBOX well. (crossing fingers)

 

in my setup i played 2 evenings and XR500 did not disconnect GS105 switch yet, will report back at 8am Pacific.

[Netduma Fraser]

24-48 hours, just happen again i put it in sleep mode last night, came home tonight and it says no network connection, i disconnect the cable from the back of the router and plugged it back in and it doesn't do anything.  I just power reset the router didn't fix it, i switched ports and it didn't fix it..  Only restarting the xbox fixes it so i guess it narrows it down to the xbox

 

Does the Xbox have a static IP? Is that mode the one where it takes its time to boot up or the quick one? 

 

I just switched my Xbox over to "instant on" it was in power save mode probably due to an update changing my settings.  I was loosing internet connection from the idle Destiny screen while in orbit but the xbox was still online.  still troubleshooting disconnects issues on the Router itself.

 

I don't think that would affect the Destiny connection, that would be an Xbox issue if anything or Destiny having some online issues. 

 

I think you suggested another mitigation until Netgear would find a bug with switching, they know how to manufacture switches. i hope they can talk to xbox team, to sort this problem out.

 

...potentially we might have 2 mitigations:

1. Configure Xbox, there was similar problem with x10, which might use similar hardware, strange thing is that Netgear ignored this issue, or did not release the fix yet.

https://community.netgear.com/t5/Nighthawk-WiFi-Routers/X10-R9000/m-p/1295894

 

2. install reliable switch similar to Netgear GS105 to offload switching to business device, still testing XR500 and GS105,  XR500 did not disconnect switch ports yet, and GS105 handles corks of XBOX well. (crossing fingers)

 

in my setup i played 2 evenings and XR500 did not disconnect GS105 switch yet, will report back at 8am Pacific.

 

Very interesting on number 1, thanks for the link!

[Bmann]

Bmann, on 02 May 2018 - 9:22 PM, said:

I just switched my Xbox over to "instant on" it was in power save mode probably due to an update changing my settings.  I was loosing internet connection from the idle Destiny screen while in orbit but the xbox was still online.  still troubleshooting disconnects issues on the Router itself.

 

I don't think that would affect the Destiny connection, that would be an Xbox issue if anything or Destiny having some online issues. 

 

 

I would agree with that but never had these issues until I installed the Netgear.  This might just be a symptom of the random disconnects from hardwired and wifi connections and i know that its the Netgear that is causing this since I can unplug the router from the equation and go direct from cable modem to PC and no issues at all and even running ping etc against the cable modem and its very stable with no spikes.

 

I have replaced cables going from CAT6 to CAT5e from router to Xbox and IP has always been static.  The frequency in the logs to me indicate that it is aggressive in its DHCP frequency and I cannot change this just to see if it has any effect.  I have also disabled 5G wifi and disabled smart connect doing this it seems to have made it crash less.  Could be all placebo effect though.

[rckrz6]

The xbox has a dynamic ip, im using the instant on where it doesn't really shut all the way down.   If i completely power it down obviously the issue won't apear but it takes 100 times longer to boot up

[Fathomless]

Update: no issue yet.

Uptime 14:50:02 up 2 days, 8 min, load average: 4.75, 4.43, 4.29

 

I forgot to mention that when xbox was directly connected to the XR500 i was kicked off the game (Call of Duty) once in a while. i did not have any second thought about it since i was not doing well and my first thought was that behavior was by design, after i connect back i would join better lobby , i do not have disconnects anymore.

 

on positive note.

... KD is still growing relatively fast 1.35 now, shot gun became my first weapon on all game plays, it seems to be contagious or people just pissed, now i see 4-5 people running with shotguns, i like pump auction

I do not play hardcore anymore, since in normal mode my shots will register well enough.

[Netduma Fraser]

 

Bmann, on 02 May 2018 - 9:22 PM, said:

 

I don't think that would affect the Destiny connection, that would be an Xbox issue if anything or Destiny having some online issues. 

 

 

I would agree with that but never had these issues until I installed the Netgear.  This might just be a symptom of the random disconnects from hardwired and wifi connections and i know that its the Netgear that is causing this since I can unplug the router from the equation and go direct from cable modem to PC and no issues at all and even running ping etc against the cable modem and its very stable with no spikes.

 

I have replaced cables going from CAT6 to CAT5e from router to Xbox and IP has always been static.  The frequency in the logs to me indicate that it is aggressive in its DHCP frequency and I cannot change this just to see if it has any effect.  I have also disabled 5G wifi and disabled smart connect doing this it seems to have made it crash less.  Could be all placebo effect though.

 

 

I will try to reproduce this with Destiny, thanks for the info.

 

The xbox has a dynamic ip, im using the instant on where it doesn't really shut all the way down.   If i completely power it down obviously the issue won't apear but it takes 100 times longer to boot up

 

Okay glad to hear we've narrowed down on this. I will try to reproduce this.

 

Update: no issue yet.

Uptime 14:50:02 up 2 days, 8 min, load average: 4.75, 4.43, 4.29

 

I forgot to mention that when xbox was directly connected to the XR500 i was kicked off the game (Call of Duty) once in a while. i did not have any second thought about it since i was not doing well and my first thought was that behavior was by design, after i connect back i would join better lobby , i do not have disconnects anymore.

 

on positive note.

... KD is still growing relatively fast 1.35 now, shot gun became my first weapon on all game plays, it seems to be contagious or people just pissed, now i see 4-5 people running with shotguns, i like pump auction

I do not play hardcore anymore, since in normal mode my shots will register well enough.

 

CoD was probably just finicky, that does happen now and again. Awesome to hear you're having a good time!

[Fathomless]

I think potentially we found 2 mitigations, there are more if we switch to PlayStation , but we need commitment from Netgear.

 

I do have instant-on feature as well and I like it, so in my case I have to have additional switch. It would be nice to know if Netgear can resolve this volubility, and it should be framed as security issue, here is why and how

 

I looked at other forums, and it seams that instant-on xbox feature casing DDOS for line of the Netgear routers or layer 3/2 devices, this is very interesting because some netgear switches(leyer 2 device on OSI model) resistant to that behavior. When switch port will get DDOSed by xbox one, such attack disables all 4 ports, they will have light and everything appear blinking, but layer 3 (networking) traffic will not flow. That would be big issue, because such behavior can compromise; small government agency, small business security system or home security system if anything sensitive connected via the netgear routers. If X10 and XR500 prong to that -> all negtear routers are the same way until proven otherwise. 

 

The attack could be mimicked, all it requires to learn normal and abnormal by capturing feedback from the router and xbox one(port mirror with xbox and look at communication, memory dump from xbox and netgear)

 

So if you want instant-on feature not killing your home security system, you have to install Firewall and that would be a simple switch such as GS105 between your XR500 and rogue device such as XBOX ONE

 

How do we file security incident with netgear? Security incident should have highest priority.

[Netduma Fraser]

I don't know how you would specificly alert them to that but I will contact them with this latest info.

[Fathomless]

I think we found mitigations, tested setup is stable if I go until Sunday/Monday without reboot, we can wait on conformation form other members and call it done, or investigate a little more.

 

From other prospective what should we do and what is right?

 

The netgear bug is not resolved, my understanding netgear is looking at DHCP(symptom not a root cause), they will close this as no repro. NG could repro it if they connect 3-4 Xboxes of different model into a switch of the router and lunch and hibernate it for 2-8hours leave overnight and in the morning save finding, but I have serious doubt they are passionate or enthusiastic about it.

 

Situation is a funny in sense that gaming console kills gaming router, and there is no official SOP on mitigation steps, it is even worse than that NG cannot acknowledge this problem for years(X10, R7800) and they don't have profiling techniques as well.

 

What would be our next step?