I think I managed to fix it. I retraced all my steps of the settings I had changed (Geo-Filter, QoS etc.) all the way to the beginning. I tried setting the default values for my network speeds 1000 down / 1000 up (in the beginning I ran the automatic test for the speeds) and after this change the issue was fixed.
Don't know if this is just luck but hey give it a shot @wrighty338 if it works for you as well.