Keep in mind first that playing on the Xbox Series X (XSX), when you select an older game, it starts some sort of Xbox 360 emulator as the old Xbox 360 start screen and music comes up. Also, it links to the traditional Xbox 360 home screen for invites and such.
Playing MW2 (2009) and older games, with any settings enabled on the router (QoS, geo, etc) the game has issues and won't connect most of the time. It not only won't find a game, but when the game starts on the XSX, it shows that I'm not even online on xbox live. Yet, when I go to the XSX home screen, I'm online and can play other games and navigate the store and xbox game pass. I had some success playing yesterday wirelessly, but obviously that isn't ideal. It still had numerous issues finding matches and would boot me from games.
If it does manage to show I'm on xbox live, I'm constantly getting a strict NAT even with port forwarding for the classic games, where I don't have that issue with the newer games.
Hence, only hooking from modem to console and bypassing the router eliminates (for the most part) those issues.