I DID NOT WRITE THIS - IT IS FROM HERE BUT I THOUGHT IT WAS AN IMPORTANT READ.
KEEP IN MIND THIS IS OLD INFORMATION, BUT THE BASIC PRINCIPLES STILL APPLY TODAY WITH DEDICATED SERVERS AND LAG.
In not so recent developments, lag has taken the number one spot as a scapegoat for a lot of players' deaths. So much so that people completely made up explanations as to how "lag comp screws you over" or whatever. On top of that, this shit is actually considered common knowledge in some places (gamefaqs, reddit, official forums, youtube, etc.). I'm sure that most of you have seen Den's quick post on lag compensation/anti-lag, but I'm going to expand upon that. I'm just trying to dispel some of the stuff surrounding that for the sake of this forum since I'm sure a good number of our users come from those sites i mentioned. Now keep in mind that we obviously do not have any networking source code. This is going over the basics of a listen server alongside more gameplay related issues such as hit detection. It's important to note that while Treyarch and IW split off from each other at CoD4 (as in, both MW2 and WAW were modded from CoD4 and they've been modding their own game since then) there isn't really any evidence of them having changed the core networking mechanics beyond deciding hosts and organizing matches*. THAT BEING SAID, this is a fairly long read. The important points of each section will be printed in gold and key terms will be in green* for those of you who don't really care all that much.
*black ops 2 has shown to have networking issues as far as artificial delays go, however there is no evidence as to why this happens, nor any discernable reason as to why it could be included. It is likely a mistake. It will covered further down.
*I apologize to the colorblind people who may have some trouble skimming because I picked green for keywords
Table of Contents
[1.0] How does networking work?
- [1.1] Network Structure
- [1.2] Transferring Data
[2.0] How does lag affect me?
- [2.1] Latency, Jitter, Three bars, etc.
- [2.2] Input Prediction
- [2.3] Hit Detection and Lag Compensation
- [2.4] Interpolation and Artificial Delays
- [2.5] Packet Loss
[3.0] How does this come into play?
- [3.1] Connection Discrepancies pt.1
- [3.2] Connection Discrepancies pt.2
- [3.3] Killcams
- [4.1] Sorting players
- [4.2] Hosts and Migrations
- [5.1] What is NAT?
- [5.2] Opening your NAT
[1.0] How does networking work?
[1.1] Network Structure
Obviously multiplayer games need a system that is capable of transferring data between the players. Virtually all modern multiplayer videogames utilize Client-Server networking or Peer-to-peer networking. In the case of Call of Duty, it uses a certain type of Client-Server based network called a Listen Server. This differs from the standard Dedicated Server (The PC versions of BO2, CoD4, and WAW use this) in that a listen server hosts both the server and client in the same location. Listen servers are generally inferior to dedicated servers in every way except for price, because it is more stressful on a single system and is limited by the host's internet connection. To simply put it, Clients send data to the server, which then organizes it and sends it out to all of the clients simultaneously, which then use that data to generate the world. Here's a slightly closer look using the Quake 3 networking model:*
*The IW engine is based off of the Quake 3 engine along with countless other shooters. They have their distinctions but the basics are pretty standard.
THE SYSTEM WORKS AS SUCH:
So the Server or Host has a Master Gamestate. This is the "official" state of the game at any given time. The host will attach a sequence number and a timestamp onto it and send it out to the clients. This is called a Snapshot. When the client receives this data. It makes a copy of the master gamestate and saves it in its memory. The client will then compare the snapshots in memory to generate a game state. If the client doesn't have any "valid" snapshots in history (either the last timestamp was too long ago due to lag issues or it just has connected and hasn't received data yet) the client will compare the snapshot with the dummy snapshot, which has every value equal to zero. The client will at the same time send inputs from the player or game (movement, hit detection, etc.) to the Server. The client will also acknowledge that it has received the previous snapshot and tell that to the server. When the server receives this data from the clients it utilizes it to create a new master gamestate, and acknowledges each client that it received the data. Lather, rinse, repeat. *
[1.2] Transferring Data
The biggest setback of online gaming is that it has to obey the laws of physics. That being, data does not travel instantly. This causes major discrepancies in the actual game, but we'll get to that later (or skip to 2.1 if you don't give a fuck about how it works).
Now all games typically choose to use TCP or UDP for data transfer. In short, TCP is very reliable at getting the information in its entirety and order from one location to another. However, the manner in which it does this is slow for various reasons. This is far too slow of a process in multiplayer games but it is perfectly fine for organizing matches, stat tracking, or pretty much anything that can afford to take more than half a second. UDP basically send the packet out with a destination stamped onto it and hopes it gets there; and it does that pretty well. The main issues with UDP is that packets tend to arrive out of order and it's quite prone to packet loss (an estimated 1-5% of the data likely won't arrive, this can of course vary with how matchmaking has places you relative to the server/host). A good layman's comparison of the two can be found here (and it goes into more detail in the following articles).
So while TCP works fine for untimely things such as playlist information or end-game stat tracking, UDP is neccessary in-game for speed reasons. So developers have to find ways to get past its faults. Now for the first issue, which is packets arriving in the wrong order, the server assigns a sequence number to outgoing data. On the client side, if any data received has a sequence number less than that of the current sequence number, the data is trashed. This effectively keeps things in order. Packet Loss is handled by each packet containing an ACK or acknowledgement. The client and server use ACKs to tell one another that they have received each other's data. Obviously any important data that wasn't sent (and needs to be) is resent.
[2.0] How does lag affect me?
[2.1] Latency, Jitter, Three bars, etc.
Alright, back to a more macro-scale of networking. So we know that the main limit to online gaming is the sheer fact that data takes time to get from point A to point B. Latency is the time delay between these points. One way latency is the time taken to get from source to destination, while round trip latency is of course the time taken for data to go from the source, to the destination, back to the source. This is the combination of numerous delays involved in the creation and transmission of a packet, but the most significant is the travel time of the packet, which is mainly what I'll focus on. Jitter or Packet Delay Variation is the standard deviation in latency from the mean latency. Ping is a network utility for accurate determining latency, in which either a client or server sends a message to the other and awaits a response. The Latency Bars (or ping bars, whatever) display the results from one's most recent ping test to the host. It's a really simple system, really:
4 Bars = less than 100 ms of latency
3 bars = 100-200 ms of latency
2 bars = 200-300 ms of latency
1 bar = over 300 ms of latency
Now in a game like CoD, 100 ms of ping is a lot. When combined with interpolation (discussed in 2.4) and short kill times yields a significant delay in what a person does. Now that's not to say it isn't still accurate, (as far as hit detection goes, CoD is fucking fantastic for a shooter) just that poor connection has a huge influence on one's performance. Due to CoD's short TTKs, connection issues are VERY apparent and have a tendency to look a lot less fair than they really are.
[2.2] Input Prediction
Alright imagine the following scenario: You press the button that makes you move forward. Notice how your character instantly responds and moves forward himself. But how? After all, that data hasn't reached the server yet. This is the effect of Input Prediction. Input prediction realizes what the player will do before the information reaches the server and back to the client, and applies it instantly in order to smooth gameplay. Basically, Input prediction executes local commands instantaneously. The alternative here is for one to input a forward move, send that info to the server, which would send it back in the next snapshot, and then you would move. You'd be a clear split second behind everything you tried to do. This system allows the game to run fluidly for every individual client while having the side effect of one's commands being asynchronous with what the server is sending out in snapshots* (which will be more obvious in a bit).
*Of course you wouldn't be synchronized with the server anyway due to latency; however, without prediction you would receive your own inputs in the same manner that everyone else is getting them.
[2.3] Hit Detection and Lag Compensation
So, we know that input prediction makes the client asynchronous with the server, but how does that effect gameplay? Well the first major issue is with hit detection. Hit detection or hit authentication is the game's way of telling whether or not you've managed to actually hit your enemy. There are two main forms of doing this: Client Side Hit Authentication and Server Side Hit Authentication.
Client Side Hit Authentication means that each individual client keeps track of when it hits an enemy, and simply send the data of it being a hit to the server. The primary benefit of Client Side hit detection is that it is 100% accurate for the player shooting. The downside is that it's incredibly weak to lagswitches and hacking. It's entirely possible for a player to use a third party program to send a "hit" to the host without seeing a player or even firing your gun. Any player would be capable of using a lagswitch to its fullest extent.*
*at the moment, only the host can truly utilize a lagswitch because they are the ones sending data to the clients. If you're platinum or higher in league then you've likely seen this in action. It is also important to note that a host can specifically block the IPs of other players in order to kick them out of games, which is another common issue at high level ranked matches.
Server Side Hit Authentication is far more complicated. Let's take the following scenario:
You shoot at the guy on the left, because that's what you see. However the target is moving to the right. You aim, and shoot. Your console/PC sends your position, direction, and the fact that you shot to the server. Now lets say that you have a latency of something nuts like 200 ms. By the time that your command reaches the host/server, the target has moved to the spot on the right. But if the game utilizes the data exactly as you sent it, then it would register as a miss because there's no player there now. This is where Lag Compensation comes into play. Lag compensation effectively "rewinds time." It digs through its memory and determines a game state that matches the client's information and implements it under those conditions. This accounts for latency, interpolation, and anything else that creates a time delay that the game can manage to quantify. The benefits of Server side lag compensation are that it is virtually immune to things such as lagswitches and various hacks (however things such as god mode hacks and aimbotting would still work). The downside of course is that it's flat out not perfect. Modern day lag compensation is very accurate for what it's worth, but CoD sends data from each client at 30 states a second all the while running at close (or more in the PC version's case) to 60 FPS on each client. Small precision errors can still occur and have very unfavorable results.
There are other methods of lag compensation, however Jon Shiring AKA slothy (Ex Infinity Ward now Respawn employee) has effectively confirmed that The IW Cods used rewind time compensation in here. It's a fairly safe assumption that the 3arc installments follow the same route, but I suppose there isn't any confirmation of that. That thread linked had some nice information in general, so it's worth looking through if you're interested.
On the server side, hit detection is accurate and lenient. Rest assured that if you shoot someone, then it will register as a hit. There are two exceptions to this rule, however. First, if the host receives information at any time that a player has been killed, then any more data it receives from that presumably dead player will not be sent to the other clients. Of course, due to latency, the server will realize that you are dead before you do. And if you were shooting someone, you are sending that information to the host. You can continue sending that information until that you receive notice from the host that you are, in fact, dead. This means that you can effectively tell the server that you have shot the enemy after the server knows that you've been killed. This would frequently result in players both killing one another due to cod's short ttks, so as a result, the server will cut off all input that it receives from you until you respawn again. If someone plays halo online then they're probably familiar with two players killing each other simultaneously. However do to the quick-dying nature of cod, this would happen in nearly every head on encounter. This will be visualized later in 3.2. The second exception to the rule involves packet loss, which will be discussed in 2.5.
[2.4] Interpolation and Artificial Delays
One thing many players will notice when looking closer at connection issues is that players have artificial delays applied to them.*This is caused by interpolation. Interpolation stores snapshots and renders them a split second later in order to keep movement fluid. Take a look at the following graph:
The line represents the timeline of the game, with each segment depicting the time delay between snapshots. It is important to know that Real Time (Server), Real time (Client), and Current Rendering Time are all occurring simultaneously, but are all at different spots on the game's timeline. Starting from the right the server sends out a snapshot. As we know, there is a delay between when it is sent and when it is received by the client. When the client receives the snapshot, it stores it in memory. It does not render the snapshot instantly. More specifically it does not render that snapshot until it has another snapshot to compare it to. However then we have to remember that we are using UDP to transfer data. If we are only using the past two snapshots, then any packet loss whatsoever would cause choppy movement as the game would need to predict where enemies are heading in order to maintain a moving picture. Thus it is standard to have interpolation span at least three snapshots, or a 100ms "lerp" it is often called. The server sends data to the clients at about about 20 snapshots/second (and 30 packets/second from client to server), which means this creates an artificial delay of roughly 100 ms.
There is another artificial delay added to the host. Since CoD4 there has always been a (reported) 20-40ms artificial delay added to the host and clients*. If 51% of the clients have a ping higher than 100ms, then the average ping of all players will be added to the host.* Robert Bowling said that this was changed so that the host's latency was equal to that of the next lowest latency player, so take that as you will.
*going off toysrme's word here and don't have any variables/dvars/values for that. However testing in CoD4 and MW2 showed this to be true. This also occurs on local splitscreen.
It is also important to note that there's been pretty good evidence of artificial delays in black ops 2. As shown here one can see that there is a major delay on splitscreen in BO2. It measures to be roughly 150ms compares to a little under 50ms for mw2/cod4. Theoretically there is absolutely no reason to have any delay on splitscreen, however that requires some reworking of game mechanics. Ignoring that there are various possibilities that could lead to a delay like this, such as interpolation having values unnecessarily high for local multiplayer or a flat delay for hosting platforms. Regardless it is worth noting that it exists and that Treyarch refused to acknowledge it during the game's lifespan.
There's another mechanic called extrapolation which predicts how the game will flow in a situation where too many snapshots were missed in a row to render a fluid scene. This is much less reliable for fluid movement and is rarely used for more than a fraction of a second in any game (if you have to extrapolate for over 250 ms then your connection's out of whack and there's no point).
[2.5] Packet Loss
Since a real time shooter like CoD is forced to use UDP, it is inherantly prone to packet loss. Now, as said earlier, data is sent from the server to the client 20 times each second and 30 times a second from client to server. Due to interpolation, missing one of these packets won't cause issues when rendering your in-game world. However, issues arrive when the exact packet containing information of a hit is dropped.
Now as said in 1.2, in order to compensate for potential packet loss, each packet sent must be confirmed by the receiver. Upon receiving information, the client or server will send out a response with an ACK to confirm that the previously sent information has reached them. If the ACK is not given within a certain period of time, then the important information from that lost packet is added onto the new one. While this system does a good job of making sure missing information goes through, it is unfortunately not very fast. It will take at least the length of one's round trip latency before the client/server knows that a packet did not get through, and in many shooters, that's often too late.
While it is certainly possible for data to get lost on the way to its destination, bandwith can be a major factor in packet loss (and is also the only factor that you can do anything about). When there is not enough room for an incoming packet to be stored or transmitted, that packet will be dropped. While CoD doesn't use too much bandwith, having multiple appliances on the same internet connection can ovwerwhelm and stifle connection from your videogame, especially if those applications primarily use a TCP connection.
[3.0] How does this come into play?
[3.1] Connection Discrepancies pt.1
Input prediction and lag will cause players to be out of sync with each other. More specifically, each client will see the other players significantly behind themselves. The difference will basically be Time it takes for the data to reach the server + Time taken from snapshot to reach another client + The time delay caused by interpolation. This stacks up to make a pretty big difference. When mixed with the fact that hit detection is designed to work properly from the shooter's perspective, this causes some wacky effects. The first major one comes from coming in and out of cover. In both these pictures: You are Red (your perspective), the enemy is Blue (his perspective), and the place where you see the enemy is on your end is a blue shadow.
Running into cover
In this situation, we have an enemy who is running away from you as you are shooting him. On his end, he has turned the corner, safe from your view. However, due to lag, the last snapshot to be rendered on your system was when he was a split second behind. due to the fact that you can act instantly and that hit detection is client side, your shots will register shooting him even though in real time on his client, he is out of your view. If you've been on the bad end of this (and you almost certainly have) you'd notice that you can actually travel a fair distance behind cover. This isn't as "bullshit" as it seems simply because for half of this time, you were already dead, your client just hadn't gotten word of it and thus input prediction allowed you to move because as far as it was concerned, you were still alive.* In actuality the difference is more reasonable and it just looks worse because the information had to get back to you.
It's important to note that this is simply dependent on a connection discrepancy. It doesn't matter who has the better connection on either side because the latencies of the players involved stack together. You do need to acknowledge the fact that in CoD, if either you or the opponent is lagging, then they have a window of opportunity after you take cover to kill you.
*it ends up being the time to send your location at time of death to server + latency between server and client + interpolation + sending it back to server + sending it back to you. Long time. Very silly. Looks weird. will visualize in 3.2
Coming out from cover
In this scenario, you are camping. Enemy comes around the corner and starts shooting you. On your end of the spectrum, he hasn't come around the corner yet. Basically, he sees you before you see him simply because he is moving into the room and you're already there. Once again, hit detection favors the attacker in this engagement; and once again, the significance is a result of the two players' combined latencies. When playing, especially on a poor connection, it's important to remember that you can manipulate this fault in networking to your advantage. However it's not consistent or reliable so it's not some be-all end-all trick either.