The World of CSGO Server Operation
I founded an Australian & New Zealand based CSGO community server called Invex Gaming, which was active from 2014 to 2019. Our community gaming servers quickly rose in popularity thanks to our amazing community, content, competitions, and custom game plugins.
There is a lot of work involved in maintaining a gaming community.
The list of responsibilities includes:
- Maintaining the infrastructure for the community forums and servers themselves.
- Managing costs and community donations.
- Adding new content like player models and fixing their hitboxes.
- Automating VIP systems and permissions.
- Writing custom, unique plugins that distinguish the community.
- Patching game exploits and bugs.
- Preventing DDOS attacks by bad actors.
- Handling abuse reports for players that break the server's rules.
- World domination.
However, the most tedious task by far is identifying and banning cheaters.
A cat-and-mouse game
Regardless of the game, there will always be cheaters. As a server operator, you can detect these cheaters using a myriad of techniques. We can use server-side code to detect actions that would be impossible for a human to perform legitimately. We can rely on the Valve Anti-Cheat (VAC) system to detect cheats, which is often criticised for its performance but does a good job at catching very blatant cheaters. We can abuse your kernel to run kernel-side anti-cheat and also mine bitcoin on the side as a "thank you" fee. Oh wait, we didn't use that last technique, but some anti-cheats certainly do!
All these detection methods are great but cannot automatically detect every cheat due to the constant back-and-forth, cat-and-mouse game that cheat developers and anti-cheat developers play together. As a last resort, we rely on manual human analysis using CSGO demos that record all in-game activity 24/7. Our admins (or a jury of your gaming peers if you prefer) make a judgement call on whether a player was cheating, leaning on the side of caution.
Once we finally catch a cheater, we must permanently ban them from our servers in the spirit of competitive integrity. The thing about cheaters is they don't like being caught. Moreover, cheaters tend to never stop cheating. I can't tell you the number of times a player was permanently banned for cheating, unbanned, then given a second chance, only to reoffend the very next day and get banned again. However, certain players are more tech-savvy and know the fundamentals of ban evasion.
Art of ban evasion
How does one evade a ban?
The core mechanics behind a ban is that some piece of identifiable, preferably unique information is used to deny you access to a service.
On the internet, this identifier is more often than not your IP address, which is essentially a number assigned to you by your ISP or server provider that allows you to interface with other machines on the internet. An example of an IPv4 IP address is 198.51.100.1.
In the Steam ecosystem, each player also needs a Steam ID to play Steam-powered games. This is a unique identifier tied to your individual Steam account that identifies you in the Steam world. An example of a Steam ID is STEAM_1:1:1111.
When a player is banned, we put their IP address and Steam ID on the naughty ban list. This means Santa won't visit you this Christmas, and also you can't play on our servers anymore if you join with this IP address or Steam ID.
Let's assume a player with the IP address (198.51.100.1) and Steam ID (STEAM_1:1:1111) is banned for cheating. We persist the details of their ban to a database and kick them off the server.
Now, the next time they re-join the server, they fail both the IP Ban check and Steam ID Ban check.
To evade this ban and continue playing, a player might decide to switch to a different steam account. Now their new Steam ID, say STEAM_1:1:2222, does not match the Steam ID they were originally banned on. However, our anti-cheat overlords have accounted for this. If a player joins with a different Steam ID but with an IP address that is already banned, the system now re-bans them, associating this new Steam ID with the previous ban. This essentially begins the process of building a fingerprint for this player. We now know this player is associated with 2 Steam IDs (STEAM_1:1:1111, STEAM_1:1:2222) and 1 IP address (198.51.100.1).
Instead, the player decides to use a VPN to change their IP address. Now their new IP address, say 203.0.113.60, does not match the IP address they were originally banned on. However, again, if a player joined with a different IP address but with a Steam ID that is already banned, the system re-bans them, associating this new IP address with the previous ban further developing the fingerprint. We now know this player is associated with 2 IP Addresses (198.51.100.1, 203.0.113.60) and 1 Steam ID (STEAM_1:1:1111).
Beyond these two identifiers, we don't have any additional identifiers for each player joining our game server. For most cheaters, this level of identification is usually enough.
🚨 BUT WAIT! 🚨
What happens if the player changes both their Steam ID and IP address in a single, decisive move?
🏳️ ggwp. Well played indeed.
In our case, the player joins with a never-before-seen IP address (100.64.50.74) and Steam ID (STEAM_1:1:3333):
As far as the server knows this is now a completely different player. There is no way to associate them with their previous bans on their Steam IDs or IP addresses because they are using fresh, never-before-seen identifiers. Unfortunately, this means the player is free to wreak havoc. If they get banned again, they simply rotate both identifiers together at once and continue ban evading.
To mitigate this, we could maintain a list of popular VPNs and their associated IP addresses but this is extremely tedious and requires constant maintenance which is something our admin team discussed and wasn't too fond of doing.
Now you might say to yourself, surely the cheaters will run out of IP addresses or Steam accounts. They did not. In fact, our analytics later revealed our most notorious cheater had access to seemingly endless geographically distributed IP addresses and over 87 Steam accounts that had purchased CSGO.
Problematic cases of IP address fingerprinting
Now, remember while the Steam ID uniquely identifies a player the IP address does not. We've encountered scenarios where two young siblings would play on our servers but only one would cheat. This resulted in both being banned as their fingerprints were entangled. One of these players claimed that "he couldn't control his brother and he shouldn't be punished unfairly for his brother's actions". This is true and I agreed!
Another issue was shared networks like universities. Sometimes a group of players would play from their university that used a single IP address for all outbound traffic. When one individual cheated and got banned on the campus network, all other players using that network were flagged as cheaters and their Steam IDs and home IP addresses were associated and unfairly banned.
We built in an exception system for these rare cases but generally advised people not to play on networks they didn't trust.
IdentityLogger
Let's face it
I'm not being funny and I mean no disrespect.
But cheaters are cunts. They're cunts now, they've always been cunts.
And the only thing that's going to change is they're going to become bigger cunts.
Maybe have some more cunt kids.
I don't ultimately get why you would cheat in a video game. Surely, that is extremely boring and a waste of your own time? Regardless of how good or bad I am at a video game, the payoff for me is to compete with others, try to improve, and achieve a sense of pride and accomplishment!
By early 2017, the ban evasion problem on our servers had worsened significantly, taking a serious toll on the admins who were performing constant manual demo reviews which were both time-consuming and, let's be honest, boring. To this day I am thankful for all the hard work the admins did in our community. Therefore, I sat down, got out my rubber ducky, put on noise-cancelling headphones and started "engineering" to figure out a way to catch them all (the cheaters not Pokémon).
Handling a simultaneous Steam ID and IP Address change
Our core issue is handling a simultaneous change of both the Steam ID and IP address. This would seemingly appear impossible since we did not have any other identifiers to rely on. At this stage I did a lot of brainstorming and analysis of the data available to us CSGO server operators. I was an active member of SourceMod, an open source modding platform for source games, and also quite comfortable with reverse engineering CSGO game binaries so I already had considerable experience in this space.
At some point I started thinking about the game's built-in web browser. CSGO offered a little basic web browser in-game (that is different from the Steam overlay browser) that allowed server operators to show a MOTD (message of the day) when people joined the server.
We'll call this browser the VGUI browser.
This browser was commonly used by server operators to display the rules of the server for new players upon joining. Interestingly, server operators had the ability to forcefully open any website on the web on a player's screen. The window could even be hidden, meaning opening a YouTube video would suddenly play audio in the background of the player spooking them. The VGUI browser was pre-authenticated with your Steam cookies so that you would automatically be logged into any Steam domains like steamcommunity.com. The browser also allowed the client-side execution of JavaScript sent from the server operator. Yes you read that right. This led to a massive Steam exploit in August 2017, in which server operators could hijack the credentials of their players via the VGUI browser but that's a topic for another day.
Cookies! 🍪
Recall how the VGUI browser was pre-authenticated with Steam cookies. This means it has native support for cookies, which are sticky and persist between browsing sessions by design. What happens if we store cookies on our own domain for each player each time they visit the server?
🔔🔔🔔 We have a winner! 🔔🔔🔔
Testing revealed that the cookies we save via the VGUI browser were persisted even after shutting down and restarting CSGO. Furthermore, there seemed to be no expiry date limitations enforced on the cookie. Modern browsers like Chrome have enforced cookie expiry limitations upper limits. However, the VGUI browser had no issues saving cookies with expiry dates exceeding 10+ years!
Another convenient feature of the VGUI browser was that the browser could be hidden entirely. This meant the server could silently force the client to visit our website where we would fingerprint you and store some cookies, all without the player ever seeing the browser window. A silent but deadly approach.
Investigations revealed the cookie data itself was persisted in a file that was located in the Steam installation directory on the user's machine. This is crucial because it means the vast majority of cheaters will never figure out how to bypass it. They are unlikely to delete this file and neither the browser nor the game had any functionality to clear cookies.
Wonderful, we have found a way to silently persist a cookie for each player as they join the server. We now have an additional third identifier to help fingerprint each player. This identifier, which we will call the Tracking ID, is tied to the installation directory of the player. Now, in order for a player to appear to us as a "fresh player" they would need to change their Steam ID, IP address and Steam installation folder. As you can imagine, no one is going to do the latter.
Implementation details
In this section, I'll explain the technical details behind the implementation of IdentityLogger (which is also open source on GitHub).
IdentityLogger is the name of the all-in-one system which fingerprints players based on their IP address, Steam ID and Tracking ID.
The implementation of the system is quite simple. The real genius lies in the idea itself.
When a player connected to one of our servers, for example at csgo.example.com:27015, we would first perform a ban check. This ban check would check a database for their IP address, Steam ID and Tracking ID. If a ban already exists, we kick the player from the server and associate any new identifiers with their existing ban.
If the player was still with us, we would then open the VGUI browser in a hidden window. The target website to visit was our own secret web page. Although unlikely anyone would ever discover the website, we used a secret in each request to authorise the request. All the requests made in the VGUI browser are client side but the traffic itself was encrypted over HTTPS. This meant that even if one were to use a packet sniffing tool like Wireshark, you would not be able to find the raw token. The only real way of finding the token would be to inject into CSGO and log VGUI browser's access. This would require not only the inkling to do this in the first place but also strong reverse engineering skills and familiarity with the CSGO binaries. Skills basically 99.99% of our cheaters would not have.
Next, our PHP script checks if a Tracking ID cookie already exists. If not, we created one and persisted it (which would ultimately store it on disk in the vgui.browser.cookies.dat file in the Steam installation directory of the player). The Tracking ID cookies were simply 64 length random alphanumeric strings such as: TeStsCOhZO1TQsumJkDUOdMMo13ReRLEngrQTg7S49LKT2rBvgPhauzSYbegscOT
All identifiers (Steam ID, IP address, Tracking ID) would also be persisted in a database whenever a player joined the server. We used this database to maintain the identities, query results and create a web panel for admins to investigate player fingerprints.
This database was also tightly integrated with our modified SourceBans plugin which is based on the popular ban management software for Source-based games.
Ban Evasion Prevention Example
Let's revisit the case where a player was banned with IP address (198.51.100.1) and Steam ID (STEAM_1:1:1111). Since this is a new player joining the server for the first time, they did not have a Tracking ID and a new one was generated and persisted for them. They are then banned for cheating and their IP address, Steam ID and newly generated Tracking ID are added to the ban database.
Later, the cheating player re-joins with a never-before-seen IP address (100.64.50.74) and Steam ID (STEAM_1:1:3333), thinking they will be able to evade their ban. WRONG!
Their new IP address and Steam ID pass the ban check as expected but their Tracking ID fails the ban check because it is already associated with a previous ban. So the player is immediately banned and their new IP address and Steam ID are associated with their prior ban.
We have effectively stopped the ban evasion attempt.
Furthermore, this player will continue to be instantly banned upon joining the server based on their Tracking ID, no matter what IP address or Steam ID they use. The cheater is left helpless and confused — afraid even. Precisely as we planned *cue diabolical laugh*.
The great ban wave of 2017
After implementing and testing the solution, we deployed the system on all Invex Gaming CSGO servers in February 2017.
The results were shocking — shockingly good.
Immediately, we noticed a massive spike in the number of cheaters getting banned — I should have bought calls. Long time well-known and trusted community members were exposed for possessing secret accounts that they were cheating on. Cheaters reached out to me personally and asked how it was possible for me to detect them as ban evading when they changed their Steam ID and IP address. Other server operators were interested in the plugin, with some offering money for it. I only shared the solution and technique with one other server operator I fully trusted based in the UK. Widespread sharing of the solution would ultimately render the technique ineffective.
It was a glorious time and I loved every minute of it. Our community earned a reputation for being "hard to cheat on" and subsequently the number of cheaters and cheater abuse reports dropped significantly on our servers.
The best part was that no one knew how we were able to do this and our admin team kept the implementation a top secret. We should have filed a patent! If cheaters learned how we were able to fingerprint them in this way, it would have been trivial for them to delete the cookies file on disk while ban evading. I am happy to report that the solution worked as expected from deployment date until October 2017 when Valve removed the VGUI browser entirely as part of their efforts to secure the game. It was only after this that I revealed the technique publicly and open-sourced the plugin.
Many years later, I remain really proud of this solution and how well it worked.
