Object Detection Based Aim Assist
This is likely my largest individual project to date, and was my 3rd year thesis project at university. I chose to create an aim assistant application for the popular first person shooter game Counter Strike: Global Offensive (CS:GO), in a fashion that did not break any rules or terms of service.
Many of Valve's (the owner and publisher of CS:GO) terms and conditions apply only to online play, Valve Anti-Cheat (VAC) secured servers, and official matchmaking with other players. However, they also explicitly disallow process tampering and any modifications to the executables and DLLs. Interestingly, in Valve's own page on their VAC system, they explicitly state they allow system hardware configurations and these will not trigger a VAC ban. Taking this at face value with a little leniency, this all suggests that a hardware configuration, running entirely independently on a separate system, that is only ever played offline on non-VAC secured servers, isn't technically breaking any rules.
Which leads me onto this abomination... This is a Raspberry Pi 4, with a USB capture card, a Google Coral USB AI accelerator, also known as a Tensor Processing Unit (TPU), and a Raspberry Pi Pico connected over UART via the GPIO pins.
I initially intended to use a Google Coral Dev Board, which is a very similar form factor device with a TPU on-board, however, I found this more difficult to work with and had significantly less community support, so I stuck with the Pi. The Pico was because USB-OTG was being unreliable and flaky, so I eventually gave up and used the Pico to emulate my Human Interface Device (HID) instead.
The whole process was fairly simple but challenging to get working effectively. The HDMI signal from my PC GPU was mirrored, with one going to my monitor and the other to the capture card. This was processed by the Coral TPU compatible TFLite model to identify the locations of enemy players in the current frame, which was accelerated to real-time performance using the USB TPU. The bounding box coordinates were sent via UART to the Pico, which would translate this into the necessary mouse movement and provide the input to the PC. This whole pipeline happened in a matter of milliseconds 30 times per second (framerate limited by the capture card).
Data collection and annotation was one of the biggest challenges in the development of this tool. To solve this, I hosted a custom, offline, non-VAC secured server running the SourceMod plugin, which allowed me to write and create my own functions that run on the server itself. I created a function that would identify the location of both the player (myself) and the enemy bot, and using those xyz coordinates and the player's current viewing angle, I could calculate the size (with Pythagorean theorem using the coordinates to calculate the distance between myself and the bot) and location of the enemy bot on the players screen . In-game, I bound a key to simultaneously take a screenshot of the game, and run my custom function, which would together create a process to automatically label any images/screenshots I take. I simply ran around in god-mode (invulnerability) and pressed the action key when the enemy was on screen, and eventually created a dataset of 10,000 labelled images.
One problem that I faced was that weapons in CS:GO have recoil, and therefore even if I moved the mouse to always be aiming at the enemy, if the gunfire was a continued spray rather than a burst, many of the bullets would likely miss. Therefore, I also implemented recoil-control. All guns in CS:GO have a recoil pattern that is followed every time the gun is fired, with minor variation added in. This meant that it was possible to offset the crosshair movement to instead ensure the bullet would go where the enemy is, rather than directly moving the crosshair there. This was performed on the Raspberry Pi Pico as it was responsible for translating the bounding box to a mouse movement, and it is a fairly computationally light task.
Coral Dev Board with built-in TPU Accelerator (seen left).
Google Coral USB TPU Accelerator (seen below).
