CS:GO OCR

As tesseract seemed overkill and is more suited to written fonts, I created my own Optical Character Recognition (OCR) system for use in this project. For the map name, the screenshot image was simply compared to a dataset of map names I created and was labelled as whichever it was most similar to. As for the money, each player was done one at a time, where the money figure was blurred, thresholded to black and white, and the leftmost white pixel suggested how many numbers long the whole thing was. This meant that each number could be segmented (including the dollar), and similarly to the map name, compared to an existing dataset of labelled images. As for the loss bonus, this was incredibly simple and merely involved segmentation of each bar, thresholding to black and white and getting the average pixel value (in case one or two were somehow bright, such as a particularly bright background shining through).

This data was then processed through a convolutional neural network that had been trained on finding the optimal strategy given the input data and outcome of rounds in previous games to determine the best strategy. Strategy outcome suggested many factors, such as how much money to spend and on what, which site to go to, and the type of execute to do to increase the odds of success.

The main limiting factor of this project was that it was unable to detect if a player already had a weapon, and therefore did not need to purchase another one. For example, if a T did not die the last round and instead saved their weapon, they would receive no money but would still have the weapon, however, this would likely cause the system to suggest an economy round as it is unable to distinguish this.

If I were to continue this project, this is the main thing I'd like to fix, which I believe could be done by also taking a screenshot of the buy menu as this shows everyone's current weapon beside their name, as well as their money. This would at least help, but could still be a problem if a weapon was dropped on the ground but not yet picked up, as this does not show up in any menu and would require some kind of object detection or manual data input.