League of Legends Gameplay Data Extraction using Computer Vision (Masters' Thesis)

Project Description:

This project focuses on data extraction from a match video of League of Legends, a widely played competitive multiplayer game, to better analyze play styles through data analytics. This is a research project supervised by Dr. Corey Clark, Director of Human & Machine Intelligence Game Lab at Southern Methodist University.

Development Specification:

  • Language: Python
  • Team Size: 2 Programmers and 1 Supervisor
  • Development Time: 6 Months
  • Domain: Computer Vision using Convolutional Neural Networks (CNNs)

Responsibilities

  • Understood the limitations of conventional computer vision techniques like template matching and feature matching.
  • Built a sample generator that generates labeled images for training the CNN.
  • Trained the CNNs on the data generated and tested the models.
  • Built a system to choose the CNN models to use to run inference on the video frames.
  • Worked with the team to define a data format (JSON).
  • Built a data post-processing system to help smooth the positional data over time. 
  • Built a bounding box visualizer and death location visualizer to validate the position data collected. 
  • Wrote a paper

Why do this?

Game analytics plays an important role in guiding teams in both traditional sports and electronic sports (eSports) by getting a better understanding of the opponent’s strategies. Unfortunately, not every eSport game has a reliable method to extract, collect and analyze the data needed for team and player analysis. For instance, League of Legends (LoL), a popular multiplayer online battle arena video game, does not provide the ability to track gameplay data like players’ position, hit points, and mana within the game without having to watch and manually encode the entire match. This project presents a methodology to extract, collect and analyze gameplay data from LoL videos via an automated computer vision and offline post processing techniques. We use template matching and feature matching to extract information from static UI elements. To track players’ position on a dynamic minimap, we utilize object detection via convolutional neural network.

Training Sample Generator

Any neural network requires training data to adjust its weights for the implementation. In this case the training data comprises of images with the icons of the heroes on them. I built a python script to generate minimap images overlaid with champion icons to ease the generation of training data. 

Inference and Data Output

Usually, a single CNN model would be trained to detect all champions. This approach led to a lot of false positives since the icons have similar features and the icons are small on the minimap. To improve the accuracy of prediction, I trained a CNN model for each champion. Although this increases the inference time 10 fold, it decreased the amount of false positives to a maximum of 5% of the frames. Before running the CNN on the video frames, we use template matching and feature matching to identify the frames to run inference on and also to identify the champions that are part of the match. This step generates a JSON file that contains frame numbers and champion states gathered from the other static UI elements. Using the JSON file created, I run the corresponding champion’s CNN model on the frames described by the frame numbers. 

Post-Processing Positional Data

While the accuracy was improved over the multi-object detector model by using individual object detectors, there were still losses in detection i.e. there would be frames in which champions would not be identified even though they were alive. These losses were mostly caused by overlaps of champion icons over other icons that show up (e.g. other champion icons, ping icon, minimap cursor, etc). The frames with these kinds of losses were tagged so that they can later be post processed to obtain a position with a marginal error. In this case, marginal error is defined as a position that shows the approximate vicinity of the actual position.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
if __name__ == '__main__':

    # Load the input json
    json_obj = None
    with open(in_json_path, 'r') as in_json_file:
        json_str = in_json_file.read()
        json_obj = json.loads(json_str)

    match_frames = json_obj["MatchFrames"]
    
    for label in labels:

        last_good_frame_idx = -1
        last_bad_frame_idx = -1
        last_good_location = []
        last_good_box = None

        # Iterate through match frames and assign position from the last good frame to the previous ones
        for frame_idx, frame_obj in enumerate(match_frames):
            location = frame_obj["Champions"][label]["Position"]
            bbox = box(frame_obj["Champions"][label]["Bbox"]["TopLeft"], frame_obj["Champions"][label]["Bbox"]["BotRight"])


            if location[0] == -1:
                last_bad_frame_idx = frame_idx
                populate_frame_info_from_last_good(last_good_location, last_good_box, last_good_frame_idx, frame_idx, label, match_frames)
                print("Reading Frame Index:", frame_idx)
            
            else:
                # Special case for first frame
                if(last_good_frame_idx == -1):
                    last_good_location = location.copy()
                    last_good_frame_idx = frame_idx
                    last_good_box = bbox
                else:
                    is_loc_good_enough = False
                    for idx in range(0, num_frames_to_check):
                        # If idx goes out of bounds, set it to the location previously in memory
                        if (idx + frame_idx) >= len(match_frames):
                            last_good_location = location.copy()
                            last_good_frame_idx = frame_idx
                            last_good_box = bbox
                            break
                        
                        else:
                            cur_frame_obj = match_frames[idx + frame_idx]
                            cur_location = cur_frame_obj["Champions"][label]["Position"]
                            if check_location_nearly_equal(location, cur_location):
                                is_loc_good_enough = True
                            else:
                                is_loc_good_enough = False
                                break

                    if is_loc_good_enough:
                        last_good_location = location.copy()
                        last_good_frame_idx = frame_idx
                        last_good_box = bbox

Validation of Results

To validate the results obtained by this method, I built a death location visualizer and a bounding box visualizer.  

Postmortem