Tuto: Visualizing game analytics with Elasticsearch and Kibana

Data gathering in Unity

We want to gather in-game data for the events we want to track,
Unity provides tools to gather core and custom data:

+ First make sure the Analytics service is enabled in your project. (https://docs.unity3d.com/Manual/UnityAnalyticsSetup.html)

+ You can then use the `Analytics Event Tracker` provided by Unity to trigger events through the editor or your custom json.
There is a limitation on size of event using unity so we broke down our data in a few different events sent at the end of each game or when the app is closed:

                                    
                                        AnalyticsEvent.Custom("EndOfGame", new Dictionary
                                        {
                                            "botsPunched": 6,
                                            "playersCount": "3",
                                            "gameDuration":"464.3931",
                                            ...
                                            }
                                        }
                                    
                                

+ Unity adds metadata on top of your custom events, timestamp, appId, userId, Country ...

Getting data from unity dashboard

For now we manually downloaded the raw data for the week of the festival from the Unity analytics dashboard.
(https://analytics.cloud.unity3d.com/)

But this should be easy to automate using the Rest API provided to get a daily dump if the analytics: https://docs.unity3d.com/Manual/UnityAnalyticsRawDataExport.html
That's an improvement we'll add when we have demos available publicly for longer periods if time.

Elaticsearch Installation and Indexing

Elasticsearch is available for free, and easy to setup:
https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

Here we are running everyhting locally, but there are plenty of resources for hosted services and deployments.
Download the ES package and run the executable, ES should be available on localhost:9200

                                    
                                     > wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-darwin-x86_64.tar.gz
                                     > tar -xvf elasticsearch-7.8.0-darwin-x86_64.tar.gz
                                     > elasticsearch-7.8.0/bin/elasticsearch
                                    
                                

The next thing we want to do is indexing our analytics file in elasticsearch.
ES is a powerful full text search engine so there is a lot to be said about the configuration and index mappings.
For now we will keep it simple, use the ES auto-mapping feature, and just make sure that the fields we want to do calculations on are in the correct formats.
Here is an example of a basic indexer written in python: It will require the elasticsearch python package
pip install elasticsearch

                                    
                                        import sys
                                        import json
                                        from elasticsearch import Elasticsearch
                                        es = Elasticsearch()

                                        int_fields = ['botsPunched', 'playersCount']
                                        float_fields = ['gameDuration']

                                        analytics_file = sys.argv[1] 

                                        with open(analytics_file, "r") as a_file:
                                        for record in a_file:
                                            json_record = json.loads(record)

                                            if json_record['custom_params']:
                                                json_record['custom_params']
                                                
                                                for field in int_fields:
                                                    if field in json_record['custom_params']:
                                                        json_record['custom_params'][field] = int(json_record['custom_params'][field])

                                                for field in float_fields:
                                                    if field in json_record['custom_params']:
                                                        json_record['custom_params'][field] = float(json_record['custom_params'][field])        

                                                data = json.dumps(json_record)

                                        res = es.index(index="my-index", body=data)
                                    
                                

We are just making sure that the integer or float fields are in the right format for ES to guess the mapping, so we can run calculations on them (avg, sums, max ...) If this is running smoothly, when the indexer is done with your file, you should see documents in your Elasticsearch index:

                                    
                                        > python indexer.py my_analytics_file.json
                                        > curl -XGET http://localhost:9200/my-index/_search
                                    
                                

Kibana Installation and Visualizations

Kibana is a great tool for analysing and visualising log files. Install it in the same way you installed ElasticSearch: https://www.elastic.co/downloads/kibana

                                    
                                     > wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.0-darwin-x86_64.tar.gz
                                     > tar -xvf kibana-7.8.0-darwin-x86_64.tar.gz
                                     > kibana-7.8.0/bin/kibana
                                    
                                

It should be accessible on localhost:5601