Analyzing telemetry
I previously covered how to connect your Unity client to a Google Drive spreadsheet and upload your telemetry data. Which is cool but not very useful if we can’t retrieve that data. One solution is to go into your Drive and download the spreadsheet manually, but that’s not really awesome. What we want is a client which not only downloads the data directly, but formats it and does cool telemetry-analytical-things to it!
Downloading the data:
Luckily, accessing the data stored in chrome, is very, very easy. The harder parts will be to reformat the data and working with it afterwards. As always, there will be a code summary at the end, but to illustrate my point in how easy the retrieval is, I want to show that part on its own. You will have to have completed the previous telemetry tutorial as well, as this is a continuation of the system which we started building then. You could potentially write your own data or download some random spreadsheet, but if this information does not adhere to the format set by the previous tutorial, we can’t work with the data.
The first thing you’ll have to do, is to open up the spreadsheet which we previously generated, then download it as a CSV-file (Comma Separated Values):
By doing this, it is now possible to access the download URL which the file came from. In Google Chrome, you can press Ctrl + J to open your Downloads tab, where the file download should be displayed alongside its source address:
For this next bit, I will create a class which uses the URL to retrieve and show the data in the console:
using System.Collections;
using UnityEngine;
using UnityEngine.Networking;
public class RetrieveData : MonoBehaviour
{
string url = "<URL goes here>";
// Start is called before the first frame update
void Start()
{
StartCoroutine(GetStuffFromWeb());
}
IEnumerator GetStuffFromWeb()
{
//Fetch the CSV file
UnityWebRequest www = UnityWebRequest.Get(url);
yield return www.SendWebRequest();
Debug.Log(www.downloadHandler.text);
}
}That’s all the code it takes, granted the fact that we have the URL. The reason why I perform the request in an IEnumerator, is that the request takes time to process. By creating a separate thread, we can pause while we wait for the request to finish, before we attempt to display it in the console.
It may look a little weird in the console at first, but if you click the output and view the expanded text, you’ll see that the data is there:
We may also look at the file which we downloaded manually from Drive, to verify that this is, indeed, the correct data:
Formatting the data:
Now that the data can be downloaded, we’ll need some way to format it. The nice thing about CSV-files is that the formatting is fairly straightforward. One thing that isn’t immediately evident (but well-known amongst those who speaketh the tongue of strings), is that each row in the data ends with an invisible character; ‘\n’ – also known as newline. Whenever you press the enter-button, you actually type in that character, but the text editor doesn’t show this as it is purely formatting and not meant to be read by humans.
The second thing to note is how the values in each row are separated by a comma-character.
Finally, the first row contains descriptions of each column. We’ll need to ignore this row during formatting.
As for the timestamp, it can be used for sorting the results. But that’s for a more complicated version, which I will make at a later time with a bunch of extra features.
Step one is to create the custom class which will represent each observation in the spreadsheet. I create this class so that I may associate the timestamp with each coordinate:
using UnityEngine;
public class DataClass
{
public System.DateTime timeStamp;
public Vector3 position;
public DataClass(string _timeStamp, string _x, string _y, string _z)
{
timeStamp = System.DateTime.ParseExact(_timeStamp, "dd/MM/yyyy HH:mm:ss", null);
position = new Vector3(float.Parse(_x), float.Parse(_y), float.Parse(_z));
}
}Note how the class constructor just takes a bunch of strings, handling the conversion to floats and DateTime values. This makes it much easier to work with on the other end, so that we just have to focus on splitting and sorting the strings in the CSV file.
Next up is splitting the CSV into rows of strings. This can be done with the .Split-function built into the string type:
string[] CSVToRow(string _inputData)
{
return _inputData.Split(new char[] { '\n' });
}This function will be accepting what was retrieved from the web server (eg. what was printed in the Debug.Log statement at the beginning of this tutorial). But before the rows are converted into the custom DataClass format, each row should first be validated. This validation is important, because it means we can delete rows from anywhere in the spreadsheet without having to worry about the consequences. If a row is deleted in the spreadsheet, it will output “,,,” to indicate that nothing is stored in that row. Plus the newline, that’s four characters. So by checking if each row has more than four characters, we’ll know if it contains any data:
string[] ValidateRows(string[] _inputRows)
{
List<string> validatedRows = new List<string>();
foreach(string row in _inputRows)
{
if(row.Length > 4)
{
validatedRows.Add(row);
}
}
return validatedRows.ToArray();
}Then each of the returned rows will have to be converted and stored as a list of the DataClass classes instead of being just strings. This where we’ll use the commas to split the string:
List<DataClass> RowsToCoordinates(string[] _inputRows)
{
//Validate the rows to ignore entries with no data
string[] _validRows = ValidateRows(_inputRows);
//Length -1 due to first row ignored
List<DataClass> toReturn = new List<DataClass>();
for (int i = 1; i < _validRows.Length; i++)
{
//Split the data in each row, into an array of strings
string[] row = _validRows[i].Split(new char[] { ',' });
toReturn.Add(new DataClass(row[0], row[1], row[2], row[3]));
}
return toReturn;
}Note how the for-loop starts at index 1. Index zero is where the column names are stored, so we’ll skip it. Going from there, it’s just a matter of initializing each element in the DataClass list with the strings from its corresponding row and bam, we have a list of DataClass objects which we can now use for visualization! The reason why I have chosen a list over an array (which otherwise has about 2.5% better performance), is due to how I will handle the data later. I will explain when I get there.
As an example, here’s a simple function which instantiates cubes at each DataClass position:
void VisualizeData(DataClass [] _data)
{
foreach(DataClass data in _data)
{
GameObject cube = GameObject.CreatePrimitive(PrimitiveType.Cube);
cube.transform.position = data.position;
}
}After generating a few data entries in the client from the previous tutorial, this function will generate something like this:
Implementing visualization:
We’ve managed to spawn a bunch of cubes now, but dare I say that’s not enough for us. While this is pretty cool, having multiple cubes overlapping each other makes it very hard to tell how many there are in one place. Consider having thousands or millions of data entries. I’d hate to be the person having to group-select a region and count the cubes in the inspector. I’d rather eat my Gryffindor tea mug, and I love that mug to bits.
To address this issue, I’d like to implement a better form of visualization. I want to generate a grid of transparent cubes, then colour them in accordance to the number of DataClass objects with positions inside of them. This will also involve a bit of shader programming to manipulate the transparency and colour values given to each cube in the grid.
First things first, we’ll need a prefab with the custom material and shader on it, to generate the grid out of. I’ve created a cube, a Standard Surface Shader and a Material using this shader. The material has been slapped onto the cube and the prefab is then stored in the Resources folder (make a folder named “Resources” and put the cube here, the exact spelling is important):
I will modify the shader later (*oof*) and add a script to this prefab to modify the shader properties. But first we’ll want to be able to create a grid of these cubes. We’ll want to be able to determine the size of the grid, as well as how many cubes will be used to populate it (eg. how precise the visualization will be). This precision parameter is important, as having too many cubes can be problematic when there’s a ton of data – each cube will go through the list and examine the coordinates.
Which brings me to why I chose the DataClass classes to be stored in a list instead of an array. Because now it is possible to remove the results which have matched a cube, so that future cubes will have to check less data. But before I get into that, let’s look at how I will actually generate the grid of cubes:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
/// <summary>
/// This script generates a grid of cubes, then uses the information from
/// the RetrieveData class to modify the look of the cubes.
/// </summary>
[RequireComponent(typeof(RetrieveData))]
public class VisualizeData : MonoBehaviour
{
[Header("Number of cubes on each axis")]
public int gridResX;
public int gridResY;
public int gridResZ;
[Header("Cube field dimensions")]
public float gridSizeX;
public float gridSizeY;
public float gridSizeZ;
//The calculated X-, Y- and Z-scale dimensions for the cubes
float scaleX, scaleY, scaleZ;
public List<GameObject> cubes;
private void Start()
{
cubes = new List<GameObject>();
//Find out the size of each cube
scaleX = gridSizeX / gridResX;
scaleY = gridSizeY / gridResY;
scaleZ = gridSizeZ / gridResZ;
SpawnGrid();
}
/// <summary>
/// Spawn the cubes using the grid information from the inspector
/// </summary>
void SpawnGrid()
{
for (int x = 0; x < gridResX; x++)
{
for (int y = 0; y < gridResY; y++)
{
for (int z = 0; z < gridResZ; z++)
{
//Debug.Log("X: " + x + " Y: " + y + " Z: " + z);
GameObject cube = Instantiate(Resources.Load("GridCube") as GameObject);
cubes.Add(cube);
cube.transform.position = transform.position;
cube.transform.localScale = new Vector3(scaleX, scaleY, scaleZ);
cube.transform.position += new Vector3(scaleX * x, scaleY * y, scaleZ * z);
}
}
}
}
}This is a fairly simple script. There are three dimensions; X, Y and Z. I want to specify how many cubes I want in each direction, and I want to specify the dimensions of the actual field of cubes in each direction. The first thing that happens in the Start function, is that I find out – with the given dimensions – how big each cube should be to fit inside those dimensions. In other words, the dimensions are dictated by how long the field is along an axis, divided by how many cubes I want to fit onto that axis. Repeat for all three axis.
In the SpawnGrid function, all I do is iterate through all of the cubes (gridResX * gridResY * gridResZ number of cubes), spawn a cube each iteration, then scale it and move it along the axis in accordance to how far along we are in the iteration (eg. the first cube “X: 0, Y: 0, Z: 0” won’t be moved at all, whereas the next “X: 0, Y: 0, Z: 1” will be moved the length of its own size, one step along the Z-axis).
Each cube is also added to a list for future reference. Here is what the script looks like in action, when writing 10 in all values (viewed in shaded wireframe in the scene view):
That is indeed a grid! But the power of this method comes with its options for customization. With the possibility of modifying the size of the grid in all three dimensions, as well as the number of cubes in each directions, we can shape it however we want and at whatever resolution is required (or possible – spawning objects at an exponential rate can be scary, don’t turn up your grid resolution too high without calculating how many cubes that will spawn, I accidentally created 1000 million cubes and instantly killed Unity).
With a customizable grid, it is time to write the logic which will change how each cube looks. It is best to start with the visual aspect, so that it becomes easier to see if the data is being registered correctly. This means starting with the shader code.
This part is quite simple. I want to give access to the transparency channel and provide three colours that can be toggled between:
Shader "Custom/GridVisual"
{
Properties
{
_ColorLight("Color", Color) = (1,1,1,1)
_CLRange("Light range", Range(0,1)) = 0
_ColorMedium("Color", Color) = (1,1,1,1)
_CMRange("Light range", Range(0,1)) = 0
_ColorHeavy("Color", Color) = (1,1,1,1)
_CHRange("Light range", Range(0,1)) = 0
_AlphaRange("Transparency", Range(0,1)) = 1
}
SubShader
{
Tags { "Queue" = "Transparent"}
CGPROGRAM
// Physically based Standard lighting model, and enable shadows on all light types
#pragma surface surf Lambert alpha:fade
struct Input
{
float2 uvMainTex;
};
float4 _ColorLight;
float4 _ColorMedium;
float4 _ColorHeavy;
half _CLRange;
half _CMRange;
half _CHRange;
half _AlphaRange;
void surf (Input IN, inout SurfaceOutput o)
{
// Albedo comes from a texture tinted by color
o.Albedo = _ColorLight.rgb * _CLRange + _ColorMedium.rgb * _CMRange + _ColorHeavy.rgb * _CHRange;
o.Alpha = _AlphaRange;
}
ENDCG
}
FallBack "Diffuse"
}Beneath each color, I have specified a range. These ranges are applied as multipliers when setting the albedo of the shader, so if any of these are set to zero, none of that colour is added – and if the range is 1, the whole colour is added. This is useful for toggling between them. Additionally, the alpha channel is modified with a range as well:
Here it is easy to check if the shader works, by modifying the properties in the inspector. However, there’s one little problem with this shader when accessing it with a script. Accessing these properties with a script will create a new instance of the material, for that script. Which means with thousands of cubes we may generate thousands of new materials in runtime and your CPU will murder you for doing that.
To fix this problem, we’re going to use a tag in the shader code called [PerRendererData]. Simply declare this in front of any property which we wish to modify with a script (eg. all of them!). The properties block in the shader will end up looking like this:
Properties
{
[PerRendererData]_ColorLight("Color", Color) = (1,1,1,1)
[PerRendererData]_CLRange("Light range", Range(0,1)) = 0
[PerRendererData]_ColorMedium("Color", Color) = (1,1,1,1)
[PerRendererData]_CMRange("Light range", Range(0,1)) = 0
[PerRendererData]_ColorHeavy("Color", Color) = (1,1,1,1)
[PerRendererData]_CHRange("Light range", Range(0,1)) = 0
[PerRendererData]_AlphaRange("Transparency", Range(0,1)) = 1
}The reason I did not write these tags immediately, is because it hides the property from the inspector – so it wouldn’t be possible to check if we had written the shader properly first (without manually modifying the default values of each property, directly in the shader code):
Which brings us to the C# part. Instead of accessing the material directly, we are going to feed a Material Property Block containing our values (what colour we want, what the scale slider values are, how transparent the object should be), to the renderer of each object. The end result will be that the shader can now be modified during runtime, via the inspector just as before, but through the script and without creating a new material for each object using the script. Put this on the GridCube prefab in the Resources folder, and configure the colours in the inspector:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class ShaderProperties : MonoBehaviour
{
//Make sure to set these in the inspector, on the prefab, or they'll default to black
public Color ColorLight, ColorMedium, ColorHeavy;
[Range(0f, 1f)]
public float RangeLight, RangeMedium, RangeHeavy, Alpha;
Renderer _renderer;
MaterialPropertyBlock _propBlock;
// Start is called before the first frame update
void Awake()
{
_propBlock = new MaterialPropertyBlock();
_renderer = GetComponent<Renderer>();
}
// Update is called once per frame
void Update()
{
//Get the current property block values from the renderer
_renderer.GetPropertyBlock(_propBlock);
//Update the values
_propBlock.SetColor("_ColorLight", ColorLight);
_propBlock.SetFloat("_CLRange", RangeLight);
_propBlock.SetColor("_ColorMedium", ColorMedium);
_propBlock.SetFloat("_CMRange", RangeMedium);
_propBlock.SetColor("_ColorHeavy", ColorHeavy);
_propBlock.SetFloat("_CHRange", RangeHeavy);
_propBlock.SetFloat("_AlphaRange", Alpha);
//Apply the new values
_renderer.SetPropertyBlock(_propBlock);
}
}And then initialize the colours in the inspector, so it looks like this (or whatever colour you prefer):
Besides from the colours and sliders being separated, this is pretty much identical to what we saw on the shader itself.
Visualizing the data:
The final piece of the puzzle is now to take our data and use it in some meaningful way. We have the data ready and the tools to modify the visuals, so next we have to come up with some rules to dictate how the data affects those visuals:
- Seeing as we have no idea how much data will be input into our system, it is good to work with percentages: “How many percent of the observations were in a cube, rather than the actual number of observations.”
- It is easy to translate the percentage number into an alpha value, making cubes with more observations, more solid.
- Then there’s the colours. Here we should simply look at the percentages and agree where the thresholds are (eg. blue below 20%, green between 20% and 40%, red above 40%), then pick one of the three colours in relation to that.
That’s pretty much it. The script should then access the ShaderProperties script which we just wrote, and modify the sliders in relation to these rules. We already have a VisualizeData script which spawns the cubes, so I am going to build this functionality into that script with the following three functions:
public void UpdateCubes()
{
//How many entries of data exists in the initial list
float totalObservations = collectedData.Count;
Debug.Log("Total observations:" + totalObservations);
foreach(GameObject cube in cubes)
{
//Count how many observations exist within the cube's boundaries
float matches = CountMatches(cube);
//Find out how much those matches make up of the total
float percentageOfTotal = matches / totalObservations;
//Update the shader of the cube with the found percentage
SubmitProperties(percentageOfTotal, cube);
}
}
int CountMatches(GameObject _cube)
{
Collider cubeBoundary = _cube.GetComponent<Collider>();
int observedMatches = 0;
//Used for deleting matches so that other cubes have less checks to make
List<DataClass> matches = new List<DataClass>();
//Count matches
foreach (DataClass observation in collectedData)
{
//If the observed point is inside the cube
if (cubeBoundary.bounds.Contains(observation.position))
{
observedMatches++;
matches.Add(observation);
}
}
//Remove the matches to increase speed on next iteration
foreach (DataClass match in matches)
{
collectedData.Remove(match);
}
Debug.Log("Observed matches: " + observedMatches);
return observedMatches;
}
void SubmitProperties(float _percentage, GameObject _cube)
{
//Access the shader-modification script of the cube
ShaderProperties cubeShaderProps = _cube.GetComponent<ShaderProperties>();
//Update the alpha value
cubeShaderProps.Alpha = 0;
switch(_percentage)
{
case float n when _percentage < 0.1f:
//Set cube to BLUE colour
cubeShaderProps.RangeLight = 1;
cubeShaderProps.RangeMedium = 0;
cubeShaderProps.RangeHeavy = 0;
//Set alpha only if there's SOME data
if(_percentage > 0)
cubeShaderProps.Alpha = 0.15f;
break;
case float n when _percentage > 0.1f && _percentage < 0.3f:
//Set cube to GREEN colour
cubeShaderProps.RangeLight = 0;
cubeShaderProps.RangeMedium = 1;
cubeShaderProps.RangeHeavy = 0;
//Set alpha
cubeShaderProps.Alpha = 0.3f;
break;
case float n when _percentage > 0.3f:
//Set cube to RED colour
cubeShaderProps.RangeLight = 0;
cubeShaderProps.RangeMedium = 0;
cubeShaderProps.RangeHeavy = 1;
//Set alpha
cubeShaderProps.Alpha = 0.7f;
break;
}
}Note that the collectedData list of DataClasses has been added here as well. The whole machinery is called from the RetrieveData script from earlier, to ensure that the data is ready before it is handled. This is done by adding two additional lines to the GetStuffFromWeb IEnumerator at the bottom:
IEnumerator GetStuffFromWeb()
{
//Fetch the CSV file
UnityWebRequest www = UnityWebRequest.Get(url);
yield return www.SendWebRequest();
Debug.Log(www.downloadHandler.text);
//Format the CSV file to rows in an array
telemetryDataRow = CSVToRow(www.downloadHandler.text);
//Convert each row to an instance of DataClass
collectedData = RowsToCoordinates(telemetryDataRow);
//VisualizeData(collectedData);
//Submit data to visualization and render the result
visualizer.collectedData = collectedData;
visualizer.UpdateCubes();
}The visualizer referenced here, is simply a reference to the VisualizeData script, which you can establish either in the Start function or via the inspector.
And boom, after generating a bit of data, I managed to pull the following visualization of almost 300 samples:
Admittedly, there’s still plenty of room for customization, especially considering how almost every parameter in the one used by the Subnautica team was customizable and the changes were live (not just shown on pressing play).
I will probably sit down and create a proper portfolio prototype with those extra elements and proper code documentation at some point – but until then, I feel like this covers the basics pretty well!