How users helped develop our new GoPro feature

First of all, we were being asked to support a CSV file that this user had generated from his GoPro Hero5, which has built-in GPS. Secondly, the data didn’t include any lap-specific data, it was just a series of GPS coordinates with timestamps and speed in meters per second.

He created this by downloading some code from Github, and running a tool manually on the command line, so this was a pretty technically savvy user, and he was following an involved process that we couldn’t really expect regular users to replicate. But the idea of being able to support imports of data from GoPro cameras was very attractive. Lots of people have them, they’re great on the track, and it would open up Serious-Racing to a whole new audience. So Lisette and I sat down and tried to think about how we could do this, and what the user experience should look like.

The user experience

The main thing we were looking for at this point was how to make it as easy as possible for users to get their data to us. So we said to ourselves, “what if a user could just select their GoPro Hero5 video file directly from our upload page and we pulled the GPS data out of it?” The developer in me started to panic. How on earth would I do this? Is this even possible? It was time to start researching…

The current process followed by our user in Brazil was to run an ffmpeg command to extract the GoPro Metadata from the video as a binary file. He then used some Go code to turn this binary file into a CSV file. So the first step was taking this CSV file and importing the data into Serious-Racing. While mostly straightforward, we’re used to getting laps and lap times from data imports, but that wasn’t the case here, so we needed a way to ask users a few questions about their upload to process it correctly:

• Which track is it on? We can usually guess correctly based on the GPS coordinates.
  
• If it’s a new track, where is the start/finish line?
  
• Is the track circular, and if not, where’s the finish line?
  

Processing binary data 

Next it was a case of trying to generate the CSV ourselves from the binary data that ffmpeg extracts from the GoPro video file. To do this, I was translating the Go code into Python that could be used by Serious-Racing, and only parsing the data types we care about.

We already process binary data from RaceChrono and some other devices, so the overall approach was pretty familiar. Working with binary data tends to be a series of frustrations followed by an epiphany, and this was no exception. Usually it’s because you’ve mixed up big-endian and little-endian, or bitwise "and"/"or" operators.

Having users all around the world is great because of the great variety of test data at our fingertips. At one point during the process after some initial imports were working fine we noticed a problem with some other imports. Inexplicably, the GPS coordinates that should have been in at Silverstone were showing up in the middle of Azerbaijan. After a bit of digging, it turned out I was using unsigned 32-bit integers when I should have been using signed 32-bit integers, so all negative numbers were incorrectly parsed. Any data north of the equator and east of the Greenwich meridian was fine, but Silverstone is west of the Greenwich meridian, so has a negative longitude. The fix was a five character change from “>IIIII” to “>iiiii”.

GPS trace

Another problem (shown in the image above) came from one user we were testing with because some GPS traces showed points that were completely out of alignment. In some cases they were a few hundred meters from the points either side of them in the data stream. In one case, they were about 80 miles away. Luckily, GoPro has open-sourced their data format, and has specifications available which show GPS fix and GPS accuracy, but more importantly give you guidelines for what values can be used to consider the GPS data correct or not. After applying these rules to the incoming data, we were able to strip out the bad data, and were left with a clean GPS trace.

Extracting video metadata with Javascript

So at this point, we could parse binary data from a GoPro video and turn that into laps on Serious-Racing. But how to get the binary data out of the video in the first place? I spent a long time looking into this and going down a few blind alleys, but eventually came across the amazing mp4box.js project. Building on this, I was able to construct a Javascript implementation that could read a local file – the video file selected by the user – and pull the binary data out of it. This means we don’t need to upload the entire video to Serious-Racing to extract its GPS data. We then use Javascript’s FormData interface to construct a virtual file containing the binary data, and upload it via Ajax to our servers for processing.

Once again, user testing was crucial. It exposed another issue which wasn’t obvious to me as I was developing the feature. This was that the way I was doing things meant that the browser was using as much RAM as the size of the file it was parsing, which could be 3-4GB. My development laptop has so much RAM that I didn’t notice this, but it became obvious using Firefox’s development tools once I knew to look for it.

The fix ended up being refactoring the code to avoid creating new objects in Javascript for every chunk of data we parsed, and to make two passes over the file. We do this to get the info about where the data is, and the data itself, but pass an option to mp4box.js to not keep segments in memory. This brought memory usage down to about 50-80MB, which is still high, but a lot more reasonable, and certainly not enough to crash modern browsers.

Next steps

At this point it is already a very slick experience: just select the video on your laptop to let Serious-Racing extract the data right in your browser and process it on our servers, magically turning your GPS trace into laps which can then be compared with friends and dissected with all our other detailed data analysis features. Other things we would love to do: 

1. Test on all GoPro models with GPS.
2. Give users the option to upload their video to Youtube and automagically add it to their laps. 
   

No rest for the wicked.. Anyway, here's another shout-out to our awesome and very patient users: we wouldn’t even have tried this without the initial suggestion from one of our users in Brazil, and it wouldn’t be working without testing by users around the world. Thanks again for inspiring us every day to keep on making Serious-Racing better.