Flight videos with Telemetry data31 Mar 2017 - Rafael Lopes
The Video Editing part
It took me approximately 18 working hours from zero knowledge about telemetry to this state: I was very very curious to know how to do that after seeing a Garmin (the GPS guys) video. They created a software that gets metadata about their cameras’ files and embeds automatically in the video, and this is a thing that I always wanted to do because watching the video with these gauges and info allows me to discover what are the impacts of my flight controls in the final result (it’s like watching yourself presenting something), and work in these aspects to make a smoother footage.
I joined Mavic Pilots and saw some telemetry posts and saw that guys use a Windows application called Dashware, which unfortunately works only for Windows. Not a problem: I partitioned my personal Mac with BootCamp, installed Windows 10 and installed the software. But GoPro guys (the company) bought Dashware and I think they hired the developers to do something for their Drone - the GoPro Karma. So the Dashware support stopped some time ago and currently the software seems bugged in a way you have to remove some DLLs, mess up with codecs and do some black magic in order to work, the downside of Dashware is that it only supports 1080p and the software crashes A LOT, there was one day that it was about 2 AM, I spent 2 hours doing the gauges and with an unsaved project the software crashed when I tried to render, reminding my 1990’s memories where hours of work would be thrown in the garbage due to software crashing.
Well, after spending 8 hours with Dashware and trying to make it work on my Windows machine, I gave up and spent some days researching if there were other ways to embed the god damn telemetry into the video files - it shouldn’t be hard. So I discovered a new software called RaceRender, I saw some YouTube videos from guys who made telemetry using it, I wrote them, added them on Facebook and we exchanged experience - one of them is a math teacher named Robert who helped me a lot on my gauges journey because he shared his template with me, which is a lovely one with lots of information, but I still thought I could do a template from the scratch.
As a first test, I ended up with something like this below, a Hoboken flight. I was very excited and I didn’t want to start the gauge design development yet before testing if the software was stable or crazy like Dashware. I found that very stable and the gauges were hi-def and the framerate was good and I felt I was good to continue. Because when you work on the video editing software, sometimes the render results may be different, and RaceRender passed the test. The Telemetry data starts at 0:10.
The idea of this video was just to check the power of RaceRender, the gauges are not that beautiful, to see the final result after some extra hours of work, continue reading.
I was decided, with the results above I purchased RaceRender ($50) and started the gauge development. The software is one of those Java software with kinda ugly GUI - not beautiful, it’s for professional usage, looks like Final Cut Pro before Apple refactored the UI (actually I read somewhere that lots of TV channels and pro guys use this software to embed telemetry into their videos as well) and this software allows 100% customization on the gauges and gives you access to their source code if you want to do something more specific (the source code is in C-style) - totally formidable for me.
Yesterday I spent like 6 hours doing the layout of my gauges and indicators to display on the video, they have different colors, shapes and they change colors according to their value. Initially I created a solid green video on FinalCut Pro to be used as a Chroma Key channel, and used that video as a background to embed the gauges in RaceRender, then export the video of the gauges with the green background and use my original drone video behind and activate the Chroma Channel with the green color, but some gauges had some transparency and I didn’t want to give up transparency, that transparency could mess up with my Chroma Key layer because the gauges had green content injected on them, so I gave up the Chroma Key idea. What I ended up doing was edit the video in FCPX, export it and use RaceRender then, using its tools to sync the data file with my edited video file - no chroma key :(
The good part about RaceRender is, as I said above, totally customizable, I got a Cessna Compass, loaded into Photoshop and created the compass gauge in RaceRender, rotating the aircraft according to the drone’s heading.
Then my friend Bruno told me to replace the airplane icon for the Mavic icon, ok!
And the final result of this gauge on the video, looks gorgeous:
The drone part:
Ok, you should be asking how can I get that flight info to embed on the video, right?
The drone sends the flight information to the DJI Go mobile app, that’s how you can have the App to provide you a flight history with all the telemetry on it, you can find it under Flight Records on the DJI mobile App. What happens is that you can export this data. There’s an “export” button on the DJI Mobile App that saves the info in the phone’s app folder. You can get that file with iTunes by manually plugging an USB cable from the phone to your computer or you can install another mobile app called HDSync, available for iPhone and Android, where you input your DJI cloud credentials and uploads the flight log to a site called healthydrones.com, which was purchased by AirData.
AirData is a very interesting website that gives you statistics about your flights, using data collected from the DJI Go App. It allows you to download CSV files containing the flight data, and guess what? RaceRender accepts these CSV files, which contains all the info flight info. So with that CSV file containing my flight metadata (a.k.a. “The Telemetry”), I get it and load to RaceRender together with my edited video file. RaceRender has a syncing tool to sync the data file with the video.
So, the tools resume:
- DJI Go mobile App is the app that you use on your phone when you are flying.
- HDSync is another mobile app to avoid you having to manually sync the files via iTunes and manually upload them to AirData.
- AirData is the website that contains the flight statistics according to what you have uploaded and allows you exporting a CSV file.
- RaceRender is a video software that renders together your original footage with the gauges overlay that is fed from the information on the CSV file.
RaceRender has a visual gauge editor that helps a lot on the gauge development. This was when I was developing the battery percentage gauge - it was a challenge because the battery measurements are decrescent: from 100 to 0, not from 0 to 100
And the good thing about RaceRender is that it supports 4K footage and high-definition gauges too. My personal Mac is a MacBook Pro 2.7Ghz 16GB RAM 512SSD, bought in 2013 and took approximately 40 minutes to render an 18-minute drone footage. I am curious to see how much time it would run in a p2.16xlarge with 16 GPU Nvidia Tesla (Amazon AWS), but the instance charges $16 per hour and I don’t want to use my employee account for that, my Mac does the job :)
I also discovered that our drones have ANOTHER micro sd card inside it. It’s a 4GB card GLUED in the drone’s mainboard that DJI uses to store the flight info. These are big binary files that you can use a CLI app to read that binary file and retrieve CSVs. This internal SD card is like the drone’s black box and this DAT file contains ALL the flight information. They are big files and a 20 min flight would give you a 400MB file, for example. These files are better than the files sent to the DJI Go app because they contain all the info (the DJI go App loses data if the radio gets out of signal), but if you had LightBridge info during the entire flight, you are safe to rely on the file in the DJI Go Mobile App.
The final result
Finally, after some hours of work, I ended up doing a video from Hoboken to Manhattan (NYC), with embedded Telemetry, take a look at how gorgeous the result is.
My interface has the following gauges:
- elevation in m/s: can be positive or negative, if it’s >0 the drone is ascending, else the drone is descending, if =0 the drone is not elevating at all;
- battery percentage;
- camera gimbal angle position, it may go from 30° to -90°: my Mavic is configured to go from 0° to -90°, where 90° is pointed to the ground. You can configure your drone to use the 0° to 30° too, but it’s not recommended because you end up filming the propellers, which is kinda useless and don’t give you a impression that you are looking to the front when flying over FPV (first person view) using the mobile app, having a false impression of what is above and on-sight it may be dangerous too, that’s why my gauge has red marks above 0°;
- horizontal speed in km/h: how much kilometers per hour (yes, metric system) the drone is traveling
- horizontal distance in meters: how much distant, horizontally, the drone is from the home point, note that this is not the distance from the drone to the radio controller;
- altitude in meters: how high the drone is;
- compass orientation tape;
- compass orientation graph;
- track map, where the red/pink parts show where I was slower than the average of the flight.