How we helped train the ML system by generating over 10.000 digital twin images of real-life scenarios
We explained the problem we wanted to solve, and we were presented with a solution that met our needs. We always felt as if they wanted to help.
-Barry Sanderson, Product Manager, TrafficNow
The Challenge
Our Client is a German-based company that produces a product that monitors traffic and generates various data. Their basic setup includes cameras in different positions, usually on highways and crossroads in cities. Images created by that video stream were used in training their ML system, with the goal of computer vision recognizing vehicle types. And that worked nicely unless Mother Earth presented challenges that day.
For example, the recognition of vehicles worked poorly once the dark set or in different weather conditions, such as rainy or snowy weather. On top of that, there was a limited amount of data with different angles and camera setups. Finally, the current system recognizes high-level types of vehicles, using trucks as an example - every truck was just a truck - and there needed to be a way to differentiate sub-types, such as tow trucks, tankers, forklifts, car transporters, trucks without trailers, etc. That problem was then propagated to all vehicle types - they couldn't recognize police cars, fire trucks, ambulance vans, and many more.
So, how do we improve the system?
The Solution Proposal
The basic idea was to generate digital twin data. Digital Twins are digital representations of real-world things—for example if we recreated the Eiffel Tower in 3D to its fullest details and proportions, we would get a Digital Twin.
But we wanted to go further. We wanted to create a dynamic system which will allow us to:
- create different roads and environments
- adjust camera positions and rotations dynamically
- set up any weather type
- generate random traffic in such an environment, consisting of many kinds of vehicles
Once we do this setup, we want to click a button that will generate hundreds of images of different cars, trucks, and buses on the streets. Next to each image, a data file with the rectangle's bounds and a description of the vehicle within that rectangle will be generated. The idea was to use those file pairs then and feed them to the ML system, train it further, and allow its computer vision to catch data more effectively in real-life camera feeds.
To put the idea to images - the idea was to create something like this:
Into this:
Note: These are actual images from our solution
The Approach
All of this sounded very nice on paper—but would it actually work? As anybody investing in improvements knows, there is always risk involved, and we spent some time discussing with the Client how to mitigate it and make sure that they got value, even if the system did not work.
So we outlined the Proof of Concept—everything we explained thus far—and with our technical team, we concluded that we could make such a system in 1 to 3 months. Due to the risk and unknowns of making something this experimental, we couldn't conclude the exact number of hours.
So we agreed on a maximum value that could be spent, and if we managed to get it running sooner, that would mark the end of the project.
And we got to work.
The Work
After some research, we concluded that NVIDIA had a great technology to support this digital twin data generation called Omniverse. In the following weeks, we've spent time modeling original 3D models, adapting bought assets, creating environments and weather conditions, writing Python scripts for the randomization of data with assets, and creating a system for exporting the data in a format that will be fed to the ML system.
We cannot share many more technical details to avoid getting into NDA-related stuff, but if this interests you or you'd like to explore similar expertise with us, do get in touch by writing to office@marbleit.rs or any of our representatives.
The Conclusion
Throughout the entire project, communication was fluent, and based on need, direct communication through messaging apps made the process very smooth. Due to this, the project was delivered on time and within scope.
-Barry Sanderson
And voilà! After 2.5 months of work, our system was done, up and running, documented, and tested by the Client.
And the result? It worked perfectly with their ML system and could be used extensively.
Another success!
