ITS-United

Year

2024

Technologies used

NVIDIA Omniverse, Python, Maya

View project

ITS-United

How we helped train the ML system by generating over 10.000 digital twin images of real-life scenarios


The Challenge

Our Client is a German-based company that produces a product that monitors traffic and generates various data. Their basic setup includes cameras in different positions, usually on highways and crossroads in cities. Images created by that video stream were used in training their ML system, with the goal of computer vision recognizing vehicle types. And that worked nicely unless Mother Earth presented challenges that day.

For example, the recognition of vehicles worked poorly once the dark set or in different weather conditions, such as rainy or snowy weather. On top of that, there was a limited amount of data with different angles and camera setups. Finally, the current system recognizes high-level types of vehicles, using trucks as an example - every truck was just a truck - and there needed to be a way to differentiate sub-types, such as tow trucks, tankers, forklifts, car transporters, trucks without trailers, etc. That problem was then propagated to all vehicle types - they couldn't recognize police cars, fire trucks, ambulance vans, and many more.

So, how do we improve the system?

The Solution Proposal

The basic idea was to generate digital twin data. Digital Twins are digital representations of real-world things—for example if we recreated the Eiffel Tower in 3D to its fullest details and proportions, we would get a Digital Twin.

But we wanted to go further. We wanted to create a dynamic system which will allow us to:

  • create different roads and environments
  • adjust camera positions and rotations dynamically
  • set up any weather type
  • generate random traffic in such an environment, consisting of many kinds of vehicles

Once we do this setup, we want to click a button that will generate hundreds of images of different cars, trucks, and buses on the streets. Next to each image, a data file with the rectangle's bounds and a description of the vehicle within that rectangle will be generated. The idea was to use those file pairs then and feed them to the ML system, train it further, and allow its computer vision to catch data more effectively in real-life camera feeds.

To put the idea to images - the idea was to create something like this:


Into this:



Note: These are actual images from our solution


The Approach

All of this sounded very nice on paper—but would it actually work? As anybody investing in improvements knows, there is always risk involved, and we spent some time discussing with the Client how to mitigate it and make sure that they got value, even if the system did not work.

So we outlined the Proof of Concept—everything we explained thus far—and with our technical team, we concluded that we could make such a system in 1 to 3 months. Due to the risk and unknowns of making something this experimental, we couldn't conclude the exact number of hours.

So we agreed on a maximum value that could be spent, and if we managed to get it running sooner, that would mark the end of the project.

And we got to work.

The Work

After some research, we concluded that NVIDIA had a great technology to support this digital twin data generation called Omniverse. In the following weeks, we've spent time modeling original 3D models, adapting bought assets, creating environments and weather conditions, writing Python scripts for the randomization of data with assets, and creating a system for exporting the data in a format that will be fed to the ML system.





We cannot share many more technical details to avoid getting into NDA-related stuff, but if this interests you or you'd like to explore similar expertise with us, do get in touch by writing to office@marbleit.rs or any of our representatives. 

The Conclusion

And voilà! After 2.5 months of work, our system was done, up and running, documented, and tested by the Client.

And the result? It worked perfectly with their ML system and could be used extensively.

Another success!



Other projects

Free Consulting

Not sure how to start implementing solution to your idea?

Do not hesitate to ask us, we'll be more than happy to schedule a meeting and help you with your questions, free of charge.

Free Consulting