Digital Twin, LLM & IIOT
IIOT for legacy and intelligent factory machines with XR and LLM feedback with a Digital Twin demonstration of real-time IOT for architecture/building applications using Omniverse.
- PIs: Sam Anand, Ming Tang
$40,000. UC Industry 4.0/5.0 Institute Consortium Research Project: 01.2024-01.2025
The project centers on the development of a Digital Twin (DT) and a multi-agent Large Language Model (LLM) framework designed to access and interpret real-time and historical data through an Industrial Internet of Things (IIoT) platform. Real-time data is sourced from legacy machines and smart machines, integrating Building Information Modeling (BIM) with environmental sensors. The multi-agent LLM framework comprises specialized agents and supports diverse user interfaces, including screen-based systems, Virtual Reality (VR) environments, and mobile devices, enabling versatile interaction, data visualization, and analysis.
The research evaluates leading DT platforms—Autodesk Tandem, NVIDIA Omniverse, and Unreal Engine—focusing on their capabilities to integrate IoT and BIM data while supporting legacy machine systems. Autodesk Tandem excelled in seamlessly combining BIM metadata with real-time IoT streams for building operations and system scalability. NVIDIA Omniverse demonstrated unmatched rendering fidelity and collaborative features through its Universal Scene Description (USD) framework. Unreal Engine, notable for its immersive visualization, proved superior for LLM integration, leveraging 3D avatars and conversational AI to enhance user interaction.
Environment Sensors for Digital Twin model. XR-Lab and SM-Lab at Digital Futures Building.
Integration of Reality capture, IOT, LLM into a digital twin model.
Digital Twin of Digital Futures Building.
Primary Objective: To develop a conversational large language modeling system that acquires data from legacy machines, digital machines, environmental data, real-time data, and historical data within an IIoT environment to create a digital twin for assisting in real-time maintenance and assistance (Application Use Case: Digital Future’s Building)
1. Autodesk DT
Live Demo with Autodesk DT. Please get in touch with Ming Tang to request a password.
2. Unreal + Omniverse DT
Demonstration videos
The original BIM model was provided by GBBN and Messer Construction. Copyright 2024.
Student: Sourabh Deshpande, Anuj Gautam , Manish Raj Aryal, Mikhail Nikolaenko, Aayush Kumar, Eian Bennett, Ahmad Alrefai
