Automatic Scene Creation for Augmented Reality Work Instructions Using Generative AI. Siemens. PI. Ming Tang. co-PI: Tianyu Jiang. $25,000. UC. 4/1/2024-12/31/2024
Students: Aayush Kumar, Mikhail Nikolaenko, Dylan Hutson.
Sponsor: Siemens through UC MME Industry 4.0/5.0 Institute
Investigate integration of LLM Gen-AI with Hololens-based training.
XR-Lab’s project was selected to join the Undergraduates Pursuing Research in Science and Engineering (UPRISE) Program
UNDERGRADUATE SUMMER RESEARCH PROGRAM IN SCIENCE AND ENGINEERING
May 6 – July 26, 2024
Student: Mikhail Nikolaenko. UC. PI: Ming Tang.
This project proposes the development of a sophisticated reinforcement learning (RL) system utilizing the robust and versatile environment of Unreal Engine 5 (UE5). The primary objective is to create a flexible and highly realistic simulation platform that can model a multitude of real-life scenarios, ranging from object recognition, urban navigation to emergency response strategies. This platform aims to significantly advance the capabilities of RL algorithms by exposing them to complex, diverse, and dynamically changing environments. Leveraging the advanced graphical and physical simulation capabilities of UE5, the project will focus on creating detailed and varied scenarios in which RL algorithms can be trained and tested. These scenarios will include, but not be limited to, urban traffic systems, natural disaster simulations, and public safety response models. The realism and intricacy of UE5’s environment will provide a challenging and rich training ground for RL models, allowing them to learn and adapt to unpredictable variables akin to those in the real world.
IIOT for legacy and intelligent factory machines with XR and LLM feedback with a Digital Twin demonstration of real-time IOT for architecture/building applications using Omniverse. 
$40,000. UC Industry 4.0/5.0 Institute Consortium Research Project: 01.2024-01.2025
The project centers on the development of a Digital Twin (DT) and a multi-agent Large Language Model (LLM) framework designed to access and interpret real-time and historical data through an Industrial Internet of Things (IIoT) platform. Real-time data is sourced from legacy machines and smart machines, integrating Building Information Modeling (BIM) with environmental sensors. The multi-agent LLM framework comprises specialized agents and supports diverse user interfaces, including screen-based systems, Virtual Reality (VR) environments, and mobile devices, enabling versatile interaction, data visualization, and analysis.
The research evaluates leading DT platforms—Autodesk Tandem, NVIDIA Omniverse, and Unreal Engine—focusing on their capabilities to integrate IoT and BIM data while supporting legacy machine systems. Autodesk Tandem excelled in seamlessly combining BIM metadata with real-time IoT streams for building operations and system scalability. NVIDIA Omniverse demonstrated unmatched rendering fidelity and collaborative features through its Universal Scene Description (USD) framework. Unreal Engine, notable for its immersive visualization, proved superior for LLM integration, leveraging 3D avatars and conversational AI to enhance user interaction.
Preface
In recent years, Artificial Intelligence (AI) has made significant inroads into the creative industries, notably in fields such as architecture design. The advent of AI-driven image creation tools and advanced language models has opened new frontiers in ideation, enabling a unique synergy between human creativity and generative AI imaging. This independent study aims to delve deep into the potential impacts of AI on the discipline of architecture and its integration into the design process. Three architecture students from the University of Cincinnati, Alex, Jackman, and Nathan, have embarked on a comprehensive investigation, utilizing cutting-edge AI tools to enhance their design capabilities and challenge and redefine traditional methodologies. Throughout their journey, three students have explored various dimensions of AI applications in architecture – from conceptualization and visualization to more complex aspects like representation and style. We invite you to explore the fruits of their experiments, which demonstrate their skillful application of AI in architecture and offer a glimpse into the future of architectural design, where human ingenuity and AI collaborate to create extraordinary works.
Ming Tang
The digital human project explores (1) high-fidelity digital human modeling and motion capture including Full body + Facial motion capture. (2) Large Language Model integration with digital human, powered by ChatGPT and Open AI.
