Posts

paper at NAMRC conference

/in /by

Anuj Gautam , Manish Raj Aryal, Sourabh Deshpande, Shailesh Padalkar, Mikhail Nikolaenko, Ming Tang, Sam Anand. IIoT-enabled Digital Twin for legacy and smart factory machines with LLM integration. 53rd SME North American Manufacturing Research Conference (NAMRC), Clemson Univ. 06/2025.

The paper is also accepted in the  Journal of Manufacturing Systems 

Abstract

The recent advancement in Large Language Models (LLMs) has significantly transformed the field of natural data interpretation, translation, and user training. However, a notable gap exists when LLMs are tasked to assist with real-time context-sensitive machine data. The paper presents a multi-agent LLM framework capable of accessing and interpreting real-time and historical data through an Industrial Internet of Things (IIoT) platform for evidence-based inferences. The real-time data is acquired from several legacy machine artifacts (such as seven-segment displays, toggle switches, and knobs), smart machines (such as 3D printers), and building data (such as sound sensors and temperature measurement devices) through MTConnect data streaming protocol. Further, a multi-agent LLM framework that consists of four specialized agents – a supervisor agent, a machine-expertise agent, a data visualization agent, and a fault-diagnostic agent is developed for context-specific manufacturing tasks. This LLM framework is then integrated into a digital twin to visualize the unstructured data in real time. The paper also explores how LLM-based digital twins can serve as real time virtual experts through an avatar, minimizing reliance on traditional manuals or supervisor-based expertise. To demonstrate the functionality and effectiveness of this framework, we present a case study consisting of legacy machine artifacts and modern machines. The results highlight the practical application of LLM to assist and infer real-time machine data in a digital twin environment.

AI and Emerging Technology Symposium

/in , , /by

Ming Tang and Mikhail Nikolaenko presented “AI-Powered Digital Humans for Enhanced Interaction in Extended Reality” at the AI and Emerging Technology Symposium, University of Cincinnati.

The day-long event explored topics around AI and robotic process automation; smart campus innovation; and extended reality, virtual reality, and augmented reality. More on UC News.

AI-Powered Talking Avatars for Enhanced Interaction in Extended Reality

Presenter. Ming Tang, Mikhail Nikolaenko. Feb. 20, 2025 in Tangeman University Center. 

This presentation explores two AI-driven talking avatars developed at the UC Extended Reality (XR) Lab, leveraging large language models (LLMs) for realistic interaction in XR environments. The XRLab Bot acts as a virtual tour guide, providing real-time engagement and navigation through the lab with spatial awareness, while the P&G Bot emulates a high-fidelity human likeness, delivering product expertise within a VR setting. These bots highlight advancements in AI, LLMs, and XR, showcasing potential applications in education, customer service, and smart campuses. The presentation will cover AI-driven navigation, multi-client architecture, and XR integration for immersive digital experiences. The session will showcase AI-driven navigation and interaction, demonstrating the bot’s capabilities in translating speech-to-text using Whisper AI, retrieving responses from ChatGPT, and interpreting real-time visitor needs and spatial data to guide users throughout XRLab. It will explore the multi-client, real-time architecture by sharing insights on managing multiple Unreal and Python clients with a central server, coordinating bot actions, face tracking, and area-specific responses in real-time. The discussion will highlight XR integration and smart campus applications, emphasizing the bot’s adaptability within XR platforms using Unreal Engine and its potential for virtual and augmented reality applications in campus tours, orientations, and educational experiences. Additionally, the session will discuss LLM-driven conversational AI, utilizing advanced models to power sophisticated, natural language interactions with users. High-fidelity 3D avatar creation will be addressed, focusing on crafting detailed, lifelike avatars capable of mimicking human expressions and movements. It will also cover customizable AI for chat avatars, enabling personalized, AI-driven avatars tailored to specific user preferences and needs. Interactive avatars with facial animation and motion capture will be demonstrated, showing how avatars can exhibit dynamic facial expressions and reactions during interactions. The session will also explore metaverse creation, showcasing the development of immersive, interconnected virtual worlds where users can interact through their avatars. Finally, the discussion will include virtual reality (VR) and augmented reality (AR) environments and experiences, highlighting their ability to blend digital content with the physical world or create entirely virtual spaces.

High-fidelity digital human.

Read more

Artistic Intelligence with AI: Gallery show at Blink

/in /by

I’m excited to be part of a group of UC DAAP faculty and student works exhibited at the 2024 Cincinnati Blink. The theme of the exhibition is “Artistic Intelligence with AI.”

Thanks to the support from Midwestcon, DisruptionNow, DAAP, and 1819 Innovation Hub.

October 18-20. Cincinnati. OH.

Read more

Redefining Space in the Age of Spatial Computing

/in , /by

ARCH 4001. Fall. 2024

Redefining Space in the Age of Spatial Computing 

Ming Tang
Professor, Registered Architect, NCARB, LEED AP, Director of Extended Reality Lab. UC Digital Futures. Leadership committee, Institute for Research in Sensing (IRiS);
School of Architecture and Interior Design, College of Design, Architecture, Art, and Planning, University of Cincinnati

In the age of spatial computing, architecture transcends the physical boundaries of bricks and mortar, evolving into an expansive realm that merges computational technologies. This studio embraces the transformative potential of AI in the architectural design process, expanding physical spaces into the metaverse and positioning architecture at the forefront of our society’s digital revolution. By serving as a platform for spatial experience, architecture redefines our interaction with space and each other. 

This studio invites students to collectively harness the power of AI to create virtual architectural spaces that inspire, connect, and elevate the human spatial experience. Students design a convention center to host various activities, from large conference events like SIGGRAPH and trade shows like CES to small lectures and social events. They research the spatial layout of a convention center and explore the possibilities of migrating these activities to a fully 3D virtual space in the metaverse. The studio also explores the intersections of AI, digital twin, and XR technologies with the metaverse, enabling students to push the boundaries of traditional architectural design. Through this exploration, students gain insights into how digital advancements can enhance and transform spatial experiences, fostering innovation and creativity in the architectural field.

 

Read more