Physical AI, Fine-Tuning VLM and Omniverse Training in Cork, Ireland

This course introduces learners to the emerging domains of Physical AI, Vision-Language Model (VLM) fine-tuning, and NVIDIA Omniverse for real-world digital simulation. Participants will explore how AI agents interact with physical systems, how multimodal AI models can be adapted for specific tasks, and how Omniverse enables digital twin creation and collaboration.

Introduction to Physical AI & Multimodal Systems

• Introduction to Physical AI

  • Definition, evolution, and real-world examples

  • Integration of perception, reasoning, and physical interaction

• Components of a Physical AI System

  • Sensors, actuators, AI agents, and robotics

• Overview of Vision-Language Models (VLMs)

  • Understanding CLIP, BLIP, Flamingo, and other foundational models

Fine-tuning Vision-Language Models (VLMs)

• Understanding the VLM fine-tuning pipeline

  • Dataset preparation: image-text pairs

  • Preprocessing and augmentation

• Tools for VLM Fine-tuning:

  • Hugging Face Transformers, PyTorch Lightning, and OpenCLIP

• Practical:

  • Fine-tuning a VLM on a custom dataset (image classification or captioning task)

  • Evaluation metrics: retrieval accuracy, text-image similarity

• Optimization Techniques:

  • LoRA, PEFT, and parameter-efficient fine-tuning

NVIDIA Omniverse and Integration with Physical AI

• Introduction to NVIDIA Omniverse

  • Overview of Omniverse Kit, Nucleus, and USD (Universal Scene Description)

  • Omniverse tools for AI and robotics simulation

• Building Digital Twins

  • Creating and importing 3D environments

  • Integrating AI agents for real-world interaction simulation

Hands-on Exercises

Summary and Conclusion