M A HAFIZ

Designed and implemented a company-grade evaluation and regression gating platform for learning-enabled controllers and classical robotics stacks. The platform converts rollouts into episode-level metrics, videos, reports and enforces an explicit SHIP / BLOCK decision before deployment.

Key features:

• Episode-level metrics & rollout video tracking
• Baseline vs candidate comparison (locked baselines)
• Automatic regression detection (success, latency, safety)
• Gate decisions enforceable via CI (GitHub Actions)
• Simulator-agnostic artifact format (not MuJoCo-specific)

Note: The failing GitHub Action in the demo is intentional and demonstrates performance-based release gating.

Related pages: Results • Architecture

Developed an end-to-end pipeline that converts real-world image collections into photorealistic Gaussian Splatting environments and imports them into NVIDIA Isaac Sim for robotics simulation and digital-twin development.

For the Gerrard Hall demonstration, the system processed 100 real-world images, reconstructed camera poses and sparse geometry using COLMAP, trained a Gaussian Splatting model, and exported the result as a PLY and NuRec-compatible USDZ scene.

Pipeline:

Real-world images
→ COLMAP feature extraction and reconstruction
→ Gaussian Splatting training
→ PLY export
→ 3DGRUT / NuRec USDZ conversion
→ NVIDIA Isaac Sim
→ physics colliders and robot integration

Current results:

• Processed and registered 100 Gerrard Hall images
• Generated approximately 482,000 Gaussian primitives
• Exported a 114 MB Gaussian Splatting PLY file
• Converted the reconstruction into a NuRec-compatible USDZ scene
• Loaded and rendered the reconstructed environment in Isaac Sim
• Added a separate physical ground collider
• Validated rigid-body interaction using a falling cube
• Integrated a Carter mobile robot into the reconstructed scene
• Built a web upload workflow using Streamlit and FastAPI
• Added Redis and Celery for queued reconstruction jobs

Hybrid simulation architecture:

Gaussian reconstruction → photorealistic visual layer
Proxy colliders         → physical interaction layer
Robot USD               → articulation and control layer
FastAPI + Celery        → reconstruction job-processing layer

The Gaussian scene provides high-quality visual reconstruction, while separate collision geometry provides physical interaction for robots and rigid objects. This hybrid approach allows realistic real-world environments to be used for robotics simulation without requiring a manually modelled visual scene.

Gerrard Hall simulation demo:

Applications: robotics testing, digital twins, autonomous navigation, sim-to-real development, robot-policy evaluation, and scalable generation of simulation environments.

Status: Active development. Current work focuses on automatic physics-proxy generation, containerized GPU workers, Docker deployment, and Kubernetes-based reconstruction jobs.

Developed an end-to-end mobile-to-simulation platform that converts real-world videos into photorealistic Gaussian Splatting environments for NVIDIA Isaac Sim.

A user can record or select a video directly from an iPhone using either a mobile web interface or an Expo Go application. The video is uploaded to a queued GPU reconstruction backend, processed through COLMAP and Nerfstudio, converted to PLY and NuRec-compatible USDZ, and made available for download from the phone.

Complete workflow:

iPhone video capture or upload
→ Streamlit mobile interface / Expo Go app
→ FastAPI upload and job creation
→ Redis queue
→ Celery GPU worker
→ video frame extraction
→ COLMAP reconstruction and quality validation
→ Nerfstudio Gaussian Splatting training
→ Gaussian PLY export
→ 3DGRUT / NuRec USDZ conversion
→ mobile artifact download
→ NVIDIA Isaac Sim

Mobile capabilities:

• Record or select a video directly from an iPhone
• Upload scenes through a mobile web interface or Expo Go app
• Monitor reconstruction progress and COLMAP quality metrics
• Review registration ratio, sparse points, and quality grade
• Approve short preview and full Gaussian training stages
• Recover the active job after Safari refresh or reconnection
• Download final PLY and Isaac Sim-compatible USDZ files

Deployment architecture:

Mobile client
→ Streamlit / Expo Go
→ FastAPI service
→ Redis broker and job store
→ Celery GPU worker
→ COLMAP + Nerfstudio + 3DGRUT
→ shared reconstruction outputs

Docker deployment:

• Containerized FastAPI backend
• Containerized Redis queue and job store
• Containerized Streamlit frontend
• Containerized NVIDIA GPU Celery worker
• One-command startup using Docker Compose
• Verified complete PLY and USDZ generation inside Docker

Kubernetes deployment:

• Deployed API, Redis, frontend, and worker as Kubernetes workloads
• Configured NVIDIA GPU scheduling with nvidia.com/gpu
• Validated RTX 5060 Ti access inside a Kubernetes pod
• Used health checks, init containers, services, and persistent Redis storage
• Added pod-local COLMAP scratch storage for reliable SQLite locking
• Completed an iPhone-submitted reconstruction through Kubernetes
• Downloaded the generated USDZ directly from the mobile interface

Validated end-to-end result:

iPhone
→ Kubernetes frontend
→ FastAPI
→ Redis / Celery
→ NVIDIA GPU worker
→ COLMAP
→ Gaussian Splatting
→ PLY
→ NuRec USDZ
→ mobile download
→ Isaac Sim

Simulation integration:

• Loaded reconstructed Gaussian environments in Isaac Sim
• Added separate ground and obstacle collision geometry
• Validated rigid-body interaction using a falling cube
• Integrated a Carter mobile robot into the reconstructed scene

Hybrid simulation architecture:

Gaussian reconstruction → photorealistic visual layer
Proxy colliders         → physical interaction layer
Robot USD               → articulation and control layer
FastAPI + Redis + Celery → reconstruction job-processing layer
Docker / Kubernetes     → deployment and GPU orchestration layer

The Gaussian scene provides the photorealistic visual environment, while separate proxy collision geometry provides physical interaction for robots and rigid objects. This approach reduces the need for manually modelling complete real-world environments before robotics simulation.

Demo video:

Applications: robotics testing, digital twins, autonomous navigation, sim-to-real development, robot-policy evaluation, warehouse reconstruction, and scalable generation of simulation environments.

Status: Working end-to-end prototype. Docker and GPU-enabled Kubernetes deployments have been completed and tested from iPhone capture through final USDZ generation and download. Current development focuses on automatic collision generation, semantic scene understanding, robot navigation, and task validation.

Developing an early-stage robot failure analysis and data-generation platform that converts failed manipulation rollouts into structured diagnostic reports, targeted retry scenarios, and training-ready datasets for robot learning systems.

The current prototype focuses on MuJoCo manipulation tasks and compares random retry generation with targeted failure-driven retry generation to evaluate whether failed episodes can be used to create more useful training and evaluation data.

Current capabilities:

• Processes robot manipulation rollouts and failure cases
• Compares random vs targeted retry strategies
• Generates scenario variations around failure conditions
• Supports analysis of robot/object/action traces
• Explores data generation for VLA, imitation learning, and reinforcement learning workflows

Startup direction:

The long-term goal is to help robotics teams upload failed robot episodes and receive automatic failure diagnosis, targeted simulation retries, and exportable training/evaluation data.

Status: Ongoing MVP / startup exploration.

Developed an early-stage surgical robotic assistance simulation prototype in Isaac Sim to explore how robots can support operating-room teams through scene understanding, instrument localization, and tool-handling assistance.

The project includes an operating-room-style simulation scene with multiple surgical instruments and a robotic assistant setup. The system was used to test concepts such as recognizing available instruments, identifying their positions, and preparing robot actions for tool assistance tasks.

Demo video:

Key elements:

• Created a simulated surgical assistance scene with multiple instrument assets
• Explored instrument localization and context-aware robot assistance
• Tested robot approach and manipulation behavior for tool handling
• Developed an early prototype for picking surgical tools such as scissors/instruments
• Investigated challenges in grasping, object physics, end-effector alignment, and simulation-to-planning consistency

This prototype is directly aligned with future work on humanoid robots as intelligent clinical assistants for instrument handover, workflow support, and human-robot interaction in surgical environments.

Status: Internal research prototype.

Built a Vision-Language-Action pick-and-place pipeline for a Franka Emika Panda robot in MuJoCo. The system converts natural-language commands into grounded objects, symbolic actions, numerical action vectors, robot-executable action sequences, and learning-ready demonstrations.

Example command:

pick the red cube and place it at x 0.55 y -0.45

Pipeline:

Natural language
→ parsed task
→ object grounding
→ symbolic action
→ numerical action vector
→ robot action sequence
→ MuJoCo execution
→ demonstration dataset
→ behavior cloning / diffusion / world model / preference / RL / video-action prototypes

Key components:

• Rule-based natural-language parser for pick-and-place commands
• Object grounding for colored cubes and named target zones
• VLA-style symbolic action encoder and numerical action vector representation
• Robot action sequence generator: approach, grasp, lift, move, place, retreat
• Demonstration dataset logger saving language, grounding, actions, and sequences as JSON
• Dataset loader for behavior cloning arrays
• RandomForest behavior cloning baseline and nearest-neighbor imitation baseline
• Tiny diffusion-style denoising action policy prototype
• Tiny world model for next-object-state prediction
• Preference learning prototype for ranking candidate actions
• RL-style reward function and random policy evaluation
• Video Action Dataset prototype with frame-level action phase labels

Action vector format:

[
  action_type_id,
  object_id,
  pick_x,
  pick_y,
  place_x,
  place_y,
  target_id,
  has_explicit_xy
]

Example action sequence:

1. APPROACH_OBJECT
2. OPEN_GRIPPER
3. DESCEND_TO_OBJECT
4. CLOSE_GRIPPER
5. LIFT_OBJECT
6. MOVE_TO_TARGET
7. DESCEND_TO_PLACE
8. OPEN_GRIPPER
9. RETREAT

This project was designed as a compact prototype for embodied AI and robot learning workflows, connecting language-conditioned manipulation with action encoding, imitation learning, diffusion-style prediction, world models, preference learning, reward evaluation, and video-action dataset preparation.

Developed a gesture-based Cartesian and rotational control system for intuitive human–robot interaction. The pipeline processes IMU signals (acc/gyro) for orientation estimation, filtering, and noise handling, combines them with EMG features, and maps gestures to stable real-time robot commands.

Validated in MuJoCo simulation and transferred to a real Franka Emika Panda robot, addressing practical constraints including latency, sensor noise, safety limits, and hardware variability.

Key contributions:

• IMU preprocessing: filtering, normalization, orientation estimation
• Gesture recognition using IMU + EMG features
• Cartesian & rotational control using IK and Jacobians
• Simulation-to-real transfer and robustness evaluation
• Contact-rich screw insertion with coordinated axial + rotational motion

Designed a MuJoCo environment that mirrors the lab setup (table, screw box, tool) used for gesture-based teleoperation and screwing / pick-and-place experiments.

Demo video:

🔗 View MuJoCo Franka Simulation on GitHub

Implemented a full pick & place pipeline in MuJoCo with a custom Tkinter GUI: detect → hover → pre-grasp → grasp → lift → place. Emphasized smooth Cartesian behavior and repeatable execution.

Demo Video:

🔗 View Project on GitHub

Built a stable Nav2 demo in simulation. Integrated a custom Behavior Tree and validated recovery behavior by blocking the robot path and observing replanning + goal completion.

Demo video:

🔗 View ROS2 Nav2 Project on GitHub

Implemented an online async SLAM pipeline using SLAM Toolbox with Gazebo + teleop + RViz2. Includes map saving for later Navigation2 experiments.

Project image:

🔗 View ROS2 SLAM Mapping Project on GitHub

Developed a landmine-detecting robot using Raspberry Pi and sensors for a national Robo-Fight competition. Implemented obstacle avoidance and metal detection logic for simulated mine detection/removal.

Demo timestamp: 2:03

GitHub repo (with teammate Md. Kaiser Raihan): 🔗 View on GitHub

Collaborated with: rahi008

Built a speed measurement setup using two IR sensors and microcontroller timer interrupts. Computed speed as Distance/Time and displayed real-time results on a 16×2 LCD.

🔗 View Project on GitHub

Designed an automatic irrigation system using a soil moisture sensor (voltage divider) and relay-driven pump. Turned the pump on/off based on moisture thresholds for efficient water usage.

🔗 View Project on GitHub

Designed and simulated an n-bit comparator in Cadence Virtuoso (schematic + layout), verified with DRC/LVS and evaluated delay/power/area trade-offs.

📄 View Full Project Report (PDF)