Autonomous surgical needle path planning using Q-Learning and DQN in a PyBullet physics simulation with a Franka Panda robotic arm.

University of Maryland, College Park | Yaxita Amin & K. Manasanjani

This project presents a reinforcement learning approach for autonomous path planning in robotic brain surgery simulation. An AI agent learns to navigate a surgical needle through complex 3D brain vasculature — avoiding blood vessels while reaching tumor targets — using Q-Learning and Deep Q-Network (DQN) algorithms.

Key highlights:

• 98–100% training success rate with tabular Q-Learning over 3,000 episodes
• 80–100% generalization to unseen tumor targets
• 0% vessel collision rate vs. 40% for straight-line approaches
• ≥4mm safety margin guaranteed from all blood vessels
• Full integration with a Franka Panda robotic arm via Inverse Kinematics

WITHOUT ROBOT ARM

WITH ROBOT ARM AND NEEDLE

• Python 3.10+
• Ubuntu (recommended) or macOS

pybullet
trimesh
scipy
numpy
torch
matplotlib

python final13.py

python brain_surgery_dqn.py

3D voxel grid position (2mm resolution) relative to tumor location.

6 discrete moves: {+X, −X, +Y, −Y, +Z, −Z} (2mm per step)

R(s, a, s') = R_goal + R_collision + R_timeout + R_shaping

• R_goal = +100 — reaching the tumor
• R_collision = -100 — vessel proximity < 4mm
• R_timeout = -50 — exceeding 100 steps
• R_shaping = Δd — distance reduction to target

• Robot: Franka Emika Panda (7 DOF, simulated in PyBullet)
• IK Solver: PyBullet damped least-squares with joint limit handling
• IK success rate: 98%
• End-effector accuracy: <0.5mm positioning error
• Average execution time: 8.5 seconds per path

brain_surgery_docker/
├── data/                       # STL models & brain vasculature data (~7.5MB)
├── final13.py                  # Q-Learning main script (path planning + simulation)
├── brain_surgery_dqn.py        # Deep Q-Network implementation
├── Dockerfile                  # Docker container configuration
├── requirements.txt            # Python dependencies
└── README.md

A Dockerfile is included for containerized, reproducible execution.

docker build -t brain-surgery-rl .

docker run --rm brain-surgery-rl

Note: PyBullet GUI visualization requires passing through a display. On Linux, use:

docker run --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix brain-surgery-rl

• Static environment (no tissue deformation modeling)
• Discrete action space limits path smoothness
• Single-needle, straight-segment paths only
• Simulated environment differs from real surgical conditions

• Extend to continuous state/action spaces using actor-critic methods
• Incorporate curved needle steering for challenging targets
• Multi-objective optimization (path length, clearance, energy)
• Dynamic replanning with real-time MRI/CT intraoperative feedback
• Sim-to-real transfer for physical Franka Panda deployment
• Training on diverse anatomical models for patient-agnostic planning

We thank Dr. Jerry Wu for guidance throughout this project, and teaching assistants Siddhant and Aswin for valuable feedback during development.

This project is intended for educational and pre-operative planning research purposes only. Not for clinical use.