HomeAboutProjectsSkillsAchievementsContact
🌙
Back to the radar

MediaPipe TouchDesigner Plugin

instagramstrong
Original post

TL;DR

mediapipe-touchdesigner is a pre-built TouchDesigner plugin (distributed as .tox files) that integrates Google's MediaPipe machine-learning framework directly into TouchDesigner, enabling real-time hand tracking, pose estimation, and other computer-vision features without writing any ML code. In the context of this post, it is used to extract fingertip XY coordinates and map them as corner pins for an interactive, hand-controlled panel. Setting up MediaPipe inside TouchDesigner from scratch requires complex Python/C++ bridging; this plugin packages everything into drag-and-drop .tox components, dramatically lowering the barrier for creative coders and visual artists who want real-time body/hand tracking in interactive installations.

What it actually is

  • What: mediapipe-touchdesigner is a pre-built TouchDesigner plugin (distributed as .tox files) that integrates Google's MediaPipe machine-learning framework directly into TouchDesigner, enabling real-time hand tracking, pose estimation, and other computer-vision features without writing any ML code. In the context of this post, it is used to extract fingertip XY coordinates and map them as corner pins for an interactive, hand-controlled panel.
  • Who built it / maintained by: Torin Blankensmith (@blankensmithing)
  • Status: stable
  • Why it matters: Setting up MediaPipe inside TouchDesigner from scratch requires complex Python/C++ bridging; this plugin packages everything into drag-and-drop .tox components, dramatically lowering the barrier for creative coders and visual artists who want real-time body/hand tracking in interactive installations.
  • How it compares to alternatives:
  • Leap Motion Controller SDK
  • Ultraleap Hand Tracking
  • OpenPose
  • ml5.js
  • Wekinator
  • KinectV2 for TouchDesigner
  • GitHub stars: 2,381 · License: MIT · Archived: no

Links

  • Repo: https://github.com/torinmb/mediapipe-touchdesigner
  • Docs: https://youtu.be/e2FtkufeErY

Kickstarter guide

Download mediapipe.tox and handtracking.tox from the GitHub releases page and drag both files into your TouchDesigner project. In the MediaPipe component's parameters under the 'Common' tab, enable 'External .tox', then wire the 'hands' output of mediapipe into the input of handtracking. From there, use Select CHOPs to pull individual fingertip XY channels and connect them to any operator parameters — such as a Corner Pin TOP — to drive real-time interactive visuals.