HomeAboutProjectsSkillsAchievementsContact
🌙
Back to the radar

MediaPipe TouchDesigner Plugin

source post: Video by prash_creates

instagramstrong
Original post

Video by prash_creates

Source: instagram · Prashansa Maurya Saved: 20260608 Tags: instagram, touchdesigner, interactivedesign, handtracking Display: MediaPipe TouchDesigner Plugin — Pre-built .tox plugin integrating Google MediaPipe into TouchDesigner for real-time hand tracking and pose estimation.

TL;DR

mediapipe-touchdesigner is a pre-built TouchDesigner plugin (distributed as .tox files) that integrates Google's MediaPipe machine-learning framework directly into TouchDesigner, enabling real-time hand tracking, pose estimation, and other computer-vision features without writing any ML code. In the context of this post, it is used to extract fingertip XY coordinates and map them as corner pins for an interactive, hand-controlled panel. Setting up MediaPipe inside TouchDesigner from scratch requires complex Python/C++ bridging; this plugin packages everything into drag-and-drop .tox components, dramatically lowering the barrier for creative coders and visual artists who want real-time body/hand tracking in interactive installations.

What the post showed

Caption: Here's how you can make your own interactive panel in touchdesigner.

1. mediapipe setup you need @blankensmithing 's mediapipe files file to handle all the hand tracking, watch his tutorial first, it's genuinely the best resource for this: https://youtu.be/e2FtkufeErY download his mediapipe files here: https://github.com/torinmb/mediapipe-touchdesigner

-drag the mediapipe.tox and handtracking.tox files into touchdesigner. -connect the hands output of mediapipe specifically into the hand tracking input, mediapipe has multiple outputs so make sure you're grabbing the right one.

2. select CHOPs add 4 select CHOPs next to your hand tracking node. the select CHOP is basically a filter it pulls only the specific data channels you need from a larger stream. pull these channels:

h1 index fingertip x + y h2 index fingertip x + y h1 thumb fingertip x + y h2 thumb fingertip x + y

3. corner pin TOP this is the operator that made everything click for me. drop it in and go to parameters → pin section. you'll see top left, top right, bottom left, bottom right. click the star icon at the bottom right corner of the select nodes- this puts it into reference mode. drag your h1 thumb x channel directly into bottom left x channel → select CHOP reference. repeat for the y. do this for all four corners:

top left - h1 index x + y top right - h2 index x + y bottom left - h1 thumb x + y bottom right - h2 thumb x + y

4. rectangle + composite create a rectangle TOP and connect it into the corner pin input. this is your panel. now add a composite TOP. feed two things into it - the corner pin output and the mediapipe camera output (top). the composite TOP layers multiple images together. go to parameters → operation dropdown → start playing with blending modes. exclude or negate gives you that inverted colour effect. you could also play with the rectangle's colours.

so there it is! a basic interactive panel. I'm still learning and exploring it myself, so id love any feedback.

built with @touchdesigner #touchdesigner #interactivedesign #handtracking

Key claims from transcript: So building this interactive panel was way easier than I thought, and here is how you can make it today. Alright, stop one, media pipe and hand tracking. So here I've used torrenting black and smith media pipe fails, and you can download it from the link in the description. Also, I recommend you to check out his videos on hand tracking, it's super helpful. So drag in the media pipe and the hand tr

What it actually is

  • What: mediapipe-touchdesigner is a pre-built TouchDesigner plugin (distributed as .tox files) that integrates Google's MediaPipe machine-learning framework directly into TouchDesigner, enabling real-time hand tracking, pose estimation, and other computer-vision features without writing any ML code. In the context of this post, it is used to extract fingertip XY coordinates and map them as corner pins for an interactive, hand-controlled panel.
  • Who built it / maintained by: Torin Blankensmith (@blankensmithing)
  • Status: stable
  • Why it matters: Setting up MediaPipe inside TouchDesigner from scratch requires complex Python/C++ bridging; this plugin packages everything into drag-and-drop .tox components, dramatically lowering the barrier for creative coders and visual artists who want real-time body/hand tracking in interactive installations.
  • How it compares to alternatives:
  • Leap Motion Controller SDK
  • Ultraleap Hand Tracking
  • OpenPose
  • ml5.js
  • Wekinator
  • KinectV2 for TouchDesigner
  • GitHub stars: 2,381 · License: MIT · Archived: no

Links

Kickstarter guide

Download mediapipe.tox and handtracking.tox from the GitHub releases page and drag both files into your TouchDesigner project. In the MediaPipe component's parameters under the 'Common' tab, enable 'External .tox', then wire the 'hands' output of mediapipe into the input of handtracking. From there, use Select CHOPs to pull individual fingertip XY channels and connect them to any operator parameters — such as a Corner Pin TOP — to drive real-time interactive visuals.