Gestures as Interface with TouchDesigner and MediaPipe

Let’s get weird with TouchDesigner! This workshop is about designing and speculating playful interactions that use face, body, and gestures as control interface. We'll work with TouchDesigner’s MediaPipe plugin implemented by Torin Blankensmith as a technical foundation, while exploring interactive, graphic, and interface design.

An introduction to perhaps unorthodox ways to think about novel game controls and screen-based interactivity, we'll prototype gestural interfaces that could exist in alternative presents or plausible futures with TouchDesigner, with a focus on embodied experiences and a 2D collage approach to making visuals.

For more workshops like this, visit Tandon at The Yard's 2025 Summer Workshops page.

• Creative coders, VJs, generative-art fans exploring non-traditional inputs
• Game & interaction designers seeking novel, embodied control schemes
• Performers & media artists wanting camera-based triggers for shows or installations
• Basic TouchDesigner navigation is helpful; no prior computer-vision experience required.
  
  

• Laptop with the latest TouchDesigner (non-commercial license OK)
• Download Torin Blankensmith’s MediaPipe for TD plug-in (link provided after sign-up)
• Built-in or USB webcam (laptop camera is fine)
  
  

10:00 – 10:30

Welcome & Ice Breaker
Quick introductions, ice-breaker that gets us moving around in the space

10:30 – 11:00

Speculative Interface
Alternative game control, OS design, and desktop performance - exploring different modalities of design and interactivity

11:00 – 12:00

MediaPipe Setup & 2D Basics in TD
Plugin installation, hand/face tracking fundamentals, layering techniques in TD that feel more like graphic design

12:00 – 12:30

Break

12:30 – 1:30

Build Session 1 – Gesture-Controlled Pixel Photo Booth
Follow-along example, building a pixel art style photobooth controlled by hand movements, working with real-time composition

1:30 – 2:15

Speculative Everything
Designing for alternative presents, relationship between tools and art, a critique of digital aesthetics, and a little bit AI for creative projects :)

2:15 – 2:30

Break

2:30 – 3:00

Design Workshop
Brainstorming and rapid feedback for your project ideas

3:00 – 4:30

Build Session 2 – Project Jam
Open experimentation time, build your project, debugging

4:30 – 5:00

Demo, share, play!

Participants will:

1. Install & run MediaPipe inside TouchDesigner for hand/face/body tracking.
2. Prototype at least one gesture-controlled visual experience.
3. Apply 2-D collage techniques for quick, graphic-style compositions.
4. Develop speculative concepts that rethink how we perform with screens.
5. Take away a project file, design sketches, and resources for further exploration.
   
   

Viola He (@v10101a) is a Shanghai/NYC based educator, performer, interdisciplinary artist, and culture organizer, exploring pathways towards alternative structures, systems and interfaces. They create immersive experiences, installations, XR theatre, performances and performance systems using algorithmic approaches to enhance, alternate, and obfuscate audio-visual assets, investigating new sonic and visual aesthetics across digital and physical mediums.

Viola is an Assistant Professor of Interactive Media Art at NYU Shanghai, as well as an organizing artist in leaderless collective Livecode.NYC, in which they continue to research, organize, teach, perform, and discuss the relationship between community and technology. They have performed in SXSW, Under the Radar Festival, Festival Tsonami, Hong Kong International Poetry Festival, as well as various clubs and venues in Shanghai, Tokyo, Los Angeles, SF, NYC, and more.

Torin Blankensmith (@blankensmithing) is a freelance creative technologist, real-time graphics artist, and educator specializing in mixed-reality installations, interactive experiences, and open-source creative tools.

Blankensmith creates and exhibits his own interactive artworks and installations while sharing his creative process and tools through educational videos on YouTube and Patreon. He’s co-created several open-source tools to empower artists working with emerging technologies, including Shader Park, a JavaScript library for interactive procedural shaders, and the MediaPipe plugin for TouchDesigner, enabling real-time motion tracking for thousands of artists.

He is the co-founder of DATLAB (Digital Art and Technology Lab), hosting a monthly NYC meetup series that brings together hundreds of artists, designers, and technologists for workshops, talks, AV performances, and critical dialogue around interactive media.

Since 2020, Blankensmith has been an adjunct professor at Parsons School of Design, teaching Advanced Creative Coding, TouchDesigner, and Adaptive Technology in both MFA and BFA programs. Previously, at Studio Elsewhere, he led the development of immersive environments to support neuroscience research and wellbeing and mental health.

His work has been shown at SIGGRAPH Asia, Art Basel’s Beyond Basel, VIDEOCITTÀ, TouchDesigner Berlin, and he has participated in NEW INC Year 6, Google Creative Lab 5, and the ITP/IMA Project Fellowship. His previous clients include The Nobel Foundation, Mt. Sinai Hospital, Google Creative Lab, ARTECHOUSE, Lab at Rockwell, Vice, Adobe, and Microsoft.

Also check out other Arts events in Brooklyn, Workshops in Brooklyn, Festivals in Brooklyn.