knot-detector

Real-time fine-motor action detection (knot-tying) via MediaPipe HandLandmarker

• heuristic pattern analysis over rolling windows of fingertip distances.

Two runtimes share one heuristic:

• web/ — pure client-side (MediaPipe Tasks Vision WASM+GPU), zero server, deployable to Hugging Face Spaces (static SDK).
• app.py / detector.py — Python + Gradio prototype for offline video analysis and local webcam.

Approach

• Track 21 landmarks per hand for both hands (numHands: 2, VIDEO mode).
• Rolling window (~45 frames) of:
  • inter-fingertip distance (thumb ↔ index across hands or intra-hand fallback)
  • combined hand bbox center + diagonal
• Heuristic “knot” = high variance of distance (normalized by bbox diagonal²)
  AND high zero-crossings of distance
  AND low drift of bbox center.
• Occlusion handling: per-handedness identity buffer with configurable hold-last-known
  window (~12 frames). Re-association by handedness (Left/Right) on reappearance.

Quality control

fixtures/ — reference videos + labeled frames.
tests/ — regression tests replaying fixtures through the detector and asserting
state transitions match golden labels.

See docs/QA.md for methodology.

Live demo

Hugging Face Space: https://huggingface.co/spaces/Dariachup/knot-detector

Local

# web (nothing to install — just open):
python3 -m http.server 8080 --directory web
# → http://localhost:8080

# python prototype:
pip install -r requirements.txt
python app.py

Roadmap

• Fixture ingestion + golden labels
• Regression test harness (pytest)
• Port heuristic to C++ core (reuse exercises-counter/cpp Port heuristic to C++ core (reuse `exercises-counter/cpp` pattern)
• iOS Vision Framework backend behind HandDetector iOS Vision Framework backend behind `HandDetector` Protocol
• Raspberry Pi 5 latency benchmark

License

MIT