Today,Project 2 Recap 3D Motion Capture Marker-based Video Based Mocap demo on Monday (2/26) Image segmentation and matting,Match Move Recap,Difficulty and Effort Varied 6 used direct linear method 3 used Tomasi-Kanade 1 used Zhangs approach 2 used home-brewed methods Data collection very important-getting good video makes all the difference Tracking was a challenge Discrete search more reliable than Lucas-Kanade for some Need to restart the tracker periodically Took a lot of fiddling with parameters to get right Rendering RenderX bugs Some used openGL, java, but not required Calibration nearly co-planar features a problem,Match Move Results,Direct Linear Method Factorization Zhangs method,Motion Capture,From “Final Fantasy” (Columbia Pictures) out Summer 2001,How is it Done?,Place ping pong balls on an actor Track the balls in 2D for each camera Triangulate to compute 3D positions Label markers Compute body pose inverse kinematics (IK) Do it all at up to 240 Hz ! Mocap demo at end of class today,Today: how to do this without markers,Hand Mocap,Capturing hands and fingers J. Rehg, T. Kanade, Model-based tracking of self-occluding articulated objects, In Proceedings of International Conference on Computer Vision, pages 612-617, Cambridge, MA, 1995. pdf, 300K Ying Wu and Thomas S. Huang, Capturing Articulated Hand Motion: A Divide-and-Conquer Approach, In Proc. IEEE Intl Conf. on Computer Vision (ICCV99), pp.606-611, Greece, Sept., 1999.,Heads and Faces,Capturing Faces Essa. I., and A. Pentland. Coding, Analysis, Interpretation and Recognition of Facial Expressions., Volume 19 (7), IEEE Computer Society Press, July, 1997. K. Toyama and G. Hager, Incremental Focus of Attention for Robust Vision-Based Tracking,IJCV(35), No. 1, November 1999, pp. 45-63. Douglas DeCarlo and Dimitris Metaxas, Optical Flow Constraints on Deformable Models with Applications to Face Tracking. In IJCV, July 2000, 38(2), pp. 99-127. PDF (695K),Human Body,Focus of today Search over 3D pose Gavrila & Davis 3D tracking Bregler & Malik Single view motion capture Leventon & Williams Brand,Gavrila and Davis,D. M. Gavrila and L. S. Davis, 3-D Model-based Tracking of Humans in Action: a Multi-view Approach, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, U.S.A., 1996.,Gavrila and Davis,Step 1: Edge Detection,Step 2: Background Edge Subtraction,Step 3: Search over 3D Parameters,Search for 3D body pose parameters Model body as segments (superquadrics), 22 DOF Discrete search over DOFs Metric: projected model should match edges Speedup: use distance transform of edge image,Divide and Conquer,Searching all pose parameters simultaneously is too hard,Stage 1:torso position,Stage 2:arm positions,Stage 3:leg positions,Bregler and Malik,Tracking approach Initialize 3D model in first frame Track over subsequent frames C. Bregler and J. Malik, Tracking People with Twists and Exponential Maps, Proc. IEEE CVPR 1998,How? Our old friend: Lukas and Kanade! But now (u,v) are functions of and,3D Tracking,Lets start with tracking a single rigid object We know 3D position X, orientation R in first frame Solve for change in 3D position and orientation,How? Our old friend,Problem: equation is not linear in and Solution (Bregler and Malik): use twist representation,Twist Representation Murray, Li, Sastry,Any rigid transformation may be expressed as a rotation about an axis and translation along that same axis w encodes the axis direction and rotation angle n the location of axis and amount of translation,Kinematic Chains As Twist Compositions,Bregler and Malik,Algorithm Overview Initialize 3D pose in first frame Compute support map pixels that are on each body part being tracked use layer extraction techniques (Wang & Adelson, EM) Apply modified Lukas-Kanade to estimate change in pose Repeat for each subsequent pose Works better with two or more views Each view gives more equations in unknown twist The more views, the better conditioned the problem is Can also do better initialization,Motion Capture from a Single Viewpoint,Basic Idea Train the system on a database of motions Bias the system to reconstruct shapes from the training set,Shadow Puppetry Brand, ICCV 99,Machine Learning Approach Model training 3D motion capture data using an HMM fancy HMM fitting method: Entropic Estimation global rotation/scale handled by duplicating HMM for each setting Find most likely path through HMM, given input data,Leventon and Freeman NIPS 2000,Train on motion capture data Break 3D motion into short snippets express motion as a linear combination of training snippets similar to Blanz and Vetter approach for face modeling track limbs in image sequence in 2D, break into 2D snippets Whole problem is cast as Bayesian estimation,Wren and Pentland,Real-time capture using learned motion models Combines inverse-kinematics with trained HMM model that predicts next move Uses “Blob Tracking” and two cameras C. Wren and A. Pentland, Dynamic Models of Human Motion, Proceedings of the Third IEEE International Conference, on Automatic Face and Gesture Recognition, April 14-16, 1998 in Nara, Japan.,