This is a note on how I created 3D characters with Three.js for performers to control with Kinect, as part of professor Shawn Van Every’s project in Future of Storytelling. The idea is creating conversations between dancers and audiences through internet, thus audiences aren’t restricted within theatre.

Inspiration

• key points: neutral characteristic for audience to easily relate to, simple and related action, the feeling of magic

( insert video inspiration )

Experiment

<–What you can get from Kinect

<–My simplified version

To represent joints data as dancers, I tried different ways: 2D – plane, 3D – separate geometry, skeletal geometry, and joint geometry, and so far the most satisfying versions are 2D plane and 3D joint geometry, and the skeletal geometry didn’t suit this project because of the hierarchy structure of it, e.g. change joint_0 will also change joint_1, because joint_0 is the parent of joint_1.

( insert video documentations )

2D plane version

Steps

1. draw texture for limbs first
2. based on the size of each picture, create THREE.PlaneGeometry
3. offset the geometry –> change the pivot to appropriate position, e.g. the head is rotating around neck, not around nose.
   • code example:
     • headTexHeight = headTex.image.height/111, headTexWidth = headTex.image.width/111;
     • var headGeo = new THREE.PlaneGeometry(headTexWidth, headTexHeight);
     • transY(headGeo, 4);
4. To make bear body part follow joint data, updating the position of bear body with joint data.
5. Use lookAt ( Rotates object to face point in space ) function of Three.js to update the rotation of bear body. !!Note!! also use matrix rotation to make sure the direction of rotation
6. To have stretchy effect, scale the bear body base on the distance changes between joints.
   • hL2.lookAt(joints[5].position);
   • hL2.position.copy(joints[0].position);
   • hL2MoveX = Math.sqrt(Math.pow((joints[0].position.x – joints[5].position.x), 2))-5;
   • hL2MoveY = Math.sqrt(Math.pow((joints[0].position.y – joints[5].position.y), 2))-2;
   • hL2.scale.set(1, 1, 1+( hL2MoveX/armULTexWidth + hL2MoveY/armULTexHeight/2 ));

3D joint version

Steps

1. Build 3D model in Maya
2. Create joints
3. Skin bind the joint with model
4. Skin weight paint to fixed the weight of binding, creating smooth transformation
5. Export as FBX file, and then import to Blender, and then export as JS file (I know there’s a lot of detouring, but it’s so far the only successful method I’ve tried)
6. Compared with 2D version’s hacky way to update the position of character’s body part, the skinned mesh is much easier to do the animation. Since the geometry is already binded with joints, what I have to do is updating the joint position correctly.
   1. elmoBone[0].position.copy( joints[5].position );
7. For the rotation part, I can get the desired rotation angle based on the relationship of each joints’ positions
   1. lengthForRot = elmoBone[3].position.distanceTo( elmoBone[2].position );
      rotForJoint = Math.asin( (elmoBone[3].position.y-elmoBone[2].position.y)/lengthForRot );
      elmoBone[3].rotation.y = rotForJoint;

And here’s a snippet!