Note on creating 3D character for Kinect
This is a note on how I created 3D characters with Three.js for performers to control with Kinect, as part of professor Shawn Van Every’s project in Future of Storytelling. The idea is creating conversations between dancers and audiences through internet, thus audiences aren’t restricted within theatre.
Inspiration
- key points: neutral characteristic for audience to easily relate to, simple and related action, the feeling of magic
( insert video inspiration )
Experiment
<–What you can get from Kinect
<–My simplified version
To represent joints data as dancers, I tried different ways: 2D – plane, 3D – separate geometry, skeletal geometry, and joint geometry, and so far the most satisfying versions are 2D plane and 3D joint geometry, and the skeletal geometry didn’t suit this project because of the hierarchy structure of it, e.g. change joint_0 will also change joint_1, because joint_0 is the parent of joint_1.
( insert video documentations )
2D plane version
Steps
- draw texture for limbs first
- based on the size of each picture, create THREE.PlaneGeometry
- offset the geometry –> change the pivot to appropriate position, e.g. the head is rotating around neck, not around nose.
- code example:
- headTexHeight = headTex.image.height/111, headTexWidth = headTex.image.width/111;
- var headGeo = new THREE.PlaneGeometry(headTexWidth, headTexHeight);
- transY(headGeo, 4);
- code example:
- To make bear body part follow joint data, updating the position of bear body with joint data.
- Use lookAt ( Rotates object to face point in space ) function of Three.js to update the rotation of bear body. !!Note!! also use matrix rotation to make sure the direction of rotation
- To have stretchy effect, scale the bear body base on the distance changes between joints.
- hL2.lookAt(joints[5].position);
- hL2.position.copy(joints[0].position);
- hL2MoveX = Math.sqrt(Math.pow((joints[0].position.x – joints[5].position.x), 2))-5;
- hL2MoveY = Math.sqrt(Math.pow((joints[0].position.y – joints[5].position.y), 2))-2;
- hL2.scale.set(1, 1, 1+( hL2MoveX/armULTexWidth + hL2MoveY/armULTexHeight/2 ));
3D joint version
Steps
- Build 3D model in Maya
- Create joints
- Skin bind the joint with model
- Skin weight paint to fixed the weight of binding, creating smooth transformation
- Export as FBX file, and then import to Blender, and then export as JS file (I know there’s a lot of detouring, but it’s so far the only successful method I’ve tried)
- Compared with 2D version’s hacky way to update the position of character’s body part, the skinned mesh is much easier to do the animation. Since the geometry is already binded with joints, what I have to do is updating the joint position correctly.
- elmoBone[0].position.copy( joints[5].position );
- For the rotation part, I can get the desired rotation angle based on the relationship of each joints’ positions
- lengthForRot = elmoBone[3].position.distanceTo( elmoBone[2].position );
rotForJoint = Math.asin( (elmoBone[3].position.y-elmoBone[2].position.y)/lengthForRot );
elmoBone[3].rotation.y = rotForJoint;
- lengthForRot = elmoBone[3].position.distanceTo( elmoBone[2].position );
And here’s a snippet!
Experiment_03
Control Room
- The first mask I’m going to make is Control Room. The user is virtually in a small room and sees the world through different windows. It creates a feeling of controlling a robot, which is the physical form of self. The distance between self and the world is created, thus decreased the threatening feeling from outside world.
- It’s also a space for your imaginative friends.
- Possible purposes
- Although hiding in control room, the user still hope to interact with others. Inside control room is just a way to deal with the social awkwardness
- Appearance: Townhouse
- Verb
- User
- Lives daily life, sees through different windows (passive v.)
- Interacts with imaginative friends (active v.)
- What kinds of interaction?
- Others
- Have few interactions with the user in general
- Windows and doors invited to be knocked to start conversations –> not like small talk but have certain level of efforts
- User
- Features to have: multi-camera input, prosthetic limbs,
And here’s a video explaining what’s going on.
Touch Event
- Copper tape extension
- For other to trigger sounds, as part of the “An Afternoon Under A Mask” experiment
But then I found out it’s super finicky if I use multiple ones and extend them T_T
An Afternoon Under A Mask
– How does it feel to live daily life under a mask exactly? What’s the reaction of others? What do I hope to happen? I’m going to try to answer those questions on Monday(2/23).
Couldn’t do it because the copper tapes are too finicky..
sound to light analog circuit!
Reference: instructables<3
I. Breadboarding
II. Eagle
Based on the tutorial of instructables, after calculation, with 9V input and yellow LEDs, I use 3 for 4 series, so total 12 LEDs, and the resistor connected to transistor will be 2.2k Ohm, and the one connected to emitter will be 22 Ohm.
The most challenging part is routing on the board. How to have an efficient route map without overlapping? Here I did a dangerous move, having a route under the electret mic, between its positive and negative hole. Don’t know if it’s doable or not.
[ UPDATED ] It’s ok to route under the mic, but should be careful not touching the pad –> adjust the width of the route to be thinner!
WTF Moment_01
Last night I found out that the idea that “Wearing mask = isolating yourself” is totally wrong. If I really want to isolate myself, I can just escape and hide myself in an empty room. On the moment that I put on the mask, I choose to stay in the environment and deal with it. Wearing mask is not isolating myself. It’s a way to interact with others relaxedly.
WTF.
I spent so much time on thinking about examples based on the “mask = isolate” rule, which is wrong.
WTF.
Oh well. At least I figured it out. Yay.
MASK fantasy
The construction of this MASK fantasy is
- Purpose of the mask.
- What does it do?
- What’s the effect of it to user?
- What’s the effect of it to the other?
And the symbol ▲ stands for under the category of SHELTER, and ▼ for IDENTITY SHIFT, and ◎ for THE SELECTED (probably).
Temporary result: ▲*5, ▼*6, ◎*3 (▲*2, ▼*1 )
Here we go!
Block annoying person (ref. tv series Black Mirror) ▲
– Blur face, muffle sound / music volume goes along with the volume of environment noise
– Ignore the person without being noticed, decrease interference
– Able to finish the unaware annoying conversation, maybe irritated because the user is wearing a mask
“I wonder what he/she is doing now?” –> { Duet } ▼
– Geolocation as blinking sphere in virtual world
– Sense of secure, feel the existence of someone under Spatial awareness
– Under the agreement, the participant might feel being watched and surveillance
&
– Change perspective of the other when you want to, shorten the distance
– Less paranoid, less nostalgic, closer relationship
– People will know the user is missing someone
Not be afraid to talk to people/be social –> { Control Room } ▲ ◎
– Virtually in a tiny room, and see world through a window on the wall, like controlling a robot (= your physical body)
– To improve social ability, the wall next to the window having top 5 news of the day for conversation start point
– Can have multiple windows on the wall, and each windows show different views of the world
– The distance between self and the world is created, thus decreased the threatening feeling.
– This is a funny guy. He/she seems know what to say. ???
Be an awesome listener –> { Awesome Listener } ▲ ◎
– Mask speaks for the user with single note yet different tune sound to represent the emotions e.g. “Oh” expresses “Yes”, “No”, “What?”, “Alright”, “Really?”, “Hah”. And the mask is made of mirror
– The user can keep on the conversation nonverbally
– The other has to guess what the user means by listening carefully to the single note sound, and analyzes to comprehend. Might be annoyed because of limited feedback
– Because of the reflection in mirror, and the universal sound as feedback, the other feels like talking to his/her subconscious. The other might have narcissistic moment during the conversations, thus will be attracted by the user
Stop comparing yourself to other people ▼
– Turn everyone “you”, replacing the head image
– Feel like talk to yourself, surprised when “you” say/do something unexpected, exploring the possibilities that you might have those hidden characters as “you”
– The other might has a “murmuring” conversation because the user is talking to “him/herself”
Provide protection from ▲
– Propaganda (of gov policy, commercial)
– Mainstream (fashion e.g. everyone wears black in NYC, brand, music, slang)
– Peer pressure
– For user, easier to maintain who he/she is
– For other, the user seems very unique and weird
Avoid judgement ▲ ◎
– Flash texts or audios to compliment the user, and the mask is made of mirror
– User’s confidence will be boosted up
– The other feels weird to talk to him/herself, might have narcissistic moment during the conversations, thus the other will be attracted by the user
Crave to be liked(as “Like” button of Facebook, opposite to block ppl) –> { Like Me } ▼ ◎
– Buttons on the mask inviting others to press, the user’s vision gets clearer once buttons being pressed, or it’ll gradually blur
– The user has to create conversations and interact with people in order to get clear vision
– The other has incentive to enter the conversation (because of lovely buttons!) and has fun
Story sharing anonymously –> { Anonymous Story } ▼
– With virtual recording button, user upload the recorded vision and audio, just as writing blog posts. It could be a diary, an instruction, or a fiction of daily life.
– The user creates movies about the user self, effortlessly. And since it’s a POV, point of view shot, the user expresses self anonymously
– It’s an unconditional way to comprehend other by literally taking non-self perspective.
Be animalized to avoid having human emotions ▼
– Provide powers of different animals, flying vision with air fan, hear only small sound, see yourself with mirror rig
– The user express him/herself as the animal, freely. No moral burden as a human
– The other will be surprised by the unusual ability of the user. Might be scared. Might ask to use the user’s superpower
Escape from reality –> { Evasive Wander } ▲
– In virtual world, real world is just a standing screen in a forest, and there are other masked creatures wandering around
– User can isolate self completely in a second, and communicate with other masked creatures. Identify self as a group
– The other might feels the existence of two world, and one of which belongs to the masked people. Difference might be widen, yet the other might want to join the virtual world as well
Train to be used to possible future distorted perception ▼
– Providing experience of Alice in Wonderland Syndrome
– Different kind of distortions let user to externalize the perception. Under the assumption that what he/she sees is not real, the user can repress instinct reaction and makes rational judgement.
– The unusual calmness of the user might terrify the other. Or the other might consult the user’s…
Experiment_02
Mask Iteration
– doodle of mask ideas
– 3D models built in Maya
– Pattern unfold by Pepakura
– Prototyping with paper
– Prototyping with fabric
– Mask iteration display –> Mask Gang
From left to right, dirty design based on the different functions:
- Spikes – being inhuman, scary and aggressive
- ChickenCow – ???
- Box – attached to body, so the mask is hard to take off
- Ears – the sound is magnified by funnels sticking out near the ears, thus user becomes more aware of the environment around
Virtual World Construction
– doodle of world: 1) Closed, 2) Open
– Coding coding coding….
Production Schedule
Surface mount soldering O_O!
2/19 update, all LEDs lit up 😀
//
Took so much time!! The issue is that the solder paste got cold so fast, and put it on top of the mug warmer would make it dry fast too! T_T
The circle part is the only LED didn’t light up because I put too less solder paste. Will try it again tomorrow 😀
Research_02
For deeper thesis research, I first looked more into Nonsense, then on the format – Helmet, and discovered some traces and had some thoughts:
- In this loud, information overloaded, rapid speedy world, people might use nonsense as a way to stand out and be noticeable, thus created this nonsense everywhere phenomenon.
- With a contrary character of hiding the face and being conspicuous, helmet might represent the contradiction/dynamic equilibrium between introversion and extroversion.
- Helmets could be utilized to bring out the good old value of nonsense in current society.
Deeper into Nonsense
—Different kinds of nonsense in history. Why did they arise? What are they about? What’s the nonsense now? Nonsense used to be able to shock people, yet now it is everywhere. What’s the impact of it currently compared to in the past? Does nonsense still have value?
{ Past }
Cubism
- 1907, Picasso
- instead of depicting objects from one viewpoint, the artist depicts the subject from a multitude of viewpoints
- unhistorical deductions
Dada
- started around 1910
- thinks the logic and reason of bourgeois caused the WWI, so express the rejection by embracing chaos and illogic
- anti-war, anti-art
- a systematic work of destruction and demoralization
Surrealism
- 1920’s
- resolve the previously contradictory conditions of dream and reality
- pure psychic automatism, expressing the subconscious
- Freud’s work with free association, dream analysis, and the unconscious
- aimed to revolutionize human experience
Fluxus
- 1960’s
- blending different artistic media and disciplines
- anti-commercialism and an anti-art sensibility
{ Now }
Frankly, I can’t define what’s the Nonsense is nowaday. But below are what I found related:
Arty Narcissism
“…pretentious, meaningless, tongue twisting nonsense spoken in some artists’ statements. The culprits are mainly the conceptual artists who need to bamboozle you with an attempt at philosophical genius in order to camouflage the insignificance of the poorly executed end product.“
“The contemporary market has been subject to extreme speculation, but there is a difference between speculation and fraud. The economic downturn will moderate the speculative bubble that has seen some less deserving artists command extreme prices.”
Justwalkingism
“A game that you just walk in. It’s not about winning, it’s not about overcoming the challenges, it’s not even about challenges anymore… Justwalkingism is giving you something to go through and what is important is that you actually go through it. You.”
“Justwalkingism is an art movement, meaning that its artists express themselves on many platforms. It’s like colouring books: the outlines are there but the finished piece is still yours to make.”
Helmet/Mask – Interconnection between Nonsense and Introversion
Introvert vs Extrovert (by Carl Jung)
- Introversion: an attitude-type characterised by orientation in life through subjective psychic contents (focus on one’s inner psychic activity)
- Extraversion: an attitude type characterised by concentration of interest on the external object (the outside world)
Mask
- The existence is the clashing of introversion and extroversion
- Identify/Mindset changing
- A shelter / lense / armor to approach the physical world
Some notes/thoughts after reading The Power of Introverts in a World That Can’t Stop Talking
- We have the ability to be whomever we want –> power of free will.
- Because I consider myself as an introvert, be careful DON’T beautify introversion.
- Solitude can be a catalyst to innovation? What about work in a group?
- If introvert and extrovert do complete each others, a person can be a “complete” one, by shifting between introvert and extrovert?