Index

A

Audio Capture

acoustic echo cancelation, 240–241

noise suppression, 238–240

sound stream

bindable properties, 235

instantiating and configuring, 236

Record and Playback buttons, 235

Record and Playback methods, 237

RecorderHelper class, 232

Recorder window, 231

Tracking Recording State, 234–235

writing to Wav file, 232–233

B

Beam tracking

BeamChanged event handler, 244

Indicator, 241

MainWindow class implementation, 243

Speech direction indicator, 242

C

Coding4Fun Kinect Toolkit, 256–258

browsable source code, 257

extension methods, 257–258

AddOne method, 257

bitmap methods, 262–263

bitmapsource methods, 263

conversion methods, 264–265

ImageExtensions class, 260–262

in MainWindow code behind, 264

pixel formats, 263

WPF project, 259–260

WinForms, 256

D

Depth image processing, 49

application, 65–66

gray shades

application output, 57

coloring, 59, 60

comparation, 61

light shade, 57

new StackPanel and Image element, 56

new version creation, 58–59

visualizations, 59

histogram

building, 62–63

MainWindow.xaml updation, 62

output image, 64

statistical distributions, 62

thresholding, 65

wall filtered out and holding newspaper front of image, 64–65

image alignment

background subtraction process, 79–80

green screening, 76

mapping method, 80

noisy pixels, 80

polling infrastructure, 77–79

stereovision, 76

XAML, 76

measurement

bit manipulation, 53

CalculatePlayerSize method, 71–73

depth bits layout, 52

different poses, 75

display depth values, 53

field of view, 51

hard-code, 54

ItemsControl, 71

mouse-up event handler, 53–54

output display, 55

output properties, 75

Depth image processing, measurement (cont.)

PlayerDepthData class, 73–75

player real world width, 70

player width and height, 69

UI, 70, 71

near mode, 81–82

player indexing

displaying in black and white, 67–68

index bits, 67

KinectExplorer, 69

raw and processed images, 67, 69

raw depth image frame, 51

steps, 49

thresholding, 61

E

Emgu CV, 272

F

Facial detection, 279

algorithm, 280

augmented reality implementation, 282

FaceFinder, 279

Pulse method, 280

setup code, 280

skeletal tracking, 281

Flexible Action and Articulated Skeleton Toolkit (FAAST), 288

G

Gestures, 167

affordance and feedback, 173–174

arbitrary and conventional, 169

aspects, 169

conventions, 173

definition, 167

in arts, 167

central distinction, 168

challenges, 168

dictionary approach, 167

in Human-Computer Communication, 168

in human-computer-interaction, 169

future aspects, 220–221

gestural idioms, 173

gesture detection

algorithms, 175

exemplar approach, 176

hand tracking (see Hand tracking, gestures)

hover button, 200–202

magnet button, 204–210

magnetic slide, 214–217

neural networks, 175–176

push button, 203–204

swipe, 210–214

universal pause, 219

vertical scroll, 217–219

wave gesture, 177–183

interaction idioms, 172

limitations, 221

Natural user interface (NUI), 170

buttons, 171

characteristics, 171

direct manipulation, 170

gestural interfaces, 171

goals, 170

posing, 171

speech interfaces, 171

touch interfaces, 171

in restaurant, 169

tasks, 167

H

Hand tracking, gestures

CursorAdorner class

Adorner Base Class method, 188–189

cursor animations, 190–191

passing coordinate positions, 189–190

visual element, 187–188

KinectButton

Base Class, 199

Base Implementation, 197–199

click event, 200

KinectCursorEventArgs

constructor overloads, 184–185

structure, 184

KinectCursorManager

constructors, 193–195

event management, 195–196

helper methods, 192–193

MapSkeletonPointToDepth method, 197

SkeletonFrameReady method, 196

UpdateCursor method, 196

KinectInput, event declaration, 185–187

Kinect to WPF data translation, 197–199

Holograms, 283

Blender, 284

configuration, 286

3D cube, 284

3D oblongs, 287

moving the camera around cude, 286

Viewport3D object, 284

I, J

Image Manipulation helper methods, 256

K

Kinect, 1

applications, 16

Explorer, 17–18

hardware

components, 10

glossy black case, 9

microphone array, 10

power source, 10

requirements, 11

zoom, 10

image processing (see Depth image processing)

installation

acoustic models, 13

drivers, 13

instructions, 12

microphone array, 13

OpenNI, 12

Microsoft Research (MSR)

microphone array, 7

motion-tracking, 5

player blob, 6

player parts, 6

Minority Report, 2

Natal device, 4

Record Audio, 20

Shape Game, 18–19

skeleton track (see Skeleton tracking)

software requirements, 11–12

speech sample, 20–21

time of flight technique, 4

vision recognition, 4

Visual Studio Project

add reference, 14

applications, 14

basic steps, 14

Kinect Depth Stream Data, 16

KinectSensor object, 15

Wii Remote, 3

Kinect3DV, 3

Kinect Math, 291

bit fields, 292

bit manipulation, 291

bit shifting, 296

bitwise AND operator, 293

bitwise NOT (complement) operator, 295

bitwise OR operator, 292

geometry, 297

trigonometry

degree and radian conversions, 298

distance between two points, 297

Law of Cosines, 299

for right triangles, 298

unit circle, 298

unit of measures, 291

Kinect sensor, 23

collection object, 24

color image stream

display, 30

enable method, 29

frame-ready event handler, 30

video camera, 28

data streams, 24

detection and monitoring, 25–27

hardware, 24

image manipulation

Kinect_ColorFrameReady event handler, 32

pixel shading, 33–34

image performance

bitmap image creation, 31

frame image, 31

image pixel updation, 32

memory allocation and deallocation, 32

loaded and unloaded events, 27

object reflection

class diagram, 36, 37

color image stream formats, 38

ImageStream class, 38

Timestamp, 39

polling application

advantages, 46

base code, 40

discover and initialization, 41, 42

OpenNextFrame method, 39

PollColorImageStream method, 42

rendering event, 41

Kinect sensor, polling application (cont.)

thread, 43–44

UI thread Updation, 45–46

snapshots

add button, 35

TakePictureButton_Click event handler, 35

test application, 36

Xbox Kinect games, 34

starting process, 28

StatusChanged event handler, 27

status values, 24

stopping process, 28

thread safety and release resources, 28

wrapper, 27

Kinect the Dots game

expansion, 112–113

feature set, 100

hand tracking

application output, 105

cursor position updation, 104

location and movements, 102

primary hand, 103, 104

SkeletonFrameReady Event Handler, 102–103

puzzle drawing, 106–108

solving puzzle, 108–112

user interface

image, 101

Viewbox and Grid elements, 101

XAML, 101

L

Libfreenect, 12

M, N

Microphone array, 223, 224

KinectAudioSource

Acoustic Echo Cancellation (AEC), 225

Acoustic Echo Suppression (AES), 225

Automatic Gain Control (AGC), 225

Beam Angle Mode, 227

BeamForming, 225

Center Clipping, 225

Echo Cancellation Mode, 226

feature properties, 226

Frame Size, 225

Gain Bounding, 225

Noise Filling, 225

Noise Suppression (NS), 225

Optibeam, 225

Single Channel, 225

Signal-to-Noise Ratio (SNR), 225

speech recognition (see also Speech recognition)

command recognition, 227

Engine object configuration, 230

free-form dictation, 227

grammars, 228–229

overloaded methods, 230

result properties, 229

SetInputToAudioStream method, 230

Voice Capture DirectX Medio Object (DMO), 224

Microsoft Kinect Audio Array Control, 12

Microsoft Kinect Camera, 12

Microsoft Kinect Device, 12

Microsoft Robotics Developer Studio, 288

O

OpenCV (Open Computer Vision), 272

P, Q, R

Proximity Detection, 265

with depth data, 269–270

_isTracking, 266

motion detection

algorithm, 275–277

configuration, 275

Emgu CV, 272

Emgu extension methods, 273–274

OpenCV, 272

polling technique, 274

strategies, 272

with player data and depth data, 271

saving the video

Record and StopRecording methods, 279

Recording Video, 278

SkeletonFrameReady event, 267

Skeleton tracking, 85, 121

application displays, 90, 91

array of brushes, 87

coordinate space, 113, 114

depth-based user interaction

hand in different position, 151

hand in same position, 151, 153

hand tracking, 150–151

layout system, 146

visual elements, 149

XAML updation, 147–149

drawing skeleton joints, 89

hit testing

Button, 126

Canvas panel, 128

definition, 125

dot proximity, 125

Grid and StackPanel, 128

hand cursors, 129

InputHitTest method, 127

layout space and bounding boxes, 127, 128

Natural User Interface design, 129

position points, 129

ShapeGame application, 125, 126

visual element layering, 126, 127

initialization, 86–87

Kinect the Dots

feature set, 100

game expansion, 112–113

hand tracking, 102–105

puzzle drawing, 106–108

solving puzzle, 108–112

user interface, 101–102

KinectSensor object, 85

mirrored effect, 115

object model

class, 96

ClippedEdges field, 98

enable and disable methods, 93

frame descriptors, 96

FrameNumber and Timestamp fields, 95–96

identifier, 97

joint object, 98–100

position field, 97

schematic representation, 91, 92

selections of, 95

SkeletonFrame objects, 95

SkeletonStream, 93

smoothing, 93–94

TrackingState values, 97

poses

execution, 157

gesture, 153

hit testing, 154

joint position calculation, 156

joint triangulation method, 156

Kinect, 154

Law of Cosines, 155

T pose, 155

type and complexity, 154

umpires, 153

S

Simon Says game

application output, 139

build infrastructure, 133–135

ChangePhase method, 139–140

commands, 140–142

enhancements, 145, 163

indicators, 145

instruction sequence, 129, 146

interactive components, 131

joint position calculation, 161

output, 164, 165

play infrastructure, 135–136

PoseAngle class, 158

pose detection, 157

pose library creation, 159

presentation, 146

ProcessGameOver updation, 160–161

processing player movements, 142–143

serialization, 163

SkeletonFrameReady Event Handler, 136–137

starting game, 137–139

timer initialization, 162

UI components, 131

UIElement object, 144

user experience, 145

user interaction, 146

user interface, 130

user movement detection, 144–145

XAML, 131–133

SkeletonViewer user control

dependency property, 116–117

drawing joints, 117–119

initialization, 119

Kinect the Dots, 119, 120

XAML, 116

space transformations, 114

stick figure creation, 87–89

user interaction

detection, 124–125

graphical user interface, 122

stick figure creation, user interaction (cont.)

touch/stylus devices, 122

WPF input system, 122–124

Speech, 223

audio capture (see also Audio Capture)

acoustic echo cancelation, 240–241

noise suppression, 238–240

sound stream, 231–238

directional microphone (see Beam tracking)

microphone array (see Microphone array)

Speech recognition

CreateAudioSource method, 247

Crosshairs user control, 245

event handlers, 251

GrammarBuilder, 252

Hand Tracking, 249

InterpretCommands method, 252

Kinect Sensor and SpeechRecognitionEngine, 248

LaunchAsMTA method, 247

MainWindow class implementation, 246

MainWindow XAML, 246

Put That There application, 245

StartSpeechRecognition method, 250

Stylus device, 122

T

Touch device, 122

U

Unity3D, 288

W, X, Y, Z

Wave gesture

constants, 179

Detection Class, 180

detection methodology, 177

helper methods, 181–183

neutral zone, 178

TrackWave method, 180–181

user waving, 177

WaveGestureState, 178

WaveGestureTracker, 178

WavePosition, 178

WPF input system

API, 122

component, 123

controls, 124

InputManager object, 123

joint positions, 124

single pixel point location, 124

touch input, 123