dp.kinect2

dp.kinect2 is a plugin for the Cycling '74 Max development environment to use the Microsoft Kinect v2 (XBox One) with a Windows computer.

Quick jump

🏃 Get started
📖 Learn

🏃 Get started

Download, trial, license

Get a download, trial, or license from the store.

System requirements

Cycling74 Max requirements
Microsoft Kinect v2 requirements can be verified with their configuration verifier
Windows 8.1 or newer (64-bit). Windows 10 is highly recommended.

Setup

Install Cycling’74 Max version 6.1.9 or newer (64-bit or 32-bit)
Download dp.kinect2 from https://hidale.com/shop/dp-kinect2/
Decompress this download and move the whole dp.kinect2 folder into your Packages folder. The default Max Packages folders are
- Max 6: home-folder/Max/Packages
- Max 7: home-folder/Max 7/Packages
- Max 8: home-folder/Max 8/Packages
Install Microsoft Kinect v2 Runtime.
Install Microsoft Visual C++ Redistributable for VS 2015 for 64-bit and/or 32-bit architecture to match your installed Cycling’74 Max architecture(s).
Plug your Kinect sensor into a USB 3.0 port. Do not plug any other USB devices into the same USB 3.0 controller because this Kinect model requires all the bandwidth of a USB 3.0 controller.
Register dp.kinect2

Optional features

Face tracking: Download the face tracking model and follow the instructions found in your folder Packages\dp.kinect2\support.
Speech recognition: Install Microsoft Speech Platform v11 Runtime. English-US is already installed. More languages are available with Kinect for Windows 2.0 Language Packs.

Register

Evaluate dp.kinect2 with a 14-day no-cost trial. When the plugin meets your needs, please visit https://hidale.com/shop/dp-kinect2/ to purchase a license from the store.
Open the help file or patch included within the dp.kinect2 download.
Click the top-left Register tab.
Optional: Start a 14-day no-cost trial by clicking register trial then skip to the last step.
Type your registration name in the field beside the register button. You received your registration name in an email from the online store. 🧐 Your registration name is usually not your email address.
Click the register button and use the dialog box that appears to select your dp.kinect2 registration key (.dpreg file)
You should see a successful registration. If not, please look at the Max console for any warnings or errors.

Collective/app

Standalone applications can be created with dp.kinect2. All standard features of dp.kinect2 will work. If you use the optional face features, you will need to copy the Kinect20.Face.xxx.dll and the NuiDatabase folder into the support subfolder of your application.

Collectives can also be created and all standard features will work. Max 6 and Max 7 have a limitation; the optional files needed for face tracking can not be directly included within the collective file. I have created a workaround in the plugin codebase. It requires you copy the Kinect20.Face.xxx.dll and the NuiDatabase folder into the same folder as the collective.

Please remember that your license for dp.kinect2 is only for one computer. You or your customers need additional licenses for each computer on which your standalone or collective runs.

Compatibility

dp.kinect2 is very compatible with dp.kinect. There are some known differences.

Kinect v2 matrices are fixed sizes. Changes to resolution attributes are ignored. Resize or crop with jit.matrix or other matrix operators if you need the same sizes as dp.kinect. Kinect v2 fixed sizes are:
- 512x424: depthmap, infrared, playermap, pointcloud
- 1920x1080: color
When the infrared (IR) output is set to the long type with @irtype 1, these values will range from [0..65535]. This is different than the [0..1023] range of dp.kinect. This wider range is due to the increased precision of the Kinect v2 sensor. If you must have the original [0..1023] range, use jit.op, jit.expr, or other matrix operators to scale the values in the matrix output. Please note, even though the Kinect v2 has more precision, the overall IR data is still degraded. See the IR degraded issue.
The face tracking pose scale value is always 1.0 until you enable @facesuau 1 or @face3dmodel. When either is enabled, the scale value will be updated.
The face tracking pose translation coordinate values is always the same as the head skeleton joint until you enable @facesuau or @face3dmodel. When either is enabled, the translation coordinate will be further refined to be a more exact head pivot point to use with 3D face modeling.
Face 2D points compatible with both dp.kinect and dp.kinect2 are @face2dpoints 2. This provides five basic 2D points: left eye, right eye, nose, left mouth corner, and right mouth corner.
Face animation units (AU) on dp.kinect2 are different than dp.kinect. Only @facesuau 2 is supported.
The @soundinfo attribute is only updated when dp.kinect2 is banged. This is different than dp.kinect which constantly updated the attribute.
The @sync attribute is not available in dp.kinect2 since all frames are synchronized by Microsoft. See the synchronization issue.
Microsoft does not support control of the IR emitter, color camera processing, or audio processing. Therefore, the following parameters are not present on dp.kinect2: iremitter, autoexposure, autowhitebalance, backlight, brightness, contrast, exposure, frameinterval, gain, gamma, hue, powerfreq, saturation, sharpness, whitebalance, autogain, echocancel.
Kinect v2 sensor works in all seated, near, and far situations while also having a 0.5-8.0 meter range. Therefore, these parameters are not available on dp.kinect2: @seated, @nearmode, @depthrange.
Kinect v2 does not have a tilt motor within itself. Therefore, the @elevation attribute only outputs the basic elevation relative to the horizon.
The following esoteric parameters are not available on dp.kinect2: @gravsmooth, @facerate, and @sensorrate.
The dumpout messages newkinect and lostkinect are not implemented. Microsoft’s API and drivers are not reliable for detecting plugged/unplugged sensors.

Known issues

Microsoft has isolated a performance problem that might occur with the Kinect v2 on Intel CPUs. Microsoft believes it is isolated to Kinect software that uses eventing to get Kinect data. dp.kinect2 does not use eventing to process core Kinect data. Therefore, I do not think dp.kinect2 is affected. I asked Microsoft for more information but have not received any to date. Microsoft does provide a workaround in this forum post.
With stress testing using a repeating Kinect Studio clip, a bug in the Microsoft Kinect v2 runtime was found when HD face tracking features are used. A private thread in their runtime accesses memory which is unavailable causing a crash in the runtime. The issue was escalated to Microsoft and they reproduced it. They have no fix. It is possible that the crash will not be seen in real-world usage. I recommend solid testing of any solutions that use HD face tracking to ensure stability.
All frames are synced by the Microsoft drivers. Out of sync or delayed frames are likely due to Microsoft limitations. Microsoft made a design decision to keep frames in sync even if it must drop the framerate to 15fps for the color camera to have enough light and therefore penalize the depth/body/ir/other frames. There is no workaround and therefore the @sync parameter is not available on dp.kinect2.
Microsoft has intentionally degraded the IR data. Intense IR regions do not linearly increase in v2 Runtime builds 1407 and higher including the final release. The newer builds oversaturate, use half the intensity bit range, and the data is poor. A bug has been logged with Microsoft in private forum and public forum. Microsoft acknowledges this is their issue and they will not fix it. The problem does not exist in 1406 or earlier releases of the Kinect v2 Runtime. There is no workaround.
Microsoft’s driver for the Kinect v2 does not dependably report the existence or absence of a Kinect. This is a known Microsoft issue reported in the private forum and public forum. Therefore, you can open a Kinect even when one isn’t plugged in. Or the driver might report there is no Kinect even though there is one plugged in. The result is sometimes the “open” will fail. Or you will get a successful open yet get repeatedly the last known data (which could be zeros). Set @unique 1 to prevent this errant data.

📖 Learn

Use search 🔍 at the top or all below to find tutorials, features, attributes, methods, and more.

Tutorials

Skeleton Joint Changes Color: Learn how to get the real-world XYZ position of your left hand relative ...
View Sensor Depthmap: Learn how to view depth data (depthmap) as a monochrome image. A depthma...

Attributes

@accel: Acceleration XYZ vector usually pointing towards gravity (where 1 unit =...
@align: Align data to the perspective of the depth or color camera. This affects...
@anim: Parent jit.anim.node name adds plugin to jit.anim.node hierarchy.
@bodyprop: Skeleton body properties output: restricted, handstate, lean, and the co...
@colorfocal: Color camera single nominal focal length in pixels. Read-only.
@colorfov: Color camera field-of-view (degrees) in horizontal, vertical, diagonal d...
@colormap: Color image output in choice of pixel format on the second outlet.
@colormapres: Resolution of color image that was enabled with @colormap
@colortype: Color component in 8-bit unsigned integer or 32-bit normalized float. Th...
@depthfocal: Depth camera single nominal focal length in pixels. Read-only.
@depthfov: Depth camera field-of-view (degrees) in horizontal, vertical, diagonal d...
@depthintrinsics: Depth camera intrinsics for dp.kinect2 as a list of seven floats. Read-o...
@depthmap: Depth output on the first outlet. This combined with resolution @depthma...
@depthmapres: Resolution of depth image that was enabled with @depthmap.
@depthonlyplayers: Filter source by tracked players. Deprecated. Use @onlyplayers.
@depthvis: Depth visibility filter removes double images and edge artifacts in imag...
@distmeter: Distance output in meters 1 or millimeters 0. This affects all distance ...
@elevation: Elevation angle (degrees) of sensor relative to the horizon; a simple X-...
@face2dpoints: Face 2D key points in color camera uv coordinates output on the fifth ou...
@face3dmodel: Face 3D model output as a mesh in local face coordinate-space. The mesh ...
@facecolors: Face and hair color for each tracked face. Output on the fifth outlet. @...
@facedll: Version numbers for face tracking DLL. Read only. Deprecated.
@faceprop: Face properties for the appearance or state of a tracked face. Output on...
@faces: Face tracking calculation and messages with 2D bounding box, rotation, t...
@facesuau: Face shape unit (SU) and animation unit (AU) output on the fifth outlet....
@facesuausmooth: Smoothing for face animation units (AU) with the same five parameter Hol...
@flipx: Flip or negate X-axis values of output (matrix images, joint positions, ...
@floor: Floor plane as x, y, z coordinates for a point on the plane and i, j, k ...
@flooronbang: Floor plane message output on each bang. More efficient than querying th...
@grammarfile: Path to the speech recognition grammar file. It is a GRXML file as defin...
@gravrot: Rotation relative to gravity message; angle (degrees) around a vector ou...
@hardware: Identify/select sensor hardware model. It is read-only on some plugins. ...
@idkinect: Bind specific sensor to the plugin. Deprecated. Use @idsensor.
@imageuntwist: Rotation (degrees) to align depth, color, etc. images to gravity. Top pi...
@irgamma: Apply gamma correction to infrared image enabled with @irmap. Use to bri...
@irmap: Infrared (IR) image output on the third outlet. This, resolution @irmapr...
@irmapres: Resolution of infrared image enabled with @irmap. Read-only. Controlled ...
@irtype: Infrared (IR) values in 8-bit unsigned integer, long signed integer, or ...
@kinectdll: Version numbers for main Kinect DLL. Read only. Deprecated.
@onlyplayers: Filter source by tracked players. Filtered data is zero-like. Possible v...
@orientformat: Skeleton joint orientation as quaternion, 4x4 matrix, hierarchical, or a...
@playermap: Player indices for each pixel output on the fourth outlet. Pixels are a ...
@pointcloud: Format of pointcloud output as three or four floating point values.
@posconfidence: Filter skeleton body, joints, and face by their confidence. This confide...
@position: Offset the origin of sensor data to this XYZ value. That is, translate s...
@quat: Rotate data by a quaternion in the form: i j k w. This is always synchro...
@register: Register a license or get the status of registration.
@rotate: Rotate data by an angle (degrees) around an XYZ vector in the form: angl...
@rotatemethod: Rotate data to compensate for gravity. This compensation is combined wit...
@rotatexyz: Rotate data by Euler angles (degrees) around the positive x, y, and z ax...
@scale: Scale the sensor data by this XYZ value. That is, multiply sensor data a...
@silenceprecise: Milliseconds of silence needed before completing a recognition that is n...
@silencevague: Milliseconds of silence needed before completing a recognition that is v...
@skelcolor: Skeleton joint output from the fifth outlet in colormap coordinates. Thi...
@skeldepth: Skeleton joint output from the fifth outlet in depthmap coordinates. Thi...
@skeleton: Skeleton, joint, and user tracking output on the fifth outlet. Output fo...
@skeletonformat: Message format for skeleton joints, users, faces, etc. Native Max messag...
@smoothing: Smoothing for skeleton joint values with either a simplified one paramet...
@soundinfo: Real-time sound loudness, position angle, and confidence float values ar...
@soundinfooutput: Real-time sound loudness, position, and confidence message output on the...
@speech: Speech recognition and speech interference output on the fifth outlet. I...
@speechadapt: Constantly adapt speech recognition model to adjust for various speaker(...
@speechconf: Filter speech based on recognition confidence. Speech with the chosen co...
@speechdll: Version numbers for Speech DLL. Read only. Deprecated.
@speechlocale: Speech recognition language and territory. US English en-us language sup...
@type: Depthmap value in signed integer, 32-bit float, or 64-bit float. This ty...
@unique: Unique-only matrices and messages; bang may not output when no changes. ...
@verbose: Verbose warnings and errors output to the Max Window for troubleshooting...
@version: Version numbers for this plugin. Read only. Deprecated.

Methods

bang: Query the plugin for updated data. Then, output matrices and messages fr...
close: Close and teardown the currently open connection to a sensor.
depthtocolor: Transform depth pixel UVZ to color UV coordinates where U is column, V i...
depthtoskel: Transform depth pixel UVZ to real-world XYZ coordinates where U is colum...
getusbidlist: Output supported sensor identifiers on the dumpout outlet. These identif...
open: Open and initialize the connection to a sensor. When @idsensor has not b...
skeltocolor: Transform real-world XYZ coordinates to color UV coordinates where U is ...
skeltodepth: Transform real-world XYZ coordinates to depth UVZ coordinates where U is...
unregister: Unregister and remove a license.