freenect2¶

The freenect2 module provides a Python interface to the libfreenect2 library. The libfreenect2 library provides a library allowing depth and RGB data to be extracted from a Kinect for Windows v2 (K4W2) device.

Installation¶

libfreenect2 must be installed prior to building this module. See the installation instructions provided by libfreenect2. The following is a brief summary of the required actions:

$ git clone https://github.com/OpenKinect/libfreenect2
...
$ cd libfreenect2; mkdir build; cd build
$ cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local .. && make all install
...
$ export PKG_CONFIG_PATH=$HOME/.local/lib/pkgconfig
$ pip install --user freenect2

pkg-config errors¶

The libfreenect2 library uses pkg-config to record where it is installed on the system. If you install libfreenect2 manually as outlined above you will need to set the PKG_CONFIG_PATH environment variable.

Simple usage¶

The library is intended to provide an easy to use Pythonic interface to libfreenect2. As an example of library usage, here is a simple infrared (IR) camera which can be used to capture an image from the Kinect’s build in IR camera:

"""
Simple IR camera using freenect2. Saves captured IR image
to output.jpg.

"""
# Import parts of freenect2 we're going to use
from freenect2 import Device, FrameType

# We use numpy to process the raw IR frame
import numpy as np

# We use the Pillow library for saving the captured image
from PIL import Image

# Open default device
device = Device()

# Start the device
with device.running():
    # For each received frame...
    for type_, frame in device:
        # ...stop only when we get an IR frame
        if type_ is FrameType.Ir:
            break

# Outside of the 'with' block, the device has been stopped again

# The received IR frame is in the range 0 -> 65535. Normalise the
# range to 0 -> 1 and take square root as a simple form of gamma
# correction.
ir_image = frame.to_array()
ir_image /= ir_image.max()
ir_image = np.sqrt(ir_image)

# Use Pillow to save the IR image.
Image.fromarray(256 * ir_image).convert('L').save('output.jpg')

The library also supports extracting real-world 3D co-ordinates from the depth maps and saving libpcl-compatible PCD files:

"""
Capture a single RGB and depth frame and save them to output.pcd in
the libpcl PCD format. View the resulting cloud with:

    pcl_viewer output.pcd

"""
from freenect2 import Device, FrameType
import numpy as np

# Open the default device and capture a color and depth frame.
device = Device()
frames = {}
with device.running():
    for type_, frame in device:
        frames[type_] = frame
        if FrameType.Color in frames and FrameType.Depth in frames:
            break

# Use the factory calibration to undistort the depth frame and register the RGB
# frame onto it.
rgb, depth = frames[FrameType.Color], frames[FrameType.Depth]
undistorted, registered, big_depth = device.registration.apply(
    rgb, depth, with_big_depth=True)

# Combine the depth and RGB data together into a single point cloud.
with open('output.pcd', 'wb') as fobj:
    device.registration.write_pcd(fobj, undistorted, registered)

with open('output_big.pcd', 'wb') as fobj:
   device.registration.write_big_pcd(fobj, big_depth, rgb)

Reference¶

class freenect2.ColorCameraParams¶

Color camera intrinsic calibration.

fx¶: Focal length for x-axis (pixels)

fy¶: Focal length for y-axis (pixels)

cx¶: Principal point for x-axis (pixels)

cy¶: Principal point for y-axis (pixels)

class freenect2.Device(c_object=None)¶

Control a single device.

If called with no arguments, the default device is opened.

Raises:: .NoDeviceError –

color_camera_params¶: (ColorCameraParams or None) A structure describing the RGB camera factory calibration. Before the start() is called, this is None since the device only reports calibration when capture begins.

ir_camera_params¶: (IrCameraParams or None) A structure describing the IR camera factory calibration. Before the start() is called, this is None since the device only reports calibration when capture begins.

close()¶: Close the device and free any associated resources.

property color_frame_listener¶: A callable called whenever a new color frame arrives from the device. The callable should take two positional arguments, the frame type (an instance of FrameType) and the frame itself (an instance of Frame).

get_next_frame(timeout=None) → tuple[FrameType, Frame]¶

Get the next frame type and frame from the device.

Parameters:: timeout (number or None) – If not-None, a positive number of seconds to wait for a frame before raising a NoFrameReceivedError exception.
Returns:: A FrameType, Frame tuple representing the type of received frame and the frame itself.

Note

This method only works if the default listener is being used. That is to say that color_frame_listener and ir_and_depth_frame_listener have not been changed from their default values.

property ir_and_depth_frame_listener¶: A callable called whenever a new IR or depth frame arrives from the device. The callable should take two positional arguments, the frame type (an instance of FrameType) and the frame itself (an instance of Frame).

property registration¶: An instance of Registration which can be used to undistort the raw depth image and register the RGB image with it.

running(*args, **kwargs)¶

A context manager which can be used to ensure that the device’s streams are stopped. Any arguments are passed to start().

from freenect2 import Device

# Open default device
device = Device()

# Start depth and color frames
with device.running():
    # ... frame listener callbacks will be called ...

# Device is now stopped

set_color_auto_exposure(exposure_compensation: float = 0)¶

Sets the RGB camera to fully automatic exposure setting. Exposure compensation: negative value gives an underexposed image, positive gives an overexposed image.

Parameters:: exposure_compensation (float) – Exposure compensation, range [-2.0, 2.0]

set_color_manual_exposure(integration_time_ms: float, analog_gain: float)¶

Manually set true exposure time and analog gain of the RGB camera.

Parameters:

integration_time_ms (float) – True shutter time in milliseconds, range (0.0, 66.0]
analog_gain (float) – Analog gain, range [1.0, 4.0]

set_color_semi_auto_exposure(pseudo_exposure_time_ms: float)¶

Sets a flicker-free exposure time of the RGB camera in pseudo-ms, value in range [0.0, 640] ms. The actual frame integration time is set to a multiple of fluorescent light period that is shorter than the requested time e.g. requesting 16 ms will set 10 ms in Australia (100Hz light flicker), 8.33 ms in USA (120Hz light flicker). The gain is automatically set to compensate for the reduced integration time, as if the gain was set to 1.0 and the integration time was the requested value.

Requesting less than a single fluorescent light period will set the integration time to the requested value, so the image brightness will flicker.

To set the shortest non-flickering integration period for any country, simply set a pseudo-exposure time of between (10.0, 16.667) ms, which will automatically drop the integration time to 10 or 8.3 ms depending on country, while setting the analog gain control to a brighter value.

Parameters:: pseudo_exposure_time_ms (float) – Pseudo-exposure time in milliseconds, range (0.0, 66.0+]

start(frame_listener=None)¶

Start depth, IR and RGB streams.

Parameters:: frame_listener (callable or None) – if not-None, this is a callable which is assigned to both color_frame_listener and ir_and_depth_frame_listener before the device is started.

stop()¶: Stop any running streams.

class freenect2.Frame(frame_ref)¶

A single frame received from the device.

These should not be constructed directly since they are usually created by the freenect2 library itself. However you may need to construct “blank” frames for use with Registration. In which case, you should use the Frame.create() class method.

property bytes_per_pixel: int¶: Number of bytes in a pixel. If format is FrameFormat.Raw, this is the buffer size.

classmethod create(width, height, bytes_per_pixel)¶: Create a blank frame with the specified width, height and bytes per pixel. Memory for the frame is automatically allocated. No other attributes are initialised.

property data¶: A buffer object pointing to the raw memory contents of the frame.

property exposure: float¶: From 0.5 (very bright) to ~60.0 (fully covered)

property format: FrameFormat¶: Byte format. Informative only, doesn’t indicate errors. An instance of FrameFormat.

property gain: float¶: From 1.0 (bright) to 1.5 (covered)

property gamma: float¶: From 1.0 (bright) to 6.4 (covered)

property height: int¶: Number of lines in the frame

property sequence: int¶: Increasing frame sequence number

property status: int¶: zero if ok; non-zero for errors

property timestamp: int¶

roughly or exactly 0.1 millisecond

Type:: Unit

to_array()¶

Convert the image to a numpy array instance.

The memory is not copied so be careful performing any operations which modify the contents of the frame.

to_image()¶: Convert the Frame to a PIL Image instance.

property width: int¶: Length of a line (in pixels)

class freenect2.FrameFormat(value, names=None, *values, module=None, qualname=None, type=None, start=1, boundary=None)¶

Pixel format.

BGRX = 4¶: 4 bytes of B, G, R, and unused per pixel

Float = 2¶: A 4-byte float per pixel

Gray = 6¶: 1 byte of gray per pixel

Invalid = 0¶: Invalid format.

RGBX = 5¶: 4 bytes of R, G, B, and unused per pixel

Raw = 1¶: Raw bitstream. ‘bytes_per_pixel’ defines the number of bytes

class freenect2.FrameType(value, names=None, *values, module=None, qualname=None, type=None, start=1, boundary=None)¶

Available types of frames.

Color = 1¶: 1920x1080. BGRX or RGBX

Depth = 4¶

millimeter. Non-positive, NaN, and infinity are invalid or missing data

Type:: 512x424 float, unit

Ir = 2¶: 512x424 float. Range is [0.0, 65535.0]

class freenect2.IrCameraParams¶

IR/depth camera intrinsic calibration.

fx¶: Focal length for x-axis (pixels)

fy¶: Focal length for y-axis (pixels)

cx¶: Principal point for x-axis (pixels)

cy¶: Principal point for y-axis (pixels)

k1¶: Radial distortion co-efficient, 1st-order

k2¶: Radial distortion co-efficient, 2nd-order

k3¶: Radial distortion co-efficient, 3rd-order

p1¶: Tangential distortion co-efficient

p2¶: Tangential distortion co-efficient

exception freenect2.NoDeviceError¶: Raised by Device when there is no default device to open.

exception freenect2.NoFrameReceivedError¶: With the default frame listener this is raised when no frame has been received from the device within a set time.

class freenect2.Registration(depth_p: Frame, rgb_p: Frame)¶

Information required to undistort raw depth frames and register RGB frames onto depth.

Do not construct this directly. Instead use the Device.registration attribute.

apply(rgb, depth, enable_filter=True, with_big_depth=False)¶

Take an RGB and Depth image and return tuple with the undistorted depth image and color image rectified onto depth.

Parameters:

rgb (Frame) – RGB frame received from device
depth (Frame) – Depth frame received from device
enable_filter (bool) – If true, filter out pixels not visible in both cameras.
with_big_depth (bool) – If true, also return a 1920x1082 mapping of depth onto the color map. The top and bottom rows are blank.

Returns:

A Frame pair representing the undistorted depth and registered RGB frames.

get_big_points_xyz_array(big_depth) → ndarray[float32]¶

Like get_points_xyz_array() but operates on the “big” depth map which can be returned from apply().

Parameters:: big_depth (Frame) – big 1920x1082 frame returned from apply().
Returns:: A 1082x1920x3 array of 3D points. The last dimension corresponding to x, y and z.

get_points_xyz(undistorted, rows, cols)¶

Retrieve real-world co-ordinates corresponding to points in the undistorted depth image. Units are millimetres.

Parameters:

undistorted (Frame) – the undistorted depth frame
rows (numpy array) – integer row indices of points in the depth frame
cols (numpy array) – integer column indices of points in the depth frame. Must be the same shape as rows.

Returns:

A three element tuple containing numpy arrays for the x-, y- and z-co-ordinates of the points. Each array has the same shape as rows.

get_points_xyz_array(undistorted) → ndarray[float32]¶

Return a 3D array of x, y, z points for each point in an undistorted frame. Invalid points are Nan-ed.

Parameters:: undistorted (Frame) – the undistorted depth frame
Returns:: A 424x512x3 array of 3D points. The last dimension corresponding to x, y and z.

write_big_pcd(file_object, big_depth, rgb=None)¶

Write depth map and (optionally) RGB data to libpcl-compatible PCD format file. Works like write_pcd() except that it works on the “big” depth map which can be returned from apply(). If the RGB frame is present, each point is coloured according to the image otherwise the points are left uncoloured.

Note

Under Python 3 the file object must be opened in binary mode.

Parameters:

file_object (file) – A file object to write PCD data to
big_depth (Frame) – the 1920x1082 de[th frame
registered (Frame) – if not-None, the RGB data from the color camera

write_pcd(file_object, undistorted, registered=None)¶

Write depth map and (optionally) RGB data to libpcl-compatible PCD format file. If the registered RGB frame is present, each point is coloured according to the image otherwise the points are left uncoloured.

Note

Under Python 3 the file object must be opened in binary mode.

Parameters:

file_object (file) – A file object to write PCD data to
undistorted (Frame) – the undistorted depth frame
registered (Frame) – if not-None, the RGB data corresponding to the depth frame.

freenect2¶

Installation¶

pkg-config errors¶

Simple usage¶

Reference¶

freenect2

Navigation

Related Topics