**Name String**

`XR_ANDROID_trackables_object`

**Extension Type**

Instance extension

**Registered Extension Number**

467

**Revision**

1

**Extension and Version Dependencies**

[`XR_ANDROID_trackables`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)

**Last Modified Date**

2024-11-01

**IP Status**

No known IP claims.

**Contributors**

Diego Tipaldi, Google

David Joseph Tan, Google

Christopher Doer, Google

Spencer Quin, Google

Jared Finder, Google

Levana Chen, Google

Kenny Vercaemer, Google

## Overview

This extension enables physical object tracking. For example, keyboards, mice,
and other objects in the environment.
| **Caution:**   
|
| **Permissions**   
|
| Android applications **must** have the `android.permission.SCENE_UNDERSTANDING_COARSE` permission listed in their manifest as this extension depends on [`XR_ANDROID_trackables`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables) and exposes the geometry of the environment. The `android.permission.SCENE_UNDERSTANDING_COARSE` permission is considered a dangerous permission.  
| (protection level: dangerous)

## Track objects

This extension adds `XR_TRACKABLE_TYPE_OBJECT_ANDROID` to
[`XrTrackableTypeANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTypeANDROID).

The application **may** create an [`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID) by calling
[`xrCreateTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#xrCreateTrackableTrackerANDROID) and specifying
`XR_TRACKABLE_TYPE_OBJECT_ANDROID` as the trackable type in
[`XrTrackableTrackerCreateInfoANDROID::trackableType`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerCreateInfoANDROID) to track objects.

The
[`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid)
structure is defined as:  

    typedef struct XrTrackableObjectConfigurationANDROID {
        XrStructureType                type;
        void*                          next;
        uint32_t                       labelCount;
        const XrObjectLabelANDROID*    activeLabels;
    } XrTrackableObjectConfigurationANDROID;

### Member Descriptions

- `type` is the [`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType) of this structure.
- `next` is `NULL` or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
- `labelCount` is the count of `activeLabels`.
- `activeLabels` is a pointer to the array of [`XRObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid) indicating the active objects in tracking.

The application **may** set an additional configuration by adding a
[`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid) to the next chain of
[`XrTrackableTrackerCreateInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerCreateInfoANDROID). The output from
[`xrGetAllTrackablesANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#xrGetAllTrackablesANDROID) will be filtered to match the `activeLabels`.

If the application does not set [`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid),
then all objects that the system has identified will be tracked.

### Valid usage (implicit)

- The [`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables) extension **must** be enabled prior to using [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- `type` **must** be `XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `next` **must** be `NULL` or a valid pointer to the [next structure in a
  structure chain](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-valid-usage-for-structure-pointer-chains)
- `activeLabels` **must** be a pointer to an array of `labelCount` valid [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid) values
- The `labelCount` parameter **must** be greater than `0`

The [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid) enum is a label for a [`XrTrackableANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableANDROID)
object.  

    typedef enum XrObjectLabelANDROID {
        XR_OBJECT_LABEL_UNKNOWN_ANDROID = 0,
        XR_OBJECT_LABEL_KEYBOARD_ANDROID = 1,
        XR_OBJECT_LABEL_MOUSE_ANDROID = 2,
        XR_OBJECT_LABEL_LAPTOP_ANDROID = 3,
        XR_OBJECT_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
    } XrObjectLabelANDROID;

## Get trackable object

The [`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid) function is defined as:  

    XrResult xrGetTrackableObjectANDROID(
        XrTrackableTrackerANDROID                   tracker,
        const XrTrackableGetInfoANDROID*            getInfo,
        XrTrackableObjectANDROID*                   objectOutput);

### Parameter Descriptions

- `tracker` is the [`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID) to query.
- `getInfo` is the [`XrTrackableGetInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID) with the information used to get the trackable object.
- `objectOutput` is a pointer to the [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid) structure in which the trackable object is returned.

`XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID` will be returned if the
trackable type of the [`XrTrackableANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableANDROID) is not
`XR_TRACKABLE_TYPE_OBJECT_ANDROID`, or if the trackable type of the
[`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID) is not `XR_TRACKABLE_TYPE_OBJECT_ANDROID`.

### Valid usage (implicit)

- The [`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables) extension **must** be enabled prior to calling [`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid)
- `tracker` **must** be a valid [`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID) handle
- `getInfo` **must** be a pointer to a valid [`XrTrackableGetInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID) structure
- `objectOutput` **must** be a pointer to an [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid) structure

### Return Codes

[**Success**](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-successcodes)

- `XR_SUCCESS`
- `XR_SESSION_LOSS_PENDING`

[**Failure**](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-errorcodes)

- `XR_ERROR_FUNCTION_UNSUPPORTED`
- `XR_ERROR_VALIDATION_FAILURE`
- `XR_ERROR_RUNTIME_FAILURE`
- `XR_ERROR_HANDLE_INVALID`
- `XR_ERROR_INSTANCE_LOST`
- `XR_ERROR_SESSION_LOST`
- `XR_ERROR_SESSION_NOT_RUNNING`
- `XR_ERROR_TIME_INVALID`
- `XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID`

The [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid) structure is defined as:  

    typedef struct XrTrackableObjectANDROID {
        XrStructureType           type;
        void*                     next;
        XrTrackingStateANDROID    trackingState;
        XrPosef                   centerPose;
        XrExtent3DfEXT            extents;
        XrObjectLabelANDROID      objectLabel;
        XrTime                    lastUpdatedTime;
    } XrTrackableObjectANDROID;

### Member Descriptions

- `type` is the [`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType) of this structure.
- `next` is `NULL` or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
- `trackingState` is the [`XrTrackingStateANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackingStateANDROID) of the object.
- `centerPose` is the [`XrPosef`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrPosef) of the object located in [`XrTrackableGetInfoANDROID::baseSpace`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID).
- `extents` is the [`XrExtent3DfEXT`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrExtent3DfEXT) dimension of the object.
- `objectLabel` is the [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid) that the runtime has determined for this object.
- `lastUpdatedTime` is the [`XrTime`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrTime) of the last update of the object.

### Valid usage (implicit)

- The [`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables) extension **must** be enabled prior to using [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- `type` **must** be `XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `next` **must** be `NULL` or a valid pointer to the next structure in a structure chain
- `trackingState` **must** be a valid [`XrTrackingStateANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackingStateANDROID) value
- `objectLabel` **must** be a valid [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid) value

## Example code for getting trackable objects

The following example code demonstrates how to get trackable objects.  

    XrSession session; // previously initialized

    // The function pointers are previously initialized using xrGetInstanceProcAddr.
    PFN_xrCreateTrackableTrackerANDROID xrCreateTrackableTrackerANDROID; // previously initialized
    PFN_xrGetAllTrackablesANDROID xrGetAllTrackablesANDROID; // previously initialized
    PFN_xrGetTrackableObjectANDROID xrGetTrackableObjectANDROID; // previously initialized
    PFN_xrDestroyTrackableTrackerANDROID xrDestroyTrackableTrackerANDROID; // previously initialized

    XrTime updateTime; // Time used for the current frame's simulation update.
    XrSpace appSpace; // Space created for XR_REFERENCE_SPACE_TYPE_LOCAL.

    XrTrackableTrackerCreateInfoANDROID
      createInfo{XR_TYPE_TRACKABLE_TRACKER_CREATE_INFO_ANDROID};
    createInfo.trackableType = XR_TRACKABLE_TYPE_OBJECT_ANDROID;
    XrTrackableTrackerANDROID objectTrackableTracker;
    XrResult result = xrCreateTrackableTrackerANDROID(
      session,
      &createInfo,
      &objectTrackableTracker);
    if (result != XR_SUCCESS) { /* Handle failures. */ }

    uint32_t trackableCountOutput = 0;
    std::vector<XrTrackableANDROID> allObjectTrackables;

    // Query the number of trackables available.
    result = xrGetAllTrackablesANDROID(
      objectTrackableTracker,
      0,
      &trackableCountOutput,
      nullptr
    );

    if (result == XR_SUCCESS) {
      allObjectTrackables.resize(trackableCountOutput, XR_NULL_TRACKABLE_ANDROID);

      // Fetch the actual trackable handles in the appropriately resized array.
      result = xrGetAllTrackablesANDROID(
        objectTrackableTracker,
        trackableCountOutput,
        &trackableCountOutput,
        allObjectTrackables.data());

      if (result == XR_SUCCESS) {
        for (XrTrackableANDROID trackable : allObjectTrackables) {

          // Object trackable query information
          XrTrackableGetInfoANDROID objectGetInfo;
          objectGetInfo.type = XR_TYPE_TRACKABLE_GET_INFO_ANDROID;
          objectGetInfo.next = nullptr;
          objectGetInfo.trackable = trackable;
          objectGetInfo.baseSpace = appSpace;
          objectGetInfo.time = updateTime;

          // Get the object trackable. Note that the tracker only returns object types.
          XrTrackableObjectANDROID object = { XR_TYPE_TRACKABLE_OBJECT_ANDROID };
          result = xrGetTrackableObjectANDROID(
            objectTrackableTracker,
            &objectGetInfo,
            &object
          );

          if (result == XR_SUCCESS) {
            /** Do Stuff with the object */
          }
        }
      }
    }

    // Release trackable tracker.
    result = xrDestroyTrackableTrackerANDROID(objectTrackableTracker);

**New Enum Constants**

[`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType) enumeration is extended with:

- `XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `XR_TYPE_TRACKABLE_OBJECT_CONFIGURATION_ANDROID`

[`XrTrackableTypeANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTypeANDROID) enumeration is extended with:

- `XR_TRACKABLE_TYPE_OBJECT_ANDROID`

**New Enums**

- [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)

**New Structures**

- [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- [`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid)

**New Functions**

- [`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid)

**Issues**

**Version History**

- Revision 1, 2024-10-03 (Kenny Vercaemer)
  - Initial extension description.

*** ** * ** ***

OpenXR™ and the OpenXR logo are trademarks owned
by The Khronos Group Inc. and are registered as a trademark in China,
the European Union, Japan and the United Kingdom.