# Frame Pacing library
Part of [Android Game Development Kit](https://developer.android.com/games/agdk/overview).

The Android Frame Pacing library, also known as Swappy, is part of the [AGDK Libraries](https://developer.android.com/games/agdk#game-libraries). It helps OpenGL and Vulkan games achieve smooth rendering and
correct frame pacing on Android. This document defines frame pacing, describes
situations where frame pacing is needed, and shows how the library addresses
these situations. If you want to jump directly to implementing frame pacing in
your game, see [Next step](https://developer.android.com/games/sdk/frame-pacing#integrate-next-step).

## Background

*Frame pacing* is the synchronization of a game's logic and rendering loop with
an OS's display subsystem and the underlying display hardware. The Android
display subsystem was designed to avoid visual artifacts (known as *tearing*)
that can occur when the display hardware switches to a new frame part-way
through an update. To avoid these artifacts, the display subsystem does the
following:

- Buffers past frames internally
- Detects late frame submissions
- Repeats the display of past frames when late frames are detected

A game informs
[SurfaceFlinger](https://source.android.com/devices/graphics/surfaceflinger-windowmanager),
the compositor within the display subsystem, that it has submitted all the draw
calls needed for a frame (by calling `eglSwapBuffers` or `vkQueuePresentKHR`).
SurfaceFlinger signals availability of a frame to the display hardware using a
latch. The display hardware then shows the given frame. The display hardware
ticks at a constant rate, for example 60 Hz, and if there is no new frame when
the hardware needs one, the hardware displays the previous frame again.

Inconsistent frame times often occur when a game render loop renders at a
different rate than the native display hardware. If a game running at 30 FPS
attempts to render on a device that natively supports 60 FPS, the game render
loop doesn't realize that a repeated frame remains on the screen for an extra 16
milliseconds. This disconnect usually creates substantial inconsistency in frame
times, such as: 49 milliseconds, 16 milliseconds, 33 milliseconds. Overly
complex scenes further compound this problem, as they cause missed frames to
occur.

## Non-optimal solutions

The following solutions for frame pacing have been employed by games in the past
and typically result in inconsistent frame times and increased input latency.

### Submit frames as quickly as the rendering API allows

This approach ties a game to variable SurfaceFlinger activity and introduces an
extra frame of latency. The display pipeline contains a queue of frames,
typically of size 2, which fills up if the game is trying to present frames too
quickly. With no more room in the queue, the game loop (or at least the
rendering thread) is blocked by an OpenGL or Vulkan call. The game is then
forced to wait for the display hardware to show a frame, and this back-pressure
synchronizes the two components. This situation is known as *buffer-stuffing* or
[*queue-stuffing*](https://developer.android.com/games/develop/gameloops#stuffing). The renderer process
doesn't realize what's going on, so framerate inconsistency gets worse. If the
game samples input before the frame, input latency gets worse.

### Use Android Choreographer by itself

Games also use Android Choreographer for synchronization. This component,
available in Java from API 16 and in C++ from API 24, delivers regular ticks at
the same frequency as the display subsystem. There are still subtleties as to
when this tick is delivered relative to the actual hardware VSYNC, and these
offsets vary by device. Buffer-stuffing may still occur for long frames.

## Advantages of the Frame Pacing library

The Frame Pacing library uses Android Choreographer for synchronization and
deals with the variability in the tick delivery for you. It uses presentation
timestamps to make sure frames are presented at the proper time and sync fences
to avoid buffer stuffing. The library uses the
[NDK Choreographer](https://developer.android.com/ndk/reference/group/choreographer) if it is available and
falls back to the [Java Choreographer](https://developer.android.com/reference/android/view/Choreographer) if
it is not.

The library handles multiple refresh rates if they are supported by the device,
which gives a game more flexibility in presenting a frame. For example, for a
device that supports a 60 Hz refresh rate as well as 90 Hz, a game that cannot
produce 60 frames per second can drop to 45 FPS instead of 30 FPS to remain
smooth. The library detects the expected game frame rate and auto-adjusts frame
presentation times accordingly. The Frame Pacing library also improves battery
life because it avoids unnecessary display updates. For example, if a game is
rendering at 60 FPS but the display is updating at 120 Hz, the screen is
updated twice for every frame. The Frame Pacing library avoids this by setting
the refresh rate to the value supported by the device that's closest to the target
frame rate.

## How it works

The following sections show how the Frame Pacing library deals with long and
short game frames in order to achieve correct frame pacing.

### Correct frame pacing at 30 Hz

When rendering at 30 Hz on a 60 Hz device, the ideal situation on Android is
shown in figure 1. SurfaceFlinger latches new graphic buffers, if present (NB in
the diagram indicates "no buffer" present and the previous one is repeated).

![Ideal frame pacing at 30 Hz on a 60 Hz device](https://developer.android.com/static/images/games/frame-pacing/frame-pacing-ideal.svg)

**Figure 1.** Ideal frame pacing at 30 Hz on a 60 Hz device.

### Short game frames lead to stuttering

On most modern devices, game engines rely on the platform choreographer
delivering ticks to drive the submission of frames. However, there is still the
possibility for poor frame pacing due to short frames, as seen in figure 2.
Short frames followed by long frames are perceived by the player as stuttering.

![Short game frames](https://developer.android.com/static/images/games/frame-pacing/frame-pacing-short-frames.svg)

**Figure 2.** Short game frame C causes frame B to present only one frame,
followed by multiple C frames.

The Frame Pacing library solves this by using presentation timestamps. The
library uses the presentation timestamp extensions
[`EGL_ANDROID_presentation_time`](https://www.khronos.org/registry/EGL/extensions/ANDROID/EGL_ANDROID_presentation_time.txt)
and
[`VK_GOOGLE_display_timing`](https://www.khronos.org/registry/vulkan/specs/1.1-extensions/man/html/VK_GOOGLE_display_timing.html)
so that frames are not presented early, as seen in figure 3.

![Presentation timestamps](https://developer.android.com/static/images/games/frame-pacing/frame-pacing-timestamps.svg)

**Figure 3.** Game frame B presented twice for a smoother display.

### Long frames lead to stuttering and latency

When the display workload takes longer than the application workload, extra
frames are added to a queue. This leads, once again, to stuttering and may also
lead to an extra frame of latency due to buffer-stuffing (see figure 4). The
library both removes the stuttering and the extra frame of latency.

![Long game frames](https://developer.android.com/static/images/games/frame-pacing/frame-pacing-long-frames.svg)

**Figure 4.** Long frame B gives incorrect pacing for 2 frames---A and B

The library solves this by using sync fences
([`EGL_KHR_fence_sync`](https://www.khronos.org/registry/EGL/extensions/KHR/EGL_KHR_fence_sync.txt)
and
[`VkFence`](https://www.khronos.org/registry/vulkan/specs/1.1-extensions/man/html/VkFence.html))
to inject waits into the application that allow the display pipeline to catch
up, rather than allowing back pressure to build up. Frame A still presents an
extra frame, but frame B now presents correctly, as seen in figure 5.

![Waits added into application layer](https://developer.android.com/static/images/games/frame-pacing/frame-pacing-waits.svg)

**Figure 5.** Frames C and D wait to present.

### Supported operating modes

You can configure the Frame Pacing library to operate in one of the three
following modes:

- Auto mode off + Pipeline
- Auto mode on + Pipeline
- Auto mode on + Auto pipeline mode (Pipeline/Non-pipeline)

#### Recommended mode

You can experiment with auto-mode and pipeline modes, but you start by turning
them off and including the following after initializing Swappy:  

      swappyAutoSwapInterval(false);
      swappyAutoPipelineMode(false);
      swappyEnableStats(false);
      swappySwapIntervalNS(1000000000L/yourPreferredFrameRateInHz);

#### Pipeline mode

To coordinate engine workloads, the library typically uses a pipelining model
which separates the CPU and GPU workloads across VSYNC boundaries.

![Pipeline mode](https://developer.android.com/static/images/games/frame-pacing/pipeline-mode.svg)

**Figure 6.** Pipeline mode.

#### Non-pipeline mode

In general, this approach results in lower, more predictable input-screen
latency. In cases where a game has a very low frame time, both the CPU and GPU
workloads may fit into a single swap interval. In this case, a non-pipelined
approach would actually deliver lower input-screen latency.

![Non-pipeline mode](https://developer.android.com/static/images/games/frame-pacing/non-pipeline-mode.svg)

**Figure 7.** Non-pipeline mode.

#### Auto mode

Most games don't know how to choose the swap interval, which is the duration for
which each frame is presented (for example, 33.3 ms for 30 Hz). On some devices,
a game can render at 60 FPS while on another it may need to drop to a lower
value. Auto mode measures CPU and GPU times in order to do the following:

- **Automatically select swap intervals**: Games which deliver 30 Hz in some scenes and 60 Hz in others can allow the library to adjust this interval dynamically.
- **Deactivate pipelining for ultra-fast frames**: Delivers optimal input-screen latency in all cases.

### Multiple refresh rates

Devices that support multiple refresh rates provide higher flexibility in
choosing a swap interval that looks smooth:

- **On 60 Hz devices**: 60 FPS / 30 FPS / 20FPS
- **On 60 Hz + 90 Hz devices**: 90 FPS / 60 FPS / 45 FPS / 30 FPS
- **On 60 Hz + 90 Hz + 120 Hz devices**: 120 FPS / 90 FPS / 60 FPS / 45 FPS / 40 FPS / 30 FPS

The library chooses the refresh rate that best matches the actual rendering
duration of a game's frames, giving a better visual experience.

For more information on multiple refresh rate frame pacing, see the
[High refresh rate rendering on Android](https://android-developers.googleblog.com/2020/04/high-refresh-rate-rendering-on-android.html)
blog post.

### Frame stats

The Frame Pacing library offers the following statistics for debugging and
profiling purposes:

- A histogram of the number of screen refreshes a frame waited in the compositor queue after rendering was completed.
- A histogram of the number of screen refreshes passed between the requested presentation time and the actual present time.
- A histogram of the number of screen refreshes passed between two consecutive frames.
- A histogram of the number of screen refreshes passed between the start of CPU work for this frame and the actual present time.

## Next step

See either of the following guides to integrate the Android Frame Pacing library
into your game:

- [Integrate Android Frame Pacing into your OpenGL renderer](https://developer.android.com/games/sdk/frame-pacing/opengl)
- [Integrate Android Frame Pacing into your Vulkan renderer](https://developer.android.com/games/sdk/frame-pacing/vulkan)

| **Note:** Unity has [integrated Android Frame Pacing](https://unity3d.com/unity/alpha/2019.2.0a6) into their engine. To enable this feature in Unity 2019.2 or higher, check the **Optimized Frame Pacing** checkbox under **Project Settings \> Player \> Settings for Android \> Resolution and Presentation**.

## Additional resources

- [Mir 2 improves rendering performance by using Swappy](https://developer.android.com/stories/games/swappy), reduces slow session rate from 40% to 10%.