Introduction

For a quick tutorial about how to use the crate, click here.

What is lightyear?

Lightyear is a networking library for games written in Bevy. It uses a client-server networking architecture, where the server is authoritative over the game state.

It is heavily inspired by naia.

What is this book about?

This book serves several purposes:

• It contains some explanations of game networking concepts, as well as how they are implemented in this crate
• provide some examples of how to use the crate
• explain some of the design decisions that were made

This book does not aim to be a polished document, or a comprehensive reference for lightyear. It is more of a collection of notes and thoughts that I had while developing the crate; a networking-related wiki that I could reference later.

Tutorial

This section will teach you how to quickly setup networking in your bevy game using this crate.

You can find an example game in the examples folder.

In this tutorial, we will reproduce the simple box example to demonstrate the features of this crate.

We will build a simple game where each client can move a "box" on the screen, the box will be replicated to the server and to all other clients.

General architecture

Lightyear essentially provides 2 plugins that will handle every networking-related concern for you: a ClientPlugin and a ServerPlugin.

The plugins will define various resources and systems that will handle the connection to the server. Some of the notable resources are:

• the TickManager: lightyear uses Ticks to handle synchronization between the client and server. The Tick is basically the fixed-timestep unit of simulation, it gets incremented by 1 every time the FixedUpdate schedule runs. The TickManager has the tick method to return the current client or server tick. (depending on which plugin you are using)
• the ClientConnectionManager or ServerConnectionManager which are used to send messages to the remote.
• the ClientConnection or ServerConnection which handle the general io connection. You can use them to get the current ClientId or to check that the connection is still alive.
• the InputManager lets you send inputs from the client to the server

There are many different sub-plugins but the most important things that lightyear handles for you are probably:

• the sending and receiving of messages.
• automatic replication of the World from the server to the client
• handling the inputs from the user.

Example code organization

In the most basic setup, you will run 2 separate apps: one for the client and one for the server. (You can also run both in the same app in what is called HostServer mode, but we will not cover that in this tutorial.)

The simple_box example has the following structure:

• main.rs: this is where we read the settings file from assets/settings.ron and create the client or server app depending on the passed CLI arguments.
• settings.rs: here we parse the settings.ron file and have helpers to create the ClientConfig and ServerConfig structs which are all that is required to build a ClientPlugin or a ServerPlugin
• protocol.rs: here we define a shared protocol, which is basically the list of messages, components and inputs that can be sent between the client and server.
• shared.rs: this is where we define shared behaviour between the client and server. For example some simulation logic (physics/movement) should be shared between the client and server.
• client.rs: this is where we define client-specific logic (input-handling, client-prediction, etc.)
• server.rs: this is where we define server-specific logic (spawning players for newly-connected clients, etc.)

Defining a protocol

First, you will need to define a protocol for your game. (see here in the example) This is where you define the "contract" of what is going to be sent across the network between your client and server.

A protocol is composed of

• Input: Defines the client's input type, i.e. the different actions that a user can perform (e.g. move, jump, shoot, etc.).
• Message: Defines the message protocol, i.e. the messages that can be exchanged between the client and server.
• Components: Defines the component protocol, i.e. the list of components that can be replicated between the client and server.
• Channels: Defines channels that are used to send messages between the client and server.

A Message is any struct that is Serialize + Deserialize + Clone.

Components

The ComponentRegistry is needed for automatic World replication: automatically replicating entities and components from the server's World to the client's World. Only the components that are defined in the ComponentRegistry will be replicated.

The ComponentRegistry is a Resource that will store metadata about which components should be replicated and how. It can also contain additional metadata for each component, such as prediction or interpolation settings. lightyear provides helper functions on the App to register components to the ComponentRegistry.

Let's define our components protocol:

#![allow(unused)]
fn main() {
/// A component that will identify which player the box belongs to
#[derive(Component, Serialize, Deserialize, Clone, Debug, PartialEq)]
pub struct PlayerId(ClientId);

/// A component that will store the position of the box. We could also directly use the `Transform` component.
#[derive(Component, Serialize, Deserialize, Clone, Debug, PartialEq)]
pub struct PlayerPosition(Vec2);

/// A component that will store the color of the box, so that each player can have a different color.
#[derive(Component, Deserialize, Serialize, Clone, Debug, PartialEq)]
pub struct PlayerColor(pub(crate) Color);

pub struct ProtocolPlugin;

impl Plugin for ProtocolPlugin{
    fn build(&self, app: &mut App) {
        app.register_component::<PlayerId>(ChannelDirection::ServerToClient)
            .add_prediction(ComponentSyncMode::Once)
            .add_interpolation(ComponentSyncMode::Once);

        app.register_component::<PlayerPosition>(ChannelDirection::ServerToClient)
            .add_prediction(ComponentSyncMode::Full)
            .add_interpolation(ComponentSyncMode::Full)
            .add_linear_interpolation_fn();

        app.register_component::<PlayerColor>(ChannelDirection::ServerToClient)
            .add_prediction(ComponentSyncMode::Once)
            .add_interpolation(ComponentSyncMode::Once);
    }
}
}

Message

Similarly, the MessageProtocol is an enum containing the list of possible Messages that can be sent over the network. When registering a message, you can specify the direction in which the message should be sent.

Let's define our message protocol:

/// We don't really use messages in the example, but here is how you would define them.
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)]
pub struct Message1(pub usize);

/// Again, you need to use the macro `message_protocol` to define a message protocol.
#[message_protocol(protocol = "MyProtocol")]
pub enum Messages {
    Message1(Message1),
}

impl Plugin for ProtocolPlugin{
  fn build(&self, app: &mut App) {
    // Register messages
    app.add_message::<Message1>(ChannelDirection::Bidirectional);
    
    // Register components
    ...
  }
}

Inputs

Lightyear handles inputs (the user actions that should be sent to the server) for you, you just need to define the list of possible inputs (like the message or component protocols).

(it is recommended to use the leafwing feature to handle inputs with leafwing-input-manager, but we will not cover that in this tutorial)

Let's define our inputs:

#![allow(unused)]
fn main() {
/// The different directions that the player can move the box
#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)]
pub struct Direction {
    pub(crate) up: bool,
    pub(crate) down: bool,
    pub(crate) left: bool,
    pub(crate) right: bool,
}

/// The `InputProtocol` needs to be an enum of the various inputs that the client can send to the server.
#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)]
pub enum Inputs {
    Direction(Direction),
    Delete,
    Spawn,
    /// NOTE: we NEED to provide a None input so that the server can distinguish between lost input packets and 'None' inputs
    None,
}

impl Plugin for ProtocolPlugin{
  fn build(&self, app: &mut App) {
    // Register inputs
    app.add_plugins(InputPlugin::<Inputs>::default());
    // Register messages
    ...
    // Register components
    ...
  }
}
}

Inputs have to implement the UserAction trait, which means that they must be Send + Sync + 'static and can be serialized.

Channels

We can also define some channels that will be used to send messages between the client and server. This is optional, since lightyear already provides some default channels for inputs and components.

A Channel defines some properties of how messages will be sent over the network:

• reliability: can the messages be lost or do we re-send them until we receive an ACK?
• ordering: do we guarantee that the messages are received in the same order that they were sent?
• priority: do we want to increase the priority of some messages in case the network is congested?

/// A channel is basically a ZST (Zero Sized Type) with the `Channel` derive macro.
#[derive(Channel)]
pub struct Channel1;

pub(crate) struct ProtocolPlugin;

impl Plugin for ProtocolPlugin {
    fn build(&self, app: &mut App) {
        // channels
        app.add_channel::<Channel1>(ChannelSettings {
          mode: ChannelMode::OrderedReliable(ReliableSettings::default()),
          ..default()
        });
        // register messages, inputs, components
        ...
    }
}

We create a channel by simply deriving the Channel trait on an empty struct.

Protocol

The complete protocol looks like this:

#![allow(unused)]
fn main() {
pub(crate) struct ProtocolPlugin;

impl Plugin for ProtocolPlugin {
    fn build(&self, app: &mut App) {
        // messages
        app.add_message::<Message1>(ChannelDirection::Bidirectional);
        // inputs
        app.add_plugins(InputPlugin::<Inputs>::default());
        // components
        app.register_component::<PlayerId>(ChannelDirection::ServerToClient)
            .add_prediction::<PlayerId>(ComponentSyncMode::Once)
            .add_interpolation::<PlayerId>(ComponentSyncMode::Once);

        app.register_component::<PlayerPosition>(ChannelDirection::ServerToClient)
            .add_prediction::<PlayerPosition>(ComponentSyncMode::Full)
            .add_interpolation::<PlayerPosition>(ComponentSyncMode::Full)
            .add_linear_interpolation_fn::<PlayerPosition>();

        app.register_component::<PlayerColor>(ChannelDirection::ServerToClient)
            .add_prediction::<PlayerColor>(ComponentSyncMode::Once)
            .add_interpolation::<PlayerColor>(ComponentSyncMode::Once);
        // channels
        app.add_channel::<Channel1>(ChannelSettings {
            mode: ChannelMode::OrderedReliable(ReliableSettings::default()),
            ..default()
        });
    }
}
}

Summary

We now have a complete Protocol that defines:

• the data that can be sent between the client and server (inputs, messages, components)
• how the data will be sent (channels)

We can now start building our client and server Plugins.

Setting up the client and server

Client

A client is simply a bevy plugin: ClientPlugin

You create it by providing a ClientConfig struct.

You can see how it is defined in the example here.

Shared Config

Some parts of the configuration must be shared between the server and the client to work correctly, so we define them in a separate function that can be re-used for both:

#![allow(unused)]
fn main() {
pub fn shared_config(mode: Mode) -> SharedConfig {
    SharedConfig {
        /// How often the client will send packets to the server (by default it is every frame).
        /// Currently, the client only works if it sends packets every frame, for proper input handling.
        client_send_interval: Duration::default(),
        /// How often the server will send packets to clients? You can reduce this to save bandwidth.
        server_send_interval: Duration::from_millis(40),
        /// The tick rate that will be used for the FixedUpdate schedule
        tick: TickConfig {
            tick_duration: Duration::from_secs_f64(1.0 / 64.0),
        },
        /// Here we make the `Mode` an argument so that we can run `lightyear` either in `Separate` mode (distinct client and server apps)
        /// or in `HostServer` mode (the server also acts as a client).
        mode,
    }
}
}

ClientConfig

The ClientConfig struct lets us configure the client. There are a lot of parameters that can be configured, but for this demo we will mostly use the defaults.

  let client_config = client::ClientConfig {
      shared: shared_config(Mode::Separate),
      net: net_config,
      ..default()
  };
  let client_plugin = client::ClientPlugin::new(client_config);

The NetConfig doesn't have any Default value and needs to be provided; it defines how (i.e. what transport layer) the client will connect to the server. There are multiple options available, but for this demo we will use the Netcode option. netcode is a standard to establish a connection between two hosts, and we can use any io layer (UDP, WebSocket, WebTransport, etc.) to send the actual bytes.

You will need to provide the IoConfig which defines the transport layer (how the raw packets are sent), with the possibility of using a LinkConditionerConfig to simulate network conditions. Here are the different possible transport options: TransportConfig

/// You can add a link conditioner to simulate network conditions
let link_conditioner = LinkConditionerConfig {
    incoming_latency: Duration::from_millis(100),
    incoming_jitter: Duration::from_millis(0),
    incoming_loss: 0.00,
};
/// Here we use the `UdpSocket` transport layer, with the link conditioner
let io_config = IoConfig::from_transport(TransportConfig::UdpSocket(addr))
    .with_conditioner(link_conditioner);

With the Netcode option, we use a ConnectToken to secure the connection. Normally, a third-party server would generate the ConnectToken and send it securely to the client.

For this demo, we will use the Manual option, which lets us manually build a ConnectToken on the client using a private key shared between the client and the server.

let server_addr = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), server_port);
let auth = Authentication::Manual {
    // server's IP address
    server_addr,
    // ID to uniquely identify the client
    client_id: client_id,
    // private key shared between the client and server
    private_key: KEY,
    // PROTOCOL_ID identifies the version of the protocol
    protocol_id: PROTOCOL_ID,
};

Now we can build the complete NetConfig:

let net_config = NetConfig::Netcode {
    auth,
    io: io_config,
    ..default()
};

Server

Building the server is very similar to building the client; we need to provide a ServerConfig struct.

let server_config = server::ServerConfig {
    shared: shared_config(Mode::Separate),
    net: net_configs,
    ..default()
};
let server_plugin = server::ServerPlugin::new(server_config);

The server can listen for client connections using multiple transports at the same time! You can do this by providing multiple NetConfig to the server.

The simple_box example generates the various NetConfigs by parsing the settings.ron file, but you can also just define them manually:

#![allow(unused)]
fn main() {
let server_addr = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), self.port);
/// You need to provide the private key and protocol id when building the `NetcodeConfig`
let netcode_config = NetcodeConfig::default()
    .with_protocol_id(PROTOCOL_ID)
    .with_key(KEY);
/// You can also add a link conditioner to simulate network conditions for packets received by the server
let link_conditioner = LinkConditionerConfig {
    incoming_latency: Duration::from_millis(100),
    incoming_jitter: Duration::from_millis(0),
    incoming_loss: 0.00,
};
let net_config = NetConfig::Netcode {
    config: netcode_config,
    io: IoConfig::from_transport(TransportConfig::UdpSocket(server_addr))
        .with_conditioner(link_conditioner),
};
let config = ServerConfig {
    shared: shared_config().clone(),
    /// Here we only provide a single net config, but you can provide multiple!
    net: vec![net_config],
    ..default()
};
/// Finally build the server plugin
let server_plugin = server::ServerPlugin::new(server_config);
}

Next we will start adding systems to the client and server.

Adding basic functionality

What we want to achieve is this:

• when a client connects to the server, the server spawns a player entity for that client
• that entity gets replicated to all clients
• a client can send inputs to move the player entity that corresponds to them

Initialization

Lightyear uses bevy States to manage the connection state. The states indicate the current state of the connection. The client states are Disconnected, Connecting, Connected; and the server states are Started, Stopped.

The connection is started/stopped using the ClientCommands and ServerCommands commands. Server:

fn init(mut commands: Commands) {
    commands.start_server();
}
app.add_systems(Startup, init);

Client:

fn init(mut commands: Commands) {
    commands.connect_client();
}
app.add_systems(Startup, init);

We also do some setup, like adding a Camera2dBundle and displaying some text to let us know if we are the server or the client.

Network events

The way you can access networking-related events is by using bevy Events. lightyear exposes a certain number of events which you can see here:

• ConnectEvent / DisconnectEvent: when a client gets connected or disconnected. This can be used to access the ClientId of the client that connected/disconnected.
• EntitySpawnEvent / EntityDespawnEvent: the receiver emits these when it spawns/despawns an entity replicated from the remote world
• ComponentInsertEvent / ComponentRemoveEvent / ComponentUpdateEvent: the receiver emits these when it inserts/removes/updates a component for an entity replicated from the remote world
• InputEvent: when a user action gets emitted. This event will be emitted on both the server and the client at the exact Tick where the input was emitted.
• MessageEvent: when a message is received from the remote machine. This is used to access the message contents.

This is what we'll use to spawn an entity on the server whenever a client connects!

/// We will maintain a mapping from client id to entity id
// so that we know which entity to despawn when the client disconnects
#[derive(Resource)]
pub(crate) struct Global {
    pub client_id_to_entity_id: HashMap<ClientId, Entity>,
}

/// Create a player entity whenever a client connects
pub(crate) fn handle_connections(
    /// Here we listen for the `ConnectEvent` event
    mut connections: EventReader<ConnectEvent>,
    mut global: ResMut<Global>,
    mut commands: Commands,
) {
    for connection in connections.read() {
        /// on the server, the `context()` method returns the `ClientId` of the client that connected
        let client_id = *connection.context();
        
        /// We add the `Replicate` bundle to start replicating the entity to clients
        /// By default, the entity will be replicated to all clients
        let replicate = Replicate::default(); 
        let entity = commands.spawn((PlayerBundle::new(client_id, Vec2::ZERO), replicate));
        
        // Add a mapping from client id to entity id
        global.client_id_to_entity_id.insert(client_id, entity.id());
    }

As you can see above, starting replicating an entity is very easy: you just need to add the Replicate bundle to the entity and it will start getting replicated.

(you can learn more in the replicate page)

The Replicate bundle is composed of several components that control the replication logic. For example ReplicationTarget specifies which clients should receive the entity. If you remove the ReplicationTarget component from an entity, any updates to that entity won't be replicated anymore. (However the remote entity won't get despawned, it will just stop getting updated)

Handle client inputs

Then we want to be able to handle inputs from the user. We need a system that reads keypresses/mouse movements and converts them into Inputs (from our Protocol). You will need to call the add_input method on the InputManager resource to send an input to the server.

There are some ordering constraints for inputs: you need to make sure that inputs are handled in the BufferInputs SystemSet, which runs in the FixedPreUpdate schedule. Then lightyear will make sure that the server will handle the input on the same tick as the client.

On the client:

pub(crate) fn buffer_input(
    /// You will need to specify the exact tick at which the input was emitted. You can use 
    /// the `TickManager` to retrieve the current tick
    tick_manager: Res<TickManager>,
    /// You will use the `InputManager` to send an input
    mut input_manager: ResMut<InputManager<Inputs>>,
    keypress: Res<ButtonInput<KeyCode>>,
) {
    let tick = tick_manager.tick();
    let mut input = Inputs::None;
    let mut direction = Direction {
        up: false,
        down: false,
        left: false,
        right: false,
    };
    if keypress.pressed(KeyCode::KeyW) || keypress.pressed(KeyCode::ArrowUp) {
        direction.up = true;
    }
    if keypress.pressed(KeyCode::KeyS) || keypress.pressed(KeyCode::ArrowDown) {
        direction.down = true;
    }
    if keypress.pressed(KeyCode::KeyA) || keypress.pressed(KeyCode::ArrowLeft) {
        direction.left = true;
    }
    if keypress.pressed(KeyCode::KeyD) || keypress.pressed(KeyCode::ArrowRight) {
        direction.right = true;
    }
    if !direction.is_none() {
        input = Inputs::Direction(direction);
    }
    input_manager.add_input(input, tick)
}
app.add_systems(FixedPreUpdate, buffer_input.in_set(InputSystemSet::BufferInputs));

Then, on the server, you will want to listen to the InputEvent event, in the FixedUpdate schedule, to move the client entity. Any changes to the entity will be replicated to all clients.

We define a function that specifies how a given input updates a given player entity. It is a good idea to define the simulation logic in a function that can be shared between the client and the server, in case we want to run the simulation ahead of time on the client (this is called client-prediction).

pub(crate) fn shared_movement_behaviour(mut position: Mut<PlayerPosition>, input: &Inputs) {
    const MOVE_SPEED: f32 = 10.0;
    match input {
        Inputs::Direction(direction) => {
            if direction.up {
                position.y += MOVE_SPEED;
            }
            if direction.down {
                position.y -= MOVE_SPEED;
            }
            if direction.left {
                position.x -= MOVE_SPEED;
            }
            if direction.right {
                position.x += MOVE_SPEED;
            }
        }
        _ => {}
    }
}

Then we can create a system that reads the inputs and applies them to the player entity. On the server:

fn movement(
    mut position_query: Query<&mut PlayerPosition>,
    /// Event that will contain the inputs for the correct tick
    mut input_reader: EventReader<InputEvent<Inputs>>,
    /// Retrieve the entity associated with a given client
    global: Res<Global>,
) {
    for input in input_reader.read() {
        let client_id = input.context();
        if let Some(input) = input.input() {
            if let Some(player_entity) = global.client_id_to_entity_id.get(client_id) {
                if let Ok(position) = position_query.get_mut(*player_entity) {
                    shared_movement_behaviour(position, input);
                }
            }
        }
    }
}
app.add_systems(FixedUpdate, movement);

Any fixed-update simulation system (physics, etc.) must run in the FixedUpdate Schedule to behave correctly.

Displaying entities

Finally we can add a system on both client and server to draw a box to show the player entity.

pub(crate) fn draw_boxes(
    mut gizmos: Gizmos,
    players: Query<(&PlayerPosition, &PlayerColor)>,
) {
    for (position, color) in &players {
        gizmos.rect(
            Vec3::new(position.x, position.y, 0.0),
            Quat::IDENTITY,
            Vec2::ONE * 50.0,
            color.0,
        );
    }
}

Now, running the server and client in parallel should give you:

• server spawns a cube when client connects
• client can send inputs to the server to control the cube
• the movements of the cube in the server world are replicated to the client (and to other clients) !

In the next section, we will see a couple more systems.

Advanced systems

In this section we will see how we can add client-prediction and entity-interpolation to make the game feel more responsive and smooth.

Client prediction

If we wait for the server to:

• receive the client input
• move the player entity
• replicate the update back to the client

We will have a delay of at least 1 RTT (round-trip-delay) before we see the impact of our inputs on the player entity. This can feel very sluggish/laggy, which is why often games will use client-side prediction. Another issue is that the entity on the client will only be updated whenever we receive a packet from the server. Usually the packet send rate is much lower than one packet per frame, for example it can be on the order of 10 packet per second. If the server's packet_send_rate is low, the entity will appear to stutter.

The solution is to run the same simulation systems on the client as on the server, but only for the entities that the client predicts. This is "client-prediction": we move the client-controlled entity immediately according to our user inputs, and then we correct the position when we receive the actual state of the entity from the server. (if there is a mismatch)

In lightyear, this is enabled by setting a prediction_target on the Replicate component, which lets you specify which clients will predict the entity.

If prediction is enabled for an entity, the client will spawn a local copy of the entity along with a marker component called Predicted. The entity that is directly replicated from the server will have a marker component called Confirmed (because it is only updated when we receive a new server update packet).

The Predicted entity lives on a different timeline than the Confirmed entity: it lives a few ticks in the future (at least 1 RTT), enough ticks so that the client inputs for tick N have time to arrive on the server before the server processes tick N.

Whenever the player sends an input, we can apply the inputs instantly to the Predicted entity; which is the only one that we show to the player. After roughly 1 RTT, we receive the actual state of the entity from the server, which is used to update the Confirmed entity. If there is a mismatch between the Confirmed and Predicted entities, we perform a rollback: we reset the Predicted entity to the state of the Confirmed entity, and re-run all the ticks that happened since the last server update was received. In particular, we will re-apply all the client inputs that were added since the last server update.

As the Confirmed and Predicted entities are 2 separate entities, you will need to specify how the components that are received on the Confirmed entity (replicated from the server) are copied to the Predicted entity. To do this, you will need to specify a ComponentSyncMode for each component in the ComponentProtocol enum. There are 3 different modes:

• Full: we apply client-side prediction with rollback
• Simple: the server-updates are copied from Confirmed to Predicted whenever we have an update
• Once: the components are copied only once from the Confirmed entity to the Predicted entity

You will need to modify your ComponentProtocol to specify the prediction behaviour for each component; if you don't specify one, the component won't be copied from the Confirmed to the Predicted entity.

On the server, we will update our entity-spawning system to add client-prediction:

/// Create a player entity whenever a client connects
pub(crate) fn handle_connections(
    /// Here we listen for the `ConnectEvent` event
    mut connections: EventReader<ConnectEvent>,
    mut global: ResMut<Global>,
    mut commands: Commands,
) {
    for connection in connections.read() {
        /// on the server, the `context()` method returns the `ClientId` of the client that connected
        let client_id = *connection.context();
        
        /// We add the `Replicate` component to start replicating the entity to clients
        /// By default, the entity will be replicated to all clients
        let replicate = Replicate {
          prediction_target: NetworkTarget::Single(client_id),
          ..default()
        };
        let entity = commands.spawn((PlayerBundle::new(client_id, Vec2::ZERO), replicate));
        
        // Add a mapping from client id to entity id
        global.client_id_to_entity_id.insert(client_id, entity.id());
    }

The only change is the line prediction_target: NetworkTarget::Single(client_id); we are specifying which clients should be predicting this entity. Those clients will spawn a copy of the entity with the Predicted component added.

Then, on the client, you need to make sure that you also run the same simulation logic as the server, for the Predicted entities. We will add a new system on the client that also applies the user inputs. It is very similar to the server system, we also listen for the InputEvent event. It also needs to run in the FixedUpdate schedule to work correctly.

On the client:

fn player_movement(
    /// Note: we only apply inputs to the `Predicted` entity
    mut position_query: Query<&mut PlayerPosition, With<Predicted>>,
    mut input_reader: EventReader<InputEvent<Inputs>>,
) {
    for input in input_reader.read() {
        if let Some(input) = input.input() {
            for position in position_query.iter_mut() {
                shared_movement_behaviour(position, input);
            }
        }
    }
}
app.add_systems(FixedUpdate, movement);

Now you can try running the server and client again; you should see 2 cubes for the client; the Predicted cube should move immediately when you send an input on the client. The Confirmed cube only moves when we receive a packet from the server.

Entity interpolation

Client-side prediction works well for entities that the player predicts, but what about entities that are not controlled by the player? There are two solutions to make updates smooth for those entities:

• predict them as well, but there might be much bigger mis-predictions because we don't have access to other player's inputs
• display those entities slightly behind the 'Confirmed' entity, and interpolate between the last two confirmed states

The second approach is called 'interpolation', and is the one we will use in this tutorial. You can read this Valve article that explains it pretty well.

You will need to modify your ComponentProtocol to specify the interpolation behaviour for each component; if you don't specify one, the component won't be copied from the Confirmed to the Interpolated entity. You will also need to provide an interpolation function; or you can use the default linear interpolation.

On the server, we will update our entity-spawning system to add entity-interpolation:

/// Create a player entity whenever a client connects
pub(crate) fn handle_connections(
    /// Here we listen for the `ConnectEvent` event
    mut connections: EventReader<ConnectEvent>,
    mut global: ResMut<Global>,
    mut commands: Commands,
) {
    for connection in connections.read() {
        /// on the server, the `context()` method returns the `ClientId` of the client that connected
        let client_id = *connection.context();
        
        /// We add the `Replicate` component to start replicating the entity to clients
        /// By default, the entity will be replicated to all clients
        let replicate = Replicate {
          prediction_target: NetworkTarget::Single(client_id),
          interpolation_target: NetworkTarget::AllExceptSingle(client_id),
          ..default()
        };
        let entity = commands.spawn((PlayerBundle::new(client_id, Vec2::ZERO), replicate));
        
        // Add a mapping from client id to entity id
        global.client_id_to_entity_id.insert(client_id, entity.id());
    }

This time we interpolate for entities that are not controlled by the player, so we use NetworkTarget::AllExceptSingle(id).

If interpolation is enabled for an entity, the client will spawn a local copy of the entity along with a marker component called Interpolated. The entity that is directly replicated from the server will have a marker component called Confirmed (because it is only updated when we receive a new server update packet).

The Interpolated entity lives on a different timeline than the Confirmed entity: it lives a few ticks in the past. We want it to live slightly in the past so that we always have at least 2 confirmed states to interpolate between.

And that's it! The ComponentSyncMode::Full mode is required on a component to run interpolation.

Now if you run a server and two clients, each player should see the other's player slightly in the past, but with movements that are interpolated smoothly between server updates.

Conclusion

We have now covered the basics of lightyear, and you should be able to build a server-authoritative multiplayer game with client-side prediction and entity interpolation!

Examples

This page contains a list of examples that you can run to see how Lightyear works. Click on the links to run the example in your browser. The examples run using WASM and WebTransport, so they might not work on all browsers (for example, they don't work on Safari currently).

Simple Box

Simple example showing how to replicate a box between the client and server.

Two boxes are actually shown:

• the red box is the Confirmed entity, which is updated at every server update (i.e. every 100ms)
• the pink box is either:
  • the Predicted entity (if the client is controlling the entity)
  • the Interpolated entity (if the client is not controlling the entity): the entity's movements are smoothed between two server updates

Replication Groups

This is an example that shows how to make Lightyear replicate multiple entities in a single message, to make sure that they are always in a consistent state (i.e. that entities in a group are all replicated on the same tick).

It also shows how lightyear can replicate components that are references to an entity. Lightyear will take care of mapping the entity from the server's World to the client's World.

Interest Management

This example shows how lightyear can perform interest management: replicate only a subset of entities to each player. Here, the server will only replicate the green dots that are close to each player.

Client Replication

This example shows how lightyear can be used to replicate entities from the client to the server (and to other clients). The replication can be client-controlled.

It also shows how to spawn entities directly on the client's predicted timeline.

Bullet Pre-spawn

This example shows how to easily pre-spawn entities on the client's predicted timeline. The bullets are created using the same system on both client and server; however when the server replicates a bullet to the client, the client will match it with the existing pre-spawned bullet (instead of creating a new entity).

Leafwing Input Prediction

This example showcases several things:

• how to integrate lightyear with leafwing_input_manager. In particular you can simply attach an ActionState and an InputMap to an Entity, and the ActionState for that Entity will be replicated automatically
• an example of how to integrate physics replication with bevy_xpbd. The physics sets have to be run in FixedUpdateSet::Main
• an example of how to run prediction for entities that are controlled by other players. (this is similar to what RocketLeague does). There is going to be a frequent number of mispredictions because the client is predicting other players without knowing their inputs. The client will just consider that other players are doing the same thing as the last time it received their inputs. You can use the parameter --predict on the server to enable this behaviour (if not, other players will be interpolated).
• The prediction behaviour can be adjusted by two parameters:
  • input_delay: the number of frames it will take for an input to be executed. If the input delay is greater than the RTT, there should be no mispredictions at all, but the game will feel more laggy.
  • correction_ticks: when there is a misprediction, we don't immediately snapback to the corrected state, but instead we visually interpolate from the current state to the corrected state. This parameter helps make mispredictions less jittery.

Priority

This examples shows how lightyear can help with bandwidth management. See this chapter of the book.

Lightyear can limit the bandwidth used by the client or the server, for example to limit server traffic costs, or because the client's connection cannot handle a very high bandwidth. You can then assign a priority score to indicate which entities/messages are important and should be sent first.

In this example, the middle row has a priority of 1.0, and the priority increases by 1.0 for each row further away from the center. (i.e. the edge rows have a priority of 7.0 and are updated 7 times more frequently than the center row)

Concepts

There are several layers that enable lightyear to act as a games networking library. Let's list them from the bottom up (closer to the wire):

• transport: how do you send/receive unreliable-unordered packets on the network (UDP, QUIC, etc.)
• connection: abstraction of a stateful connection between two peers
• channels/reliability: how do you add ordering and reliability to packets
• replication: how do you replicate components between the client and server
• advanced replication: prediction, interpolation, etc.

Transport

The Transport trait is the trait that is used to send and receive raw data on the network.

It is very general:

pub trait PacketSender: Send + Sync {
    /// Send data on the socket to the remote address
    fn send(&mut self, payload: &[u8], address: &SocketAddr) -> Result<()>;
}
pub trait PacketReceiver: Send + Sync {
    /// Receive a packet from the socket. Returns the data read and the origin.
    ///
    /// Returns Ok(None) if no data is available
    fn recv(&mut self) -> Result<Option<(&mut [u8], SocketAddr)>>;
}

The trait currently has 4 implementations:

• UDP sockets
• WebTransport (using QUIC)
• WebSocket
• crossbeam-channels: used for internal testing

Serialization

We use the bitcode library to serialize and deserialize messages. Bitcode is a very compact serialization format that uses bit-packing (a bool will be serialized as a single bit).

When sending messages, we start by serializing the message early into a Bytes structure.

This allows us to:

• know the size of the message right away (which helps with packet fragmentation)
• cheaply copy the message if we need to send it multiple times (for reliable channels) However:
• it is much more expensive and inefficient to call serialize on each individual message compared with the final packet, and the serialized bytes compress less efficiently

Buffers

We use a Buffer to serialize/deserialize messages in order to re-use memory allocations.

When we receive a packet (&[u8]), we create a ReadBuffer from it, which starts by copying the bytes into the buffer.

Packet

On top of the transport layer (which lets us send some arbitrary bytes) we have the packet layer.

A packet is a structure that contains some data and some metadata (inside the header).

Packet header

The packet header will contain the same data as described in the Gaffer On Games articles:

• the packet type (single vs fragmented)
• the packet id (a wrapping u16)
• the last ack-ed packet id received by the sender
• an ack bitfield containing the ack of the last 32 packets before last_ack_packet_id
• the current tick

Packet data

The data will be a list of Messages that are contained in the packet.

A message is a structure that knows how to serialize/deserialize itself.

This is how we store messages into packets:

• the message get serialized into raw bytes
• if the message is over the packet limit size (roughly 1200 bytes), it gets fragmented into multiple parts
• we build a packet by iterating through the channels in order of priority, and then storing as many messages we can into the packet

Connection

Introduction

Our transport layer only allows us to send/receive raw packets to a remote address. But we want to be able to create a stateful 'connection' where we know that two peers are connected.

To establish that connection, that needs to be some machinery that runs on top of the transport layer and takes care of:

• sending handshake packets to authenticate the connection
• sending keep-alive packets to check that the connection is still open
• storing the list of connected remote peers
• etc.

lightyear uses the traits NetClient and NetServer to abstract over the connection logic.

Multiple implementations are provided:

• Netcode
• Steam
• Local

Netcode

This implementation is based on the netcode.io standard created by Glenn Fiedler (of GafferOnGames fame). It describes a protocol to establish a secure connection between two peers, provided that there is an unoredered unreliable transport layer to exchange packets.

For my purpose I am using this Rust implementation of the standard.

You can use the Netcode connection by using the NetcodeClient and NetcodeServer structs, coupled with any of the available transports (Udp, WebTransport, etc.)

To connect to a game server, the client needs to send a ConnectToken to the game server to start the connection process.

There are several ways to obtain a ConnectToken:

• the client can request a ConnectToken via a secure (e.g. HTTPS) connection from a backend server. The server must use the same protocol_id and private_key as the game servers. The backend server could be a dedicated webserver; or the game server itself, if it has a way to establish secure connection.
• when testing, it can be convenient for the client to create its own ConnectToken manually. You can use Authentication::Manual for those cases.

Currently lightyear does not provide any functionality to let a game server send a ConnectToken securely to a client. You will have to handle this logic youself.

Steam

This implementation is based on the Steamworks SDK.

Local

Local connections are used when running in host-server mode: the server and the client are running in the same bevy App. No packets are actually sent over the network since the client and server share the same World.

Multi connection

In lightyear, the server can handle multiple connection protocol as the same time. This means that the server could:

• open a port to establish steam socket connections
• open another port for UDP connections
• open another port for WebTransport connections
• etc.

and have all these connections running at the same time.

You can therefore have cross-play between different platforms.

Another potential usage is to have a "ListenServer" setup where a client acts as the "host":

• the Client and the Server run in the same process
• the Server has multiple connection protocols:
  • one is based on local channels to talk to the Client that is running in the same process
  • the other could be for example a UDP connection to allow other clients to connect to the server

To achieve this, you can just provide multiple NetConfig when creating the ServerConfig that will be used to create the server.

Reliability

In this layer we add some mechanisms to be able to send and receive messages reliably or in a given order.

It is similar to the reliable layer created by Glenn Fiedler on top of his netcode.io code.

This layer introduces:

• reliability: make sure a packets is received by the remote peer
• ordering: make sure packets are received in the same order they were sent
• channels: allow to send packets on different channels, which can have different reliability and ordering guarantees

PacketHeader

Channels

Lightyear introduces the concept of a Channel to handle reliability.

A Channel is a way to send packets with specific reliability, ordering and priority guarantees.

You can add a channel to your protocol like so:

#[derive(Channel)]
struct MyChannel;

pub fn protocol() -> MyProtocol {
    let mut p = MyProtocol::default();
    p.add_channel::<MyChannel>(ChannelSettings {
        mode: ChannelMode::OrderedReliable(ReliableSettings::default()),
        direction: ChannelDirection::Bidirectional,
    });
    p
}

Mode

The mode field of ChannelSettings defines the reliability/ordering guarantees of the channel.

Reliability:

• Unreliable: packets are not guaranteed to arrive
• Reliable: packets are guaranteed to arrive. We will resend the packet until we receive an acknowledgement from the remote. You can define how often we resend the packet via the ReliableSettings field.

Ordering:

• Ordered: packets are guaranteed to arrive in the order they were sent (client sends 1,2,3,4,5, server receives 1,2,3,4,5)
• Unordered: packets are not guaranteed to arrive in the order they were sent (client sends 1,2,3,4,5, server receives 1,3,2,5,4)
• Sequenced: packets are not guaranteed to arrive in the order they were sent, but we will discard packets that are older than the last received packet (client sends 1,2,3,4,5, server receives 1,3,5 (2 and 4 are discarded))

Direction

The direction field can be used to restrict a Channel from sending packets from client->server or server->client.

Replication

Protocol

Overview

The Protocol module in this library is responsible for defining the communication protocol used to send messages between the client and server. The Protocol must be shared between client and server, so that the messages can be serialized and deserialized correctly.

Key Concepts

Protocol Trait

A Protocol contains multiple sub-parts:

• Input: Defines the user inputs, which is an enum of all the inputs that the client can send to the server. Input handling can be added by adding the InputPlugin plugin: app.add_plugins(InputPlugin::<I>::default());

• LeafwingInput: (only if the feature leafwing is enabled) Defines the leafwing ActionState that the client can send to the server. Input handling can be added by adding the LeafwingInputPlugin plugin: app.add_plugins(LeafwingInputPlugin::<I>::default());

• MessageRegistry: Will hold metadata about the all the messages that can be sent over the network. Each message must be Serializable + Deserializeable + Clone. You can register a message with the command app.add_message::<Message1>(ChannelDirection::Bidirectional);

• Components: Defines the component protocol, which is an enum of all the components that can be replicated between the client and server. Each component must be Serializable + Clone + Component. You can register a component with:

  app.register_component::<PlayerId>(ChannelDirection::ServerToClient)
    .add_prediction::<PlayerId>(ComponentSyncMode::Once)
    .add_interpolation::<PlayerId>(ComponentSyncMode::Once);

  (You can specify additional behaviour for the component, such as prediction or interpolation.)

• Channels: the protocol should also contain a list of channels to be used to send messages. A Channel defines guarantees about how the packets will be sent over the network: reliably? in-order? etc. You can register a channel with:

  app.add_channel::<Channel1>(ChannelSettings {
      mode: ChannelMode::OrderedReliable(ReliableSettings::default()),
      ..default()
  });

Replication

You can use the Replicate bundle to initiate replicating an entity from the local World to the remote World.

It is composed of multiple smaller components that each control an aspect of replication:

• ReplicationTarget to decide who to replicate to
• VisibilityMode to enable interest management
• ControlledBy so the server can track which entity is owned by each client
• ReplicationGroup to know which entity updates should be sent together in the same message
• ReplicateHierarchy to control if the children of an entity should also be replicated
• DisabledComponent<C> to disable replication for a specific component
• ReplicateOnceComponent<C> to specify that some components should not replicate updates, only inserts/removals
• OverrideTargetComponent<C> to override the replication target for a specific component

By default, every component in the entity that is part of the ComponentRegistry will be replicated. Any changes in those components will be replicated. However the entity state will always be 'consistent': the remote entity will always contain the exact same combination of components as the local entity, even if it's a bit delayed.

You can remove the ReplicationTarget component to pause the replication. This can be useful when you want to despawn the entity on the server without replicating the despawn. (e.g. an entity can be despawned immediately on the server, but needs to remain alive on the client to play a dying animation)

You can find some of the other usages in the advanced_replication section.

Replicating resources

You can also replicate bevy Resources. This is useful when you want to update a Resource on the server and keep synced copies on the client. This only works for Resources that also implement Clone, and should be limited to resources which are cheap to clone.

• Then, to replicate a Resource, you can use the commands.replicate_resource::<R>(replicate) method. You will need to provide an instance of the Replicate struct to specify how the replication should be done (e.g. to which clients should the resource be replicated). To stop replicating a Resource, you can use the commands.stop_replicate_resource::<R>() method. Note that this won't delete the resource from the client, but it will stop updating it.

System order

Lightyear provides several SystemSets that you can use to run your systems in the correct order.

The main things to keep in mind are:

• All packets are read during the PreUpdate schedule. This is also where all components that were received are replicated to the Client World.
• All packets are sent during the PostUpdate schedule. All messages that were buffered are then sent to the remote, and all replication messages (entity spawn, component updated, etc.) are also sent
• There are 2 SystemSets that you should interact with:
  • BufferInputs: this is where you should be running client.add_inputs() so that they are buffered and sent to the server correctly
  • Main: this is where all your FixedUpdate Schedule systems (physics, etc.) should be run, so that they interact correctly with client-side prediction, etc.

Here is a simplified version of the system order:

---
title: Simplified SystemSet order
---
stateDiagram-v2

   classDef flush font-style:italic;
   
   ReceiveFlush: Flush

   
   PreUpdate --> FixedUpdate
   FixedUpdate --> PostUpdate 
   state PreUpdate {
      Receive --> ReceiveFlush
      ReceiveFlush --> Prediction
      ReceiveFlush --> Interpolation
   }
   state FixedUpdate {
      TickUpdate --> BufferInputs
      BufferInputs --> Main
   }
   state PostUpdate {
       Send
   }

Full system order

---
title: SystemSet order
---
stateDiagram-v2

   classDef flush font-style:italic;
   
   SpawnPredictionHistory : SpawnHistory
   SpawnInterpolationHistory : SpawnHistory
   SpawnPredictionHistoryFlush : Flush
   SpawnInterpolationHistoryFlush : Flush
   SpawnPredictionFlush : Flush
   SpawnInterpolationFlush: Flush
   CheckRollbackFlush: Flush
   DespawnFlush: Flush
   ReceiveFlush: Flush
   FixedUpdatePrediction: Prediction
   
   PreUpdate --> FixedUpdate
   FixedUpdate --> PostUpdate 
   state PreUpdate {
      Receive --> ReceiveFlush
      ReceiveFlush --> Prediction
      ReceiveFlush --> Interpolation
   }
   state Prediction {
      SpawnPrediction --> SpawnPredictionFlush
      SpawnPredictionFlush --> SpawnPredictionHistory
      SpawnPredictionHistory --> SpawnPredictionHistoryFlush
      SpawnPredictionHistoryFlush --> CheckRollback
      CheckRollback --> CheckRollbackFlush
      CheckRollbackFlush --> Rollback
   }
   state Interpolation {
       SpawnInterpolation --> SpawnInterpolationFlush
       SpawnInterpolationFlush --> SpawnInterpolationHistory
       SpawnInterpolationHistory --> SpawnInterpolationHistoryFlush
       SpawnInterpolationHistoryFlush --> Despawn
       Despawn --> DespawnFlush
       DespawnFlush --> Interpolate
   }
   state FixedUpdate {
      TickUpdate --> BufferInputs
      BufferInputs --> WriteInputEvent
      WriteInputEvent --> Main
      Main --> ClearInputEvent
      Main --> FixedUpdatePrediction
   }
   state FixedUpdatePrediction {
      PredictionEntityDespawn --> PredictionEntityDespawnFlush
      PredictionEntityDespawnFlush --> UpdatePredictionHistory
      UpdatePredictionHistory --> IncrementRollbackTick : if rollback
   }
   state PostUpdate {
        state Send {
            SendEntityUpdates --> SendComponentUpdates
            SendComponentUpdates --> SendInputMessage
            SendInputMessage --> SendPackets
        }
        --
        Sync
   }

Authority

Networked entities can be simulated on a client or on a server. We define by 'Authority' the decision of which peer is simulating an entity. The authoritative peer (client or server) is the only one that is allowed to send replication updates for an entity, and it won't accept updates from a non-authoritative peer.

Only one peer can be the authority over an entity at a given time.

Benefits of distributed client-authority

Client authority means that the client is directly responsible for simulating an entity and sending replication updates for that entity.

Cons:

• high exposure to cheating.
• lower latency Pros:
• less CPU load on the server since the client is simulating some entities

How it works

We have 2 components:

• HasAuthority: this is a marker component that you can use as a filter in queries to check if the current peer has authority over the entity.
  • on clients:
    • a client will not accept any replication updates from the server if it has HasAuthority for an entity
    • a client will send replication updates for an entity only if it has HasAuthority for that entity
  • on server:
    • this component is just used as an indicator for convenience, but the server can still send replication updates even if it doesn't have HasAuthority for an entity. (because it's broadcasting the updates coming from a client)
• AuthorityPeer: this component is only present on the server, and it indicates to the server which peer currently holds authority over an entity. (None, Server or a Client). The server will only accept replication updates for an entity if the sender matches the AuthorityPeer.

Authority Transfer

On the server, you can use the EntityCommand transfer_authority to transfer the authority for an entity to a different peer. The command is simply commands.entity(entity).transfer_authority(new_owner) to transfer the authority of entity to the AuthorityPeer new_owner.

Under the hood, authority transfers do two things:

• on the server, the transfer is applied immediately (i.e. the HasAuthority and AuthorityPeer components are updated instantly)
• than the server sends messages to clients to notify them of an authority change. Upon receiving the message, the client will add or remove the HasAuthority component as needed.

Implementation details

• There could be a time where both the client and server have authority at the same time

  • server is transferring authority from itself to a client: there is a period of time where no peer has authority, which is ok.

  • server is transferring authority from a client to itself: there is a period of time where both the client and server have authority. The client's updates won't be accepted by the server because it has authority, and the server's updates won't be accepted by the client because it has authority, so no updates will be applied.

  • server is transferring authority from client C1 to client C2:

    • if C1 receives the message first, then for a short period of time no client has authority, which is ok
    • if C2 receives the message first, then for a short period of time both clients have authority. However the AuthorityPeer is immediately updated on the server, so the server will only accept updates from C2, and will discard the updates from C1.

• We have to be careful on the server about how updates are re-broadcasted to other clients. If a client 1 has authority and the server broadcasts the updates to all entities, we keep the ReplicationTarget as NetworkTarget::All (it would be tedious to keep track of how the replication target needs to be updated as we change authority again), but instead the server never sends updates to the client that has authority.

• One thing that we have to be careful about is that lightyear used to only apply entity mapping on the receiver side. The reason is that the receiver receives a 'Spawn' message with the remote entity id so it knows how to map from the local to the remote id. In this case, the authority can now be transferred to the receiver. The receiver will now send replication updates, but the peer who was originally the spawner of the entity doesn't have an entity mapping. This means that the new sender (who was originally the receiver) must do the entity mapping on the send side.

  • the Entity in EntityUpdates or EntityActions can now be mapped by the sender, if there is a mapping detected in local_to_remote entity map
  • the entity mappers used on the send side and the receiver side are not the same anymore. To avoid possible conflicts, on the send side we flip a bit to indicate that we did a local->remote mapping so that the receiver doesn't potentially reapply a remote->local mapping. The send entity_map flips the bit, and the remote entity_map checks the bit.
  • since we are now potentially doing entity mapping on the send side, we cannot just replicate a component &C because we might have to update the component to do entity mapping. Therefore if the component implements MapEntities, we clone it first and then apply entity mapping.
    • TODO: this is potentially inefficient because it should be quite rare that the sender needs to do entity mapping (it's only if the authority over an entity was transferred). However components that contain other entities should change pretty infrequently so this clone should be ok. Still, it would be nice if we could avoid it

• We want the Interpolated entity to still get updated even if the client has authority over the Confirmed entity. To do this, we populate the ConfirmedHistory with the server's updates when we don't have authority, and with the client's Confirmed updates if we have authority. This makes sense because Interpolated should just interpolate between ground truth states.

TODO:

• what to do with prepredicted?
  • client spawns an entity with PrePredicted
  • server receives it, adds Replicate
  • currently: server replicates a spawn, which will become the Confirmed entity on the client.
    • if the Spawn has entity mapping, then we're screwed! (because it maps to the client entity)
    • if the Spawn has no entity mapping, but the Components don't, we're screwed (it will be interpreted as 2 different actions)
    • sol 1: use the local entity for bookkeeping and apply entity mapping at the end for the entire action. If the action has a spawn, no mapping. (because it's a new entity)
    • sol 2: we change how PrePredicted works. It spawns a Confirmed AND a Predicted on client; and replicates the Confirmed. Then the server transfers authority to the client upon receipt.
• test with conflict (both client and server spawn entity E and replicate it to the remote)

TODO:

• maybe let the client always accept updates from the server, even if the client has HasAuthority? What is the goal of disallowing the client to accept updates from the server if it has HasAuthority?
• maybe include a timestamp/tick to the ChangeAuthority messages so that any in-flight replication updates can be handled correctly?
  • authority changes from C1 to C2 on tick 7. All updates from C1 that are previous to tick 7 are accepted by the server. Updates after that are discarded. We receive updates from C2 as soon as it receives the ChangeAuthority message.
  • authority changes from C1 to S on tick 7. All updates from C1 that are previous to tick 7 are accepted by the server.
• how do we deal with Predicted?
  • if Confirmed has authority, we probably want to disable rollback and set the predicted state to be equal to the confirmed state?
  • ideally, if an entity is client-authoritative, then it should interact with 0 delay with the client predicted entities. But currently only the Confirmed entity would get the Authority. Would we also want to sync the HasAuthority component so that it gets added to Predicted?
• maybe have an API request_authority where the client requests the authority? and receives a response from the server telling it if the request is accepted or not? Look at this page: https://docs-multiplayer.unity3d.com/netcode/current/basics/ownership/

Bandwidth management

By default, lightyear sends all messages (created by the user, or messages created from replication updates) every send_interval (this interval is configurable) without any regard for the bandwidth available to the client.

But in some situations you might want to limit the bandwidth used by the client or the server, for example to limit server traffic costs, or because the client's connection cannot handle a very high bandwidth.

This page will explain how to do that. There are several options to choose from.

Limiting the number of replication objects

The simplest thing you can do is to carefully choose which entities and components you need to replicate. For example, rendering-related components (particles, assets, etc.) do not need to spawned on the server and replicated to the client. They can be created on the client and only the necessary information (position, rotation, etc.) can be replicated.

This also saves CPU costs on the server.

Updating the send interval

Another thing you can do is to update the send_interval of the client or server. This will reduce the number of times the SystemSet::Send systems will run. This SystemSet is responsible for aggregating all the messages that were buffered and are ready to send, as well as generating all the replication-messages (entity-actions, entity-updates) that should be sent.

NOTE: Currently lightyear expects send_interval to be 0 on the client (i.e. the client sends all updates immediately) to manage client inputs properly.

This will also reduce the CPU usage of the server as it runs the replication-send logic less often.

TODO: Updating the replication rate per replication group

You can also override the replication rate per replication group. For some entities it might not be important to run replication at a very high rate, so you can reduce the rate for those entities.

NOTE: this is currently not possible

Prioritizing replication groups

Even so, there might be situations where you have more messages to send than the bandwidth available to you. In that case you can set a priority to indicate which messages are important and should be sent first.

Every time the server (or client) is ready to send messages, it will first:

• aggregate the list of messages that should be sent
• then sort them by priority. The priority is computed with the formula channel_priority * message_priority.
• it will send messages in order of priority until all the bandwidth is used
• it will then discard the remaining messages
  • note that this means that discarded messages via an unreliable channel will simply not be sent
  • for entity updates, we still try to send an update until the remote world is consistent with the local world, so we will keep trying sending updates until we receive an ack from the remote that it received the updates.

Only the relative priority values matter, not their absolute value: an entity with priority 10 will be replicated twice as often as an entity with priority 5.

To avoid having some replication groups entities be starved of updates (because their priority is always too low), we do priority accumulation:

• every send_interval, we accumulate the priority of all messages: accumulated_priority += priority
• if a replication groups successfully sends an update or an action, we reset the accumulated priority to 0. (note that it's not guaranteed that the message was received by the remote, just that the message was sent)
• for reliable channels, we also keep accumulating the priority until we receive an ack from the remote that the message was successfully received

Replication Logic

This page explains how replication works and what guarantees can be made.

Replication makes a distinction between:

• Entity Actions (entity spawn/despawn, component insert/remove): these events change the archetype of an entity
• Entity Updates (component update): these events don't change the archetype of an entity but simply update the value of some components. Most (90%+) replication messages should be Entity Updates.

Those two are handled differently by the replication system.

Invariants

There are certain invariants/guarantees that we wish to maintain with replication.

Rule #1a: we would like a replicated entity to be in a consistent state compared to what it was on the server: at no point do we want a situation where a given component is on tick T1 but another component of the same entity is on tick T2. The replicated entity should be equal to a version of the remote entity in the past. Similarly, we would not want one component of an entity to be inserted later than other components. This could be disastrous because some other system could depend on both components being present together!

Rule #2: we want to be able to extend this guarantee to multiple entities. I will give two relevant examples:

• client prediction: for client-prediction, we want to rollback if a receives server-state doesn't match with the predicted history. If we are running client-prediction for multiple entities that are not in the same tick, we could have situations where we need to rollback one entity starting from tick T1 and another entity starting from tick T2. This can be fairly hard to achieve, so we'd like to have all predicted entities be on the same tick.
• hierarchies: some entities have relationships. For example you could have an entity with a component Head, and an entity Body with a component HasParent(Entity) which points to the Head entity. If we want to replicate this hierarchy, we need to make sure that the Head entity is replicated before the Body entity. (otherwise the Entity pointed to in HasParent would be invalid on the client). Therefore we need to make sure that all updates for both the parent and the head are in sync.

The only way to guarantee that these rules are respected is to send all the updates for a given "replication group" as a single message. (if we send multiple messages, they could be added to multiple packets, and therefore arrive in a different time/order on the client because of jitter and packet loss)

Lightyear introduces the concept of a ReplicationGroup which is a group of entity whose EntityActions and EntityUpdates will be sent over the network as a single message. It is guaranteed that the state of all entities in a given ReplicationGroup will be consistent on the client, i.e. will be equivalent to the state of the group on the server at a given previous tick T.

Entity Actions

For each ReplicationGroup, Entity Actions are replicated in an OrderedReliable manner.

Send

Whenever there are any actions for a given ReplicationGroup, we send them as a single message AND we include any updates for this group as well. This is to guarantee consistency; if we sent them as 2 separate messages, the packet containing the updates could get lost and we would be in an inconsistent state. Each message for a given [ReplicationGroup] is associated with a message id (a monotonically increasing number) that is used to order the messages on the client.

Receive

On the receive side, we buffer the EntityActions that we receive, so that we can read them in order (message id 1, 2, 3, 4, etc.) We keep track of the next message id that we should receive.

Entity Updates

Send

We gather all updates since the last time we got an ACK from the client that the EntityUpdates was received

The reason for this is:

• we could be gathering all the component changes since the last time we sent EntityActions, but then it could be wasteful if the last time we had any entity actions was a long time ago and many components got updated since.
• we could be gathering all the component changes since the last time we sent a message, but then we could have a situation where:
  • we send changes for C1 on tick 1
  • we send changes for C2 on tick 2
  • packet for C1 gets lost, and we apply the C2 changes -> the entity is now in an inconsistent state at C2

Receive

For each ReplicationGroup, Entity Updates are replicated in a SequencedUnreliable manner. We have some additional constraints:

• we only apply EntityUpdates if we have already applied all the EntityActions for the given ReplicationGroup that were sent when the Updates were sent.
  • for example we send A1, U2, A3, U4; we receive U4 first, but we only apply it if we have applied A3, as those are the latest EntityActions sent when U4 was sent
• if we received a more recent update that can be applied, we discard the older one (Sequencing)
  • for example if we send A1, U2, U3 and we receive A1 then U3, we discard U2 because it is older than U3

Input handling

Lightyear handles inputs for you by:

• buffering the last few inputs on both client and server
• re-using the inputs from past ticks during rollback
• sending client inputs to the server with redundancy

Client-side

Input handling is currently done in the FixedUpdate schedule.

There are multiple SystemSets involved that should run in the following order:

• BufferInputs: the user must add their inputs for the given tick in this system set
• WriteInputEvents: we get the inputs for the current tick and return them as the bevy event InputEvent<I>
  • notably, during rollback we get the inputs for the older rollback tick
• ClearInputEvents: we clear the bevy events.
• SendInputMessage: we prepare a message with the last few inputs. For redundancy, we will send the inputs of the last few frames, so that the server can still get the correct input for a given tick even if some packets are lost.

Server-side

Input handling is also done in the FixedUpdate schedule. These are the relevant SystemSets:

• WriteInputEvents: we receive the input message from the client, add the inputs into an internal buffer. Then in this SystemSet we retrieve the inputs for the current tick for the given client. The retrieved inputs will be returned as InputEvent<I>
• ClearInputEvents: we clear the events

Interpolation

Introduction

Interpolation means that we will store replicated entities in a buffer, and then interpolate between the last two states to get a smoother movement.

See this excellent explanation from Valve: link or this one from Gabriel Gambetta: link

Implementation

In lightyear, interpolation can be automatically managed for you.

Every replicated entity can specify to which clients it should be interpolated to:

Replicate {
    interpolation_target: NetworkTarget::AllExcept(vec![id]),
    ..default()
},

This means that all clients except for the one with id id will interpolate this entity. In practice, it means that they will store in a buffer the history for all components that are enabled for Interpolation.

Component Sync Mode

Not all components in the protocol are necessarily interpolated. Each component can implement a ComponentSyncMode that defines how it gets handled for the Predicted and Interpolated entities.

Only components that have ComponentSyncMode::Full will be interpolated.

Interpolation function

By default, the implementation function for a given component will be linear interpolation. It is also possibly to override this behaviour by implementing a custom interpolation function.

Here is an example:

    #[derive(Component, Message, Serialize, Deserialize, Debug, PartialEq, Clone)]
    pub struct Component1(pub f32);
    #[derive(Component, Message, Serialize, Deserialize, Debug, PartialEq, Clone)]
    pub struct Component2(pub f32);

    #[component_protocol(protocol = "MyProtocol")]
    pub enum MyComponentProtocol {
        #[sync(full)]
        Component1(Component1),
        #[sync(full, lerp = "MyCustomInterpFn")]
        Component2(Component2),
    }

    // custom interpolation logic
    pub struct MyCustomInterpFn;
    impl<C> InterpFn<C> for MyCustomInterpFn {
        fn lerp(start: C, _other: C, _t: f32) -> C {
            start
        }
    }

You will have to add the attribute lerp = "TYPE_NAME" to the component. The TYPE_NAME must be a type that implements the InterpFn trait.

pub trait InterpFn<C> {
    fn lerp(start: C, other: C, t: f32) -> C;
}

Complex interpolation

In some cases, the interpolation logic can be more complex than a simple linear interpolation. For example, we might want to have different interpolation functions for different entities, even if they have the same component type. Or we might want to do interpolation based on multiple comments (applying some cubic spline interpolation that relies not only on the position, but also on the velocity and acceleration).

In those cases, you can disable the default per-component interpolation logic and provide your own custom logic. rust,noplayground

Client-side Prediction

Introduction

Client-side prediction means that some entities are on the 'client' timeline instead of the 'server' timeline: they are updated instantly on the client.

The way that it works in lightyear is that for each replicated entity from the server, the client can choose to spawn 2 entities:

• a Confirmed entity that simply replicates the server updates for that entity
• a Predicted entity that is updated instantly on the client, and is then corrected by the server updates

The main difference between the two is that, if you do an action on the client (for example move a character), the action will be applied instantly on the Predicted entity, but will be applied on the Confirmed entity only after the server executed the action and replicated the result back to the client.

Wrong predictions and rollback

Sometimes, the client will predict something, but the server's version won't match what the client has predicted. For example the client moves their character by 1 unit, but the server doesn't move the character because it detects that the character was actually stunned by another player at that time and couldn't move. (the client could not have predicted this because the 'stun' action from the other player hasn't been replicated yet).

In those cases the client will have to perform a rollback. Let's say the client entity is now at tick T', but the client is only receiving the server update for tick T. (T < T') Every time the client receives an update for the Confirmed entity at tick T, it will:

• check for each updated component if it matches what the predicted version for tick T was
• if it doesn't, it will restore all the components to the confirmed version at tick T
• then the client will replay all the systems for the predicted entity from tick T to T'

Pre-predicted entities

In some cases, you might want to spawn a player-controlled entity right away on the client, without waiting for it to be replicated from the server. In this case, you can spawn an entity directly on the client with the PrePredicted component. You also need to add Replicate on the client entity so that the entity gets replicated to the server.

Then on the server, you can replicate back the entity to the original client. The client will receive the entity, but instead of spawning a new separate Predicted entity, it will re-use the existing entity that had the PrePredicted component!

Edge cases

Component removal on predicted

Client removes a component on the predicted entity, but the server doesn't remove it. There should be a rollback and the client should re-add that component on the predicted entity.

Status: added unit test. Need to reconfirm that it works.

Component removal on confirmed

Server removes a component on the confirmed entity, but the Predicted entity had that component. There should be a rollback where the component gets removed from the Predicted entity.

Status: added unit test. Need to reconfirm that it works.

Component added on predicted

The client adds a component on the Predicted entity, but the Confirmed entity doesn't add it. There should be a rollback and that component gets removed from the Predicted entity.

Status: added unit test. Need to reconfirm that it works.

Component added on confirmed

The server receives an update where a new component gets added to the Confirmed entity. If it was not also added on the Predicted entity, there should be a rollback, where the component gets added to the Confirmed entity.

Status: added unit test. Need to reconfirm that it works.

Pre-predicted entity gets spawned

See more information in the client-replication section.

Status:

• the pre-predicted entity get spawned. Upon server replication, we re-use it as Predicted entity: no unit tests but tested in an example that it works.
• the pre-predicted entity gets spawned. The server doesn't agree that an entity should be spawned, the pre-spawned entity should get despawned: not handled currently.

Confirmed entity gets despawned

We never want to directly modify the Confirmed entity on the client; the Confirmed entity will get despawned only when the server despawns the entity and the despawn is replicated.

When that happens:

• Then the predicted entity should get despawned as well.
• Pre-predicted entities should still get attached to the confirmed entity on spawn, become Predicted entities and get despawned only when the confirmed entity gets despawned.

Status: no unit tests but tested in an example that it works.

Predicted entity gets despawned

There are several options:

OPTION A: Despawn predicted immediately but leave the possibility to rollback and re-spawn it.

We could despawn the predicted entity immediately on the client timeline. If it turns out that the server doesn't despawn the confirmed entity, we then have to rollback and re-spawn the predicted entity with all its components. We can achieve this by using the trait

pub trait PredictionCommandsExt {
    fn prediction_despawn<P: Protocol>(&mut self);
}

that is implemented for EntityCommands. Instead of actually despawning the entity, we will just remove all the synced components, but keep the entity and the components' histories. If it turns out that the confirmed entity was not despawned, we can then rollback and re-add all the components for that entity.

The main benefit is that this is very responsive: the entity will get despawned immediately on the client timeline, but respawning it (during rollback) can be jarring. This can be improved somewhat by animations: instead of the entity disappearing it can just start a death animation. If the death is cancelled, we can simply cancel the animation.

Status:

• predicted despawn, server doesn't despawn, rollback: no unit tests but tested in an example that it works.
  • TODO: this needs to be improved! See note below.
  • NOTE: the way it works now is not perfect. We rely on getting a rollback (where we can see that the confirmed entity does not match the fact that the predicted entity was despawned). However we only initiate rollbacks on receiving server updates, and it's possible that we are not receiving any updates because the confirmed entity is not changing, or because of packet loss! One option would be that predicted_despawn sends a message Re-Replicate(Entity) to the server, which will answer back by replicating the entity again. Let's wait to see how big of an issue this is first.
• predicted despawn, server despawns, we should not rollback but instead despawn both confirmed/predicted when the server despawn gets replicated: no unit tests but tested in an example that it works

OPTION B: despawn the confirmed entity and wait for that to be replicated

If we want to avoid the jarring effect of respawning the entity, we can instead wait for the server to confirm the despawn. In that case, we will just wait for the Confirmed entity to get despawned. When that despawn is propagated, the client entity will despawned as well.

Status: no unit tests but tested in example.

There is no jarring effect, but the despawn will be delayed by 1 RTT.

OPTION C: despawn predicted immediately and don't allow rollback

If you don't care about rollback and just want to get rid of the Predicted entity, you can just call despawn on it normally.

Status: no unit tests but tested in example.

Pre-predicted entity gets despawned

Same thing as Predicted entity getting despawned, but this time we are despawning the pre-predicted entity before we even received the server's confirmation. (this can happen if the entity is spawned and despawned soon after)

Status:

• pre-predicted despawn before we have received the server's replication, server doesn't despawn, rollback:
  • no unit tests but tested in an example that it works
  • TODO: same problem as with normal predicted entities: only works if we get a rollback, which is not guaranteed
• pre-predicted despawn before we have received the server's replication, server despawns, no rollback:
  • the Predicted entity should visually get despawned (all components removed). When the server entity gets replicated, it should start re-using the Predicted entity, initiate a rollback, and see at the end of the rollback that the entity should indeed be despawned.
  • no unit tests but tested in an example that it works

Visual Interpolation

Usually, you will want your simulation (physics, etc.) to run during the FixedUpdate schedule.

The reason is that if you run this logic during the Update schedule, the simulation will run at a rate that is influenced by the frame rate of the client. This can lead to inconsistent results between clients: a beefy machine might have a higher FPS, which would translate into a "faster" game/simulation.

This article by GafferOnGames talks a bit more about this: Fix Your Timestep

The issue is that this can cause the movement of your entity to appear jittery (if the entity only moves during the FixedUpdate schedule). There could be frames where the FixedUpdate schedule does not run at all, and frames where the FixedUpdate schedule runs multiple times in a row.

To solve this, lightyear provides the VisualInterpolation plugin.

The plugin will take care of interpolating the position of the entity between the last two FixedUpdate ticks, thus making sure that the entity is making smooth progress on every frame.

Three Approaches to Visual Interpolation

How Lightyear Does it

lerp(previous_tick_value, current_tick_value, time.overstep_fraction())

Using the time overstep, it lerps between the current and previous value generated during FixedUpdate ticks in accordance with how much time has passed.

The interpolated values are written during PostUpdate (see below). The original / canonical value, which was typically set by the physics logic in FixedUpdate, is stored, to be written back to the component in PreUpdate on the next tick. This means the rendering code should "just work" without being aware interpolation is happening.

PROS:

• relatively simple to implement

CONS:

• introduces a visual delay of 1 simulation tick
• need to store the previous and current value (so extra component clones)

Alternative A

lerp(current_tick_value, future_tick_value, time.overstep_fraction())

Simulate an extra step during FixedUpdate to compute the future_tick_value, then interpolate between the current_tick_value and the future_tick_value

PROS:

• simulation completely up-to-date, and accurate (if we have inputs for the future tick)

CONS:

• could be less accurate in some cases (inputs didn't arrive in time)
• need to store the previous and current value (so extra component clones)

Alternative B

Do not interpolate, but instead run the simulation (FixedUpdate schedule) for one 'partial' tick, i.e. we use (time.overstep_fraction() * fixed_timestep) as the timestep for the simulation.

PROS:

• no visual delay
• no need to store copies of the components

CONS:

• we might run many extra simulation steps if we run an extra partial step in every frame

VisualInterpolationPlugin systems

There are 3 main systems:

• during FixedUpdate, we run update_visual_interpolation_status after the FixedUpdate::Main set (meaning that the simulation has run). In that system we keep track of the value of the component on the current tick and the previous tick
• during PostUpdate, we run visual_interpolation which will interpolate the value of the component between the last two tick values stored in the update_visual_interpolation_status system. We use the Time<Fixed>::overstep_percentage() to determine how much to interpolate between the two ticks
• during PreUpdate, we run restore_from_visual_interpolation to restore the component value to what it was before visual interpolation. This is necessary because the interpolated value is not the "real" value of the component for the simulation, it's just a visual representation of the component. We need to restore the real value before the simulation runs again.

Example

• you have a component that gets incremented by 1.0 at every fixed update step (and starts at 0.0)
• the fixed-update step takes 9ms, and the frame takes 12ms

Frame 0:

• tick is 0, the component is at 0.0

Frame 1:

• We run FixedUpdate once in this frame:
  • tick is 1
  • update_visual_interpolation_status: set current_value for tick 1 is 1.0, previous_value is None
• visual_interpolation: we do not interpolate because we don't have 2 ticks to interpolate between yet. So the component value is 1.0
• the time is at 12ms, the overstep_percentage is 0.33

Frame 2:

• restore_from_visual_interpolation: we restore the component to 1.0
• We run FixedUpdate once in this frame:
  • tick is 2
  • update_visual_interpolation_status: set current_value for tick 2 is 2.0, previous_value is 1.0
• visual_interpolation: the time is 24ms, the overstep percentage is 0.667, we interpolate between 1.0 and 2.0 so the component is now at 1.667

Frame 3:

• restore_from_visual_interpolation: we restore the component to 2.0
• We run FixedUpdate twice in this frame:
  • tick is 3
  • update_visual_interpolation_status: set current_value for tick 3 is 3.0, previous_value is 2.0
  • tick is 4
  • update_visual_interpolation_status: set current_value for tick 4 is 4.0, previous_value is 3.0
• visual_interpolation: the time is 36, the overstep percentage is 0.0, we interpolate between 3.0 and 4.0 so the component is now at 3.0

Frame 4:

• restore_from_visual_interpolation: we restore the component to 4.0
• We run FixedUpdate once in this frame:
  • tick is 5
  • update_visual_interpolation_status: set current_value for tick 5 is 5.0, previous_value is 4.0
• visual_interpolation: the time is 48, the overstep percentage is 0.33, we interpolate between 5.0 and 4.0 so the component is now at 4.33

So overall the component value progresses by 1.33 every frame, which is what we expect because a frame duration (12ms) is 1.33 times the fixed update duration (9ms).

Usage

Visual interpolation is currently only available per component, and you need to enable it by adding a plugin:

app.add_plugins(VisualInterpolationPlugin::<Position>::default());

You will also need to add the VisualInterpolateState component to any entity you want to enable visual interpolation for:

fn spawn_entity(mut commands: Commands) {
    commands.spawn().insert(VisualInterpolateState::<Position>::default());
}

Usage with Avian Physics and FixedUpdate

Here's how you might enable Visual Interpolation for Avian's Position and Rotation:

app.add_plugins(VisualInterpolationPlugin::<Position>::default());
app.add_plugins(VisualInterpolationPlugin::<Rotation>::default());

app.observe(add_visual_interpolation_components::<Position>);
app.observe(add_visual_interpolation_components::<Rotation>);

// ...

fn add_visual_interpolation_components<T: Component>(
    trigger: Trigger<OnAdd, T>,
    q: Query<&RigidBody, With<T>>,
    mut commands: Commands,
) {
    let Ok(rigid_body) = q.get(trigger.entity()) else {
        return;
    };
    // No need to interp static bodies
    if matches!(rigid_body, RigidBody::Static) {
        return;
    }
    // triggering change detection necessary for SyncPlugin to work
    commands
        .entity(trigger.entity())
        .insert(VisualInterpolateStatus::<T> {
            trigger_change_detection: true,
            ..default()
        });
}

If you draw your entities with gizmos based on their Position and Rotation components, this is all you need.

However, if you have meshes, sprites, or anything that depends on Transform (as is the norm), you need to ensure that changes made by the visual interpolation systems are synched to the transforms.

Avian's SyncPlugin does this for you, but beware if you use FixedUpdate, you need to run the SyncPlugin in PostUpdate, otherwise you won't be syncing changes correctly. Visual interp happens in PostUpdate, even if physics runs in FixedUpdate.

// Run physics in FixedUpdate, but run the SyncPlugin in PostUpdate
app
  .add_plugins(
    PhysicsPlugins::new(FixedUpdate)
      .build()
      .disable::<SyncPlugin>(),
  )
  .add_plugins(SyncPlugin::new(PostUpdate));

Be aware that moving the SyncPlugin to PostUpdate could cause issues if you rely on modifying Transforms during FixedUpdate – they will no longer be synced back to the Avian components during FixedUpdate. If you manipulate physics objects by changing the Avian components directly, it should be fine.

Avian's SyncConfig

Using Avian's SyncConfig you can control how position and transform are synced. You might wish to disable transform_to_position, depending on how you game is built.

Caveats

• The VisualInterpolationPlugin is currently only available for components that are present in the protocol. This is because the plugin needs to know how interpolation will be performed. The interpolation function is registered on the protocol directly instead of the component to circumvent the orphan rule (we want users to be able to define a custom interpolation function for external components)

  • NOTE: This will probably be changed in the future, by letting the user provide a custom interpolation function when creating the plugin.

• The interpolation doesn't progress at the same rate at the very beginning, because we wait until there are two ticks available before we start doing the interpolation. (you can see in the example above that on the first frame the component is 1.0, and on the second frame 1.667, so the component value didn't progress by 1.33 like it will in the other frames).

  • NOTE: it's probably possible to avoid this by just not displaying the component until we have 2 ticks available, but I haven't implemented this yet.

Prespawning

Introduction

There are several types of entities you might want to create on the client (predicted) timeline:

• normal ("delayed") predicted entities: they are spawned on the server and then replicated to the client. The client creates a Confirmed entity, and performs a rollback to create a corresponding Predicted entity on the client timeline.
• client-issued pre-predicted entities: the entity is first created on the client, and then replicated to the server. The server then receives the entity, decides if it's valid, and if so, replicates back the original client.
• prespawned entities: the entity is created on the client (in the predicted timeline) and the server using the same system. When the server replicates the entity back to the client, the client creates a Confirmed entity, but instead of creating a new Predicted entity, it re-uses the pre-spawned entity.

This section focuses about the third type of predicted entity: prespawned entities.

How does it work

You can find an example of prespawning in the prespawned example.

Let's say you want to spawn a bullet when the client shoots. You could just spawn the bullet on the server and wait for it to be replicated + predicted on the client. However that would introduce a delay between clicking on the 'shoot' button and seeing the bullet spawned.

So instead you run the same system on the client to prespawn the bullet in the predicted timeline. The only thing you need to do is add the PreSpawnedPlayerObject component to the entity spawned (on both the client and server).

commands.spawn((BulletBundle::default(), PreSpawnedPlayerObject));

That's it!

• The client will assign a hash to the entity, based on its components and the tick at which it was spawned. You can also override the hash to use a custom one.
• When the client receives a server entity that has PreSpawnedPlayerObject, it will check if the hash matches any of its pre-spawned entities. If it does, it will remove the PreSpawnedPlayerObject component and add the Predicted component. If it doesn't, it will just spawn a normal predicted entity.

In-depth

The various system-sets for prespawning are:

• PreUpdate schedule:

  • PredictionSet::SpawnPrediction: we first run the prespawn match system to match the pre-spawned entities with their corresponding server entity. If there is a match, we remove the PreSpawnedPlayerObject component and add the Predicted/Confirmed components. We then run an apply_deferred, and we run the normal predicted spawn system, which will skip all confirmed entities that already have a predicted counterpart (i.e. were matched)

• FixedUpdate schedule:

  • FixedUpdate::Main: prespawn the entity
  • FixedUpdate::SetPreSpawnedHash: we compute the hash of the prespawned entity based on its archetype (only the components that are present in the ComponentProtocol) + spawn tick. We store the hash and the spawn tick in the PredictionManager (not in the PreSpawnedPlayerObject component).
  • FixedUpdate::SpawnHistory: add a PredictionHistory for each component of the pre-spawned entity. We need this to:
    • not rollback immediately when we get the corresponding server entity
    • do rollbacks correctly for pre-spawned entities

• PostUpdate schedule:

  • we cleanup any pre-spawned entity no the clients that were not matched with any server entity. We do the cleanup when the (spawn_tick - interpolation_tick) * 2 ticks have elapsed. Normally at interpolation tick we should have received all the matching replication messages, but it doesn't seem like it's the case for some reason.. To be investigated.

One thing to note is that we updated the rollback logic for pre-spawned entities. The normal rollback logic is:

• we receive a confirmed update
• we check if the confirmed update matches the predicted history
• if not, we initiate a rollback, and restore the predicted history to the confirmed state. (Thanks to replication group, all components of all entities in the replication group are guaranteed to be on the same confirmed tick)

However for pre-spawned entities, we do not have a confirmed entity yet! So instead we need to rollback to history of the pre-spawned entity.

• we compute the prediction history of all components during FixedUpdate
• when we have a rollback, we also rollback all prespawned entities to their history
• Edge cases:
  • if the prespawned entity didn't exist at the rollback tick, we despawn it
  • if a component didn't exist at the rollback tick, we remove it
  • if a component existed at the rollback tick but not anymore, we re-spawn it
  • TODO: if the preentity existed at the rollback tick but not anymore, we re-spawn it This one is NOT handled (or maybe it is via prediction_despawn(), check!)

Caveats

There are some things to be careful of:

• the entity must be spawned in a system that runs in the FixedUpdate::Main SystemSet, because only then are you guaranteed to have exactly the same tick between client and server.
  • If you spawn the prespawned entity in the Update schedule, it won't be registered correctly for rollbacks, and also the tick associated with the entity spawn might be incorrect.

Interest management

Interest management is the concept of only replicating to clients the entities that they need.

For example: in a MMORPG, replicating only the entities that are "close" to the player.

There are two main advantages:

• bandwidth savings: it is pointless to replicate entities that are far away from the player, or that the player cannot interact with. Those bandwidth savings become especially important when you have a lot of concurrent connected clients.
• prevent cheating: if you replicate entities that the player is not supposed to see, there is a risk that clients read that data and use it to cheat. For example, in a RTS, you can avoid replicating units that are in fog-of-war.

Implementation

VisibilityMode

The first step is to think about the NetworkRelevanceMode of your entities. It is defined on the Replicate component.

#[derive(Default)]
pub enum VisibilityMode {
  /// We will replicate this entity to all clients that are present in the [`NetworkTarget`] AND use relevance on top of that
  InterestManagement,
  /// We will replicate this entity to all clients that are present in the [`NetworkTarget`]
  #[default]
  All
}

If NetworkRelevanceMode::All, you have a coarse way of doing interest management, which is to use the replication_target to specify which clients will receive client updates. The replication_target is a NetworkTarget which is a list of clients that we should replicate to.

In some cases, you might want to use NetworkRelevanceMode::InterestManagement, which is a more fine-grained way of doing interest management. This adds additional constraints on top of the replication_target, we will never send updates for a client that is not in the replication_target of your entity.

Interest management

If you set NetworkRelevanceMode::InterestManagement, we will add a ReplicateVisibility component to your entity, which is a cached list of clients that should receive replication updates about this entity.

There are several ways to update the relevance of an entity:

• you can either update the relevance directly with the RelevanceManager resource
• we also provide a more static way of updating the relevance with the concept of Rooms and the RoomManager resource.

Immediate relevance update

You can simply directly update the relevance of an entity/client pair with the RelevanceManager resource.

#![allow(unused)]
fn main() {
use bevy::prelude::*;
use lightyear::prelude::*;
use lightyear::prelude::server::*;

fn my_system(
    mut relevance_manager: ResMut<RelevanceManager>,
) {
    // you can update the relevance like so
    relevance_manager.gain_relevance(ClientId::Netcode(1), Entity::PLACEHOLDER);
    relevance_manager.lose_relevance(ClientId::Netcode(2), Entity::PLACEHOLDER);
}
}

Rooms

An entity can join one or more rooms, and clients can similarly join one or more rooms.

We then compute which entities should be replicated to which clients by looking at which rooms they are both in.

To summarize:

• if a client is in a room but the entity is not (or vice-versa), we will not replicate that entity to that client
• if the client and entity are both in the same room, we will replicate that entity to that client
• if a client leaves a room that the entity is in (or an entity leaves a room that the client is in), we will despawn that entity for that client
• if a client joins a room that the entity is in (or an entity joins a room that the client is in), we will spawn that entity for that client

This can be useful for games where you have physical instances of rooms:

• a RPG where you can have different rooms (tavern, cave, city, etc.)
• a server could have multiple lobbies, and each lobby is in its own room
• a map could be divided into a grid of 2D squares, where each square is its own room

#![allow(unused)]
fn main() {
use bevy::prelude::*;
use lightyear::prelude::*;
use lightyear::prelude::server::*;

fn room_system(mut manager: ResMut<RoomManager>) {
   // the entity will now be visible to the client
   manager.add_client(ClientId::Netcode(0), RoomId(0));
   manager.add_entity(Entity::PLACEHOLDER, RoomId(0));
}
}

Client replication

It is also possible to use lightyear to replicate entities from the client to the server. There are different possibilities.

Client authoritative

To replicate a client-entity to the server, it is exactly the same as for a server-entity. Just add the Replicate component to the entity and it will be replicated to the server.

#![allow(unused)]
fn main() {
fn handle_connection(
    mut connection_event: EventReader<ConnectEvent>,
    mut commands: Commands,
) {
    for event in connection_event.read() {
        let local_client_id = event.client_id();
        commands.spawn((
            /* your other components here */
            Replicate {
                replication_target: NetworkTarget::All,
                interpolation_target: NetworkTarget::AllExcept(vec![local_client_id]),
                ..default()
            },
        ));
    }
}
}

Note that prediction_target and interpolation_target will be unused as the server doesn't do any prediction or interpolation.

If you want to then broadcast that entity to other clients, you will have to add a Replicate component on the server entity. This should generally happen in the ServerReplicationSet::ClientReplication SystemSet on the server, so that it happens right after receiving the client entity.

Be careful to not replicate the entity back to the original client, as it would create a duplicate entity on the client.

Example flow:

---
title: Client Authoritative
---
sequenceDiagram
    participant Client1
    participant Server
    participant Client2
    participant Client3
    Client1->>Server: Connect()
    Server->>Client1: Connected
    Client1->>Client1: ConnectEvent
    Client1->>Client1: SpawnPredicted(PlayerID: 1)
    Client1->>Server: Replicate(PlayerID: 1)
    Server-->>Client2: Replicate(PlayerID: 1)
    Client2->>Client2: SpawnConfirmed(PlayerID: 1)
    Server-->>Client3: Replicate(PlayerID: 1)
    Client3->>Client3: SpawnConfirmed(PlayerID: 1)

Pre-spawned predicted entities

Sometimes you might want to spawn a predicted entity on the client, but then replicate it to the server and let the server get control over the entity (server-authoritative).

For example you might have a predicted character that spawns another predicted entity (a projectile that other clients should see, for example).

You could wait for the user input to reach the server, then for the server to spawn the projectile and for the projectile to be replicated back to the client, but that would mean a delay of 1-RTT on the client to see the projectile when they pres a button.

A solution is to spawn the predicted projectile on the client; but then replicate back to the server. When the server replicates back the projectile, it creates a Confirmed entity on the client, but will re-use the existing predicted projectile as the Predicted entity.

The way to do this is to add a PrePredicted component on the client entity that you want to predict. On the server, to replicate back the entity to the client, you will need to manually add a Replicate component to the entity to specify to which clients you want to rebroadcast it. Note that you must add the Replicate component in the ServerReplicationSet::ClientReplication SystemSet for proper handling of pre-spawned predicted entities!

When the server replicates back the entity, the client will check if the entity has a PrePredicted component. If not present, that means this is not a pre-spawned Predicted entity, and the client will spawn both the Confirmed and Predicted entities. If it's present, the client will spawn a new Confirmed entity, but will re-use the entity as the Predicted entity.

Note that pre-spawned predicted entities will give authority to the server's entity immediately, the client to server replication will stop immediately after the initial replication, and the server entity should be the authoritative one.

Example flow:

---
title: Client PrePredicted
---
sequenceDiagram
    participant Client1
    participant Server
    participant Client2
    participant Client3
    Client1->>Server: Connect()
    Server->>Client1: Connected
    Client1->>Client1: ConnectEvent
    Client1->>Client1: SpawnPredicted(PlayerID: 1)
    Client1->>Server: Replicate(PlayerID: 1)
    Server-->>Client1: Replicate(PlayerID: 1)
    Client1->>Client1: SpawnConfirmed(PlayerID: 1)
    Server-->>Client2: Replicate(PlayerID: 1)
    Client2->>Client2: SpawnConfirmed(PlayerID: 1)
    Server-->>Client3: Replicate(PlayerID: 1)
    Client3->>Client3: SpawnConfirmed(PlayerID: 1)

Notes to myself:

• one thing to be careful for is that we want to immediately stop replicating updates from the pre-spawned predicted entity to the server; because that entity should be server-authoritative. Right after the first time the Send SystemSet runs, run clean_prespawned_entities to remove Replicate from those entities.
• another thing we have to be careful of is this: let's say we receive on the server a pre-predicted entity with ShouldBePredicted(1). Then we rebroadcast it to other clients. If an entity 1 already exists on other clients; we will start using that entity as our Prediction target! That means that we should:
  • even if pre-spawned replication, require users to set the prediction_target correctly
  • only broadcast ShouldBePredicted to the clients who have prediction_target set.
  • be careful that ShouldBePredicted can be added once during spawn, and once from replication. In that case the second one should win out!

Connecting to a remote server

You've tested your multiplayer locally, and now you want to try connecting to a remote server.

This is a quick guide mostly for myself, as I knew nothing about networking and didn't know where to start.

Set up a server

From what I understand you can either:

• rent a dedicated node with a cloud provider. Smaller cloud providers (Linode, DigitalOcean) seem cheaper than bigger ones (Google Cloud, AWS) unless you have free credits. I used Kamatera, which worked excellently
• rent a VPS, which is a virtual machine on a dedicated node. I believe this means that you are sharing the resources of the nodes with other customers.

You need to get the public ip address of your server: S-IP.

Connect using UDP

On the client:

• You will need to create a local UDP socket that binds your local client ip address
  • I believe that when the server receives a packet on their UDP socket, they will receive the public ip address of the client. By specifying the local client ip address, you enable the server to do NAT-traversal? I'm not sure.
  • You can either manually specify your local client ip address, or you can use 0.0.0.0 (INADDR_ANY), which binds to any of the local ip addresses of your machine. I think that if your computer only has 1 network card, then it will bind to that one. (source: Beej's networking guide)

On the server:

• Same thing, you will need to create UDP socket that binds to your local server ip address.
  • Again, you can either manually specify your local server ip address, or you can use 0.0.0.0 (INADDR_ANY) if your machine only has 1 network card, or if you don't care which of the local ip addresses the socket binds to
  • Keep track of the port used for the server: S-P
• You will need to keep track of the public ip address of your server: S-IP
• Start the lightyear process on the server, for example with cargo run --example interest_management -- server --headless

On the client:

• connect to the server by specifying the server public ip address and the server port: cargo run --example interest_management -- client -c 1 --server-addr=S-IP --server-port=S-P

Appendix