Networking

Needle Engine includes a full networking solution for multiplayer experiences.
A shared world state, voice chat, session persistence, and more can be achieved with our networking components and APIs. You can network your own components with a choice of automatic or manual networking.

Networking in Needle Engine is based on Websockets. Automatic networking uses JSON data for ease of use. For complex usecases and high-performance requirements, we use Flatbuffers.

Access to core networking functionality can be obtained by using this.context.connection from a component. The default backend server connects users to rooms. Users in the same room will share state and receive messages from each other.

Networking Concepts

Rooms and State

At the heart of networking in Needle Engine is the concept of synchronized rooms. Each room has an ID, and users connect to a room by providing this ID. Rooms are stored on a server, and users can join and leave rooms at any time.
When a user joins a room, they receive the current state of the room, apply that current state to their scene, and then listen for changes to the room state.
When a user leaves a room, they stop listening for changes to the room state.

Room state is stored as JSON data on the server, so all changes are persistent. This means that room state is not only useful for networking, but also for persisting actions of a single user.

Needle can provide view-only IDs for rooms. When accessing a room with a view-only ID, the user will not be able to interact with the room, but will be able to see the current state and get live updates. This is useful for presentations or demonstrations.

Ownership

Objects in a room can be owned by a user. This means that only the owner of an object can change its state. By default, objects have no owner. Components like DragControls will request ownership of an object before actually moving it. In custom components, you can control how ownership is handled. There may be no ownership required, ownership may be allowed to be transferred to another user automatically, or ownership may be transferred only by a specific action.

When a user leaves a room, objects owned by that user will either be deleted or have ownership reset, depending on how the object was created.

Enable Networking for your project

1. Add a SyncedRoom component to your scene. By default, this will use networking infrastructure provided by Needle.
2. Add a SyncedTransform component to a object whose movement you want to synchronize across the network.
3. Add a DragControls component to the same object.
4. Run the project. In the browser, click on "Join Room" and copy the URL.
5. Open a new browser window and paste the URL. You should now see the same object in both windows. Try dragging the object in one window and see it move in the other window.

The DragControls component, like many other Needle components, has built-in networking support. Ownership will be transferred to whoever starts dragging the object.

Built-In Components with Networking Support

Automatic Networking for custom Components

Fields in your own components can be networked very easily. Changes to the field will automatically be detected and sent to all users in the room. The changes are also persisted as part of the Room State, so users that join the room later will receive the current state of the field as well, ensuring everyone sees the same data.

To automatically network a field in a component, decorate it with the @syncField() decorator:

::::code-group :::code-group-item Sync a number

import { Behaviour, syncField, IPointerClickHandler } from "@needle-tools/engine"

export class SyncedNumber extends Behaviour implements IPointerClickHandler {

    // Use `@syncField` to automatically network a field. 
    // You can optionally assign a method or method name to be called when the value changes.
    @syncField("myValueChanged")
    mySyncedValue?: number = 1;
    
    private myValueChanged() {
       console.log("My value changed", this.mySyncedValue);
    }
    
    onPointerClick() {
       this.mySyncedValue = Math.random();
    }
}

::: :::code-group-item Sync an object's color

import { Behaviour, IPointerClickHandler, PointerEventData, Renderer, RoomEvents, delay, serializable, showBalloonMessage, syncField } from "@needle-tools/engine";
import { Color } from "three"

export class Networking_ClickToChangeColor extends Behaviour implements IPointerClickHandler {

    // START MARKER network color change syncField
    /** syncField does automatically send a property value when it changes */
    @syncField(Networking_ClickToChangeColor.prototype.onColorChanged)
    @serializable(Color)
    color!: Color;

    private onColorChanged() {
        // syncField will network the color as a number, so we need to convert it back to a Color when we receive it
        if (typeof this.color === "number")
            this.color = new Color(this.color);
        this.setColorToMaterials();
    }
    // END MARKER network color change syncField

    /** called when the object is clicked and does generate a random color */
    onPointerClick(_: PointerEventData) {
        const randomColor = new Color(Math.random(), Math.random(), Math.random());
        this.color = randomColor;
    }

    onEnable() {
        this.setColorToMaterials();
    }

    private setColorToMaterials() {
        const renderer = this.gameObject.getComponent(Renderer);
        if (renderer) {
            for (let i = 0; i < renderer.sharedMaterials.length; i++) {
                // we clone the material so that we don't change the original material
                // just for demonstration purposes, you can also change the original material
                const mat = renderer.sharedMaterials[i]?.clone();
                renderer.sharedMaterials[i] = mat;
                if (mat && "color" in mat)
                    mat.color = this.color;
            }
        }
        else console.warn("No renderer found", this.gameObject)
    }

}

::: ::::

Note that syncField has an optional parameter to specify a method that should be called when the value changes. This method should be defined in the same class.

Creating and destroying objects

Often, you want to create and destroy objects at runtime, and of course these changes should be synchronized across the network.

The PlayerSync component simplifies this process by automatically instantiating a specific object for each connected player. When a player leaves the room, the object is destroyed for all users.

Additionally, Needle Engine provides two high-level methods:

• syncInstantiate() to duplicate objects across the network.
• syncDestroy() to destroy objects across the network.

🏗️ Code Samples Under Construction

Manual Networking

Needle Engine also provides a low-level API for sending and receiving messages. We call this "manual networking". The principles are the same, but you're in full control for sending and receiving messages and how to handle them.

Sending Messages

Send a message to all users in the same room:

this.context.connection.send(key: string, data: IModel | object | boolean | string | number | null);

Receiving Messages

You can subscribe to events in the room using a specific key. Typically, you want to match this with unsubscribing:

• subscribe in onEnable and unsubscribe in onDisable
  With this approach, no messages will be received while the object is disabled.

• or subscribe in start and unsubscribe in onDestroy
  With this approach, messages will still be received while the object is disabled.

this.context.connection.beginListen(key:string, callback:(data) => void)

Unsubscribe from events:

this.context.connection.stopListen(key:string)

Controlling message persistence

When sending network messages, the low-level API allows you to decide whether that message should be persistet (saved in the room state) or not (only sent to users currently in the room). To persist a message, make sure it has a guid field. This field is typically used to apply the message data to a specific object, by providing that object's guid. If you want target a specific object (and thus, include a guid field) but want the data to not be persisted, set the dontSave field to true in your message.

All persistent messages are saved in the room state and will be sent to users that connect at a later point. Non-persistent messages are only sent to users currently in the room, which is useful for effects (like playing a sound effect) that don't make sense to play for users that are currently not in the room. Optionally, you can include a deleteOnDisconnect field in your message to delete this particular message when the user disconnects.

// This message will be sent to all users currently in the room,
// AND to users that join the room later.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue" });

// This message will be sent to all users currently in the room,
// but NOT be sent to users that join the room later.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue", dontSave: true });

// This message will be sent to all users currently in the room,
// but NOT be sent to users that join the room later.
this.context.connection.send("my-message", { myData: "myValue" });

// This message will be sent to all users currently in the room,
// AND to users that join the room later,
// but will be deleted from the room state when the user disconnects.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue", deleteOnDisconnect: true });

To delete state for a specific guid from the backend storage, set the message key to delete-state and target a specific object with its guid: { guid: "guid_to_delete" } .

this.context.connection.send("delete-state", { guid: "guid_to_delete" });

Using debug flags to understand network messages

There are several debug flags that can be used to dive deeper into network messages. These can be appended the the page URL, like https://localhost:3000/?debugnet.

Networking Lifecycle Events

The following events are available to listen to in your components. They describe common network events that you might want to react to in your components, like yourself or another user joining or leaving a room.

// Listen to the event when *you* have joined a networked room
this.context.beginListen(RoomEvents.JoinedRoom, ({room, viewId, allowEditing, inRoom}) => { ... });

// Listen to the event when *you* have left a networked room
this.context.beginListen(RoomEvents.LeftRoom, ({room}) => { ... });

// Listen to the event when *another user* has joined your networked room
this.context.beginListen(RoomEvents.UserJoinedRoom, ({userId}) => { ... });

// Listen to the event when *another user* has left your networked room
this.context.beginListen(RoomEvents.UserLeftRoom, ({userId}) => { ... });

// This event is received after all current room state has been sent to the client
this.context.beginListen(RoomEvents.RoomStateSent, () => { ... });

• See all Room Events in the API docs
• See all Ownership Events in the API docs
• See all Connection Events in the API docs

Using Needle Networking Servers

By default, networked Needle scenes connect to cloud infrastructure managed and provided by Needle. There is no additional setup needed, and currently no additional cost for using this service.

Typically, this will work fine for around 15-20 users in the same room. Once your project matures, you can upgrade to a bigger/better/stronger networking solution, by hosting your own networking server.

Hosting your own Networking Server

You might want to host your own networking server for larger deployments or to have more control over the networking infrastructure and implementation.

Our networking server is available on NPM own networking package as node.js package. The package contains pre-configured integrations for the popular web frameworks Fastify and Express, and can be integrated into other Node.js server frameworks as well.

::::code-group :::code-group-item Fastify

import networking from "@needle-tools/needle-networking";
networking.startServerFastify(fastifyApp, { endpoint: "/socket" });

::: :::code-group-item Express

import networking from "@needle-tools/needle-networking";
networking.startServerExpress(expressApp, { endpoint: "/socket" });

::: :::code-group-item Custom Integration

import { init, onConnection } from "@needle-tools/networking";

// Add your framework-specific websocket implementation here. 
// You can view the fastify and express implementations in server.js for reference.
class WebsocketConnector {
    constructor(frameworkWebsocket) {
        // Your implementation.
    }
    on(event, callback) {
        // Your implementation. When receiving a message in the websocket connection, call the callback.
        // 'event' can be 'message' or 'close'.
    }
    send(key, value) {
        // Your implementation. Pass the message along to the websocket connection.
    }
}
const options = { endpoint: "/socket" };
init(options);
yourFramework.createWebsocketRoute(options.endpoint, frameworkWebsocket => {
    onConnection(new WebsocketConnector(frameworkWebsocket));
});

::: ::::

Configuration

The following options are available:

The networking server will automatically manage connecting and disconnecting users, receiving and sending messages, and persisting room state.

Custom networking servers can be deployed anywhere, for example on Google Cloud. For further instructions please refer to this repository: Local Needle Networking Server

State Storage

Network state is by default stored to disk on the server as JSON files in the /.data directory.
Each room has its own file, and the state is sent to connecting clients when they join a room.

Optionally, the networking state can be stored with an S3 compatible storage provider. Use the following environment variables to enable S3 storage:

NEEDLE_NETWORKING_S3_ENDPOINT=
NEEDLE_NETWORKING_S3_REGION=
NEEDLE_NETWORKING_S3_BUCKET=
NEEDLE_NETWORKING_S3_ACCESS_KEY_ID=
NEEDLE_NETWORKING_S3_ACCESS_KEY=
NEEDLE_NETWORKING_S3_PREFIX= # all state saved in the bucket will be prefixed with this string. This can be a path e.g. `my_state/` or a unique id `server_123_`

Local Networking Server

For testing and development purposes, you can run the Needle Engine networking package on a local server. We have prepared a repository that is set up to host the websocket package and to make that easy for you.

1. Download the local server sample from github.com/needle-tools/networking-local
2. Follow the instructions in the README to set up the server. The server will run on wss://localhost:9001/socket by default.
3. Add the Networking component to your scene.
4. Paste the local server address into the Localhost field on the Networking component.

Advanced: Customizing WebRTC settings for peer.js

Needle Engine Screencapture (Screensharing) and VoIP (Voice communication) components use peer.js for networking audio and video. Peer.js uses WebRTC under the hood.

Needle Engine uses reasonable defaults for peerjs. If you want to modify those defaults, you can call

setPeerOptions(opts: PeerjsOptions);

with your custom settings. This can be used to modify the hosting provider for ICE/STUN/TURN servers, for example when you use your own WebRTC servers.

Advanced: Server and Client Message Formats

Messages are sent in JSON format. They always have a key field that describes the type of message, and a data field that contains the message payload. The data field can be any JSON-serializable object.

Built-in Room Events

::::code-group :::code-group-item Join

// Sent to the server to attempt joining a room.
{
    "key": "join-room",
    "data": {
        "room": string,
        "viewOnly": boolean,
    }
}

::: :::code-group-item Leave

// Sent to the server to leave a room.
{
    "key": "leave-room",
    "data": {
        "room": string
    }
}

::: :::code-group-item JoinedRoom

// Sent to the client when the local user has joined a room.
// Type: JoinedRoomResponse
{
    "key": "joined-room",
    "room": string,
    "viewId": string,
    "allowEditing": boolean,
    "inRoom": string[] // connection IDs
}

::: :::code-group-item LeftRoom

// Sent to the client when the local user has left a room.
// Type: LeftRoomResponse
{
    "key": "left-room",
    "room": string
}

::: :::code-group-item UserJoinedRoom

// Sent to the client when any user has joined a room.
// Type: UserJoinedOrLeftRoomModel
{
    "key": "user-joined-room",
    "data": {
        "userId": string // connection ID
    }
}

::: :::code-group-item UserLeftRoom

// Sent to the client when any user has left a room.
// Type: UserJoinedOrLeftRoomModel
{
    "key": "user-left-room",
    "data": {
        "userId": string // connection ID
    }
}

::: :::code-group-item RoomStateSent

// Sent to the client after the complete room state has been sent.
{
    "key": "room-state-sent",
    "room": string // room name 
}

::: ::::

Built-In Utility Events

::::code-group :::code-group-item ConnectionInfo

// Sent to the client when the connection is established.
{
    "key": "connection-start-info",
    "data": {
        "id": string // connection ID
    }
}

::: :::code-group-item syncInstantiate

// Used by the syncInstantiate() API to create a new instance of an asset.
// Type: NewInstanceModel
{
    "key": "new-instance-created",
    "data": {
        "guid": string,
        "originalGuid": string,
        "seed": number | undefined,
        "visible": boolean | undefined,
        "dontSave": boolean | undefined,

        "parent": string | undefined,
        "position": { x: number, y: number, z: number } | undefined,
        "rotation": { x: number, y: number, z: number, w: number } | undefined,
        "scale": { x: number, y: number, z: number } | undefined,

        "deleteStateOnDisconnect": boolean | undefined
    }

::: :::code-group-item syncDestroy

// Used by the syncDestroy() API to destroy an instance of an asset.
// Type: DestroyInstanceModel
{
    "key": "instance-destroyed",
    "data": {
        "guid": string,
        "dontSave": boolean | undefined
    }
}

::: :::code-group-item Ping

// Sent to the server every few seconds to keep the connection alive.
{
    "key": "ping",
    "data": {}
}

::: :::code-group-item Pong

// Sent by the server in response to a ping.
{
    "key": "pong",
    "data": {}
}

::: :::code-group-item DeleteState

// Sent to the server to delete state for a specific guid.
{
    "key": "delete-state",
    "data": {
        "guid": <string>
    }
}

::: :::code-group-item DeleteAllState

// Sent to the server to delete ALL current room state.
{
    "key": "delete-all-state",
    "data": {}
}

::::

Built-In Ownership Events

::::code-group :::code-group-item OwnershipRequest

{
    "key": 
      "request-has-owner" | 
      "request-ownership" | 
      "remove-ownership",
    "data": {
        "guid": string
    }
}

::: :::code-group-item OwnershipResponse // Type: OwnershipResponse

{
    "key": 
      "response-has-owner",
    "data": {
        "guid": string,
        "value": boolean
    }
}

::: ::: code-group-item OwnershipBroadcastResponse

{
    "key": 
      "gained-ownership" | 
      "lost-ownership" | 
      "gained-ownership-broadcast" | 
      "lost-ownership-broadcast",
    "data": {
        "guid": string,
        "owner": string
    }
}

::: ::::

Built-In Flatbuffer Schemas

Flatbuffer messages are sent directly as binary messages.

::::code-group :::code-group-item SyncedTransform ('STRS')

include "vec.fbs";

struct Transform {
    position:Vec3;
    rotation:Vec3;
    scale:Vec3;
}

table SyncedTransformModel {
    guid:string;
    fast:bool;
    transform: Transform;
    dont_save:bool;
}

root_type SyncedTransformModel;

::: :::code-group-item SyncedCamera ('SCAM')

include "vec.fbs";

table SyncedCameraModel {
    user_id:string;
    guid:string;
    dont_save:bool;
    pos:Vec3;
    rot:Vec3;
}

root_type SyncedCameraModel;

::: :::code-group-item Vec2|3|4

struct Vec2 {
  x:float;
  y:float;
}

struct Vec3 {
  x:float;
  y:float;
  z:float;
}

struct Vec4 {
  x:float;
  y:float;
  z:float;
  w:float;
}

::: ::::

Advanced: Binary Messages in the Flatbuffer format

JSON messages are easy to use and understand, but are typically larger in memory and bandwidth. For large amounts of data, or when sending fast updates, binary messages are faster and more efficient. You can use Flatbuffers messages in Needle Engine for cases where that is required. Using Flatbuffers requires additional setup steps like defining and compiling a message schema, and is harder to debug since you're dealing with binary messages.

Note that when sending and receiving flatbuffer messages, there is no key field – the message type is part of the Flatbuffer schema. What you send and receive over the Websocket connection is a single binary buffer.

Send a binary message to all users in the same room:

this.context.connection.sendBinary(byteArray: Uint8Array);

Subscribe to binary messages in Flatbuffer format:

this.context.connection.beginListenBinary(identifier:string, callback:(data : ByteBuffer) => void);

Unsubscribe from binary messages:

this.context.connection.stopListenBinary(identifier:string);

Flatbuffers Sample Code

Before you can send and receive Flatbuffer messages, you need to define a schema and compile it to TypeScript. Then, register the schema with the networking system and use the generated schema methods to create and parse messages.

• Built-in Flatbuffer schemas in Needle Engine
• Generating a schema
• Using the schema compiler
• Flatbuffers in Typescript

::::code-group :::code-group-item Register a schema

// Register a new Flatbuffer schema with the networking system
import { registerBinaryType } from '@needle-tools/engine';
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";

registerBinaryType(MySchemaIdentifier, MyDataModel.getRootAsSyncedTransformModel);

::: :::code-group-item Send Messages

// Prepare data for sending by creating a Flatbuffer message:
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";
const builder = new flatbuffers.Builder();

// Construct a Flatbuffer message
function createMyCustomModel(somePayload: string): Uint8Array {
    builder.clear();
    MyDataModel.startMyDataModel(builder);    
    const guidObj = builder.createString(guid);
    MyDataModel.addSomePayload(builder, guidObj);
    const res = MyDataModel.endMyDataModel(builder);
    builder.finish(res, MySchemaIdentifier);
    return builder.asUint8Array();
}

// Send the data
function sendData() {
    const data = createMyCustomModel("your-payload", this, true);
    this.context.connection.sendBinary(data);
}

::: :::code-group-item Receive Messages

// Subscribe to receive this specific message type:
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";

this.context.connection.beginListenBinary(MySchemaIdentifier, (data) => {
    const model = MyDataModel.getRootAsMyDataModel(data);
    console.log("Received binary message", model, model.somePayload());
});

::: ::::

Summary

Needle Engine makes the complex topic of networking approachable and easy to use. You can get started with automatic networking for your components with just a few lines of code, and you can dive deeper into manual networking when you need more control.