Networking

Needle Engine includes a full networking solution for multiplayer experiences.
A shared world state, voice chat, session persistence, and more can be achieved with our networking components and APIs. You can network your own components with a choice of automatic or manual networking.

Networking in Needle Engine is based on Websockets. Automatic networking uses JSON data for ease of use. For complex usecases and high-performance requirements, we use Flatbuffers.

Access to core networking functionality can be obtained by using this.context.connection from a component. The default backend server connects users to rooms. Users in the same room will share state and receive messages from each other.

Networking Concepts

Rooms and State

At the heart of networking in Needle Engine is the concept of synchronized rooms. Each room has an ID, and users connect to a room by providing this ID. Rooms are stored on a server, and users can join and leave rooms at any time.
When a user joins a room, they receive the current state of the room, apply that current state to their scene, and then listen for changes to the room state.
When a user leaves a room, they stop listening for changes to the room state.

Room state is stored as JSON data on the server, so all changes are persistent. This means that room state is not only useful for networking, but also for persisting actions of a single user.

Needle can provide view-only IDs for rooms. When accessing a room with a view-only ID, the user will not be able to interact with the room, but will be able to see the current state and get live updates. This is useful for presentations or demonstrations.

Ownership

Objects in a room can be owned by a user. This means that only the owner of an object can change its state. By default, objects have no owner. Components like DragControls will request ownership of an object before actually moving it. In custom components, you can control how ownership is handled. There may be no ownership required, ownership may be allowed to be transferred to another user automatically, or ownership may be transferred only by a specific action.

When a user leaves a room, objects owned by that user will either be deleted or have ownership reset, depending on how the object was created.

Enable Networking for your project

1. Add a SyncedRoom component to your scene. By default, this will use networking infrastructure provided by Needle.
2. Add a SyncedTransform component to a object whose movement you want to synchronize across the network.
3. Add a DragControls component to the same object.
4. Run the project. In the browser, click on "Join Room" and copy the URL.
5. Open a new browser window and paste the URL. You should now see the same object in both windows. Try dragging the object in one window and see it move in the other window.

The DragControls component, like many other Needle components, has built-in networking support. Ownership will be transferred to whoever starts dragging the object.

Built-In Components with Networking Support

Component	Description
SyncedRoom	Handles networking connection and connection to a room.
SyncedTransform	Handles synchronizing transforms.
SyncedCamera	Spawns a prefab for any user connected to the room which will follow their position.
VoIP	Handles voice-over-IP audio connections, microphone access etc. between users.
ScreenCapture	Handles screensharing via web APIs.
Networking	Use to customize the server backend URL. Also allows setting a local server for development.
DragControls	Handles dragging objects. Ownership will automatically be passed to the last user dragging an object.
Duplicatable	Handles duplicating objects. Duplicated objects are instantiated for everyone in the room.
Deletable	Handles deleting objects. Deletions are synchronized across the network.
DeleteBox	Handles deleting objects that have a "Deletable" component when they are dragged into a box volume.
PlayerSync	Powerful component that instantiates a specific object for each connected player.
PlayerState	Add this component to objects that are assigned to PlayerSync.
PlayerColor	Simple component for player-specific colors. Each user gets assigned a random color upon joining a room. This component assigns that color to the object's main material.
WebXR	Handles synchronizing user avatars (hands and heads).

Automatic Networking for custom Components

Fields in your own components can be networked very easily. Changes to the field will automatically be detected and sent to all users in the room. The changes are also persisted as part of the Room State, so users that join the room later will receive the current state of the field as well, ensuring everyone sees the same data.

To automatically network a field in a component, decorate it with the @syncField() decorator:

import { Behaviour, syncField, IPointerClickHandler } from "@needle-tools/engine"

export class SyncedNumber extends Behaviour implements IPointerClickHandler {

    // Use `@syncField` to automatically network a field. 
    // You can optionally assign a method or method name to be called when the value changes.
    @syncField("myValueChanged")
    mySyncedValue?: number = 1;
    
    private myValueChanged() {
       console.log("My value changed", this.mySyncedValue);
    }
    
    onPointerClick() {
       this.mySyncedValue = Math.random();
    }
}

import { Behaviour, IPointerClickHandler, PointerEventData, Renderer, RoomEvents, delay, serializable, showBalloonMessage, syncField } from "@needle-tools/engine";
import { Color } from "three"

export class Networking_ClickToChangeColor extends Behaviour implements IPointerClickHandler {

    // START MARKER network color change syncField
    /** syncField does automatically send a property value when it changes */
    @syncField(Networking_ClickToChangeColor.prototype.onColorChanged)
    @serializable(Color)
    color!: Color;

    private onColorChanged() {
        // syncField will network the color as a number, so we need to convert it back to a Color when we receive it
        if (typeof this.color === "number")
            this.color = new Color(this.color);
        this.setColorToMaterials();
    }
    // END MARKER network color change syncField

    /** called when the object is clicked and does generate a random color */
    onPointerClick(_: PointerEventData) {
        const randomColor = new Color(Math.random(), Math.random(), Math.random());
        this.color = randomColor;
    }

    onEnable() {
        this.setColorToMaterials();
    }

    private setColorToMaterials() {
        const renderer = this.gameObject.getComponent(Renderer);
        if (renderer) {
            for (let i = 0; i < renderer.sharedMaterials.length; i++) {
                // we clone the material so that we don't change the original material
                // just for demonstration purposes, you can also change the original material
                const mat = renderer.sharedMaterials[i]?.clone();
                renderer.sharedMaterials[i] = mat;
                if (mat && "color" in mat)
                    mat.color = this.color;
            }
        }
        else console.warn("No renderer found", this.gameObject)
    }

}

Note that syncField has an optional parameter to specify a method that should be called when the value changes. This method should be defined in the same class.

Custom Project Setup

If you're using a custom project setup, you need to have experimentalDecorators: true in your tsconfig.json file for syncField decorators to work. Projects created with Needle Starters have this enabled by default.

Creating and destroying objects

Often, you want to create and destroy objects at runtime, and of course these changes should be synchronized across the network.

The PlayerSync component simplifies this process by automatically instantiating a specific object for each connected player. When a player leaves the room, the object is destroyed for all users.

Additionally, Needle Engine provides two high-level methods:

• syncInstantiate() to duplicate objects across the network.
• syncDestroy() to destroy objects across the network.

🏗️ Code Samples Under Construction

Manual Networking

Needle Engine also provides a low-level API for sending and receiving messages. We call this "manual networking". The principles are the same, but you're in full control for sending and receiving messages and how to handle them.

Sending Messages

Send a message to all users in the same room:

this.context.connection.send(key: string, data: IModel | object | boolean | string | number | null);

Receiving Messages

You can subscribe to events in the room using a specific key. Typically, you want to match this with unsubscribing:

• subscribe in onEnable and unsubscribe in onDisable
  With this approach, no messages will be received while the object is disabled.

• or subscribe in start and unsubscribe in onDestroy
  With this approach, messages will still be received while the object is disabled.

this.context.connection.beginListen(key:string, callback:(data) => void)

Unsubscribe from events:

this.context.connection.stopListen(key:string)

Controlling message persistence

When sending network messages, the low-level API allows you to decide whether that message should be persistet (saved in the room state) or not (only sent to users currently in the room). To persist a message, make sure it has a guid field. This field is typically used to apply the message data to a specific object, by providing that object's guid. If you want target a specific object (and thus, include a guid field) but want the data to not be persisted, set the dontSave field to true in your message.

All persistent messages are saved in the room state and will be sent to users that connect at a later point. Non-persistent messages are only sent to users currently in the room, which is useful for effects (like playing a sound effect) that don't make sense to play for users that are currently not in the room. Optionally, you can include a deleteOnDisconnect field in your message to delete this particular message when the user disconnects.

// This message will be sent to all users currently in the room,
// AND to users that join the room later.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue" });

// This message will be sent to all users currently in the room,
// but NOT be sent to users that join the room later.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue", dontSave: true });

// This message will be sent to all users currently in the room,
// but NOT be sent to users that join the room later.
this.context.connection.send("my-message", { myData: "myValue" });

// This message will be sent to all users currently in the room,
// AND to users that join the room later,
// but will be deleted from the room state when the user disconnects.
this.context.connection.send("my-message", { guid: this.guid, myData: "myValue", deleteOnDisconnect: true });

To delete state for a specific guid from the backend storage, set the message key to delete-state and target a specific object with its guid: { guid: "guid_to_delete" } .

this.context.connection.send("delete-state", { guid: "guid_to_delete" });

Using debug flags to understand network messages

There are several debug flags that can be used to dive deeper into network messages. These can be appended the the page URL, like https://localhost:3000/?debugnet.

Flag	Description
?debugnet	Log all incoming and outgoing network messages to the console
?debugowner	Log all ownership requests and changes to the console
?debugnetbin	Log additional information for incoming and outgoing binary messages

Networking Lifecycle Events

The following events are available to listen to in your components. They describe common network events that you might want to react to in your components, like yourself or another user joining or leaving a room.

// Listen to the event when *you* have joined a networked room
this.context.beginListen(RoomEvents.JoinedRoom, ({room, viewId, allowEditing, inRoom}) => { ... });

// Listen to the event when *you* have left a networked room
this.context.beginListen(RoomEvents.LeftRoom, ({room}) => { ... });

// Listen to the event when *another user* has joined your networked room
this.context.beginListen(RoomEvents.UserJoinedRoom, ({userId}) => { ... });

// Listen to the event when *another user* has left your networked room
this.context.beginListen(RoomEvents.UserLeftRoom, ({userId}) => { ... });

// This event is received after all current room state has been sent to the client
this.context.beginListen(RoomEvents.RoomStateSent, () => { ... });

• See all Room Events in the API docs
• See all Ownership Events in the API docs
• See all Connection Events in the API docs

Using Needle Networking Servers

By default, networked Needle scenes connect to cloud infrastructure managed and provided by Needle. There is no additional setup needed, and currently no additional cost for using this service.

Typically, this will work fine for around 15-20 users in the same room. Once your project matures, you can upgrade to a bigger/better/stronger networking solution, by hosting your own networking server.

Hosting your own Networking Server

You might want to host your own networking server for larger deployments or to have more control over the networking infrastructure and implementation.

Our networking server is available on NPM own networking package as node.js package. The package contains pre-configured integrations for the popular web frameworks Fastify and Express, and can be integrated into other Node.js server frameworks as well.

For quick experiments: Remix on Glitch

You can remix a simple networking server running on Glitch from this page: needle-networking.glitch.me by clicking the button in the bottom right corner.

The default Glitch server instance is small and can only handle a limited amount of users. If you expect more than 15-20 people to be in your scene at the same time, you should consider hosting your networking server elsewhere (like on Google Cloud or AWS).

import networking from "@needle-tools/needle-networking";
networking.startServerFastify(fastifyApp, { endpoint: "/socket" });

import networking from "@needle-tools/needle-networking";
networking.startServerExpress(expressApp, { endpoint: "/socket" });

import { init, onConnection } from "@needle-tools/networking";

// Add your framework-specific websocket implementation here. 
// You can view the fastify and express implementations in server.js for reference.
class WebsocketConnector {
    constructor(frameworkWebsocket) {
        // Your implementation.
    }
    on(event, callback) {
        // Your implementation. When receiving a message in the websocket connection, call the callback.
        // 'event' can be 'message' or 'close'.
    }
    send(key, value) {
        // Your implementation. Pass the message along to the websocket connection.
    }
}
const options = { endpoint: "/socket" };
init(options);
yourFramework.createWebsocketRoute(options.endpoint, frameworkWebsocket => {
    onConnection(new WebsocketConnector(frameworkWebsocket));
});

Example on Glitch.com

See the code on glitch.com/edit/#!/needle-networking for an example of how to integrate Needle Networking with an Express server.

Configuration

The following options are available:

Option	Description
options.endpoint string	Optional. Relative server endpoint. For example, /socket will start the websocket endpoint on yourserver/socket. Defaults to /.
options.maxUsers number	Maximum number of concurrent users on a server. Defaults to 50.
options.defaultUserTimeout number	Time in seconds after which a user is considered disconnected. Defaults to 30.
process.env.VIEW_ONLY_SALT string	Salt value used for generating view-only room IDs from regular room IDs. Defaults to a predefined salt value.
process.env.NEEDLE_NETWORKING_S3_* string	Enable S3 storage. See below for the full list of environment variables you need to set for this. When not set, the default storage is used (JSON files on disk).

The networking server will automatically manage connecting and disconnecting users, receiving and sending messages, and persisting room state.

Custom networking servers can be deployed anywhere, for example on Google Cloud. For further instructions please refer to this repository: Local Needle Networking Server

Different server locations for local and hosted development

If you're working on custom networking code, you may want to use different server locations for local development and the hosted app. You can set individual server URLs in the Networking component:

State Storage

Network state is by default stored to disk on the server as JSON files in the /.data directory.
Each room has its own file, and the state is sent to connecting clients when they join a room.

Optionally, the networking state can be stored with an S3 compatible storage provider. Use the following environment variables to enable S3 storage:

NEEDLE_NETWORKING_S3_ENDPOINT=
NEEDLE_NETWORKING_S3_REGION=
NEEDLE_NETWORKING_S3_BUCKET=
NEEDLE_NETWORKING_S3_ACCESS_KEY_ID=
NEEDLE_NETWORKING_S3_ACCESS_KEY=
NEEDLE_NETWORKING_S3_PREFIX= # all state saved in the bucket will be prefixed with this string. This can be a path e.g. `my_state/` or a unique id `server_123_`

Local Networking Server

For testing and development purposes, you can run the Needle Engine networking package on a local server. We have prepared a repository that is set up to host the websocket package and to make that easy for you.

1. Download the local server sample from github.com/needle-tools/networking-local
2. Follow the instructions in the README to set up the server. The server will run on wss://localhost:9001/socket by default.
3. Add the Networking component to your scene.
4. Paste the local server address into the Localhost field on the Networking component.

Advanced: Customizing WebRTC settings for peer.js

Needle Engine Screencapture (Screensharing) and VoIP (Voice communication) components use peer.js for networking audio and video. Peer.js uses WebRTC under the hood.

Needle Engine uses reasonable defaults for peerjs. If you want to modify those defaults, you can call

setPeerOptions(opts: PeerjsOptions);

with your custom settings. This can be used to modify the hosting provider for ICE/STUN/TURN servers, for example when you use your own WebRTC servers.

Advanced: Server and Client Message Formats

For informational purposes. Use the APIs provided by Needle Engine instead.

Typically, you do not need to interact with these message formats directly, as the low-level networking API already handles parsing messages and giving you the correct types. The information here is provided for advanced users who want to understand the underlying message formats or implement their own networking solutions.

Messages are sent in JSON format. They always have a key field that describes the type of message, and a data field that contains the message payload. The data field can be any JSON-serializable object.

Built-in Room Events

// Sent to the server to attempt joining a room.
{
    "key": "join-room",
    "data": {
        "room": string,
        "viewOnly": boolean,
    }
}

// Sent to the server to leave a room.
{
    "key": "leave-room",
    "data": {
        "room": string
    }
}

// Sent to the client when the local user has joined a room.
// Type: JoinedRoomResponse
{
    "key": "joined-room",
    "data": {
        "room": string,
        "viewId": string,
        "allowEditing": boolean,
        "inRoom": string[] // connection IDs
    }
}

// Sent to the client when the local user has left a room.
// Type: LeftRoomResponse
{
    "key": "left-room",
    "data": {
        "room": string
    }
}

// Sent to the client when any user has joined a room.
// Type: UserJoinedOrLeftRoomModel
{
    "key": "user-joined-room",
    "data": {
        "userId": string // connection ID
    }
}

// Sent to the client when any user has left a room.
// Type: UserJoinedOrLeftRoomModel
{
    "key": "user-left-room",
    "data": {
        "userId": string // connection ID
    }
}

// Sent to the client after the complete room state has been sent.
{
    "key": "room-state-sent",
    "data": {}
}

Built-In Utility Events

// Sent to the client when the connection is established.
{
    "key": "connection-start-info",
    "data": {
        "id": string // connection ID
    }
}

// Used by the syncInstantiate() API to create a new instance of an asset.
// Type: NewInstanceModel
{
    "key": "new-instance-created",
    "data": {
        "guid": string,
        "originalGuid": string,
        "seed": number | undefined,
        "visible": boolean | undefined,
        "dontSave": boolean | undefined,

        "parent": string | undefined,
        "position": { x: number, y: number, z: number } | undefined,
        "rotation": { x: number, y: number, z: number, w: number } | undefined,
        "scale": { x: number, y: number, z: number } | undefined,

        "deleteStateOnDisconnect": boolean | undefined
    }

// Used by the syncDestroy() API to destroy an instance of an asset.
// Type: DestroyInstanceModel
{
    "key": "instance-destroyed",
    "data": {
        "guid": string,
        "dontSave": boolean | undefined
    }
}

// Sent to the server every few seconds to keep the connection alive.
{
    "key": "ping",
    "data": {}
}

// Sent by the server in response to a ping.
{
    "key": "pong",
    "data": {}
}

// Sent to the server to delete state for a specific guid.
{
    "key": "delete-state",
    "data": {
        "guid": <string>
    }
}

// Sent to the server to delete ALL current room state.
{
    "key": "delete-all-state",
    "data": {}
}

Built-In Ownership Events

{
    "key": 
      "request-has-owner" | 
      "request-ownership" | 
      "remove-ownership",
    "data": {
        "guid": string
    }
}

// Type: OwnershipResponse

{
    "key": 
      "response-has-owner",
    "data": {
        "guid": string,
        "value": boolean
    }
}

{
    "key": 
      "gained-ownership" | 
      "lost-ownership" | 
      "gained-ownership-broadcast" | 
      "lost-ownership-broadcast",
    "data": {
        "guid": string,
        "owner": string
    }
}

Built-In Flatbuffer Schemas

Flatbuffer messages are sent directly as binary messages.

include "vec.fbs";

struct Transform {
    position:Vec3;
    rotation:Vec3;
    scale:Vec3;
}

table SyncedTransformModel {
    guid:string;
    fast:bool;
    transform: Transform;
    dont_save:bool;
}

root_type SyncedTransformModel;

include "vec.fbs";

table SyncedCameraModel {
    user_id:string;
    guid:string;
    dont_save:bool;
    pos:Vec3;
    rot:Vec3;
}

root_type SyncedCameraModel;

struct Vec2 {
  x:float;
  y:float;
}

struct Vec3 {
  x:float;
  y:float;
  z:float;
}

struct Vec4 {
  x:float;
  y:float;
  z:float;
  w:float;
}

Advanced: Binary Messages in the Flatbuffer format

JSON messages are easy to use and understand, but are typically larger in memory and bandwidth. For large amounts of data, or when sending fast updates, binary messages are faster and more efficient. You can use Flatbuffers messages in Needle Engine for cases where that is required. Using Flatbuffers requires additional setup steps like defining and compiling a message schema, and is harder to debug since you're dealing with binary messages.

Note that when sending and receiving flatbuffer messages, there is no key field – the message type is part of the Flatbuffer schema. What you send and receive over the Websocket connection is a single binary buffer.

Send a binary message to all users in the same room:

this.context.connection.sendBinary(byteArray: Uint8Array);

Subscribe to binary messages in Flatbuffer format:

this.context.connection.beginListenBinary(identifier:string, callback:(data : ByteBuffer) => void);

Unsubscribe from binary messages:

this.context.connection.stopListenBinary(identifier:string);

Flatbuffers Sample Code

Before you can send and receive Flatbuffer messages, you need to define a schema and compile it to TypeScript. Then, register the schema with the networking system and use the generated schema methods to create and parse messages.

• Built-in Flatbuffer schemas in Needle Engine
• Generating a schema
• Using the schema compiler
• Flatbuffers in Typescript

// Register a new Flatbuffer schema with the networking system
import { registerBinaryType } from '@needle-tools/engine';
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";

registerBinaryType(MySchemaIdentifier, MyDataModel.getRootAsSyncedTransformModel);

// Prepare data for sending by creating a Flatbuffer message:
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";
const builder = new flatbuffers.Builder();

// Construct a Flatbuffer message
function createMyCustomModel(somePayload: string): Uint8Array {
    builder.clear();
    MyDataModel.startMyDataModel(builder);    
    const guidObj = builder.createString(guid);
    MyDataModel.addSomePayload(builder, guidObj);
    const res = MyDataModel.endMyDataModel(builder);
    builder.finish(res, MySchemaIdentifier);
    return builder.asUint8Array();
}

// Send the data
function sendData() {
    const data = createMyCustomModel("your-payload", this, true);
    this.context.connection.sendBinary(data);
}

// Subscribe to receive this specific message type:
import { MyDataModel } from 'my-data-model.js';

const MySchemaIdentifier = "MYSC";

this.context.connection.beginListenBinary(MySchemaIdentifier, (data) => {
    const model = MyDataModel.getRootAsMyDataModel(data);
    console.log("Received binary message", model, model.somePayload());
});

Custom Flatbuffer messages and persistence

Currently, custom binary messages can't be persisted on the networking server. Modify the networking server and add your custom flatbuffer schemas to ensure the guid property can be processed.

Summary

Needle Engine makes the complex topic of networking approachable and easy to use. You can get started with automatic networking for your components with just a few lines of code, and you can dive deeper into manual networking when you need more control.