Reconciliation

Reconciliation is how Trailer keeps what is actually running on a host in step with what you asked for. You describe the state you want through the web client or the API. The agent on each host continuously works to make that host match.

Desired state and observed state

There are two pictures of the world:

Desired state is what you configured: which workspaces, images, networks, and volumes should exist, and how each is set up. The server stores this in its database.
Observed state is what actually exists on the host right now: the running containers, their images, networks, and volumes.

Reconciliation is the process of comparing the two and making the changes that bring the host in line with the desired state.

A pull model

The agent pulls work from the server rather than the server pushing changes to the agent. On a fixed interval, the agent asks the server for the desired state of its host, compares that against what is running, and applies the difference. Because the agent pulls on a schedule, a brief network interruption or a restart is self-correcting: the next cycle simply picks up wherever things stand.

sequenceDiagram
    participant U as User
    participant S as Server
    participant A as Agent
    participant R as Container runtime
    U->>S: create or edit a workspace
    S->>S: validate and store desired state
    loop every reconciliation interval
        A->>S: ask for this host's desired state
        S-->>A: desired workspaces, images, networks, volumes
        A->>R: create, update, or remove resources
        A->>S: report status and messages
    end
    S-->>U: live status updates

The reconciliation cycle

Within a single cycle the agent works in a deliberate order so that each resource exists before anything that depends on it. Independent resources within a step are handled at the same time.

flowchart TD
    A["Cleanup<br/>(remove resources no longer wanted)"] --> B["Shared prerequisites<br/>(GPU support, reverse proxy)"]
    B --> C["Networks and volumes"]
    C --> D["Workspaces<br/>(create, update, or recreate)"]
    D --> E["Snapshots<br/>(capture running workspaces)"]
    B -.-> F["Images<br/>(build or pull, in parallel)"]

Cleanup. Resources that are no longer in the desired state are removed first, as soon as the agent has fetched the freshest desired state. Cleanup never touches anything another step is actively working on (see Staying stable).
Shared prerequisites. Host-wide services such as GPU support and the request router are set up before workspaces that may rely on them.
Networks and volumes. Created before any workspace attaches to a network or mounts a volume.
Images. Built or pulled alongside the rest of the cycle. A long build never delays the other steps: a workspace waiting on a brand-new image simply deploys on a later cycle once the build finishes, and running workspaces are untouched until their new image is fully ready.
Workspaces. Created, updated, or recreated to match their configuration.
Snapshots. Running workspaces are captured into new images once deployments settle.

Applying changes to a workspace

When a workspace already exists, the agent decides whether the change can be applied to the running container or whether the container has to be replaced. Most properties of a container can only be set when it is created, so changing them means removing the old container and creating a fresh one in its place.

A recreate keeps the workspace’s identity, name, and attached storage. Attached volumes and bind mounts, and the data on them, survive. What does not survive is anything written inside the container’s own filesystem outside a mounted volume: that lives only in the container being replaced, so it is lost. Keep anything that must persist on a volume.

On a Docker-based host, the following changes replace the container and therefore discard that in-container data:

The workspace image, including rebuilding the same image to a newer version
The container runtime
CPU, memory, or shared-memory limits
Environment variables
Published ports
Attaching or detaching a volume or bind mount, or changing a mount point
Hardware acceleration, nested virtualization, or GPU access being turned on or off
The init-process setting
The startup command or arguments
A workspace URL, but only when the selected host uses Traefik as its reverse proxy in label mode. In that mode a URL is expressed as labels on the container, which can only be set at creation, so changing a URL replaces the container (see Workspace URLs and routing).

The following changes are applied to the running container without replacing it, so in-container data is kept:

Attaching or detaching a network, or changing a network’s static IP: the container is reconnected in place.
Renaming the workspace: the existing container is renamed.
A workspace URL, on any host that does not use Traefik in label mode. When the host serves URLs through Traefik configuration files, the files are regenerated in place. When it uses the built-in proxy, the change is picked up without touching the container. Only Traefik-in-label-mode forces a recreate on a URL change.

Images and builds

Images move through their own lifecycle, which you can watch live on the image detail page:

stateDiagram-v2
    [*] --> Pending
    Pending --> Building
    Building --> Finished
    Building --> Failed
    Building --> Cancelled
    Finished --> [*]

Custom images are built from your configuration on the host. Build output streams back so you can follow progress and read any errors.
External images are pulled from a registry, with credentials when the registry needs them.
Snapshots capture a running workspace into a new image after deployments in the cycle have completed.

A build that fails is reported with its error and can be retried. A build can also be cancelled while it is running.

Reporting status

The agent reports back over two channels:

Per-resource updates are sent the moment a step finishes: a container created, a build completed, an error raised. This is what drives the live status badges and messages you see in the web client, so a workspace that starts, stops, or fails updates without a refresh. Workspaces move through states such as Deployment pending, Deploying, Starting, Running, Stopping, and Stopped, or Error if something goes wrong.
Heartbeats are sent on their own interval and carry the host’s overall health: available runtimes, and, when enabled, host, GPU, and per-workspace metrics. The server’s reply can adjust the host’s settings on the fly.

stateDiagram-v2
    dp: Deployment pending
    d: Deploying
    s: Starting
    r: Running
    st: Stopping
    sd: Stopped
    sn: Snapshotting
    e: Error
    [*] --> dp
    dp --> d
    d --> s
    s --> r
    r --> st
    st --> sd
    sd --> d: redeploy
    r --> sn
    sn --> r
    d --> e
    s --> e
    e --> d: retry

Staying stable and idempotent

Reconciliation runs continuously, so it is designed to be safe to repeat. Two ideas keep it from causing needless churn:

Thorough comparison. Before recreating anything, the agent compares the desired and observed configuration in detail rather than at a glance. Differences that do not actually matter, such as a different ordering of the same values, do not trigger a rebuild. This prevents a workspace from being restarted on every cycle for no reason.
Overlapping-cycle safety. Cycles are allowed to overlap: a slow one (a large image build, for example) does not block the next from starting. Every cycle shares the freshest desired state it has fetched, and both cleanup and deployment decisions always consult the newest shared view rather than the possibly outdated one a slow cycle started with. A slow cycle can therefore never remove something a newer cycle just created, and never recreates something you have deleted in the meantime. Cleanup itself runs at most once at a time. Overlapping cycles simply leave it to the one already running, which sees their fresher state anyway.

Because cleanup runs at the start of every cycle and never waits behind builds or deployments, deleting a workspace frees its resources within a few cycles even while the host is busy building images. The same holds for snapshots: capturing a workspace does not wait for unrelated image builds to finish.

When things go wrong

Failures are contained and self-healing:

A failure on one resource is reported and does not stop the rest of the cycle from proceeding.
A failed cycle does not stop the loop. The next interval simply tries again with the current desired state.
Status updates are retried until they reach the server, so the web client eventually reflects the true state.

Tuning reconciliation

The reconciliation interval, the heartbeat interval, and whether a host is enabled at all are per-host settings you can change from the host’s detail page. They take effect without redeploying the agent. Disabling a host tells its agent to stop the host’s workspaces on the next cycle. See the Configuration Reference for the full list of host settings.