Operate a model server¶

loom-serve separates process lifetime, model registration, request admission and parameter residency. A model can stay registered while drained or cold; changing one of these states need not restart the server or discard other models. The first-output walkthrough establishes a running server and a vectors deployment for the commands below.

Find the state you need¶

curl --fail --silent --show-error http://127.0.0.1:8080/v1/status
curl --fail --silent --show-error http://127.0.0.1:8080/v1/models

/v1/status includes every registration and the server's actual resources. /v1/models lists prepared deployments and their capabilities; a prepared but drained model remains listed. /readyz describes the server, not readiness of every model. The status fields answer different questions:

Field on each model	Question
`loom.registration`	Is this registration declared, preparing, ready, retiring, failed or awaiting a release retry?
`loom.status.admission`	Is the gate open, draining, drained or removing?
`loom.status.phase`	What is the modality scheduler doing?
`loom.parameter_operation`	Is an explicit activation/deactivation waiting or in progress?
`loom.parameters`	What reloadable weight plans are active and how much retained backing do they account for?
`loom.variants`	Which native executable choice is preferred, and how many choices are retained?
`loom.error`	Did preparation, execution, or a lifecycle operation fail?

The registration wire values are registered, preparing, ready, retiring, failed and release_failed. Scheduler phases are modality-specific and are not measurements of device utilization.

Register without immediately consuming model memory¶

A model document with "admission":"drained" registers metadata without reading programs or weights or allocating model storage. It still occupies one registration slot and has an observation stream. This is useful for a catalog larger than the set of models that can be resident simultaneously.

The normal default is "open". Registration returns HTTP 202 after accepting server-owned intent. Preparation and JIT happen asynchronously. A client disconnect does not cancel that registration; /v1/status reports its outcome. A failed source package remains inspectable without stopping healthy peers. After correcting its inputs, unregistering and registering again starts an explicit new attempt with a new incarnation.

A terminal execution or weight-loading error also retires the affected model, while healthy models continue serving. For example, a native allocation failure returns HTTP 503 for the affected image or embedding request; the failed model remains in /v1/status with loom.error, but is removed from /v1/models. Unregistering and registering it again starts an explicit recovery attempt after the cause has been addressed. Sessions and checkpoints owned by that failed registration are not preserved, and requests are not automatically replayed. An error in a shared resource can affect several models and requires the server to drain instead; model-local recovery does not reset a failed device.

The model-document reference applies equally to startup --model arguments and HTTP registration. HTTP changes live state; the client owns persistence of those changes into its launch configuration.

Drain, resume or remove¶

curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/drain \
  -H 'Content-Type: application/json' -d '{"model":"vectors"}'
curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/undrain \
  -H 'Content-Type: application/json' -d '{"model":"vectors"}'

Drain closes admission when the owner processes the request. Accepted model work, including queued requests, continues. New inference, tokenizer, checkpoint pinning, and checkpoint suspension requests wait outside the model within the holding budget. Checkpoint deletion remains available so clients can release retained history while inference is drained. Control and observation routes remain available. A successful drain response means the gate changed; admission:"drained" confirms accepted work has finished.

Drain is reversible and does not evict weights, erase sessions or remove the registration. Undrain reopens admission; held requests enter that modality's normal bounded admission policy. Repeated drain/undrain is idempotent.

Unregister is irreversible:

curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/unregister \
  -H 'Content-Type: application/json' -d '{"model":"vectors","mode":"drain"}'

mode:"drain" is the default and lets accepted requests finish. mode:"abort" cancels their intent. Both wait for actual CPU/device borrowers to retire before releasing memory, and return HTTP 202 for accepted removal. Graceful removal can be escalated to abort but cannot be undone by undrain.

The name stays occupied during retirement; a duplicate registration returns 409. The model leaves /v1/models as soon as removal is accepted, but remains in /v1/status until retirement finishes and its name is released. Absence from /v1/status confirms completion. With concurrent administrators, compare loom.incarnation: the same name may already identify a new registration.

Held arrivals receive 410 during removal, and requests for the absent name receive 404 after retirement. A quiescent platform release failure remains visible in /v1/status as release_failed, retaining the resources needed to retry. Another unregister explicitly retries that release.

Control weight residency independently¶

Explicit activation requires a healthy registration with admission closed:

curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/drain \
  -H 'Content-Type: application/json' -d '{"model":"vectors"}'
curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/activate \
  -H 'Content-Type: application/json' -d '{"model":"vectors"}'
curl --fail --silent --show-error http://127.0.0.1:8080/v1/status

This prepares and warms the primary computation without opening admission. It leaves a text deployment's optional image projector dormant. HTTP 202 accepts intent; loom.parameter_operation advances through waiting and activating to idle. Check loom.error and wait for idle before undrain. loom.parameters.primary.active alone means weight activations have been enqueued, not that device work has completed.

For a text deployment with image input, explicit projector warmup uses:

curl --fail --silent --show-error http://127.0.0.1:8080/v1/models/activate \
  -H 'Content-Type: application/json' -d '{"model":"scout","component":"image"}'

The registration must be drained first. This prepares the projector lazily, then warms its parameters. loom.parameter_component identifies the current or last operation; check loom.parameters.image.error as well as loom.error when it becomes idle. A projector source/preparation error remains local to image input; ordinary text can still run after undrain.

With elastic backing, /v1/models/deactivate releases reloadable weights from a healthy, drained, unpinned model while retaining compiled programs, sessions and checkpoints. It accepts the same {"model":"name"} body and returns 202. Wait for parameter_operation:"idle", check loom.error, and inspect loom.parameters.primary.active and committed_bytes. Fixed primary backing cannot deactivate and returns 409; a metadata-only drained registration is already cold. With component:"image", deactivation releases projector weights on elastic backing or retires the projector on fixed backing. Both preserve text histories and checkpoints. Fresh image demand reloads weights or prepares the retired projector; a retained image-prefix hit needs neither.

Native weight allocation and release run independently of the serving scheduler: another ready model can continue serving during either transition. Retained memory reports pending_bytes for accepted allocations and releasing_bytes for releases still in flight. Releasing bytes remain included in committed_bytes until native cleanup completes; they are not free capacity. Requests waiting for that capacity resume from its completion notification.

Undrain and conflicting residency operations return 409 while an operation is pending. Ordinary inference can reload evicted weights on demand without re-JIT. Successful activation does not itself pin weights against later pressure.

/v1/models/pin and /v1/models/unpin accept the same optional component selector and change only that component's eviction eligibility. Pinning is not activation: a cold component remembers the policy without loading its weights. Status exposes it as loom.parameters.primary.pinned or loom.parameters.image.pinned. A component with pending deactivation cannot be pinned. Unpinning never removes protection held by accepted work.

Warmup releases its temporary protection when complete; it does not create a persistent pin. Pinning the primary language model can prevent its projector from fitting on a constrained device, even when the projector is unpinned. Pin only components that must remain resident within the deployment's budget.

Return idle backing without removing models¶

Automatic reclamation is opt-in. For example, this server releases unused command workspace after 30 seconds and unpinned weights after five minutes:

loom-serve --device=amdgpu --model=/srv/models/scout/model.json \
  --workspace_idle_ms=30000 --parameter_idle_ms=300000

Each duration is independent; 0 disables that policy. Workspace expiry applies to each pool's shared command workspace after its last allocation actually retires. Weight expiry starts when a model's last invocation pin or explicit parameter pin is released. Polling status, receiving telemetry, and leaving a client connected do not keep idle resources hot. Explicitly pinned weights and active invocations remain protected. Fixed-backed weights cannot deactivate; workspace expiry works with either backing policy.

Expiry queues native release without draining the server or waiting on unrelated requests. The next invocation reloads weights and reuses the compiled programs. Registrations, sessions, KV/prefix history, and named checkpoints stay intact: these policies release reloadable backing, not conversation history. They therefore need not reduce an idle process to its minimum possible footprint.

With --devices=devices.json, each pool supplies its own workspace_idle_ms and parameter_idle_ms instead of global flags. /v1/status publishes the durations in each server.resources.pools entry's idle.workspace_ms and idle.parameter_ms. Pool workspace commitment and per-model parameter activity/accounting show reclamation and subsequent reloads. Expiry checks use the smaller enabled duration, capped at one second; physical release finishes asynchronously. When nothing awaits retirement or expiry, maintenance stops rearming its timer. Heartbeat reporting has an independent interval and can be disabled without disabling reclamation.

Replace native code without reloading weights¶

A prepared text deployment can JIT replacement command programs and kernels while keeping its weights, control VM, tokenizer, sessions and checkpoints. Staging and selection are separate: several candidates can compile without changing inference. Explicit selection makes a ready choice preferred at a completed execution boundary. There is no unregister/register cycle or required admission drain.

The author asserts that replacements preserve the meaning and layout of retained state and parameters. The server checks the native binding contract, not mathematical equivalence. Preparation declarations, shape catalog, control VM and tokenizer remain fixed; changing them requires a new registration. For a text deployment named scout with variant capacity enabled, these commands use jq to carry server-issued IDs:

original=$(curl --fail --silent --show-error \
  http://127.0.0.1:8080/v1/models/variants -H 'X-Loom-Model: scout' |
  jq -er '.selected')

candidate=$(curl --fail-with-body --silent --show-error \
  http://127.0.0.1:8080/v1/models/variants \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"scout\",\"base\":$original,\"base_directory\":\"/srv/models/scout/candidate\",\"sources\":[\"config.loom\",\"advance.loom\"]}" |
  jq -er 'select(.variant.phase == "ready") | .variant.id')

curl --fail-with-body --silent --show-error \
  http://127.0.0.1:8080/v1/models/variants/select \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"scout\",\"variant\":$candidate,\"expected\":$original}"

These JSON-only commands use provider files already on the server. A multipart source package instead carries client-local providers in the request. Staging waits asynchronously by default and returns retained compiler diagnostics on failure; "wait":false instead returns an ID immediately for later inspection. Existing inference can progress during JIT. Selection briefly quiesces that model's accepted CPU/device work before rebinding native stages; it neither reloads weights nor copies KV state. Subsequent text epochs use the new code, including epochs advancing already-admitted requests. This is a model-wide preference, not a request-local A/B selector.

Keeping the original choice makes rollback another selection:

curl --fail-with-body --silent --show-error \
  http://127.0.0.1:8080/v1/models/variants/select \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"scout\",\"variant\":$original,\"expected\":$candidate}"

Expected-current protection prevents another author's update from being silently overwritten. After an interrupted response, inventory establishes the actual preference; disconnect does not cancel accepted work. Rollback restores code, not a conversation's earlier state. Named checkpoints provide independent state forks when comparing continuations. Native-update compatibility must hold for every retained session and checkpoint that either choice will consume.

Unused terminal candidates can be explicitly removed to reclaim cache credit. Pending and failed attempts count against variant_capacity; capacity exhaustion rejects new staging instead of evicting an arbitrary rollback choice. The HTTP reference defines inspection, selection and removal responses. The live development workflow combines these controls with remote source uploads, checked kernel changes and pinned continuation trials.

Bound memory and admission¶

Process limits and model capacities constrain different resources:

Limit	Resource it bounds
`--connections`	HTTP connections shared by all listeners.
`--held_requests`	Requests waiting outside closed/unprepared model gates.
`--model_capacity`	All registered names, including failed and retiring models.
Model `pending_requests`	Modality request credit; text/image count waiting requests, finite services include the active cohort.
Text `rows`	Concurrently advancing execution rows.
Text `context_capacity`	Logical token ceiling of one context.
Text `variant_capacity`	Native choices, including the original, pending JITs and failed attempts.
Pool `memory_bytes` / shortcut `--memory_bytes`	Shared elastic physical backing for parameters, permanent buffers, sparse state, invocation I/O, and workspace.

Holding capacity stays below connection capacity so held inference cannot consume every control connection. Live log streams and waiting variant-management requests share an additional combined bound with held requests. Full credit returns 429. Incomplete requests and active response streams still consume transport credit; these bounds are not a promise that arbitrary slow clients leave connections free.

Text cache addressability follows pool backing. Fixed backing reserves one model context shared across its rows. Elastic backing sizes its sparse address range from the selected pool's physical capacity, any tighter memory_bytes budget, and the model's declared cache bytes per page. Increasing addressability does not commit the corresponding cache or allocate its whole page directory. Physical admission uses the same ceiling across all models in the pool, whether it comes from the selected source or a tighter memory_bytes budget. Total addressability is not a promise that every model can fill it simultaneously. A backend reporting no physical capacity requires an explicit pool budget.

Text admission reserves a request's completion high-water. Short contexts grow by blocks rather than eagerly reserving each row's maximum physical slab. When an oldest request cannot fit, admission_overtakes permits a bounded number of smaller fitting requests to pass; its default is the execution-row count. Zero selects FIFO. This bounds successful overtakes, not elapsed waiting time: a slow active reader can retain its reservation.

Elastic parameters, permanent model buffers, sparse mutable state, invocation input/output, and command scratch share the deployment's selected pool. Admission uses the model's prepared working-set peak and outstanding work, rather than probing free HAL memory. Command scratch can be reused after its ordered release; an escaping output remains charged until both its producer and final consumer retire, even if its model is removed. Permanent buffers remain charged while alive, including overlapping old and new allocations during resize. Host snapshots, other host allocations, and driver overhead need headroom when sizing the pool; memory_bytes is not a whole-device cap.

Elastic pressure first reclaims idle allocator backing, then idle unpinned weights and eligible idle text history across models sharing the same pool. Accepted work, prepared continuations, and explicit checkpoint references remain protected. Before discarding history, admission checks a minimum memory requirement that allows legal cache-slot reuse and KV compaction. If even that minimum cannot fit beside retained state, the request is rejected without evicting conversations. For example, a named checkpoint can prevent a second model from fitting even when the first model's automatic session cache is eligible for eviction. Releasing the checkpoint can make the same request admissible. Passing the minimum check is not a reservation: actual placement and backing still determine whether the request can run. An evicted session can replay its full input history; a session name is not durable storage. Pool pressure does not automatically serialize conversations to disk. If all capacity is protected and no accepted work can release it, admission reports overload rather than reclaiming live storage.

Trim without changing the budget¶

curl --fail --silent --show-error http://127.0.0.1:8080/v1/pools/trim \
  -H 'Content-Type: application/json' -d '{"pool":"pool","target_bytes":0}'

Here pool is the key assigned by the single-device shortcut. With a named configuration, use a pool key shown in /v1/status, such as interactive. This requests reduction of eligible serving-managed commitment toward the target. It releases idle workspace and unpinned reloadable parameters, not active work or retained conversations. HTTP 200 returns current accounting, including accepted native releases that have not completed. /v1/status reports subsequent completion; protected storage can prevent reaching the target. Fixed weights are not reclaimable. Trim does not change the configured limit or discard compiled code.

Compiled products belong to execution devices, independently of the storage pools used by their models. To discard cached generations with no live model catalog, select the device separately:

curl --fail --silent --show-error http://127.0.0.1:8080/v1/devices/trim \
  -H 'Content-Type: application/json' -d '{"device":"device"}'

Here device is the single-device shortcut's key. This operation neither changes pool residency nor interrupts live models. It also prunes unused tokenizers from the group-wide cache; live models and pending loads preserve their tokenizer references. A subsequent registration may need to compile products or reload tokenizers that were removed.

The observation workflow explains shared backing and its workspace subset. Adding those counters double-counts workspace; neither is total VRAM or process RAM. Reported free bytes are not a guarantee that the next request can be admitted.

Shut down safely¶

SIGINT/SIGTERM stops application admission and joins accepted asynchronous work before destruction. Cancellation does not make an already-submitted device operation disappear. There is no timeout that authorizes freeing live buffers.

Invalid requests and isolated source failures stay scoped to their requests or registrations. A native failure affecting shared backing instead stops admission and drains accepted work before reporting the terminal error. Completion callbacks publish outcomes; they do not tear down shared devices or pools.

HTTP and source loading assume a trusted deployment. TCP/Unix listeners share the same authority; socket permissions, tunnels or an external proxy supply access control. Command options describe binding and resource configuration.