A remote worker runs a pool's runs on a machine other than the server, such
as a coding machine with spare CPU, another server or a laptop. The worker
talks only to morca ui's API, over HTTPS. Nothing on it holds database
credentials. The scheduler stays the only brain. It fires, senses, queues,
retries and runs the watchdog, and keeps all state in its Postgres. A worker
only runs what it is given.
server worker (bgent)
────── ──────────────
morca scheduler ── queues runs, held morca worker --pool bgent
"waiting for a worker" │ polls POST /api/worker/claim
morca ui ◀──────── HTTPS (Cloudflare Tunnel) ─────┤ gets a run's spec
│ /api/worker/claim, /exited, /release │ starts <binary> resume <run>
│ /api/worker/store/<call> ◀── the run's state calls, with the token
▼ ▼
Postgres MORCA_WAREHOUSE, MORCA_LOGS
(Object Storage, straight from the worker)
[pipelines.mojo_releases]
placement = "worker:bgent"Or morca register --placement worker:bgent. A pool's name is 1 to 32 of
a-z, 0-9, - and _. The scheduler never starts these runs. They stay
queued, held "waiting for a worker of pool bgent", and the pipeline's page
says so. max_runs and morca pause apply as for any pipeline: a worker is
given a run only while its pipeline isn't paused and has fewer than
max_runs live runs.
A pool-placed pipeline with fills = "all" fills its range as a local one
does (morca.toml): the scheduler queues its fill runs
in the same order (fill_order), within the same fill_window, and never
over a partition that is done or in a queued or running run, or that a fill
run ever took. It doesn't start them: they stay queued, held "waiting for
a worker of pool ", until a worker claims one.
The budget. fill_max_runs caps the pipeline's live fill runs across
the whole pool: the ones running on any worker and the ones queued for
one. So a fill never has more than fill_max_runs runs going, however many
workers the pool has, and the scheduler queues a new one only as one ends.
What one machine takes is that worker's own --max-runs and, for a
pipeline with a request, what its host can grant. Fill runs don't count
against max_runs, and the pipeline's other runs are claimed first; across
the pool's pipelines, the highest priority
first, then every pipeline's other runs before any fill runs. A
worker's machine runs anyway, so a pool's fill runs cost nothing against
fill_budget_per_day. The pipeline page's Fill line and the coverage
calendar count the queued fill runs as live.
A two-machine pool, the server and a Mac, filling gharchive two at a time, oldest first:
[pipelines.gharchive]
placement = "worker:gharchive"
fills = "all"
fill_order = "oldest"
fill_max_runs = 2 # across the pool# On the server, once: a token per machine.
morca worker-token create --pool gharchive --label hetzner
morca worker-token create --pool gharchive --label bgent
# On each machine, its own token in worker.env (below), one run at a time:
docker run -d --name orca-worker --restart always --env-file worker.env \
-v "$PWD/secrets:/etc/orca/secrets:ro" -v "$PWD/secret-fetch:/etc/orca/secret-fetch:ro" \
ghcr.io/<owner>/<image>:<tag> worker --pool gharchive --max-runs 1 \
--name hetzner --remote https://morca.example.com # --name bgent on the MacWith fill_max_runs = 2 and --max-runs 1 on each, each machine runs one
fill run; with --max-runs 2 on both, still two at once in all, wherever
they land. Give each machine its own token: the scheduler tells which runs
a worker still has by what its token's worker last said.
When a worker dies, its fill runs are queued again for the pool (below), so the fill goes on on the other machine.
A pipeline with dista tasks can spread one run's cluster over several
workers of its pool (#268). workers = N (or --workers N) on a pipeline
placed worker:<pool> lets each run use up to N more machines than the one
that starts it; without it, a run stays on one machine, as before. This is
what Pipeline[workers=] means on a pool. A local run forks MORCA_WORKERS
dista workers (default 4) and ignores the pipeline's workers; workers
applies on EC2 (worker instances), on Kubernetes (dista workers) and on a
pool (extra machines).
[pipelines.fanout]
placement = "worker:lab"
workers = 2 # up to 2 more machines join each run- The worker that claims the run starts it as a cluster of its own: a
dista head on a free port, a dista worker, and the run's driver joining
them. It reports the head's address, the one it was started with
--advertise <address>(orMORCA_WORKER_ADVERTISE), to the UI. - Another worker of the pool with room and nothing to start is given the run to join: it starts a dista worker against that head, sized as a run of the pipeline would be on its machine and counted against its room as one. dista uses a worker that joins a running job at once.
- A cluster token. morca makes one for each run start and gives it, as
DISTA_TOKEN, to the run's head, its driver and every member, the members over the worker API's TLS. dista refuses any connection without it (dista#124), so nothing else on the network joins the run, submits to it or fetches its results. It authenticates, it doesn't encrypt: keep the pool's machines on a private network. - When the run ends, each member is told to stop on its worker's next
claim, and a member's dista worker gives up on a head that's gone within
10 s anyway.
orca.run_memberskeeps who joined each run start, with its grant, and when and why it ended.
A worker without --advertise neither starts a run as a cluster nor joins
one: such a run runs on its machine alone.
On a pool morca grows ([pools.<name>]), this is set up for you:
- every Hetzner machine joins a private network of the stack's, and its worker advertises its address there, read from the provider's metadata (EC2: its private address);
- its worker runs with host networking, so dista's ports are on that address;
- a running run that wants more members than have joined counts as the
pool's demand, so the pool grows for it (up to
max_machines), and the new machine joins the run; - a machine running only a member is busy, not idle.
On the server:
morca worker-token create --pool bgent --label bgent # prints the token, once
morca worker-token list # id, pool, label, state, worker, last seen
morca worker-token revoke 3A worker token is an API token, orca_ and 64 hex digits, like morca login's. The server keeps only its SHA-256, in orca.workers. It is scoped
to one pool. It can claim that pool's runs and make the state calls of the
runs it claimed. It can do nothing else, and a user's token gets nothing
from the worker API. It lasts --days, by default and at most
MORCA_WORKER_TOKEN_DAYS (365). A revoked token stays listed as revoked, and
the worker using it is refused and exits. Each claim is a row of
orca.audit, user worker:<label>, source worker.
MORCA_WORKER_TOKEN=orca_... MORCA_IMAGE=ghcr.io/morca-dev/census:0.11.0 \
MORCA_WAREHOUSE=s3://... MORCA_LOGS=s3://... MORCA_SECRET_FETCH=/etc/orca/fetch-secret \
morca worker --pool bgent --remote https://orca.mlake.dev| Flag | Default | |
|---|---|---|
--pool |
(required) | the pool, which must be the token's |
--remote |
MORCA_REMOTE |
the UI; https, or http to this machine only |
--token-file |
MORCA_WORKER_TOKEN |
the token, the file's first line |
--max-runs |
1 | runs at once on this worker |
--poll-seconds |
2 | how often it asks for a run, and beats |
--dir |
$TMPDIR/morca-worker |
where a run's output is staged |
--image |
MORCA_IMAGE |
the worker's version (below) |
--name |
the host name | how Ops and the run's page name it |
A worker with MORCA_DATABASE_URL set refuses to start.
A run it is given starts the way the local runner starts one. The pipeline's
binary runs at the path it was registered with, which the worker's image
must have too, as <binary> resume <run>, in a process group of its own.
It gets the pipeline's env, its secrets read on the worker with the
worker's own MORCA_SECRET_FETCH (the spec names them, never their values),
and MORCA_STATE_URL and MORCA_STATE_TOKEN in place of a database. Data and
logs go straight from the worker to MORCA_WAREHOUSE and MORCA_LOGS, as
any run's do. When the process ends, <binary> keep-log appends its output,
masked, to the run's driver.log.
A pipeline with a request (cpus, memory_gb: morca.toml)
is admitted against the worker's own host, as the scheduler admits a local
run: each claim says what the worker can grant (its CPUs and memory, the
cgroup's limits if set, less MORCA_RESERVE_CPUS, 0, and
MORCA_RESERVE_MEMORY_MB, 512, of the worker's environment), and the UI
gives it a run only if the run's minimum fits less what the worker's
running runs were granted. The run gets its grant in MORCA_GRANT_CPUS and
MORCA_GRANT_MEMORY_MB, recorded on the run. One that doesn't fit stays
queued, held saying which worker had no room, for another worker of the
pool. A worker from before requests says nothing of its room, and its runs
are started as before.
Its runs yield the CPU to the worker and the rest of its host as the
scheduler's do (hetzner.md): a run's cgroup
has cpu.weight MORCA_RUN_CPU_WEIGHT (20) and, with a CPU grant,
cpu.max; without one the run starts at nice MORCA_RUN_NICE (10).
A worker also reads its own host's memory pressure
each poll (the same files, in the same order, and the same
MORCA_ADMIT_MEMORY_PRESSURE of its own environment) and claims nothing while
some avg10 is over the limit; the runs it has go on. It sends the reading
with its heartbeat, so its pool's queued runs say waiting: memory pressure 23% (limit 10%) on worker <name>, and /api/workers (memory_pressure)
show it. A hold that lasts MORCA_ADMIT_PRESSURE_GRACE_SECONDS (600) with
none of its runs going lets it claim one at a time, and raises the
memory-pressure alert once. Its runs get MORCA_MEMORY_PRESSURE_FILE too.
The simplest worker is the server's own image in Docker, at the same tag,
so the pipeline binaries are at the paths they were registered with
(/opt/morca/pipelines/<name> in a project image).
On the server, once, a token for the pool:
morca worker-token create --pool bgent --label bgentOn the worker, worker.env (mode 600) holds the token, the bucket and
where secrets come from:
MORCA_WORKER_TOKEN=orca_...
MORCA_WAREHOUSE=s3://<bucket>/warehouse
MORCA_LOGS=s3://<bucket>/logs
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=<location>
AWS_ENDPOINT_URL_S3=https://<location>.your-objectstorage.com # S3-compatible storage only
MORCA_SECRET_FETCH=sh /etc/orca/secret-fetchThen:
docker run -d --name orca-worker --restart always --env-file worker.env \
-v "$PWD/secrets:/etc/orca/secrets:ro" -v "$PWD/secret-fetch:/etc/orca/secret-fetch:ro" \
ghcr.io/<owner>/<image>:<tag> worker --pool bgent --remote https://morca.example.comsecret-fetchcan be a copy of the server's (deploy/hetzner/server/secret-fetch): it prints/etc/orca/secrets/<id>, so a pipeline'sGITHUB_TOKEN = "orca/github-token"readssecrets/orca/github-token.- Pass
--image(orMORCA_IMAGE) only if the pipelines were registered with--image, and then the same value: Versions. - The worker needs only outbound HTTPS, to the UI and to the bucket.
docker logs orca-workershows each run it starts and how it exited; Ops > Pools shows its heartbeat.
Deploying. The worker runs the code of its own image, so keep it on
the server's tag: deploy the server first (deploy.sh <tag>), then
restart the worker on the same tag (docker rm -f orca-worker, then the
docker run above with the new tag). In between, a worker on the old tag
runs the old code, unless the pipelines are registered with --image: then
it is given nothing until it matches.
A pipeline binary with MORCA_STATE_URL records its run through
morca._internal.http_store.HttpStore, the HTTP implementation of the RunStore
trait that PgStore and SqliteStore implement. Each call is a POST /api/worker/store/<call> of JSON. The UI makes the same call on PgStore,
so every check is the same code as for a run that writes to Postgres
itself: the run is still this process's, it isn't cancelled, and its
heartbeat is fresh enough to commit. The calls are run_info, claim,
version, ran_version, interrupt, prior, steps, start_attempt,
finish_attempt, skip, pinned_reads, record (reads and commits, so
lineage and find_commit's retry guard work as before), fence,
set_log, finish_run, defer_run, cancel_requested and beat.
Encoded inputs and outputs travel as hex.
A call that can't reach the UI, or gets a 5xx, is retried for
MORCA_DB_RETRY_SECONDS, as Postgres is. A call that is refused (401, 403,
or 409 for a lost or cancelled run) is final, and the run stops. A fence
that can't be made within half the stale limit refuses the commit.
- A dead worker. Each run's heartbeat thread beats over the API. It
also exits its process once the worker that started it (
MORCA_WORKER_PID) isn't its parent any more. So when a worker is killed, or its machine goes to sleep, the run's heartbeat goes stale. A run whose worker is gone is queued again for the pool, whatever its trigger (a manual, UI or rerun run too, which on other runners failsinterrupted): one whose heartbeat is stale and whose worker hasn't been seen for half the stale limit (MORCA_STALE_SECONDS), or has been seen since without it (a restarted worker). Its next start resumes it after its last step done. A run a worker claimed and never started has its start given back. Each is a row oforca.audit(userscheduler, action/api/worker/requeue, saying why) and a line of the scheduler's log. A run that started counts that start, and pastrun_retriesit fails as on any runner, so a run that takes its worker down can't loop. A run whose process stopped while its worker goes on is the dead-run rule every runner has: queued again withinrun_retriesif the scheduler retries its trigger, else failedinterrupted: its process stopped, resumable from the UI. A machine that wakes up finds its run lost at its next call, and a commit is fenced. - A process that never claimed its run (a missing binary, a
resumethat refused) has the run failed, saying why (/api/worker/exited). - What a run uses. The worker samples each run's process tree every
poll, as the scheduler does a local run's, and sends the samples with
/api/worker/exited; the UI records each attempt's peak memory and CPU from them (the UI). Beside each run's process it also starts<binary> sample, as the scheduler does beside a local run, which writes the machine's metrics to<MORCA_LOGS>/<pipeline>/<run>/metrics/worker-<name>.jsonl, so the run's page has its Machines card (MORCA_SAMPLE_SECONDS=0in the worker's environment turns it off). - Versions. A run needs its pipeline's image (
morca register --image;morca config show <pipeline>prints it) to be the worker's (--image,MORCA_IMAGE), both empty included, so there's no silent drift. A worker of another image isn't given the run, and the run stays queued, held saying why. The worker checks the spec again and puts back a run that isn't for it (/api/worker/release). Pin the same tag indeploy.shand the worker's script.
Ops > Pools (/ops/pools?kind=self-hosted, and /api/pools as JSON; see
ui.md) has a row per worker token, the machine its worker
runs on, in pool worker:<pool>: its name, state (live, or gone when
the token is revoked or expired or the worker hasn't been heard from for
MORCA_STALE_SECONDS), image, CPUs, memory, current runs and last
heartbeat, and the worker's token and label. Its page lists the runs it
claimed, and draws the machine's samples over its latest run (the
sampler the worker starts beside each run). /api/workers is still each token as JSON, and /ops/workers
redirects to Pools. A run's page says where it ran: the worker's name, and
the token's id and label.
Outbound HTTPS only, through the server's existing Cloudflare Tunnel. No firewall change is needed, and Postgres stays reachable only on the server.
By default morca ui answers the worker API (/api/worker/...) on the same
port as its pages, so a page that is slow to render (a large pipeline's
Overview, Control's charts) can hold up a worker's claims and heartbeats,
and its runs' store calls, behind it. Set MORCA_WORKER_ADDR (host:port)
and morca ui also serves the worker API on that address, from a thread of
its own:
MORCA_UI_ADDR=127.0.0.1:8270 MORCA_WORKER_ADDR=127.0.0.1:8271 morca uiPoint morca worker --remote at it (through the tunnel, a hostname of its
own for that port); the runs it starts record themselves there too. That
address answers only the worker API: anything else is 404. The UI's own
port keeps answering the worker API as well, so workers can move over one
at a time. Unset, nothing changes.