Daytona is the only execution provider in the first StackShip release. StackShip does not ship a customer-operated or self-hosted execution agent today.
Cloudflare Workers and Workflows coordinate execution; OpenTofu plan, policy, and apply processes run inside a disposable Daytona sandbox. A Worker never runs OpenTofu itself.
Bound execution
Every attempt binds:
- one run and attempt id;
- one phase: plan, policy, or apply;
- one exact OpenTofu version;
- one execution region;
- one logical runner release identity and one standard Daytona base profile;
- immutable configuration, lock-file, plan, and state digests; and
- one sandbox identity.
StackShip accepts any exact stable OpenTofu release from 1.6.0 up. Aliases,
prereleases, and version ranges are rejected. A version becomes runnable once
its immutable toolchain pin exists: the pin is created the first time any
workspace selects the version, from the official upstream SHA256SUMS and a
checksum-verified read of the linux_amd64 archive. Every sandbox bootstrap
verifies the downloaded archive and extracted binary against that pin, and the
runner re-verifies the installed binary against the run’s immutable manifest
plus an exact tofu version -json check.
Production configuration retains a schema-v3 runner catalog holding one
logical release digest for all versions. The regional runner name is a pure
function of version, digest, and execution region, for example
stackship-runner-opentofu-1-12-5-<digest-prefix>-us. A malformed catalog,
mutable name, or name/digest/region mismatch disables managed execution. The
run persists the full digest, regional name, and execution region, so a later
catalog release cannot retarget an existing run. Launch-control allowlists
accept either exact regional names or the full release digest, which admits
every version derived from that digest. These compatibility values identify
reviewed StackShip runner releases; they are not Daytona snapshot names or
OCI image digests.
StackShip does not publish custom Daytona snapshots. Every attempt starts from
Daytona’s standard daytona-medium base. The orchestrator transfers each
generated StackShip runner bundle and the policy-capabilities manifest through
bounded, gzip-compressed Daytona process-command chunks, reconstructs each file
at a temporary path, verifies it against the digest of the reviewed
Worker-embedded bytes, and only then moves it into place. This avoids relying
on the serverless multipart uploader for runner assets. Each command carries
at most 256 KiB of encoded compressed data; an uncompressed asset is capped at
3 MiB and its compressed encoded transfer at 4 MiB. An oversized release fails
closed before transfer. Every chunk is written to its own deterministic
staging path, so a lost Daytona command response can be retried once without
duplicating bytes. Only thrown transport failures are retried; a nonzero
command, digest mismatch, or install failure still fails closed. Bootstrap
removes the staging files after verifying the reviewed bundle bytes.
It downloads the selected OpenTofu archive and OPA
binary from their official GitHub releases, verifies the committed archive and
binary SHA-256 values, and installs the toolchain before runner start. The
sandbox therefore receives the requested OpenTofu version on demand without a
custom image, runner registry, or snapshot-release pipeline.
Production launch controls independently gate new runs and apply. Sandbox creation rechecks the stored region immediately before calling Daytona and never silently falls back to another region. Daytona owns the standard base’s resource shape: 2 CPU, 4 GiB memory, and 8 GiB disk. StackShip does not send resource overrides when creating the sandbox.
Before public activation, an audited dark-canary workspace allowlist can admit
only exact organization-and-workspace pairs while the public gate stays
closed. A ws_ ID alone is insufficient and cannot authorize the same
workspace ID in another organization. The allowlist does not bypass new-run,
apply, provider, region, suspension, capacity, policy, or logical runner
release controls.
It is cleared after canary evidence is collected and before readiness is
signed.
Bootstrap first verifies the uploaded main runner, policy builder, and
capabilities manifest byte-for-byte. It then verifies the downloaded OpenTofu
archive, extracted tofu executable, and OPA executable against committed
SHA-256 values, followed by their reported versions and the bundled runner
protocol. The phase runner independently hashes its selected executable
against the immutable manifest before OpenTofu or OPA execution. A truncated
bundle or mismatched toolchain therefore fails before tofu init, plan,
policy evaluation, or apply.
After initialization, the runner records resolved modules and providers in a
deterministic dependency bundle. Modules resolved from the private
module registry are captured the same way: the plan
authenticates to the registry with the injected TF_TOKEN_stackship_run
run credential, and the apply replays the recorded bytes offline without
contacting the registry. OpenTofu may omit its module manifest for a
root-only configuration; in that case StackShip records the canonical empty
manifest instead of rejecting an otherwise valid plan. Transient provider
installation lock markers are excluded while every resolved provider payload
remains digest-bound. The bundle also records the plan sandbox’s
Linux-enriched .terraform.lock.hcl. This matters when a configuration was
uploaded with a lock file generated on macOS or by Terraform: the plan can
verify and add the Linux package checksum while resolving providers, and the
apply must reuse that verified result instead of trying to install an
already-bundled provider directory onto itself.
The apply sandbox first verifies the complete dependency bundle. Only after
that verification succeeds does it atomically replace the uploaded lock file
with the plan-enriched copy, then run tofu init -lockfile=readonly against
the local provider mirror. The original uploaded lock-file digest remains part
of the immutable run input, while the enriched lock-file bytes and digest are
bound by the approved dependency bundle. StackShip does not remove checksum
verification or fall back to a provider registry during apply.
Capacity
Each sandbox-create intent atomically acquires one regional capacity lease in PlanetScale. Unique regional slots prevent concurrent creates from exceeding the configured ceiling. Replaying an attempt reuses its lease rather than reserving another.
The Daytona provider identity is attached to the lease immediately after creation and before runner upload or toolchain preparation. A failed upload or bootstrap therefore retains the exact cleanup authority instead of leaving an anonymous sandbox behind. Successful cleanup releases the lease in the same durable termination path. The five-minute reconciler asks PlanetScale for at most 32 terminal, not-yet-cleaned attempts whose exact identity still owns an unreleased lease. It never derives cleanup authority from Daytona’s global sandbox list.
For an ordinary terminal attempt, the orchestrator compares the complete
recorded provider, region, sandbox id, name, and labels, stops and deletes that
exact sandbox, and proves a subsequent provider inspection is deleted.
Only then does the API atomically mark the attempt cleaned and release its
exact capacity lease. A failed acknowledgement is safe to retry because
provider absence and the database update are both idempotent. When that
acknowledgement completes the last attempted sandbox for a terminal plan or
policy failure, the same transaction also releases the run’s exact managed
state lock and writer reservation. A provider or acknowledgement failure
quarantines only that target: the bounded pass continues later leases, then
raises one sanitized aggregate failure so the cron remains observable without
logging provider messages or identities.
Run-level failure cleanup follows the same order for every recorded identity: prove provider absence, acknowledge that exact attempt, then finalize the run. Deleting a sandbox without the matching database acknowledgement is never reported as complete.
Daytona may accept stop for an ephemeral sandbox before its automatic
deletion leaves the transitional state. The phase cleanup Workflow therefore
retries the same recorded identity every 30 seconds with constant backoff for
at most 20 attempts; each provider call has a ten-minute timeout. It never
creates a replacement, changes region, or releases capacity while deletion is
unproven. Exhausting those attempts records the sanitized cleanup failure and
leaves the exact lease for the five-minute reconciler.
An uncertain apply is stricter. It becomes eligible only after its runner
session is absent, revoked, or expired, and reconciliation may acknowledge it
only when exact provider inspection already reports deleted. If the sandbox
still exists, or Daytona reports creating, transitional, unknown, or
identity-mismatched state, the attempt stays quarantined with its lease and
recovery evidence intact. Explicit run recovery changes the preserved attempt
to an ordinary terminal target, after which the same exact cleanup path may
delete a remaining sandbox. Operators never release a capacity lease through
a direct database update.
Streaming
Configuration archives, lock files, plan artifacts, logs, and Cloud-backend
state uploads move through Worker streams and encrypted R2 objects. The API enforces a
100_000_000-byte plaintext launch limit and bounded read-ahead. It does not
materialize a complete project or artifact in Worker memory.
OpenTofu reads its run-pinned predecessor and publishes its successor directly
through /api/v2. The runner does not receive or upload a state artifact and
does not run tofu output. After a zero-exit apply it requests only the exact
session-bound state id, checksum, serial, and fence; any missing or mismatched
publication is apply uncertainty.
Runner logs are sanitized before persistence. Customer credentials, variable plaintext, internal capability tokens, and raw state are not included in operational analytics. The API validates the complete session-bound log event but persists only its redacted, display-ready message bytes in the encrypted log object. Protocol-envelope metadata is not mixed into terminal output. Dashboard and OpenTofu log reads therefore stream the original human-readable output, while an exact plaintext digest prevents a same-sequence replay from substituting different content.
The streaming redactor holds any suffix that could be the start of a known secret across UTF-8 chunk and nonterminal flush boundaries. When a stream ends, an unmatched suffix is ordinary output: the runner emits it exactly once, waits for that final log append, and only then submits the terminal result. A complete known secret remains replaced even when it spans chunks. Repeated-prefix and self-overlapping secrets keep their masking state while a possible suffix is held, so finalization cannot expose an already matched portion.
Apply stdout and stderr have an additional recovery guard before the ordinary
log pipeline. OpenTofu’s final persistence fallback can print an entire state
JSON object to human-view stderr when both the Cloud backend and local
errored.tfstate write fail. JSON view emits an Emergency state dump
envelope containing the same raw state on stdout. The runner holds every
JSON-looking object from both streams in a private 0600 disk spool until its
complete top-level shape is known, independent of key order. A state object or
emergency envelope is never emitted as a log event; its state bytes are
streamed into the bounded emergency-state artifact instead. Non-state JSON is
replayed from disk with backpressure and without a full-file read. A truncated
object, a second state object, a destination conflict, excessive nesting, or
byte 100_000_001 fails closed as apply uncertainty and suppresses the
ambiguous remainder. Stdout and stderr use separate private 0600 candidates;
exactly one may be promoted. Explicit same-stream or cross-stream ambiguity
removes every guard-owned candidate so raw state and provider secrets do not
remain in a reusable sandbox. An unrelated later log-delivery failure keeps a
single classified candidate available while the apply remains uncertain.
An errored.tfstate file or guarded output candidate comes from the same
customer-code sandbox trust domain as providers, provisioners, and
local-exec. It is untrusted emergency evidence, not OpenTofu-attested or
verified state, regardless of which bounded capture path found it. StackShip
records its exact digest and byte count but never makes it current
automatically. Import remains an explicit operator recovery decision with a
typed reason, attestation, provider reconciliation, and predecessor, lineage,
fence, and successor checks.
One attempt-wide queue assigns and delivers stdout, stderr, and system events
in a single global sequence. Only one log append is in flight at a time.
Network failures, 408, 429, and server errors retry the exact serialized
event at most three times; a conflict or another terminal client error does
not advance to a later sequence. If delivery remains unavailable, the runner
still submits a bound terminal result instead of losing the callback. Plan and
policy fail safely, and a non-successful apply retains the recovery boundary.
Credentials
Cloud federation is preferred. Verified AWS, Azure, and Google Cloud credential profiles issue phase-bound identity material through the internal credential broker.
A generic-secret manifest contains only credential and version ids, environment names, expiry, and a one-time handle. After exchanging its attempt-bound runner session, the runner presents those same fields to the API. The API reloads the envelope from PlanetScale and the broker permits one redemption bound to the exact organization, workspace, run, attempt, phase, sandbox, input digest, profile version, names, and expiry. Plaintext is returned only to that runner and merged into the OpenTofu process environment.
The envelope and plaintext are not Daytona sandbox-creation parameters or Workflow state. Generic values join the runner’s known-secret set before OpenTofu starts, so stdout and stderr redaction happens before any log event is sent back for persistence. The redemption document is bounded by the 64 KiB plaintext limit; project archives and artifacts continue to use streaming rather than Worker buffering.
A speculative pull request receives no credential by default. A fork receives
none unconditionally. A same-repository speculative run may use only a
separate, verified speculative_plan federation profile. A normal plan or
plan-and-apply profile cannot cross that boundary. Generic-secret profiles
never support speculative runs.
Network policy
Daytona sandboxes have unrestricted public outbound network access. Bootstrap requires official OpenTofu and OPA release endpoints, and customer operations may require providers, modules, registries, VCS sources, provisioners, and other public endpoints that StackShip cannot enumerate correctly. StackShip does not treat runner egress as a security boundary.
AWS GovCloud and AWS China federation are rejected in the first release. Their audiences, trust principals, regional token exchanges, endpoint suffixes, and canaries require a separate partition-aware identity contract. That restriction does not come from sandbox networking.
Unrestricted egress means any provider, data source, module, or process such as
local-exec can attempt to transmit every secret exposed to that run.
Customers must trust the repository and its dependencies, prefer
provider-native short-lived federation, and scope credentials to the minimum
authority and lifetime. StackShip still keeps credentials out of sandbox
creation parameters and binds their redemption to one immutable attempt.
Private endpoints are not included in this release; private connectivity is future private-runner work. Public egress from a Daytona sandbox does not give the sandbox access to a customer’s private network.
Policy runner
Policy publication and evaluation use the same standard Daytona base and unrestricted sandbox egress needed by bootstrap. The runner receives only the encrypted source or sanitized policy input needed for that phase and the fixed OPA capabilities contract. The capabilities contract, input isolation, and absence of customer cloud credentials constrain policy execution; the sandbox network is not a policy security boundary.
Production-dark evidence
The protected production-dark canary exercises one distinct dedicated
workspace in every Daytona region enabled for the organization after the
exact candidate build is deployed. The current Daytona organization exposes
the required container class in us; eu remains launch-disabled until
Daytona makes that class available there. A GitHub
nonce-file commit triggers the normal App webhook and source-ingestion
Workflow. The harness uses only bounded authenticated public API reads and the
exact approval action; it never invokes an internal Workflow endpoint and
never reads state bodies, archives, or artifacts.
For each run, version 2 evidence cross-links the ready source ingestion,
source/run Workflow identities, both executing Workflows’ self-attested build
SHAs, one authoritative apply process-start record, the starting and
resulting state serials, provider cleanup, and the terminal GitHub Check.
Success requires both observed build SHAs to equal the candidate, exactly one
apply start backed by process_started runner and start-grant evidence, state
serial exactly +1, no live Daytona sandbox with the run label after bounded
polling, and one successful Check whose external_id is the run ID. The
exact run review must match the configured workspace and region both before
approval and at terminal observation. Cleanup polling and evidence use that
verified terminal region, so evidence from one region cannot satisfy another
region’s gate.
Local tests prove this harness fails closed, but they are not production evidence. Only the protected Action’s exact-build document, validated before artifact upload, can satisfy the execution portion of the release canary.
Failure behavior
Temporary Daytona, R2, database, or Workflow errors retry from durable identities. Labels and stored provider ids reconcile a lost create response. Runner start uses one deterministic execution nonce and Daytona session id. The broker retains only the original bootstrap authority until its existing expiry. Workflow history stores only the attempt id: the runner-start service reacquires and redeems the exact attempt-, phase-, and sandbox-bound handle in Worker memory. Reissuing the same authority does not extend its original expiry, and any changed non-expiry binding is denied. Every exact retry carries the same bootstrap JTI, and the API permits that JTI to create at most one runner session. A retry therefore reconciles the recorded Daytona process instead of minting another active credential or starting a second logical runner. Daytona can expose that deterministic session before it contains a command. StackShip treats this as a durable intermediate state and starts the expected command in the existing session; it creates a session only after a not-found response and never deletes or recreates an empty session. More than one command or a command that does not match the expected runner fails closed as reconciliation uncertainty. Apply ambiguity is never converted into success; it produces an explicit uncertain or recovery status.