Deployment

StackShip deploys six Workers to the Cloudflare account that owns the stackship.run zone. The orchestrator and credential broker are internal service-binding targets and do not have public domains.

Domains

Wrangler custom domains create the required DNS records and certificates. A hostname cannot already have a conflicting CNAME.

r.stackship.run is a separately provisioned PostHog-managed US reverse proxy, not a Worker custom domain. Its DNS record is a DNS-only CNAME to the unique PostHog-managed proxyhog.com target issued for the production organization. The dashboard keeps the public phc_ project token and uses this first-party origin as api_host; no privileged PostHog key is stored in a Worker.

Do not use GET / as the relay health check: the managed proxy intentionally returns 404 there. Verify GET /static/array.js and the production project’s /array/<public-project-token>/config.js path instead; both must return JavaScript successfully, and an OPTIONS /e/ preflight must advertise GET,POST,OPTIONS.

The credential broker and orchestrator must set workers_dev = false and must not declare routes. They are reachable only through explicit Worker service and Workflow bindings; an accidental *.workers.dev deployment is a release-blocking public surface.

Database

The PlanetScale PostgreSQL database is linked through Hyperdrive configuration 397048f06dbc43c1a921c92258384a40, bound to the API Worker as HYPERDRIVE. Query caching is disabled on this configuration because it serves Better Auth sessions, permissions, organizations, and other consistency-sensitive reads.

Do not use SQL comments as cache hints. Cloudflare does not document them as a cache-control API. If product reads later benefit from caching, create a second cache-enabled Hyperdrive configuration and keep identity traffic on this cache-disabled binding.

The repository root contains an ignored .env.migrations file with the direct administrative connection:

DATABASE_URL=postgresql://…

Migration commands use Node 24’s optional env-file flag. A local command loads this file when it exists; a clean CI checkout without it continues with its already-exported DATABASE_URL:

pnpm db:generate
pnpm db:migrate

Drizzle reads the shared schema from packages/db/src/schema/ and writes one ordered migration stream to packages/db/migrations/. Platform tables repeat organization_id directly, and tenant-owned unique and lookup indexes begin with that column. Repository methods require an explicit tenant context and query with both the organization ID and the object ID; an active organization stored in a browser session is never a database authorization boundary.

Migration 0021_noisy_karen_page adds the global GitHub installation lifecycle fence, tenant-free delivery identity authority, and global delivery constraint. It rejects ambiguous legacy IDs and backfills every tenant delivery before adding a restrictive foreign key. Insert and update triggers mirror legacy Worker writes into the global identity, reject identity changes, and prevent a tenant write from claiming an existing global-only result. Apply it before deploying Workers that use global delivery identities. The migration intentionally retains the older tenant-scoped unique index beside the global index so old and new Workers remain compatible during a rolling deployment.

Migration 0026_tofu_oauth_callback_range repairs the fixed public tofu-cli OAuth client on databases that already applied migration 0006. It expands the exact loopback callback range from ports 10000–10009 to ports 10000–10010 without changing the historical migration. The update is conditional and can be replayed safely; deploy it before relying on OpenTofu’s port-10010 login listener.

Migration 0030_generic_secret_envelope_constraint replaces oversized PostgreSQL regular-expression repetition bounds with explicit length checks for generic-secret ciphertext and wrapped keys. PostgreSQL limits bounded regular-expression repetitions to 255, so the earlier constraint raised a database error only when a non-null encrypted envelope was inserted. Apply 0030 before deploying or enabling generic-secret credential versions. The migration keeps the constraint name, validates existing rows, and never changes or decrypts stored envelope material.

Migration 0031_organization_membership_uniqueness enforces one authoritative Better Auth membership per user and organization. Before applying it, run this read-only preflight; it must return no rows:

SELECT organization_id, user_id, count(*) AS membership_count
FROM member
GROUP BY organization_id, user_id
HAVING count(*) > 1;

The migration fails closed if duplicate membership records already exist. It does not choose a role, delete a record, or otherwise rewrite identity data. Stop the rollout and reconcile any returned records through an audited identity operation before retrying the migration.

Migration 0032_authentication_rate_limits creates the durable Better Auth rate-limit counter table. Apply it before deploying the API build that enables database-backed rate limiting. The table contains normalized client IP and authentication-route keys, counters, and request timestamps; it does not contain credentials, request bodies, email addresses, or bearer tokens. Treat the table as operational personal data and retain the existing database access boundary.

Migration 0033_authentication_rate_limit_identities upgrades that table to Better Auth 1.6.25’s Drizzle adapter contract. It assigns each existing counter an adapter id equal to its already-unique key, makes that id the primary key, and retains a unique constraint on the request key. Existing counts and timestamps are preserved. Apply 0033 before deploying the API schema that includes database-backed rate limiting; otherwise Better Auth fails closed during adapter initialization because the id field is absent.

Migration 0034_cold_omega_red permits a policy evaluation’s input_digest to be null only while the evaluation is running. This supports atomically reserving the deterministic policy-check row during a successful plan commit, before the policy runner has produced its sanitized-input digest. The existing digest-format constraint still applies once a digest is present, and the completion constraint requires a non-null digest plus complete outcome, composition, result-count, and timestamp evidence. Apply 0034 before deploying the API build that reserves pending policy checks; older Workers continue to read existing non-null evaluations during the rolling deployment.

Runs whose successful plan committed before this API build may not yet have a policy-evaluation row. The compatibility path is intentionally limited to a run whose persisted workflow_build_sha is a valid earlier build SHA and differs from the currently deployed APPLICATION_BUILD_SHA. It derives the policy-check ID from the exact run ID, then binds the same tenant-scoped plan, policy snapshot, and plan digest used by normal reservation. A current-build run with a missing reservation, a changed identifier, or mismatched evidence still fails closed. This bridge creates no broad migration backfill and does not change the atomic reservation invariant for new plan commits.

Never point .env.migrations at Hyperdrive or commit it to source control. PlanetScale’s sslrootcert=system URL parameter is normalized in memory for node-postgres; sslmode=verify-full remains enabled.

PlanetScale’s stable postgres role must own the application tables and the drizzle.__drizzle_migrations schema. The credential in .env.migrations must inherit PlanetScale’s postgres permission so Drizzle can perform DDL, but the credential itself must not become the long-lived object owner.

If a Hyperdrive or other integration role bootstraps the schema, use PlanetScale’s Reassign objects action to transfer its objects to postgres before running later migrations. Verify the transfer and run pnpm db:migrate before rotating or deleting the original role.

Secrets

Set Worker secrets without printing them:

wrangler secret put BETTER_AUTH_SECRET --config apps/api/wrangler.jsonc
wrangler secret put CLOUD_TOKEN_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put ARTIFACT_CAPABILITY_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put GOOGLE_CLIENT_ID --config apps/api/wrangler.jsonc
wrangler secret put GOOGLE_CLIENT_SECRET --config apps/api/wrangler.jsonc
wrangler secret put GITHUB_APP_ID --config apps/api/wrangler.jsonc
wrangler secret put GITHUB_APP_PRIVATE_KEY --config apps/api/wrangler.jsonc
wrangler secret put GITHUB_WEBHOOK_SECRET --config apps/api/wrangler.jsonc
wrangler secret put IDEMPOTENCY_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put LOG_CAPABILITY_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put INTERNAL_GRANT_PRIVATE_JWK --config apps/api/wrangler.jsonc
wrangler secret put INTERNAL_GRANT_PUBLIC_JWK --config apps/credential-broker/wrangler.jsonc
wrangler secret put PLATFORM_KEK_V1 --config apps/credential-broker/wrangler.jsonc
wrangler secret put RUNNER_BOOTSTRAP_PRIVATE_JWK --config apps/credential-broker/wrangler.jsonc
wrangler secret put RUNNER_BOOTSTRAP_PUBLIC_JWK --config apps/api/wrangler.jsonc
wrangler secret put RUNNER_CALLBACK_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put RUNNER_RUN_TOKEN_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put REGISTRY_DOWNLOAD_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put WORKLOAD_IDENTITY_PRIVATE_JWK --config apps/credential-broker/wrangler.jsonc
wrangler secret put WORKLOAD_IDENTITY_PUBLIC_JWK --config apps/api/wrangler.jsonc
# Set only while old workload-identity keys overlap the active key.
wrangler secret put WORKLOAD_IDENTITY_RETIRING_PUBLIC_JWKS --config apps/api/wrangler.jsonc
wrangler secret put BROKER_CONTROL_SHARED_SECRET --config apps/api/wrangler.jsonc
wrangler secret put BROKER_CONTROL_SHARED_SECRET --config apps/orchestrator/wrangler.jsonc
wrangler secret put BROKER_CONTROL_SHARED_SECRET --config apps/credential-broker/wrangler.jsonc
wrangler secret put CONTROL_PLANE_SHARED_SECRET --config apps/api/wrangler.jsonc
wrangler secret put CONTROL_PLANE_SHARED_SECRET --config apps/orchestrator/wrangler.jsonc
wrangler secret put DAYTONA_API_KEY --config apps/orchestrator/wrangler.jsonc
wrangler secret put POLICY_BUILD_HMAC_KEY --config apps/api/wrangler.jsonc
wrangler secret put PLATFORM_ADMIN_PRINCIPAL_IDS --config apps/api/wrangler.jsonc
wrangler secret put RUNNER_SNAPSHOT_CATALOG --config apps/api/wrangler.jsonc

The production GitHub App slug is the non-secret Wrangler variable GITHUB_APP_SLUG=stackship-run. Configure its setup URL as https://stackship.run/api/control-plane/v1/github/installations/callback, its webhook URL as https://api.stackship.run/v1/github/webhooks, and do not enable GitHub user OAuth. Grant only Checks read/write, Contents read, Metadata read, and Pull requests read. Select only push and pull_request under manual event subscriptions. GitHub automatically sends installation and installation_repositories lifecycle events to every GitHub App; there is no manual subscription checkbox for those lifecycle events.

CLOUD_TOKEN_HMAC_KEY is a dedicated high-entropy secret used only for keyed service-account-token lookup. Generate it independently, do not reuse BETTER_AUTH_SECRET, and rotate it through the documented token migration because changing it immediately invalidates every service-account token.

IDEMPOTENCY_HMAC_KEY is a separate API-only secret of at least 32 UTF-8 bytes. It keys the deterministic fingerprint used for exact retries of generic-secret version creation, so PlanetScale never receives either the plaintext environment map or an offline-guessable hash of it. Generate it independently and keep it stable. If it must rotate, wait for the 24-hour idempotency retention window or require clients to use new idempotency keys; old-key replays fail closed.

The API also requires five independent high-entropy HMAC secrets: ARTIFACT_CAPABILITY_HMAC_KEY, LOG_CAPABILITY_HMAC_KEY, RUNNER_CALLBACK_HMAC_KEY, RUNNER_RUN_TOKEN_HMAC_KEY, and REGISTRY_DOWNLOAD_HMAC_KEY. They bind artifact transfers, first-release bounded log reads, terminal runner callbacks, runner control requests, and the five-minute module-registry download capability URLs, respectively. Generate each from at least 32 random bytes. Do not reuse the cloud-token, idempotency, control-plane, broker-control, policy-build, Better Auth, or wrapping keys. Rotation invalidates outstanding capabilities for that exact purpose, so disable new runs and wait through the documented maximum capability lifetime before replacement.

Migration 0018_keyed_generic_secret_fingerprints removes every pre-HMAC generic_secret profile version because StackShip has no plaintext read path with which to re-encrypt it. It also removes only the idempotency rows for the credential-version creation route. Generic profile records, federated provider versions, audit records, and all unrelated idempotency routes remain. This pre-customer migration is intentionally destructive: configure IDEMPOTENCY_HMAC_KEY, apply the migration, deploy the updated credential broker and API, then recreate each generic-secret version through the write-only form.

Migration 0022_wet_black_queen makes the manual-review rule a database invariant. Before applying it to a pre-release environment, disable new runs and confirm that no nonterminal row has auto_apply=true; a durable workflow may already have replayed that old value. The migration normalizes all historical flags to false and then adds the runs_auto_apply_disabled check constraint. Deploy the matching API and orchestrator build before re-enabling run creation.

Identity provider callbacks

Configure Google with the complete secret pair and register the dashboard callback, not the API issuer callback:

https://stackship.run/api/auth/callback/google

Register the shared enterprise OIDC callback as:

https://stackship.run/api/auth/sso/callback

For each operator-provisioned SAML connection, replace <provider-id> with the exact Better Auth provider identifier and register:

https://stackship.run/api/auth/sso/saml2/sp/acs/<provider-id>

Use that same URI as the connection’s samlConfig.callbackUrl. Do not register an api.stackship.run callback: browser identity exchanges must terminate through the dashboard auth boundary so the session remains a host-only stackship.run cookie.

BROKER_CONTROL_SHARED_SECRET is a separate, high-entropy value of at least 32 bytes. The API, orchestrator, and credential broker receive the same value; it must not reuse an auth, token-HMAC, or encryption key. Capability requests also carry an exact X-StackShip-Caller service identity, and the broker accepts each operation only from its expected internal caller.

CONTROL_PLANE_SHARED_SECRET is another independent high-entropy value shared only by the API and orchestrator. It authenticates internal Workflow control calls, including the scheduled run-signal, Workflow-outbox, and artifact-orphan reconcilers. Never reuse the broker, Better Auth, token-HMAC, or encryption secret.

Scheduled recovery

The orchestrator Worker must retain the committed */5 * * * * recovery cron, the 17 3 * * * daily artifact-orphan cron, and its CONTROL_PLANE service binding to stackship-api. Every five-minute tick performs Daytona cleanup reconciliation, run-signal delivery, and a bounded Workflow outbox pass in parallel. Daytona cleanup is database-first: the authenticated API returns no more than 32 terminal attempt identities that still own their exact active capacity lease. The orchestrator proves the complete recorded identity before any provider mutation. Ordinary terminal sandboxes are stopped, deleted, and re-inspected as deleted before the API atomically records cleaned_at and releases the exact lease. A terminal uncertain apply with no live runner session uses absence-only mode: an existing, transitional, unknown, or mismatched sandbox remains quarantined and is never deleted as part of that check. Provider absence can release capacity without deleting the run’s durable recovery records or encrypted artifacts. No provider-list result and no operator database update can substitute for the authenticated acknowledgement. One target failure does not starve the rest of the bounded page: reconciliation continues without acknowledging that target, then raises a sanitized aggregate sentinel containing only bounded internal failure codes.

The outbox pass leases no more than 25 due run, source-ingestion, registry-ingestion, or policy-publication dispatches for 60 seconds and releases failures with bounded exponential backoff. A missing cron trigger, service binding, or matching control-plane secret leaves committed work pending rather than weakening authentication or starting unbound work.

The daily cron creates one deterministic ArtifactOrphanSweepWorkflowV1 instance per UTC date. Each durable step asks the API to scan at most 1,000 R2 objects and carries only the opaque next cursor and aggregate counts. The API deletes a key only when it has a managed artifact shape, is older than 24 hours, and is absent from the exact committed R2-key set in PlanetScale. Do not shorten that grace period during an incident; uncertain metadata acknowledgements intentionally preserve their object until the delayed proof.

POLICY_BUILD_HMAC_KEY is an API-only, independent high-entropy secret of at least 32 bytes. It signs the 15-minute bootstrap used by one Daytona policy build and must not reuse the Better Auth, service-token, control-plane, or artifact-encryption keys.

PLATFORM_ADMIN_PRINCIPAL_IDS is a comma-separated, bounded allowlist of Better Auth principal IDs authorized to read or mutate internal launch controls. A caller must also hold organization administration permissions. Missing or malformed configuration denies every control request. Control snapshots and mutations belong in PlanetScale, never Wrangler variables.

RUNNER_SNAPSHOT_CATALOG is schema-v3 compatibility configuration:

{ "schemaVersion": 3, "imageDigest": "sha256:<64 lowercase hex>" }

One logical release digest covers every OpenTofu version. The regional runner name persisted on a run is derived from the workspace version, this digest’s first 12 hex characters, and the execution region, ending in -us or -eu. These values preserve the existing database and launch-control contract; they are not OCI image digests or Daytona snapshot names. The catalog contains no credential, but install it through wrangler secret put so a runner release does not require a committed Wrangler change. Managed execution and policy publication fail closed when the document is missing, malformed, carries an unknown schema version, or has extra fields. Rotate the secret atomically with the deploy that changes its schema; a schema-v2 document is rejected by schema-v3 code and vice versa. Per-version OpenTofu digests live in the opentofu_toolchains database table (seeded by migration 0043), not in this catalog.

StackShip does not create or publish custom runner snapshots. The orchestrator creates each private, ephemeral sandbox from daytona-medium, transfers its two generated runner bundles and policy-capabilities manifest through bounded gzip-compressed Daytona process-command chunks, and installs the manifest’s exact OpenTofu version and pinned OPA release. Each asset is reconstructed at a temporary path, verified against the digest of the reviewed Worker-embedded bytes, and only then moved into place. Bootstrap then verifies the OpenTofu archive and extracted binary plus the OPA binary against committed SHA-256 values before runner start. Runner bootstrap does not use Daytona’s serverless multipart uploader. Each command carries at most 256 KiB of encoded compressed data; an uncompressed asset is capped at 3 MiB and its compressed encoded transfer at 4 MiB. A larger generated asset disables managed execution until the release is reduced or the reviewed transfer contract changes. Chunks use deterministic per-index staging files so the orchestrator can replay a command once after a thrown Daytona transport error without appending duplicate bytes. Nonzero exits, digest failures, and install failures are not converted into successful retries, and verified bootstrap removes the staging files. The orchestrator records the exact sandbox identity and capacity lease before the first transfer, preserving cleanup authority when preparation fails. The orchestrator generates the bundles before build, test, development, and deployment, so the deployed Worker and uploaded runner code come from the same reviewed checkout. The Daytona API key is needed only by the orchestrator Worker and never belongs in Wrangler vars, Workflow state, a runner bundle, or the catalog.

RUNNER_BOOTSTRAP_PRIVATE_JWK and RUNNER_BOOTSTRAP_PUBLIC_JWK are the private and public sides of one EC P-256 key pair used only to sign and verify five-minute runner bootstrap tokens. The private side exists only in the credential broker; the API receives only the public side. Set kid to runner-v1 on the public JWK. INTERNAL_GRANT_PRIVATE_JWK and INTERNAL_GRANT_PUBLIC_JWK are the two sides of the separate ES256 data-key grant signer. PLATFORM_KEK_V1 is a base64url-encoded 32-byte platform key-encryption key used by the broker to wrap random artifact, variable, and generic-secret data keys. Generic-secret wrapping and authenticated data bind the organization, workspace, profile-version id, and environment-name set. Only the credential broker receives the KEK. Its ACTIVE_PLATFORM_KEY_VERSION must be 1 for the currently configured PLATFORM_KEK_V1; a version must remain configured while any envelope names it. Generate all three key sets independently and never place a private value in Wrangler vars.

WORKLOAD_IDENTITY_PRIVATE_JWK and WORKLOAD_IDENTITY_PUBLIC_JWK are an independent RSA signing pair with a modulus of at least 2048 bits. The private side exists only in the credential broker. Set WORKLOAD_IDENTITY_KEY_ID to the same rotation version in the API and broker Wrangler vars; never reuse the runner-bootstrap or data-key-grant keys. Before enabling credential profiles, confirm that https://api.stackship.run/workload-identity/jwks.json publishes that exact public key and that every cloud provider can fetch it.

RSA modulus values must use canonical base64url unsigned-integer encoding: leading zero octets are rejected. Both Workers also import the configured key and independently require the resulting Web Crypto RSA modulus length to be at least 2048 bits; padding a smaller modulus cannot satisfy the check.

Rotate the workload-identity signing key

The broker signs with exactly one active private key. The API publishes that key first and may also publish up to four retiring public keys from WORKLOAD_IDENTITY_RETIRING_PUBLIC_JWKS. The optional value is a standard JWKS object, is limited to 16 KiB, and has this shape:

{
  "keys": [
    {
      "alg": "RS256",
      "e": "AQAB",
      "kid": "workload-v1",
      "kty": "RSA",
      "n": "<old-public-modulus>",
      "use": "sig"
    }
  ]
}

Every kid must be unique, including the active WORKLOAD_IDENTITY_KEY_ID. A malformed, duplicate, non-RSA, non-RS256, non-signing, or private key makes the JWKS endpoint return 503 with Cache-Control: no-store; StackShip never publishes a partial key set. Retiring private keys must not be placed in either Worker.

Use this order so cached provider JWKS documents and already-issued tokens remain valid:

1. Generate a new independent RSA key pair of at least 2048 bits and retain the old public key and kid.
2. Prepare one API Worker version with the new WORKLOAD_IDENTITY_PUBLIC_JWK, the new WORKLOAD_IDENTITY_KEY_ID, and a retiring JWKS containing the old public key with its old kid. Deploy those three values together. A non-atomic partial update safely returns 503, but must not be left serving.
3. Confirm the public JWKS contains the new key first and the old key second. Verify that neither entry contains d, p, q, dp, dq, qi, or oth.
4. Wait at least five minutes, the published JWKS cache lifetime, before changing the signer. This gives provider caches time to learn the new key.
5. Prepare and deploy one credential-broker version with the new WORKLOAD_IDENTITY_PRIVATE_JWK and matching WORKLOAD_IDENTITY_KEY_ID. Do not configure the old private key there.
6. Decode a newly issued test token and confirm its header uses the new kid; complete one provider exchange before enabling normal issuance.
7. Keep the old public key in the retiring set for at least six minutes after the final token signed by the old key: the five-minute token lifetime plus the bounded clock-skew allowance.
8. Remove the old key by deploying the API with WORKLOAD_IDENTITY_RETIRING_PUBLIC_JWKS unset or exactly {"keys":[]}. Recheck the endpoint before deleting the retired private material from the secure rotation workspace.

Never switch the broker first. A token signed by a key that provider caches have not observed fails federation even when the API is corrected moments later. For emergency revocation, stop credential issuance first and remove the compromised public key immediately; availability is secondary to preventing acceptance of newly forged identities.

The credential broker binds CAPABILITY_HANDLES to its CapabilityHandleStore Durable Object. Bootstrap, runner-secret, and apply-start handles are stored there with an alarm-backed expiry. Generic secret handles store only exact execution ids and digests, environment names, expiry, and the opaque handle; they never store the database envelope or plaintext. Runner-secret and generic-secret records are deleted atomically on first exact redemption. Bootstrap and apply-start redemption retain their exact replay authority until the existing alarm expires, allowing recovery from a lost response without authorizing changed bindings or extending the original expiry. Handle records contain immutable identifiers, digests, encrypted-secret version references, and non-secret workload-federation metadata such as role ARNs or application ids. Plaintext workspace variables and temporary provider credentials are never persisted in a handle or Cloudflare Workflow history. Workflow step results also exclude raw capability handles, bootstrap tokens, execution and callback nonces, and bearer credentials. The orchestrator reacquires a bootstrap handle only inside the authenticated runner-start service call, redeems it in Worker memory, and persists only the attempt id as step evidence. Generic-secret envelopes remain in PlanetScale and never enter Workflow history.

This boundary prevents capability material from entering newly executed step results; deployment does not rewrite older Cloudflare Workflow histories. Treat pre-fix histories as containing expired authority until Cloudflare retention removes them, keep Workflow inspection access restricted, and verify that their runner sessions and apply authorizations are expired or revoked.

Internal resource topology

Provision the resources and deploy their owners before deploying Workers that bind to them.

For a new account, create R2, Hyperdrive, Analytics Engine, and Email Sending resources first. Then deploy the credential broker, apply database migrations, and deploy the one-time orchestrator bootstrap only when the orchestrator Worker does not exist. That bootstrap registers the same Workflow owners but has no cron trigger or API/credential-broker service bindings. Next deploy the API, the full orchestrator, and the API again if its Workflow bindings could not be resolved during the first pass. Deploy the dashboard, MCP server, and documentation last. Do not start a run or request policy publication until the API readiness check passes and all Workflow bindings resolve.

The production Daytona account must support unrestricted public outbound network access with domainAllowList omitted. Bootstrap needs the official OpenTofu and OPA GitHub release endpoints, and customer plan/apply operations need representative registry and provider endpoints. Policy sandboxes use the same unrestricted network setting; fixed OPA capabilities, sanitized inputs, and the absence of customer cloud credentials are their execution boundaries. Public egress does not provide connectivity to customer private networks.

The current Daytona organization exposes the required container class only in us. Keep the production eu launch control disabled. Do not silently create an EU workspace in US or count a US canary as EU evidence. Enabling EU requires Daytona to expose the same class there, a successful daytona-medium bootstrap, and a dedicated EU production-dark canary before the audited region switch changes.

Migration 0019_thick_goliath is the immutable historical migration that introduced runner_snapshot_name; never edit it. New databases apply it in sequence before the regional migration. Existing installations retain its logical version-and-digest evidence as the input to the next forward migration.

Migration 0027_regional_runner_snapshot_names is a later forward-only contract change. Before applying it, disable new runs, recovery dispatch, applies, and policy publication; drain all nonterminal run and policy execution; and export a stable inventory including execution_region. Install the complete schema-v2 catalog and exact regional runner and policy allowlists, then compare the database inventory again. Apply 0027 only when the two inventories match. The migration appends each row’s persisted execution region and replaces the database constraint so version, binary digest, logical release digest, logical runner name, and region remain one tuple. The historical runner_image_digest and runner_snapshot_name column names do not make those compatibility values an OCI image or Daytona snapshot.

After 0027, rollback is roll-forward and schema-v2 only. Do not restore a schema-v1 catalog, remove regional suffixes, or revert persisted rows. Keep execution disabled while repairing the base-profile bootstrap, runner bundle, or checksum-verified toolchain installation, or while installing a prior known-good complete schema-v2 catalog and regional allowlists.

Migration 0028_policy_version_execution_identity is a writer-contract cutover and requires a policy-publication maintenance window. An old API version does not write policy_versions.workspace_settings_version_id; after 0028 makes that column NOT NULL, any policy version it creates fails. Do not use the ordinary migrate-before-API deployment order for this migration unless the publication endpoints are already fenced.

Use this order:

1. At API ingress, stop both policy-version creation routes: POST /v1/policy-sets/{policySetId}/versions and POST /v1/workspaces/{workspaceId}/policy-sets/{policySetId}/versions. Drain already accepted policy publications, including their durable dispatch records and completion callbacks, while keeping both routes fenced.

2. Run the following read-only preflight against the same database and search path that the migration will use. It must return no rows:

   SELECT
     policy_version.organization_id,
     policy_version.public_id,
     policy_version.source_artifact_id
   FROM policy_versions AS policy_version
   WHERE NOT EXISTS (
     SELECT 1
     FROM artifacts AS artifact
     INNER JOIN workspaces AS workspace
       ON workspace.organization_id = artifact.organization_id
      AND workspace.id = artifact.workspace_id
     INNER JOIN workspace_settings_versions AS settings
       ON settings.organization_id = workspace.organization_id
      AND settings.workspace_id = workspace.id
     WHERE artifact.organization_id = policy_version.organization_id
       AND artifact.public_id = policy_version.source_artifact_id
       AND settings.created_at <= policy_version.created_at
   )
   ORDER BY
     policy_version.organization_id,
     policy_version.public_id;

   If it returns any row, stop and repair the unresolved historical identity. The migration fails closed with POLICY_VERSION_EXECUTION_IDENTITY_BACKFILL_FAILED and rolls back instead of guessing.

3. Apply 0028 while the old writer remains fenced. The backfill chooses the latest tenant- and workspace-matching settings row at or before policy creation, breaking timestamp ties by settings version and then UUID.

4. Deploy the API build that always writes the pinned settings identity, pass readiness checks, and only then remove the ingress fence.

After 0028, do not roll the API back to a writer from before this contract. Database rollback is roll-forward: keep publication fenced and deploy a compatible writer. Reads and unrelated execution can remain available during the cutover.

Migration 0043_opentofu_open_versions opens the OpenTofu version set. It creates the opentofu_toolchains pin table seeded with the three launch releases, replaces the closed IN-list workspace-settings constraint with a format constraint, and removes the hard-coded (version, sha256) pair table from runs_digests_valid while keeping the snapshot-name derivation identity. No rows change, and the previous three-version code still satisfies the new constraints, so the ordinary migrate-before-API order applies. Deploy the API, orchestrator, and web builds together with the schema-v3 RUNNER_SNAPSHOT_CATALOG secret rotation: schema-v2 documents are rejected by the new code and schema-v3 documents by the old code. Rollback before the catalog rotation is a normal redeploy; after it, restore the schema-v2 secret together with the previous build. To admit newly pinned versions without per-version allowlist entries, add the full sha256: release digest to the launch-control runner snapshot allowlist; retained exact regional names keep working.

Transactional email

In Compute → Email Service → Email Sending, onboard stackship.run and publish every Cloudflare-provided bounce MX, SPF, DKIM, and DMARC record. Wait until the sending domain and all authentication records show as verified before exercising an auth flow.

The API Worker exposes only the restricted EMAIL binding and may send only from no-reply@stackship.run. Do not create an SMTP credential or a long-lived Email Service API token for application code. Remove development recipient restrictions before production canaries so account mail can reach arbitrary users.

Disable Email preview in the sending-domain settings before sending any verification or reset canary. New Cloudflare sending domains may enable preview by default; when enabled, rendered bodies and bearer links are retained in the Email Service activity log for approximately seven days.

Deploy

pnpm install --frozen-lockfile
pnpm verify:bindings
pnpm verify:github-workflows
pnpm verify:workers
pnpm check
pnpm test
pnpm build

Those commands build and verify the candidate without changing production. Production deploys use the protected workflow, which serializes the credential broker, database migration, API, Workflow owner, API binding refresh, MCP, OpenNext dashboard, and Nimbus documentation. Do not use the root pnpm deploy command for production: Turbo may schedule independent Worker owners concurrently.

The dashboard keeps OpenNext’s asset-first routing so Cloudflare Static Assets serves static requests before the Worker route. Its checked-in public/_headers file must be present in .open-next/assets/_headers after the OpenNext build. Validate the security-header baseline on both a prerendered page and an immutable /_next/static/* asset; the static manifest must not contain a document Content Security Policy.

The documentation Worker is an assets-only Nimbus deployment. Its checked-in apps/docs/public/_headers file must be present as apps/docs/dist/_headers after the Astro build. Validate the security-header baseline on both a rendered documentation page and a static asset; the manifest intentionally uses Referrer-Policy: no-referrer and must not contain a document Content Security Policy.

The API and orchestrator are one provenance-coupled execution release. Their supported repository deployment entry point is:

STACKSHIP_BUILD_SHA="$(git rev-parse HEAD)" pnpm deploy:execution-workers

The wrapper requires a lowercase 40-character Git commit SHA, requires it to equal the checked-out HEAD, and rejects staged, unstaged, or untracked worktree changes. It deploys API, orchestrator, then API again with one identical APPLICATION_BUILD_SHA, --strict, and --keep-vars. The final API pass refreshes Workflow bindings after the owner is installed. The protected workflow supplies ${{ github.sha }} as STACKSHIP_BUILD_SHA and calls this same wrapper.

pnpm deploy:api and pnpm deploy:orchestrator use the same validation for a bounded recovery operation, but they do not constitute a complete execution release. Do not bypass the wrapper with pnpm --filter ... deploy or bare Wrangler: a caller that omits the candidate SHA can preserve an old value or erase the non-configured production variable, desynchronizing durable Workflow provenance from API dispatch evidence. Cloudflare and database credentials remain environment-only and are never command arguments or wrapper output.

The first basic deployment can ship public health surfaces before execution is enabled. In that state, auth or readiness failures return 503, MCP returns 401 without a valid audience-bound token, and managed runs fail closed.

Quiesced execution-contract cutover

An execution release that changes the runner manifest, result, session, or state-publication contract without supporting both deployed contract shapes must use a quiesced cutover. Before the first Worker deployment, replace the production launch control through its audited compare-and-swap interface with both newRunsEnabled and applyEnabled set to false, while preserving the complete prior control value for restoration.

Reject or otherwise terminalize every Workflow created by the previous execution release, then verify that no old run Workflow is waiting or running and that no runner attempt remains active. Only then run deploy:execution-workers, verify that the API and orchestrator expose the same expected APPLICATION_BUILD_SHA, and restore the exact prior launch control through another audited compare-and-swap mutation. If draining, deployment, provenance verification, or restoration fails, keep execution disabled and follow the recovery runbook. Never rely on an API-first mixed version interval to make an incompatible execution contract safe.

When the same release adds a field to a strict dashboard response schema, make the new reader accept the field as optional and retain a fail-closed fallback for an old API. Deploy and verify that dashboard reader first, then perform the quiesced API and orchestrator cutover that starts returning the field. Do not deploy a strict API writer before every public reader accepts its response shape.

Release delivery gate

Pull requests and main run the pinned GitHub Actions CI workflow with a frozen pnpm install, generated API and deployment-boundary checks, full typecheck/test/build, real PostgreSQL transaction tests, focused GitHub/policy/speculative/fault/security/canary suites, pinned browser and OpenTofu gates, accessibility tests, the OpenTofu matrix, Nimbus lint, and patch-hygiene validation. These checks validate the external-canary harness but do not claim that external services passed.

Production deployment is manual and uses GitHub’s protected production environment. Every dispatch explicitly checks out github.sha and runs the complete release gate again in the protected deployment job. It does not trust a prior CI result, reuse a different revision’s build, or delegate validation to another job. Before the first production mutation, that same job performs:

1. OpenAPI, binding, email-domain, GitHub-workflow, and Worker build-boundary verification.
2. Full typecheck, test, and build.
3. The critical transaction suite against its own healthy PostgreSQL 17 service.
4. The focused GitHub, policy, speculative, fault, security, and canary suites.
5. Checksum-verified OpenTofu 1.12.5 and OPA 1.18.2 installation, the real rooted-bundle compatibility gate, browser and accessibility gates, the native OpenTofu matrix, Nimbus lint, and git diff --check.

The OPA compatibility test is optional for ordinary local unit-test runs. Release workflows set STACKSHIP_OPA_REQUIRED=1, install the exact pinned Linux binary after verifying its SHA-256 digest, and export its absolute path as STACKSHIP_OPA_BINARY. A missing binary, checksum mismatch, invalid rooted bundle, unredacted policy input, or unexpected decision fails the release before deployment.

The workflow has a 60-minute timeout; a failed or timed-out validation step prevents every delivery command from starting.

Store the Cloudflare account ID, least-privilege Cloudflare API token, and direct TLS-enforced PlanetScale DATABASE_URL only as protected environment secrets. The workflow passes the Cloudflare credentials only to Worker delivery steps and DATABASE_URL only to the migration step; validation and certification steps receive none of them. A dispatch with certify=false orders the credential broker, reviewed Drizzle migrations, the optional fresh-account Workflow bootstrap, API, full Workflow owner, API binding refresh, then public read surfaces.

The API and full Workflow-owner deployments both receive the exact APPLICATION_BUILD_SHA. The API persists that value as expected pre-dispatch evidence. At its first durable step, each source-ingestion and run Workflow reports the build SHA from its own Worker environment through the authenticated internal service binding. The database stores that observed execution SHA separately and rejects a mismatch before source fetch or sandbox execution.

When a run reports a mismatched build, the API also finalizes that unstarted run as a terminal plan failure before it returns the conflict. The Workflow still fails closed, while OpenTofu receives a terminal Cloud run instead of polling an orphaned plan_queued run indefinitely.

The checked-in bootstrap executor first lists deployments for the exact stackship-orchestrator Worker. A successful JSON array, empty or non-empty, means no bootstrap. Only process exit 1 with Cloudflare code 10007, the exact orchestrator deployments endpoint, and Cloudflare’s missing-Worker message permits one restricted bootstrap deployment. Authentication, permission, network, rate-limit, malformed-output, unknown-code, and bootstrap deployment failures stop the release without a fallback or retry.

A dispatch with certify=true runs the same complete, exact-SHA release gate and then only exact-SHA canary and readiness validation; it never deploys a Worker, applies a database or Durable Object migration, or creates a Worker version ID. Neither path changes the server-side public launch control.

After the dark deployment, an operator adds only dedicated canary entries to darkCanaryWorkspaceAllowlist through the audited, compare-and-swap launch-control API. Each entry is the exact object { "organizationId": "org_<32 lowercase hex>", "workspaceId": "ws_<32 lowercase hex>" }; workspace public IDs are organization-local, so a ws_ value alone is never an authorization boundary. publicLaunchEnabled remains false. Only those exact organization-and-workspace pairs may cross the public gate; new-run, apply, federation-provider, Daytona-region, suspension, capacity, and logical runner release checks still apply at every phase. The list accepts at most 16 unique pairs and rejects malformed entries, duplicate pairs, and unknown entry fields.

The operator then dispatches the protected, manual production-dark-canary.yml workflow for the exact deployed SHA. It is serialized, has a 60-minute timeout, checks out exactly github.sha, and has only repository contents: read permission. Protected environment configuration currently identifies one dedicated branch, pre-existing nonce file, and workspace in Daytona us. EU canary configuration remains absent while that region is launch-disabled. Step-scoped secrets provide a canary-tenant StackShip token with only workspace/state read and run read/approve, a disposable-repository GitHub token with contents-update and Checks-read authority, a least-privilege Daytona list credential, and the existing federation/email operator attestations.

Updating each pre-existing nonce file is the only canary mutation outside StackShip’s normal run lifecycle. The GitHub App webhook is the only run trigger. The harness does not create runs directly and does not call internal Workflow, internal service, deployment, migration, or launch-control endpoints. Through bounded public responses it reads the starting state serial, discovers the exact run ID only from the new commit’s public GitHub Check external_id, then polls only /v1/runs/{runId}/review. It observes the exact ready source ingestion and separately attested source/run Workflow execution SHAs. Before approval and again at terminal observation, the review must report the configured workspace ID and execution region. The harness approves the exact binding only after the first identity check and waits for the real apply. It never requests logs, state bodies, artifacts, or source archives.

Version 2 evidence contains one cross-linked execution record for every enabled Daytona region. Each record requires the exact source commit, both observed Workflow execution SHAs equal to the candidate, exactly one apply process start derived from persisted runner start-grant evidence, state serial exactly +1, and one successful StackShip GitHub Check whose external_id is the run ID. The harness polls exact run-labelled Daytona inventory until it is empty or the shared deadline expires, using the execution region verified from the terminal run review. Evidence records that verified workspace and region; it does not derive them from display labels. The workflow validates that exact-build document before artifact upload. AWS/GCP/Azure federation and Cloudflare Email fields retain their existing operator-attested semantics until dedicated live harnesses exist: short-lived credentials, deleted disposable resources, two email deliveries, disabled Email preview, and absence of bearer links from prohibited stores remain required. Missing or stale evidence fails closed.

After collecting evidence, remove every canary organization-and-workspace pair from the allowlist with a second audited mutation and confirm publicLaunchEnabled is still false before signing readiness. The canary Action never performs that control mutation.

See the repository deployment and transactional-email runbooks for rollback, evidence fields, and the separately audited first-release activation procedure.

The repository incident runbooks also cover state and fence recovery, uncertain apply containment, Daytona regional outages and sandbox leaks, signing/HMAC and wrapping-key rotation limits, GitHub App compromise, and policy evaluator incidents. These procedures preserve reads and recovery, never retry apply, never fail over a pinned Daytona region, and never weaken policy or identity checks.

Production checks

• GET https://api.stackship.run/health checks Worker liveness and configured bindings.
• GET https://api.stackship.run/ready first requires the complete identity configuration as one coarse, fail-closed gate: Better Auth, the Google client pair, Cloudflare Email, and an independent cloud-token HMAC key of at least 32 UTF-8 bytes. Managed-run authentication additionally requires a distinct RUNNER_RUN_TOKEN_HMAC_KEY of at least 32 UTF-8 bytes. It then connects through Hyperdrive and checks the core Better Auth tables plus the runner_sessions.run_token_digest migration. It returns 503 with identity: configuration_required for an incomplete identity boundary, or schema: migration_required until the generated migration is applied. It never names a missing secret or returns configuration values.
• GET https://stackship.run/.well-known/terraform.json and GET https://api.stackship.run/.well-known/terraform.json return the OpenTofu service-discovery document. Both must advertise modules.v1 at https://api.stackship.run/registry/modules/v1/ for the private module registry, alongside the existing login and /api/v2 keys.
• GET https://mcp.stackship.run/health checks MCP Worker liveness.
• GET https://mcp.stackship.run/.well-known/oauth-protected-resource returns the OAuth protected-resource metadata.

Before any public activation, confirm the production launch_controls row exists, publicLaunchEnabled remains false, and darkCanaryWorkspaceAllowlist is empty. Confirm the us Daytona region is enabled, eu is disabled, and a private daytona-medium sandbox in US can upload the reviewed runner bundles, install the checksum-verified toolchain, execute the canary, and clean up. Exercise the new-run disable, apply disable, regional disable, and sandbox leak runbooks in staging. Production activation is a later separately audited mutation after the readiness record is signed.