Code Apps operator runbook

Operate Cuitty Code Apps, the App Market, SpiceDB, and Airflow in self-hosted deployments.

This runbook covers local and self-hosted operation for Code Apps, the App Market, SpiceDB authorization, and Airflow execution. Cuitty Registry now has a separate Registry operator runbook for package protocols, indexes, upload sessions, and artifact storage.

Required services

Service	Purpose	Local default
Cuitty Code API	REST API, Git HTTP, app, and callback endpoints	`http://localhost:4351`
Cuitty Code frontend	Browser UI	`http://localhost:4350`
Cuitty Auth	OAuth/OIDC sign-in	`http://localhost:7705`
Database	Persistent metadata and run state	Deployment-specific
SpiceDB	Relationship authorization	`http://localhost:50051`
Airflow	App run orchestration	`http://localhost:8080`
Cuitty Registry	Optional standalone package registry for app artifact workflows	Deployment-specific
Object storage	Optional backing storage for large artifacts and logs	Deployment-specific

Startup order

Start the database.
Start SpiceDB.
Load the SpiceDB schema.
Start Cuitty Auth.
Start Airflow.
Run Cuitty Code API migrations, then start the API.
Start the Cuitty Code frontend.
Start Cuitty Registry separately if Code App workflows need package publish or install targets.

Environment variables

For self-hosted production, commit Safe references instead of raw secret values. The reference syntax is cuitty-safe:account/safe/secret; only the resolver sees the decrypted value.

PUBLIC_API_URL=https://code.example.com
PUBLIC_CUITTY_AUTH_URL=https://auth.example.com
AUTH_ISSUER=https://auth.example.com
AUTH_CLIENT_ID=cuitty-code
CUITTY_PUBLIC_URL=https://code.example.com
CUITTY_CODE_SECRET_KEY=cuitty-safe:acme/prod/cuitty-code-secret-key
SPICEDB_ENDPOINT=https://spicedb.example.com:50051
SPICEDB_PRESHARED_KEY=cuitty-safe:acme/prod/spicedb-preshared-key
AIRFLOW_URL=https://airflow.example.com
AIRFLOW_USERNAME=airflow-service
AIRFLOW_PASSWORD=cuitty-safe:acme/prod/airflow-password
CUITTY_APP_EXECUTION_MODE=airflow
CUITTY_APP_RUN_CALLBACK_SECRET=cuitty-safe:acme/prod/app-run-callback-secret
REGISTRY_STORAGE_URL=s3://cuitty-registry/prod

For an isolated local development stack only, use disposable placeholder values and local endpoints:

PUBLIC_API_URL=http://localhost:4351
PUBLIC_CUITTY_AUTH_URL=http://localhost:7705
AUTH_ISSUER=http://localhost:7705
AUTH_CLIENT_ID=cuitty-code
CUITTY_PUBLIC_URL=http://localhost:4350
CUITTY_CODE_SECRET_KEY=<local-dev-only-code-secret-key>
SPICEDB_ENDPOINT=http://localhost:50051
SPICEDB_PRESHARED_KEY=<local-dev-only-spicedb-key>
AIRFLOW_URL=http://localhost:8080
AIRFLOW_USERNAME=airflow
AIRFLOW_PASSWORD=<local-dev-only-airflow-password>
CUITTY_APP_EXECUTION_MODE=airflow
CUITTY_APP_RUN_CALLBACK_SECRET=<local-dev-only-callback-secret>
PUBLIC_REGISTRY_URL=http://localhost:4371
REGISTRY_STORAGE_URL=file:///var/lib/cuitty/registry

Migrations

Run migrations before traffic reaches a new API build:

cargo run -p code-server -- migrate

Verify the API starts with the expected database URL and no pending migration errors.

SpiceDB checks

zed schema write spicedb/schema.zed
zed permission check user:alice read repository:acme/demo
zed permission check user:alice install app_listing:acme/review-bot

Inspect failed authorization relationship writes:

SELECT id, operation_kind, resource_kind, resource_id, status, attempts, last_error
FROM authz_outbox
WHERE status IN ('pending', 'processing', 'failed_retryable', 'failed_terminal')
ORDER BY created_at ASC
LIMIT 50;

Registry integration checks

If Code Apps publish or install packages, run Cuitty Registry separately and verify Code discovers it through PUBLIC_REGISTRY_URL or product endpoint discovery. Use the Registry operator runbook for npm, Cargo, PyPI, and OCI smoke checks.

Airflow checks

Submit or trigger an app run, then confirm the run has an Airflow DAG ID and DAG run ID. Reconcile submitted runs when callbacks are delayed:

curl -X POST http://localhost:4351/api/v1/apps/runs/reconcile   -H "Authorization: Bearer $CUITTY_TOKEN"

Callback failures usually mean the timestamp is stale, the x-cuitty-signature HMAC is wrong, or Airflow is signing with a value that does not match the resolved CUITTY_APP_RUN_CALLBACK_SECRET reference.

Playwright E2E

From the Cuitty Code tests directory, run the mocked app-market and integration specs:

bun run test:e2e:apps-registry -- --project=chromium
bun run test:e2e:apps-registry:headed

The specs set browser auth state and mock API responses by path, so they cover both direct API calls on http://localhost:4351/api/v1/... and frontend-proxied calls on http://localhost:4350/api/v1/....

Failure modes and recovery

SpiceDB unavailable: pause security-expanding app, package-target, and transfer writes; restore SpiceDB; drain the authz outbox.
Authz outbox backlog: inspect last_error, reset stale processing rows to pending only after confirming no dispatcher is active, and restart the dispatcher.
Airflow DAG submission failure: check AIRFLOW_URL, credentials, DAG ID derivation, API logs, and Airflow scheduler/webserver logs.
App run stuck in submitted or running: poll Airflow for the DAG run ID, check task logs, verify callback delivery, and run reconciliation.
Registry integration unavailable: keep Code App package workflows queued or failed retryable, verify PUBLIC_REGISTRY_URL, then run Registry smoke checks before replaying work.