{
  "slug": "code/operator-runbook",
  "title": "Code Apps operator runbook",
  "description": "Operate Cuitty Code Apps, the App Market, SpiceDB, and Airflow in self-hosted deployments.",
  "url": "https://cuitty.com/docs/code/operator-runbook",
  "markdown_url": "https://cuitty.com/docs/code/operator-runbook.md",
  "json_url": "https://cuitty.com/docs/code/operator-runbook.json",
  "frontmatter": {
    "title": "Code Apps operator runbook",
    "description": "Operate Cuitty Code Apps, the App Market, SpiceDB, and Airflow in self-hosted deployments.",
    "order": 41,
    "section": "Code",
    "updatedAt": "2026-06-09"
  },
  "headings": [
    {
      "depth": 2,
      "slug": "required-services",
      "text": "Required services"
    },
    {
      "depth": 2,
      "slug": "startup-order",
      "text": "Startup order"
    },
    {
      "depth": 2,
      "slug": "environment-variables",
      "text": "Environment variables"
    },
    {
      "depth": 2,
      "slug": "migrations",
      "text": "Migrations"
    },
    {
      "depth": 2,
      "slug": "spicedb-checks",
      "text": "SpiceDB checks"
    },
    {
      "depth": 2,
      "slug": "registry-integration-checks",
      "text": "Registry integration checks"
    },
    {
      "depth": 2,
      "slug": "airflow-checks",
      "text": "Airflow checks"
    },
    {
      "depth": 2,
      "slug": "playwright-e2e",
      "text": "Playwright E2E"
    },
    {
      "depth": 2,
      "slug": "failure-modes-and-recovery",
      "text": "Failure modes and recovery"
    },
    {
      "depth": 2,
      "slug": "related-pages",
      "text": "Related pages"
    }
  ],
  "body_markdown": "This runbook covers local and self-hosted operation for Code Apps, the App Market, SpiceDB authorization, and Airflow execution. Cuitty Registry now has a separate [Registry operator runbook](/docs/registry/operator-runbook) for package protocols, indexes, upload sessions, and artifact storage.\n\n## Required services\n\n| Service | Purpose | Local default |\n| --- | --- | --- |\n| Cuitty Code API | REST API, Git HTTP, app, and callback endpoints | `http://localhost:4351` |\n| Cuitty Code frontend | Browser UI | `http://localhost:4350` |\n| Cuitty Auth | OAuth/OIDC sign-in | `http://localhost:7705` |\n| Database | Persistent metadata and run state | Deployment-specific |\n| SpiceDB | Relationship authorization | `http://localhost:50051` |\n| Airflow | App run orchestration | `http://localhost:8080` |\n| Cuitty Registry | Optional standalone package registry for app artifact workflows | Deployment-specific |\n| Object storage | Optional backing storage for large artifacts and logs | Deployment-specific |\n\n## Startup order\n\n1. Start the database.\n2. Start SpiceDB.\n3. Load the SpiceDB schema.\n4. Start Cuitty Auth.\n5. Start Airflow.\n6. Run Cuitty Code API migrations, then start the API.\n7. Start the Cuitty Code frontend.\n8. Start Cuitty Registry separately if Code App workflows need package publish or install targets.\n\n## Environment variables\n\nFor self-hosted production, commit Safe references instead of raw secret values. The reference syntax is `cuitty-safe:account/safe/secret`; only the resolver sees the decrypted value.\n\n```dotenv\nPUBLIC_API_URL=https://code.example.com\nPUBLIC_CUITTY_AUTH_URL=https://auth.example.com\nAUTH_ISSUER=https://auth.example.com\nAUTH_CLIENT_ID=cuitty-code\nCUITTY_PUBLIC_URL=https://code.example.com\nCUITTY_CODE_SECRET_KEY=cuitty-safe:acme/prod/cuitty-code-secret-key\nSPICEDB_ENDPOINT=https://spicedb.example.com:50051\nSPICEDB_PRESHARED_KEY=cuitty-safe:acme/prod/spicedb-preshared-key\nAIRFLOW_URL=https://airflow.example.com\nAIRFLOW_USERNAME=airflow-service\nAIRFLOW_PASSWORD=cuitty-safe:acme/prod/airflow-password\nCUITTY_APP_EXECUTION_MODE=airflow\nCUITTY_APP_RUN_CALLBACK_SECRET=cuitty-safe:acme/prod/app-run-callback-secret\nREGISTRY_STORAGE_URL=s3://cuitty-registry/prod\n```\n\nFor an isolated local development stack only, use disposable placeholder values and local endpoints:\n\n```dotenv\nPUBLIC_API_URL=http://localhost:4351\nPUBLIC_CUITTY_AUTH_URL=http://localhost:7705\nAUTH_ISSUER=http://localhost:7705\nAUTH_CLIENT_ID=cuitty-code\nCUITTY_PUBLIC_URL=http://localhost:4350\nCUITTY_CODE_SECRET_KEY=<local-dev-only-code-secret-key>\nSPICEDB_ENDPOINT=http://localhost:50051\nSPICEDB_PRESHARED_KEY=<local-dev-only-spicedb-key>\nAIRFLOW_URL=http://localhost:8080\nAIRFLOW_USERNAME=airflow\nAIRFLOW_PASSWORD=<local-dev-only-airflow-password>\nCUITTY_APP_EXECUTION_MODE=airflow\nCUITTY_APP_RUN_CALLBACK_SECRET=<local-dev-only-callback-secret>\nPUBLIC_REGISTRY_URL=http://localhost:4371\nREGISTRY_STORAGE_URL=file:///var/lib/cuitty/registry\n```\n\n## Migrations\n\nRun migrations before traffic reaches a new API build:\n\n```bash\ncargo run -p code-server -- migrate\n```\n\nVerify the API starts with the expected database URL and no pending migration errors.\n\n## SpiceDB checks\n\n```bash\nzed schema write spicedb/schema.zed\nzed permission check user:alice read repository:acme/demo\nzed permission check user:alice install app_listing:acme/review-bot\n```\n\nInspect failed authorization relationship writes:\n\n```sql\nSELECT id, operation_kind, resource_kind, resource_id, status, attempts, last_error\nFROM authz_outbox\nWHERE status IN ('pending', 'processing', 'failed_retryable', 'failed_terminal')\nORDER BY created_at ASC\nLIMIT 50;\n```\n\n## Registry integration checks\n\nIf Code Apps publish or install packages, run Cuitty Registry separately and verify Code discovers it through `PUBLIC_REGISTRY_URL` or product endpoint discovery. Use the [Registry operator runbook](/docs/registry/operator-runbook) for npm, Cargo, PyPI, and OCI smoke checks.\n\n## Airflow checks\n\nSubmit or trigger an app run, then confirm the run has an Airflow DAG ID and DAG run ID. Reconcile submitted runs when callbacks are delayed:\n\n```bash\ncurl -X POST http://localhost:4351/api/v1/apps/runs/reconcile   -H \"Authorization: Bearer $CUITTY_TOKEN\"\n```\n\nCallback failures usually mean the timestamp is stale, the `x-cuitty-signature` HMAC is wrong, or Airflow is signing with a value that does not match the resolved `CUITTY_APP_RUN_CALLBACK_SECRET` reference.\n\n## Playwright E2E\n\nFrom the Cuitty Code `tests` directory, run the mocked app-market and integration specs:\n\n```bash\nbun run test:e2e:apps-registry -- --project=chromium\nbun run test:e2e:apps-registry:headed\n```\n\nThe specs set browser auth state and mock API responses by path, so they cover both direct API calls on `http://localhost:4351/api/v1/...` and frontend-proxied calls on `http://localhost:4350/api/v1/...`.\n\n## Failure modes and recovery\n\n- SpiceDB unavailable: pause security-expanding app, package-target, and transfer writes; restore SpiceDB; drain the authz outbox.\n- Authz outbox backlog: inspect `last_error`, reset stale `processing` rows to `pending` only after confirming no dispatcher is active, and restart the dispatcher.\n- Airflow DAG submission failure: check `AIRFLOW_URL`, credentials, DAG ID derivation, API logs, and Airflow scheduler/webserver logs.\n- App run stuck in `submitted` or `running`: poll Airflow for the DAG run ID, check task logs, verify callback delivery, and run reconciliation.\n- Registry integration unavailable: keep Code App package workflows queued or failed retryable, verify `PUBLIC_REGISTRY_URL`, then run Registry smoke checks before replaying work.\n\n## Related pages\n\n- [Registry operator runbook](/docs/registry/operator-runbook)\n- [Cuitty Code Apps](/docs/code/apps)\n- [Code App Market](/docs/code/app-market)\n- [Code integration](/docs/registry/code-integration)\n- [Authorization with SpiceDB](/docs/code/authz-spicedb)\n- [App execution with Airflow](/docs/code/app-execution-airflow)",
  "links_out": [
    "/docs/registry/operator-runbook",
    "/docs/code/apps",
    "/docs/code/app-market",
    "/docs/registry/code-integration",
    "/docs/code/authz-spicedb",
    "/docs/code/app-execution-airflow"
  ]
}