Skip to content

Paperclip Control Plane Audit

Date: 2026-07-02 (release/build facts refreshed 2026-07-10)

Sources Checked

  • Paperclip repo: https://github.com/paperclipai/paperclip
  • Latest stable release checked live: v2026.707.0, published 2026-07-07.
  • npm package checked live: [email protected], latest=2026.707.0, canary=2026.710.0-canary.6.
  • Docs repo checked live: https://github.com/paperclipai/paperclip-docs
  • Local source clones used for this audit:
  • .workingdir2/upstream/paperclip

Full v2026.707.0 Alignment Recheck (2026-07-10)

This was a source-and-docs recheck, not an assumption from the old pilot. The stable release, published npm dist, current docs repository (through docs commit 773f72838a5205c62a5010876d2bb00e9289ebbe), OpenAPI validators, plugin SDK, first-party orchestration plugin example, and the actual heartbeat claim path were compared. The following supported surfaces are now reflected in the Northlight reconciliation:

  • managed instruction bundles and per-agent files;
  • company skills plus explicit agent skill assignment/sync verification;
  • projects, isolated workspaces, goals, routines, pipelines, review stages, secret definitions, and export/import boundaries;
  • OpenCode custom providers, global MCP configuration, and per-run runtime config generation;
  • plugin jobs, events, webhooks, tools, scoped state/database access, issue orchestration summaries, and plugin-requested wakeups.

The resource-dispatch gap is real in v2026.707.0:

  • Agent heartbeat policy has a per-agent maxConcurrentRuns; routines have coalesce/enqueue/skip policies. Neither models a global cap, resource pool, exclusive device/model lease, backend health, or cross-agent fairness.
  • Budgets can block invocation from current billed-cent policy state at company, project, or agent scope. They do not reserve projected cost or scarce hardware capacity atomically across concurrent claims.
  • The scheduler resumes queued runs and every assignment, comment, dependency, routine, pipeline, recovery, direct invoke, and plugin wake eventually uses the same internal queued-to-running claim. Native wakes therefore bypass an external dispatcher unless admission is enforced at that claim.
  • The plugin SDK can inspect orchestration/run summaries and request wakes, but v2026.707.0 exposes no before-claim veto/admission hook. A plugin alone cannot prevent a separate native wake from starting. The SDK docs also say local dynamic plugin installation is not yet a complete multi-instance artifact distribution solution.
  • The API docs explicitly leave deployment rate limiting to infrastructure.

Therefore the immediate Cauda fix is a fail-closed compiled-dist patch at the single queued-to-running transition: short-timeout reachability probes, PostgreSQL advisory-lock serialization, global/backend/pool/agent checks, immutable run admission metadata, model freezing, and a fair global queue drain when capacity returns. It is a safety bridge, not the desired public end state. Paperclip backlog CAU-8 tracks a general resource-aware admission plugin (hardware, provider concurrency, rate/token/cash limits, licensed seats, and other finite resources) plus the upstream hook/proposal required to make that plugin authoritative without a private dist patch. The inspected documentation snapshot remains in .workingdir2/upstream/paperclip-docs.

The upstream docs repo is licensed separately from the MIT app repo, so raw upstream docs stay in .workingdir2/ scratch. This file is the durable local decision record.

What Paperclip Is

Paperclip is an agent control plane: UI, API, issue/task model, agent org chart, heartbeats, budgets, approvals, routines, plugin system, and adapter runtime. It is useful for coordinating coding agents and custom agents across machines.

Paperclip is not a GPU inference scheduler and should not replace Kubernetes, KServe, OVMS, ComfyUI, Open WebUI, Ollama, Argo Workflows, or the device plugins. Those still own model serving, batch jobs, and GPU placement.

The useful split for this cluster is:

  • Paperclip owns agent work routing, governance, budgets, watchdogs, and audit.
  • Kubernetes owns node/GPU scheduling and workload isolation.
  • Existing AI services own inference: OVMS/KServe for OpenAI-compatible model endpoints, Open WebUI for chat/RAG, ComfyUI for image workflows, and app CronWorkflows for narrow automation.
  • Paperclip adapters invoke those services or run Codex/Claude/local agents on worker hosts.

Current Cluster Fit

Live node allocatable GPU resources on 2026-07-02:

  • k8s-worker-home-1: gpu.intel.com/i915=5
  • k8s-worker-home-2: gpu.intel.com/xe=5
  • No live node advertises nvidia.com/gpu.

That means the first integration should target Intel GPU and existing model services, not CUDA assumptions. Paperclip agents that need local GPUs should run through a Kubernetes sandbox/worker path pinned to the home GPU workers, while plain coding agents can run on non-GPU nodes or off-cluster machines.

Deployment Findings

  • Upstream docs describe local, VPS/Fly, Docker, external Postgres, local encrypted secrets, adapters, routines, and sandbox providers.
  • Upstream docs do not ship a Kubernetes Helm chart.
  • The Docker docs use a locally built paperclip-local image.
  • GitHub Packages did not expose a published container package via the GitHub packages API check.
  • Upstream's Dockerfile currently installs some runtime CLIs with @latest. That is incompatible with this repo's pinning discipline for production.
  • The stable deployable app package is currently npm [email protected].

Do not deploy upstream's image recipe unchanged. Build and publish an internal image first, pinned by version and digest.

Initial app shape:

  • Namespace/project: ai.
  • Hostname: paperclip.printing.cauda.dev or paperclip.ai.cauda.dev, routed through Cilium Gateway.
  • Auth mode: authenticated + private/public as required, protected by Authentik or Paperclip's own login until OIDC support is verified.
  • Database: CNPG pg-cluster with a dedicated paperclip role/database and a sealed DATABASE_URL.
  • Storage: persistent PVC for PAPERCLIP_HOME, likely longhorn-home for the first pilot because agent workers and model endpoints live on home GPU nodes.
  • Secrets: enable PAPERCLIP_SECRETS_STRICT_MODE=true; bind provider keys as Paperclip secret refs, not inline config.
  • Telemetry: set PAPERCLIP_TELEMETRY_DISABLED=1 unless explicitly enabled.
  • Routines: leave timer heartbeats off by default; use routines/webhooks for scheduled work to avoid noisy token burn.

Worker/adapter path:

  • Phase 1: deploy control plane only, no GPU request.
  • Phase 2: configure Codex/Claude/local agents on selected always-on worker machines, with per-agent homes and budgets.
  • Phase 3: add HTTP adapter workers for existing cluster automation and model endpoints.
  • Phase 4: test Paperclip's Kubernetes sandbox provider in job backend first. Treat sandbox-cr and kubernetes-sigs/agent-sandbox as alpha until proven stable here.
  • Phase 5: if a worker needs local Intel GPU access, run it as a dedicated Kubernetes workload with explicit gpu.intel.com/xe or gpu.intel.com/i915 requests, not as a generic Paperclip server pod.

Immediate Blockers Before Deploy

  1. Create a pinned internal image build.
  2. Base app: published compiled package [email protected]; the tracked image patcher deliberately avoids the source-build layout.
  3. Pin Codex/Claude/Gemini/OpenCode adapter CLIs instead of @latest.

  4. Publish to Harbor/GHCR and use digest-pinned image references.

  5. Add a first-class chart.

  6. Deployment, Service, PVC, NetworkPolicy/CiliumNetworkPolicy, resources, sealed secret references, and health probes.

  7. Disable automounting service account tokens unless the Kubernetes sandbox provider is enabled.

  8. Codify database ownership.

  9. Add CNPG managed role and Database CR for paperclip.

  10. Reflect or seal the app connection secret into ai.

  11. Choose the first worker lane.

  12. For coding agents: external/off-cluster workers are safer initially.
  13. For cluster agents: HTTP adapter workers can call existing in-cluster services with narrow service accounts.

  14. For GPU agents: use explicit Kubernetes workloads on Intel GPU nodes.

  15. Security review.

  16. Verify auth/OIDC behavior.
  17. Verify outbound egress policy for provider APIs and internal services.
  18. Verify how Paperclip stores secrets in shared/container deployments.
  19. Keep raw kubeconfig out of Paperclip unless using a restricted service account or in-cluster auth.

Decision

Use Paperclip as an agent control plane, but only after the pinned image and minimal chart are in place. It should sit above the current AI stack and GitOps automation, not replace them.

The first useful milestone is a private Paperclip pilot with one Codex agent, one cluster-readonly HTTP worker, and one routine that opens or updates GitHub issues from real cluster evidence. GPU workers come after that path is stable.

Pilot Phase 1 — DONE (2026-07-02)

Configured live via the API (issue #2103); everything readback-verified with GETs after each write.

  • Bootstrap: first user signed up via POST /api/auth/sign-up/email, claimed instance_admin via POST /api/bootstrap/claim (/api/health now reports bootstrapStatus: ready). Credentials sealed as paperclip-bootstrap-admin (ns ai). Sign-up disabled afterwards via PAPERCLIP_AUTH_DISABLE_SIGN_UP=true in the deployment.
  • Company: Cauda Cluster Ops, issue prefix CAU.
  • Cluster-readonly HTTP worker: agent prometheus-readonly-worker (built-in http adapter — API-only, hidden in the UI dropdown). POSTs the ready-node count query to the in-cluster Prometheus HTTP API. Adapter environment test passes from the server pod.
  • Routine: Weekly cluster heartbeat (Prometheus ready-node probe) assigned to the HTTP worker; weekly schedule trigger (0 6 * * 1 Europe/Berlin) created but left disabled. One manual run demonstrated the loop: run → execution issue CAU-1 → webhook delivered → source run succeeded. Finding: the http adapter is fire-and-forget — the remote endpoint must call back into Paperclip (PAPERCLIP_API_URL + agent API key) to record an issue disposition; without that, the missing-disposition watchdog blocks each execution issue. The GitHub-issue routine from the milestone therefore needs a phase-3 callback-capable worker service, not a bare Prometheus URL.
  • LLM provider verdict: Paperclip has no instance-level LLM provider setting (the llm config block is only a CLI doctor check for claude/openai keys). LLMs are wired per agent through adapters, and custom OpenAI-compatible endpoints ARE supported: opencode_local via PAPERCLIP_OPENCODE_PROVIDERS (+ OPENCODE_ALLOW_ALL_MODELS, PAPERCLIP_OPENCODE_SMALL_MODEL) and codex_local via PAPERCLIP_CODEX_PROVIDERS. The pilot agent qwen3-b60-worker (opencode_local, model b60/qwen3-8b) targets the home B60 through http://qwen3-8b-router.ai.svc.cluster.local/v3; the adapter environment test passes end-to-end (hello probe answered by qwen3-8b on the B60, 6 models discovered). Pod-specific fix: the agent env pins HOME=/paperclip + XDG_{DATA,CACHE,STATE}_HOME because the pod root FS is read-only and OpenCode defaults to /home/node.
  • Not started (later phases): Codex/off-cluster coding agents, callback-capable HTTP worker service, GitHub issue automation, Kubernetes sandbox provider, GPU workers.