Agent channel configuration

dark-factory serve starts the agent port and the run scheduler. Settings come from the instance configuration file (--config, default /etc/dark-factory/config.yaml).

Configuration

The Helm chart sets agent.tls to the cert-manager secret mounted at /etc/dark-factory/tls.

The three files are checked for changes by the first connection that arrives at least 10 s after the last check; changed material applies to new connections without a restart, and material that does not parse is ignored (logged) until fixed. At most 256 TLS handshakes run at the same time.

Run lifecycle

    stateDiagram-v2
    [*] --> pending: task queued, no active run
    pending --> provisioning: volume ensured
    pending --> failed: provisioning_failed
    provisioning --> connecting: provider Started (pod IP)
    provisioning --> failed: provisioning_failed
    connecting --> preparing: Hello
    connecting --> failed: connect_timeout
    preparing --> running: PhaseChanged RUNNING
    running --> awaiting_input: PhaseChanged AWAITING_INPUT (turn ended)
    awaiting_input --> running: PhaseChanged RUNNING (follow-up prompt)
    awaiting_input --> finalizing: idle deadline (Release) or mark done (Cancel)
    running --> finalizing: PhaseChanged FINALIZING
    finalizing --> succeeded: RunReport completed / end_turn / limit_reached / rate_limited
    finalizing --> cancelled: RunReport cancelled
    finalizing --> failed: RunReport failed
    preparing --> failed: sandbox_lost
    running --> failed: sandbox_lost, hard stop
    preparing --> finalizing: human stop (Cancel)
    running --> finalizing: human stop (Cancel)
    awaiting_input --> finalizing: human stop (Cancel)
    pending --> cancelled: human stop
    provisioning --> cancelled: human stop
    connecting --> cancelled: human stop
    finalizing --> cancelled: stop timeout (120 s)
  

A RunReport is accepted in any non-terminal state. Cancel is sent by the replica holding the stream at connect + maxWallClock + cancelGrace (and at the handshake if that moment has passed). The scheduler tears the sandbox down at connect + maxWallClock + 2 x cancelGrace. Prompts and Cancel are not sent once the run has ended; the prompt stays queued for the next run.

If prompts are still undelivered when a run ends with an outcome other than failed-class or rate_limited, the task becomes queued and the next run starts with them. A task a board member marked done (completed) keeps its status when its run ends, unless prompts are undelivered.

Human stop and cancel

A board member stops the active run (POST /tasks/{key}/stop) or cancels the task (POST /tasks/{key}/cancel, which stops the run too). Both set runs.stop_requested_at.

• A connected run in preparing, running or awaiting_input moves to finalizing; the session holding the stream sends Cancel, the runtime reports cancelled and the task becomes waiting. A run that is already finalizing is not marked by a stop (cancelling the task still marks it).
• A run without a connection (pending, provisioning, connecting) is ended cancelled by the scheduler on the next notification or tick and its sandbox is torn down. The task volume is kept.
• A stopped run that has not ended 120 s after the stop is torn down and ends cancelled with detail stop-timeout (volume kept).
• Task status cancelled is kept when the run ends. A message reopens the task (queued); the next run starts after the stopped run ended.
• Only messages queued after the stop requeue the task at run end; older undelivered messages leave it waiting. Cancelling discards the undelivered messages and closes the pending requests (task-cancelled).
• No pull request is opened or marked ready for a cancelled task; an existing one is left as it is.

Idle run (keep-alive)

After a normal turn end the runtime reports AWAITING_INPUT: the run becomes awaiting_input and the task waiting (it stays running while a message is undelivered). The replica holding the stream sends Release at the last persisted TurnEnded + runs.keepAliveSeconds; a replica that takes over, or a run that reconnects after the deadline, computes the same deadline. Before it sends Release (or Cancel for a task marked done) the control plane moves the run to finalizing; no prompt is sent to a finalizing run, a message queued meanwhile makes the task queued and the run end starts the next run. A message for an awaiting_input run sets the task running, starts no run and is forwarded at once.

Protocol behavior

• Identity: the client certificate must carry the URI SAN spiffe://dark-factory/ns/<ns>/pod/<name>/<uid>; SandboxProvider::resolve_identity maps SAN and peer IP to a run. Terminal runs are rejected with UNAUTHENTICATED.
• Handshake: Hello first (else FAILED_PRECONDITION), then HelloAck, HarnessCredentials, Assign, then the prompts: undelivered prompts that are already forwarded (task_messages.forwarded_at set) are sent again, held-back ones are claimed (marked forwarded) and sent only while no turn is open (no persisted TurnStarted without a later TurnEnded). After a persisted TurnEnded the held-back prompts are sent. See “Queued messages reach the runtime between turns” in contract/README.md. HarnessCredentials precede Assign because the runtime ignores credentials that arrive later.
• Timeouts: enforce_timeouts ends every overdue run it can; a run that fails to end (or whose row cannot be read) is logged and does not stop the others.
• Ordering and delivery: messages are persisted in batches of up to 50 per transaction, duplicates (seq <= runs.last_persisted_seq, blobs included) are dropped, a gap closes the stream with DATA_LOSS, and the cumulative ack is sent after commit. Effects of the messages before a gap (teardown, pull request) still run. A message with an unknown body keeps its sequence number and is ignored.
• Tool calls and credential requests that arrive after the run ended (for example after a hard stop) are refused: an error result and DeniedReason, nothing is executed or granted.
• Pull request work after a PushReport or RunReport runs beside the stream (60 s limit each), so a slow forge delays neither acks, Cancel nor prompts. If marking the pull request ready fails, the completion summary is still posted.
• Shutdown (SIGTERM): the scheduler lets provisioning in flight finish (10 s); sessions stop reading, finish their in-flight tool calls and credential requests (up to 20 s, then they are aborted) and end their streams; the listeners get 30 s before they are aborted.
• A runtime failing before Hello is logged and answered with its gRPC status.
• Limits: 16 MiB per message (RESOURCE_EXHAUSTED above), HTTP/2 keepalive 30 s / 20 s, 10 s for the TLS handshake.
• Tool calls and credential requests run off the stream loop and are awaited (up to 60 s) when the stream ends. A tool call is claimed by call_id (tool_calls row) before it runs and executed at most once: a repeat gets the stored result, a call still running gets an error “in progress”, and a claim without a result after 5 minutes is closed with an error and never executed. The forge is called without an open database transaction. Executed calls are audited as tool_call.execute.
• Git credentials are granted only when protocol, host and path equal the project’s repository URL (a trailing .git is optional). Secret values are never logged or stored.

Database changes

20261019000001_task_completion_summary.sql adds tasks.completion_summary (summary passed to complete_task, posted on the pull request when the run completes) and runs.connected_at (first Hello; start of the wall-clock budget).