Skip to main content

Designing a Deterministic Agentic Coding Orchestrator

How skeleton-parallel migrated from phase-based parallel execution to a task-based agent loop with bounded retries, checkpoints, and quality gates.

5 min read·Intermediate·Concept·Jul 17, 2026
agentic developmentorchestrationquality gatesdeterministic systems

Designing a Deterministic Agentic Coding Orchestrator

What Was Built​

skeleton-parallel is a reusable baseline for building deterministic software with AI agents. Version 1.0 replaced a phase-based parallel orchestrator (run_parallel.sh + config/phases.yaml) with a task-based agentic CLI (skeleton run + docs/PLAN.md). The system runs autonomous implementation tasks through a bounded retry pipeline with Git checkpoint rollback and automated quality gates.

The Problem​

AI agents can implement software features, but unconstrained agent loops are non-deterministic, expensive, and prone to infinite retry. You need an orchestrator that:

  • Defines work as discrete, verifiable tasks.
  • Runs agents with bounded retries and explicit rollback.
  • Validates architecture rules (DTO immutability, module boundaries).
  • Terminates in a known state — success, rollback, or remediation_failed.

Why This Problem Is Difficult​

  1. Parallelism vs context — parallel agents are fast but merge conflicts are costly.
  2. Agent non-determinism — same prompt can produce different code.
  3. Architecture drift — agents violate module boundaries without enforcement.
  4. Unbounded retries — failed agents can loop forever without checkpoint rollback.
  5. Migration — existing projects using phase-based config need a compatibility path.

Beginner Mental Model​

Think of a construction site manager with a task list (docs/PLAN.md). For each task, the manager sends workers (AI agents) through a fixed checklist: build → validate DTOs → check integration → run quality gates. Before starting each task, the manager places a checkpoint flag (Git tag). If workers fail after 5 attempts, the manager resets to the flag and marks the task failed. The site always ends in a known state.

Requirements and Constraints​

Requirementv1.0 implementation
Work definitiondocs/PLAN.md with ### Task N sections
Orchestratorskeleton run CLI reading config/skeleton.yaml
Bounded retriesPer-stage limits in YAML (retries.task_runner, etc.)
Checkpoint rollbackGit tags before each task; git reset --hard on failure
Quality gatesscripts/hooks/quality-gates.sh (project-specific)
Provider flexibilityrouter_http, cli_subscription, sdk_cursor drivers
Backward compatibilityrun_parallel.sh shim delegates to skeleton run for one release

Architecture Overview​

Three execution modes balance speed and cost:

ModeParallelismBest for
--parallelFull (separate worktrees)Independent tasks, deadline pressure
--sequentialNone (shared context)Cost-sensitive, dependent tasks
Default (hybrid)Groups in parallel, sequential within groupMost sessions

Execution Flow​

  1. Operator defines tasks in docs/PLAN.md with dependencies and complexity.
  2. skeleton run <task-ids> creates Git branches/worktrees per mode.
  3. Checkpoint tag placed: checkpoint-<task>-pre.
  4. Agent pipeline executes per task:
    • task-runner — implements the task (up to N retries).
    • dto-guardian — validates contracts/ immutability (STRICT).
    • integration — checks cross-module wiring.
    • refactor — fixes quality gate failures (bounded retries).
  5. On retry exhaustion: git reset --hard to checkpoint; task marked failed.
  6. Successful tasks auto-merge via union strategy; conflict-resolver agent handles merges.
  7. Post-merge: merge-reviewer validates DTO flow and module boundaries.
  8. Global validation runs quality-gates.sh; remediation agent fixes failures.
  9. gh pr create opens a reviewable pull request.

Important Components​

ComponentResponsibility
skeleton runTask orchestration CLI entry point
config/skeleton.yamlRetry limits, model routing, execution drivers
.skeleton-dev/run-status.jsonPer-task state tracking
.skeleton-dev/events.jsonlAppend-only event log (rollbacks, failures)
scripts/hooks/quality-gates.shProject-specific lint, test, architecture checks
scripts/hooks/setup-env.shDependency installation per worktree
run_parallel.shDeprecated shim → skeleton run

Simplified Implementation Examples​

Universal retry pattern (from project docs):

execute → validate → fix → re-validate → bounded retry → success OR rollback

Task state values:

{
  "task-1": {
    "state": "complete",
    "model": "claude-sonnet-4.6",
    "exit_code": 0
  }
}

States: running → complete | failed | timed_out

Reliability and Idempotency​

  • State storage: .skeleton-dev/run-status.json and events.jsonl.
  • Synchronous agent stages: Each pipeline stage completes before the next begins.
  • Parallel task groups: Independent worktrees isolate file conflicts.
  • Checkpoint rollback: Git tags provide deterministic undo — no partial file state.
  • Guaranteed termination: All retry limits are bounded; system cannot loop forever.

Failure Modes​

FailureBehaviour
Task exceeds retry limitRollback to checkpoint; other parallel tasks continue
DTO validation fails (STRICT)Rollback; DTO changes are never partially merged
Merge conflictconflict-resolver agent (up to 5 retries)
Global validation failsrefactor remediation (up to 5 retries) → remediation_failed
Agent timeout (30 min default)Task marked timed_out; treated as failure
Partial group failure (hybrid mode)Auto-merge skipped; operator runs skeleton run merge manually

Trade-offs and Rejected Alternatives​

ChoiceWhyRejected alternative
Task-based (PLAN.md)Human-readable; maps to PR-sized work unitsPhase YAML — harder to read and migrate
Git checkpoint rollbackSimple, proven, auditableFilesystem snapshots — not version-controlled
Hook-based quality gatesLanguage-agnostic orchestratorHardcoded Python/Go checks in orchestrator
Bounded retries everywhereGuaranteed terminationUnlimited retry — infinite loops, token waste
One-release shimSmooth migration for existing usersHard cutover — breaks CI pipelines

Testing​

Quality gate chain: make test → make lint → make check. The orchestrator delegates all language-specific validation to hook scripts, keeping the core orchestrator project-agnostic.

Operations and Observability​

skeleton run 1 2 3               # hybrid mode (default)
skeleton run --parallel 1 2 3     # full parallel
skeleton run --sequential 1 2 3   # token-optimized
skeleton run --full               # entire PLAN
skeleton doctor                   # verify installation

Status persisted to .skeleton-dev/run-status.json with per-task model, exit code, and timestamps. Event log in .skeleton-dev/events.jsonl captures rollbacks.

Lessons Learned​

  1. Tasks beat phases — ### Task N in Markdown is more portable than YAML phase configs and maps naturally to PR scope.
  2. Checkpoints before agents, not after — rollback must be instant and trustworthy.
  3. Strict DTO enforcement — architecture violations caught early save expensive integration debugging.
  4. Shim migrations — one-release compatibility shims prevent breaking existing automation while new CLI stabilizes.

Sources​