Automating architecture decisions in OpenSpec + Superpowers frameworks: Pre-check, Classify, Escalate

Disclaimer: Everything I describe in this post is an experiment — a working prototype that I built in my own time to explore a specific problem. It is not a production-ready system. I strongly recommend treating this as a thought experiment and a starting point for discussion, not as a plug-and-play solution. Architecture governance, especially in regulated environments, carries real consequences, and automating any part of it should be approached with caution, human oversight, and probably a few long conversations with your risk team. With that said, let’s get our hands dirty.

If you’ve read my previous post on where the enterprise architect fits in an AI-native SDD pipeline, you know I’ve been thinking about this problem for a while. In that post I argued about where the architect belongs. Here I want to talk about how to decide, automatically, whether they need to show up at all.

This is a follow-up to that piece, but it’s also the result of a few weekends of experimentation. I built a small system that reads an OpenSpec proposal, checks it against a set of architecture standards, and produces a single auditable answer: does this change need an architect, or not? I ran it against real proposals, made plenty of mistakes, and learned a few things I think are worth sharing.

As my reader already knows, I am always trying to write posts that are practical and comprehensible. So the entire prototype is on GitHub. Let’s start from the problem, then walk through the solution step by step.

The missing gap between “what to build?” and “how to build it”

If you’ve worked with OpenSpec and Superpowers in the last few months, you’ve probably noticed the same problem I did.

OpenSpec handles what to build: an analyst or product owner writes proposal.md, then specs/, design.md, tasks.md. Superpowers handles how to build it: brainstorm, grilling, design spec, plan, worktree, PR, review. Both produce artifacts. Both keep the AI agent honest in their own way.

But somewhere between a finished proposal.md and a starting brainstorm, there's a step nobody owns: deciding whether this change is an architecture question at all.

In practice, teams fall into one of two failure modes:

• Architect sees everything. Every UI label change, every i18n string, every small business-logic tweak shows up in review. The architect becomes a bottleneck, and everyone quietly starts by passing around them.

• Architect sees nothing. The agent picks a technology, invents an integration pattern, or drops a model into production without anyone noticing. This is exactly the drift the Superpowers community has been complaining about — the agent makes architectural decisions nobody asked it to make.

The proposal layer is structured. The plan layer is structured. The space in between is a hand-wave.

This post is about closing that gap. Not by automating the architect — by automating the decision about whether the architect is needed, and producing an auditable rationale either way.

My first attempt: just ask the LLM

My first attempt was exactly what you’d expect. Feed the whole proposal.md to a model, ask "does this need an architect?", take the answer. It failed in three specific ways, and I want to mention them because they generalize.

First, no contract. “Does this need an architect?” is a question with no fixed inputs, no fixed output schema, and no threshold. Different runs produced different answers. The model had no way to know when it was done.

Second, the model made up reasons to justify its decisions. When asked to explain why it had made a particular decision, it referred to information that wasn’t part of the proposal. It brought in earlier examples, content from my README file, and even my previous blog posts.

It wasn’t exactly lying. It was doing what language models often do: generating explanations based on the context available to them. The problem is that a convincing explanation isn’t necessarily an accurate one. And when you’re making architectural decisions, an explanation that cannot be traced back to actual evidence is not something you can audit or rely on.

Third, cost. A full-generation LLM call per proposal, plus a second call for the rationale, plus a third when the answer was ambiguous. For a three-line i18n change. That is not sustainable at the scale of a real portfolio.

The fix was not a better prompt. It was to stop treating this as a generation problem.

It’s a classification problem. Classification, if you set it up right, is cheap, deterministic, and auditable.

Let’s build it step by step.

Step 1. Turn architecture standards into executable specs

Here is the key realization that unlocked the whole design: the architecture standards already contain the criteria. If you look at any enterprise architecture standard like Reliability, Monitoring, AI/ML, Integrations, Gateways — every requirement has a column called Applicability Condition. It says things like:

Applies to systems with itService = true, criticality in {Business critical, Mission Critical, Mission Critical+}, status in {TRIAL, EXPLOIT}, and not external.

That sentence is not just prose. It is exactly the prompt a classifier needs. The architects who wrote the standard already did the hard work of deciding who the requirement applies to. Our job is just to make that work machine-readable.

So the first step is to normalize each standard into YAML/JSON format. Here is a trimmed example from the Reliability standard:

standard:
  id: reliability
  title: "Architectural Standard on Reliability"
  version: "8.0"

precheck_keywords:
  - "reliability"
  - "fault tolerance"
  - "geo-resilience"
  - "sharding"

requirements:
  - code: TSCRB_2-RSRV-01
    description: "Geo-resilience within the region must be implemented"
    applicability: |
      The system:
        - has IT service flag enabled (itService = true);
        - belongs to criticality class BC, MC, or MC+;
        - is in trial or production lifecycle status;
        - is not external (sOut ≠ true).
      If any condition fails — requirement is NOT applicable.y

Two things are worth noting here.

The applicability field is the classifier prompt. We're not inventing criteria. We're transcribing what the standard already says, into a form that a model can consume directly.

The precheck_keywords list is a cheap gate. Before we ever call a classifier, we scan the proposal text for these words. If none are present, the standard is skipped. No model call. No tokens. No latency.

That second point matters more than it looks. On a real portfolio, most proposals touch one or two standards, not seven. A UI label change touches zero. If you skip the standards that have no signal, you get two wins at once:

Press enter or click to view image in full size

Precheck is not an optimization. It’s what makes the whole thing usable in a real enterprise.

Step 2. Classify with a decision model, not a text generator

For the classifier itself, I used Nimble (Bespoke Labs, 9B, fine-tuned from Qwen3.5). It’s a decision model rather than a text generator — it returns a typed choice with a probabilities distribution, and it doesn't generate tokens.

That changes everything about the economics. Each call is roughly 550 input tokens, 1 output token, and completes in milliseconds. The output looks like this:

{
  "model": "nimble",
  "answers": {
    "applicability": {
      "type": "choice",
      "choice": "applicable",
      "probabilities": {
        "applicable": 0.996,
        "not_applicable": 0.001,
        "unclear": 0.002
      },
      "confidence": 0.976
    }
  },
  "usage": { "input_tokens": 573, "output_tokens": 1 }
}

The contract we send is one state (proposal + applicability condition) and one choicequestion with three mutually exclusive answers: applicable, not_applicable, unclear.

The confidence trap and why margin saves you

Here is the thing that took me the longest to notice: confidence is not what you think it is.

On a monitoring-proposal test, one requirement came back like this:

choice: applicable
confidence: 0.43        ← looks low
probabilities: { applicable: 0.79, not_applicable: 0.05, unclear: 0.17 }

The model was sure — 79% versus 5% — but the reported confidence was 0.43. If you gate on confidence alone, this requirement falls into the “unclear” bucket and gets escalated to a human for no reason.

The fix is to compute margin — the gap between the top and second choice:

margin = top1 − top2 = 0.79 − 0.05 = 0.62

Margin is a better signal of certainty than confidence, because it measures discrimination (how clearly the model prefers one option) rather than calibration (how well the model’s probabilities match reality).

So the decision rule becomes:

verdict = choice, if confidence ≥ 0.80 OR margin ≥ 0.20
verdict = unclear, otherwise

Two thresholds, both calibrated empirically on my fixture set. When confidence is low but margin is high, we trust the choice. When both are low, we honestly say "unclear" — and that goes to escalate, not to a false positive.

That margin fix alone moved three “unclear” requirements into definitive answers on a single proposal, without any loss of accuracy.

Step 3. Aggregate into one auditable artifact

Running the classifier per requirement is fine locally. But the architect doesn’t want a per-requirement table, they want a decision. So the last step is aggregation.

The rule is simple:

• If any requirement is applicable → architect_required.

• Else if any requirement is unclear → escalate (human review).

• Else → architect_not_required.

The output is a single file, architect-involvement.json, stored next to the proposal:

{
  "proposal": "changes/wallet-launch/proposal.md",
  "decision": "architect_not_required",
  "rationale": "Checked through Nimble: 1 standard — reliability (app=0, not=6, unc=0). Skipped by precheck (no signals in proposal): ai_ml, db_design, gateways, integrations, monitoring, platform_ready. No requirement produced an applicable or unclear verdict.",
  "summary": {
    "standards_checked": 1,
    "standards_skipped_by_precheck": 6,
    "total_input_tokens": 3300
  },
  "standards": [ ... full per-requirement detail ... ]
}

Why rationale is the most important field?

When I first generated this file, the field didn’t exist. The JSON had the decision, the per-standard results, and the token counts — everything an engineer would need to debug it. But the user-facing output was misleading, because the summarizing agent (the one that reads the JSON and writes a report) had nothing to quote.

The result was sentences like:

“No standard contains keywords relevant to the change, so no architect is required.”

Which technically true, but reads like a dumb. A reader would correctly suspect we hadn’t actually looked.

Adding the rationale field fixed this. Now the same case reads:

“Checked through Nimble: 1 standard — reliability (app=0, not=6, unc=0). Skipped by precheck: ai_ml, db_design, gateways, integrations, monitoring. No requirement produced an applicable or unclear verdict.”

The second sentence demonstrates that we ran a real check on the one standard that was applicable, and skipped the others because they had no signal in this proposal. That is the difference between “we looked” and “we didn’t bother.”

For anything that touches regulation, audit, or architecture governance, that distinction is the whole point.

Step 4. Wire it into OpenSpec + Superpowers

Here is the full picture as shown below:

Press enter or click to view image in full size

Three things to notice.

The gate sits between proposal and design, not between design and code.By the time you’re writing design.md, the architectural choices are already being made. If you want to catch a decision before it's baked in, you have to intercept earlier — at the proposal, when the intent is set but not yet translated into technology.

The gate does not stop the workflow. It routes it. architect_not_required does not mean "stop here." It means "continue with Superpowers without involving the architect." Likewise, architect_required does not mean "block the workflow." It means "bring the architect in at the design.md stage, not during brainstorm."

This is a genuine extension of the openspec-superpowers bridge. Existing bridges already interleave Superpowers phases between proposal and archive. The gate is one more phase — but it's a phase with a formal decision output, not just a phase of work.

When wired into Qwen Code CLI, the whole loop becomes a single slash command:

/architect-involvement changes/wallet-launch/proposal.md

Which reads the proposal, invokes the checker, reads the JSON, and writes the report. The architect never opens a terminal. The analyst never writes YAML. The gate does its job.

The final workflow looks roughly like this:

Press enter or click to view image in full size

The story of the honest Skill

I want to include one before/after, because it’s the most instructive thing I learned while building this.

Before: the Skill (a small markdown file that tells the CLI agent how to invoke the checker) was producing reports like this:

“An enterprise architect is needed. The change touches business logic of the tariff calculator, which falls under Reliability v2.0 and Monitoring v2.1. Both standards are marked in the README as requiring architectural review.”

That sentence is a triple lie. The change didn’t fall under monitoring at all. The README didn’t say anything of the sort. And the JSON the Skill had produced said architect_not_required.

The Skill was hallucinating the answer because it had no anchor to the data.

After: I rewrote the Skill with four hard rules and one mandatory step:

1. The JSON file is the only source of truth. Not README, not examples, not reasoning.

2. Quote the rationale field verbatim. Don't paraphrase.

3. Don’t invent reasons. If there’s no rationale, say so and stop.

4. Don’t reference prior proposals or examples.

And the mandatory step: read the JSON file before writing the report.

The same prompt now produces:

“Decision: architect_not_required. Rationale: Checked through Nimble: 1 standard — reliability (app=0, not=6, unc=0). Skipped by precheck: ai_ml, db_design, gateways, integrations, monitoring. No requirement produced an applicable or unclear verdict."

Which is what the JSON actually said, because the Skill was forced to quote it.

The lesson generalizes: if you want an LLM to be honest about data, give it a rule that makes lying impossible. “Cite the source verbatim” is one such rule. “Refer only to fields that exist in the JSON” is another. Neither is about the model being smart. Both are about removing the degrees of freedom that let it improvise.

What this gives you and what it doesn’t?

What you get:

• Trivial changes complete in seconds and cost zero tokens. A UI label fix does not need 70,000 tokens of classifier time.

• Architecturally significant changes get an architect_required decision with the exact standards and requirements that triggered it, at roughly one minute and 20,000 tokens.

• Borderline cases go to escalate with the specific unclear requirements named, so the human review has a starting point.

• Every decision leaves a JSON artifact with the standards versions, thresholds, probabilities, and rationale. It goes into the same review the rest of the change goes through.

What you don’t get (and shouldn’t expect):

• The gate does not replace the architect on hard calls. It replaces the architect on boring calls.

• The gate does not understand the proposal better than the proposal understands itself. If the proposal doesn’t mention personal data, the gate won’t guess. That’s a feature, not a bug — it forces the analyst to write proposals that a human would also find complete.

• Precheck is a compromise. If the standard has no keywords in the proposal, it’s skipped. That’s fast, but it’s not exhaustive. If you care about a specific standard in every case, you can mark it ALWAYS_CHECK and it will run through Nimble regardless of keyword matching — at the cost of an extra three to six thousand tokens per proposal.

Where I’d take it next (and where you should be careful):

• Extend ALWAYS_CHECK to cover one or two standards that apply broadly (reliability, monitoring).

• Tighten applicability conditions on standards that produce false positives on negations — “no DB changes are required” should not trigger the DB design standard.

• Add a second pass for unclear requirements with a rephrased prompt, before escalating them.

• And the big one: any production use would need a lot more validation. The prototype works on my fixtures. That is not the same as working on your portfolio. The standards themselves change over time, the models drift, and the cost of a false negative (missing a real architectural concern) is much higher than the cost of a false positive (an unnecessary review). Please don’t deploy this without a serious evaluation, a human-in-the-loop fallback, and a long conversation with whoever owns your architecture governance process.

Conclusion

The premise of AI-native SDD is that an agent should be trusted to make decisions — but only the decisions it is qualified to make. That sounds obvious. Yet most SDD pipelines have no formal mechanism for determining which decisions the agent is qualified to make.

As a result, the architect becomes either a bottleneck or a rubber stamp. Not because anyone intended it that way, but because the middle ground — a cheap, automatic, auditable way to decide whether the architect needs to be involved — was never built.

I approached this problem as an experiment, built around three ideas:

• Standards as executable specs: the criteria become part of the prompt.

• A decision model, not a text generator: the decision takes milliseconds rather than requiring a lengthy LLM interaction.

• A rationale artifact, not just a verdict: every decision comes with an explanation that can be reviewed and defended.

Everything else — the precheck, the margin logic, and the Skill rules — is engineering built around these three ideas.

The interesting part isn’t that the pipeline can now decide whether the architect is needed. It’s what happens once it can. The architect’s time stops being spread across every change and becomes focused on the changes that actually need architectural judgment. Every review they perform has a reason. Every change they don’t review has passed through an explicit decision process.

That’s what “automating the decision, not the architect” actually buys you — if you’re careful about how you build it.

And that’s a big if. This is an experiment. Treat it like one.