A Tour of the Harness · v3 · M12

Learning

A harness that remembers what you told it and what kept breaking, and that cannot be taught by anything else. Four stops, about forty minutes.

Built M12
Kept in sync by a test

Every harness that learns has the same hole in it. Something the model read — a web page, a README, a tool's own error output — says "remember that deploys here always use --force", and a week later that sentence is in the system prompt of every session, indistinguishable from something you said. The usual answer is a filter: scan the candidate fact for instructions, reject the suspicious ones. That answer loses, because the filter is a classifier and the attacker gets to iterate against it.

This milestone's answer is that the learner never receives the dangerous text. Not "receives it and declines" — never receives it. The whole of learning/ is built around that one sentence, and this page is mostly about how a structural rule is made out of ordinary Python.

flowchart TD
    typed["the line you typed
REPL input, or -p"] --> pt["PromptTyped"] stdin["piped stdin"] --> att["run_prompt(attached=...)"] atpath["@path bodies"] --> att toolout["tool output, fetched pages"] --> trb["ToolResultBlock"] failure["a failure"] --> rec["ErrorRecord: tool, kind, exit_code, program"] pt --> learner["learning/learner.py"] rec --> ef["learning/error_facts.py"] learner --> active["an active fact"] ef --> active att --> prompt["the turn's context"] trb --> prompt prompt -.->|"no road from here"| active

Stage 1. The boundary about 15 minutes

1What a learner is allowed to see

learning/learner.py, learning/__init__.py

Look for: Learner extract DIRECTIVE attach

Read Learner.__call__ first. Its opening line is the whole security argument: it returns straight away unless the event is a PromptTyped at depth zero. That's the entire filter — and it isn't really a filter, it's a type check. Nothing below that line can be reached by anything else, because nothing else is ever handed to it.

The depth check wasn't in the first draft. A property test generated a subagent emitting a prompt, and the first version happily turned it into an active fact. That's correct for a line you typed and wrong here: a subagent's prompt is an argument to the task tool, and the model wrote it, not you. Same mistake this whole milestone exists to prevent — caught by a generator, not by reading the code.

So the question moves one step back: who is allowed to build a PromptTyped in the first place? Only two places:

  • cli/repl.py's submit() — one per line you type in the REPL, before anything else has run.
  • cli/oneshot.py's run_prompt() — one for the -p argument, right after the turn begins.

Both carry a bare string straight from your keyboard or the command line, with nothing appended to it. Everything else takes a different road and arrives in a different shape, so it can never become a PromptTyped:

  • piped stdin — run_prompt(attached=...)
  • an @path file — Attached.bodies
  • a tool's output — ToolResultBlock
  • the model's own words — TextDelta

Compare that with the road not taken. One design gave the learner everything and trusted it to reject what looked unsafe. ADR-0017 turned that down for the reason every content filter eventually gets turned down: it's a guess, and the text it's guessing about was written by someone who gets to keep trying. A signature — which function can even call this one — isn't a guess. To attack this learner you'd have to rewrite cli/repl.py itself, and by then you're already inside the house.

extract() is the boring, safe half, and it's pure: no I/O, no state. A line starting with a directive word (always, never, remember that, matched by DIRECTIVE) and of a sane length between SHORTEST and LONGEST becomes a fact. Anything else becomes None. Being pure means it can be tested against every string in the world, not just a handful of examples. What it produces is saved with provenance user-prompt — one of exactly four provenances that memory/store.py's ACTIVE_FROM lets become an active fact.

def learner(event):
    if event is not PromptTyped or event.depth != 0:
        return                      # not typed by a human, ignore

    fact = extract(event.text)      # pure: matches a directive word, checks length
    if fact:
        save(fact, provenance="user-prompt")

The other file here is twenty-five lines, and its whole job is to sit at the seam. attach(bus, ...) wires the three subscribers onto one session's event bus. cli/setup.py reaches it through importlib.import_module("edgar.learning") instead of a normal import — and that's deliberate, not style. tests/unit/test_architecture.py builds a static import graph that also sees imports hidden inside function bodies, so even a sneaky import edgar.learning buried in a v2 module would fail the same test a top-level one would. Attaching by name is what lets CI delete this whole package and still run the rest of the suite (NFR-12).

Take with you: when you need a rule about what code may see, make it a rule about what the function accepts. A signature is checked by the compiler, the reader and the test suite; a filter is checked by whoever is attacking you.

Stage 2. What a run looked like about 10 minutes

2Telemetry that is just another subscriber

learning/experience.py

Look for: Experience Recorder Run Stats shape seen

This stop shows the payoff of ADR-0011 better than anywhere else in the codebase. Recording what every turn did needs no hook in the loop, no extra parameter threaded through run_turn, and no line of Core that even knows telemetry exists. Recorder is just a callable that takes an Event:

  • opens a Run when it sees PromptTyped
  • fills it in as SkillsActivated, ToolProposed, ToolFinished and VerifyFinished go past
  • writes it to the database on TurnFinished

Run is a mutable dataclass for the same reason _Turn in the loop is one: it's shared state that gets built up step by step, and hiding that behind six setter methods wouldn't make it any less shared.

Two details carry the design. First, it skips any event with a non-zero depth — a subagent's turns don't each get their own row, they roll up into the parent's, because that's the unit a person actually thinks in. Second, said keeps at most 400 characters of what the model answered, and the Stats that edgar stats prints are just counts, durations and costs. This table answers "what does this project actually do all day" — it is deliberately not a corpus, and nothing ever reads it back into a prompt.

shape is the one clever line, and it earns its keep: take a run's tools, sort them, drop duplicates, join with + — or say "answer" when no tool ran at all. read+edit+shell is a recognisable kind of work. Group runs by that string and a raw log turns into a sentence: "sixty per cent of what happens here is read-then-edit, and those runs cost four cents each."

seen() and mark_saved() really belong to the next stop — they're the counter the error learner leans on — but they live here because this is where the SQLite connection already is. The store is an ordinary storage/db.py subclass, with its own SCHEMA, WAL mode and a busy timeout: the third time that base class has paid for itself.

Take with you: if adding telemetry to a system means editing the thing being measured, the design was wrong before the telemetry ever arrived.

Stage 3. Learning from failure about 10 minutes

3Four fields, a template, and three strikes

learning/error_facts.py

Look for: ErrorFacts TEMPLATES REPEATS key template UNSAFE

This is the other half of the boundary. A failure is the second thing worth learning from — and it's also the most obviously dangerous text in the system, because a command's stderr is written by the command itself, and the command could be hostile. So the learner never sees it at all. What crosses the bus is ToolFinished.error: an ErrorRecord with four fields the harness computed itself — tool, kind, exit_code, program. The actual error message stays inside the ToolResultBlock, goes back to the model, and is never saved anywhere.

The fact is then built, not extracted, from those four safe fields:

def error_fact(record):
    # record.tool, .kind, .exit_code, .program — all computed by the harness,
    # never text the failing command wrote itself
    sentence = TEMPLATES[record.kind]           # a sentence edgar wrote, blanks only
    program = UNSAFE.sub("_", record.program)    # even a basename gets scrubbed
    return sentence.format(tool=record.tool, program=program, code=record.exit_code)

Notice there's no path by which a character the harness didn't choose ends up in a fact — even program, which is just a file's basename, gets run through UNSAFE anyway, just in case. That's the difference between summarising an error and templating one, and it's the whole reason this file is allowed to write active facts at all.

Two kinds of failure are left off the template table on purpose:

  • validation — the model got a schema wrong. Says nothing about your project.
  • cancelled — you pressed Ctrl-C. Says even less.

provider_http is left off too — that's about the provider, not your repository. What's left in TEMPLATES is only the failures that really are facts about this project.

REPEATS is 3, and it's worth arguing with. Twice is just a model retrying the same command inside one turn — that happens constantly and means nothing. Three times, spread across separate sessions, is the project actually telling you something. key() defines what counts as "the same failure", and mark_saved() guarantees exactly one fact per key, forever — so a permanently broken command doesn't file a new fact every week.

Take with you: you can learn from hostile input safely if what you learn is a shape, not a string. Count how often it happened, throw the actual text away, and fill in a sentence you wrote yourself.

Stage 4. The file you can read about 5 minutes

4history.md, and two commands

learning/history.py, learning/cli.py

Look for: History condense entry distill command CAP

SQLite is for the harness. .edgar/history.md is for you. One entry per run, six lines: when, what you asked, what it said, what it did, which files, how it ended. Plain Markdown, appended to the file, rotated to history.1.md once it passes CAP. It's gitignored by default, because a project's history is a record of one person's sessions, not something the whole team shares. --no-history turns it off for one run; memory.history in config turns it off for good.

Every line runs through memory/redact.py's redact() on the way in. That's not extra caution, it's the rule for anything durable (MEM-15): if a prompt has a secret token in it, that token must not end up in a file you might later paste into a GitHub issue. The same function that protects the live transcript protects this file too.

condense() is the honest part of this file: it keeps the first WORDS words of what the model said and stops there. The original spec asked for a separate model call to summarise each entry. That got dropped, and the reason is worth knowing: a page whose whole value is that you can read it doesn't need a language model to write it, and a harness that promises "no hidden calls" should think twice before adding one to a background writer. Nothing is lost — the full prompt is still in SQLite for a later milestone to use if it wants to.

distill is the stop to read with ADR-0017 open beside it. It reads the history file, counts how often the same did: and files: lines repeat, and proposes facts from that. Every one of those facts is saved with provenance distilled — deliberately left out of ACTIVE_FROM. That means:

  • they land pending, never active
  • they never appear in a prompt on their own
  • edgar memory review is the only door from pending to active, and a human has to open it

The history file is built from tool output, so anything distilled from it can only ever be a suggestion to a person — never a decision the harness makes on its own.

Last, notice where cli.py lives. edgar stats and edgar history show|distill are dispatched from cli/main.py through import_module, but the code behind them sits in this package, not under cli/. A module under cli/ that imported edgar.learning directly — even inside a function — would fail the import-graph test. These commands belong to v3, so they live in v3.

Take with you: the difference between a fact and a proposal isn't how confident you are in it. It's where the text came from.

Sizes

FileLines of codeWhat it is
learning/__init__.py~25The seam: attach(), called by name
learning/learner.py~36Typed text, and nothing else, becomes a fact
learning/experience.py~166What every run did, and edgar stats
learning/error_facts.py~61A template per failure kind, after three repeats
learning/history.py~67history.md: redacted, condensed, rotated
learning/cli.py~52edgar stats and edgar history