Seeing and searching

1One new block

core/message.py

Look for: ImageBlock images

ImageBlock has exactly four fields: ref, media_type, width, height. Frozen and slotted, like every other block in the vocabulary, and deliberately nothing more than that.

• ref, media_type — what an adapter needs to build a request to a provider
• width, height — what image_tokens needs to price the picture

The test for whether a field belongs here was narrow on purpose: does serialisation or token counting actually read it? A filename, a caption, a checksum, a source URL — all sounded useful, none of them had a real caller, so none of them made the cut. This file is the frozen vocabulary the whole harness shares. A field added "just in case" has to be carried by every adapter, every test and every stored session forever after, and taking one away later means a migration.

Notice what's missing most of all: the bytes. An ImageBlock is only a reference to a file under sessions/<id>/blobs/. So the JSONL record of a session never contains base64 — which keeps a transcript grep-able, and keeps a session with twenty screenshots in it a small file.

The second half of the change is Message.images, a convenience property that reaches into both a message's own blocks and its tool results. Everything downstream — counting, eliding, serialising — reads that one property instead of walking the blocks itself.

Take with you: in a vocabulary everything else depends on, the fields you leave out are the design.

2Where the bytes go

tools/spill.py

Look for: spill spill_image SIGNATURES

Read spill first — M5 wrote it for oversized tool output. It writes a file under the session's blobs folder and hands back a path. spill_image sits right beside it and shares the file-writing, but not the rule for when to write:

• Text spills conditionally, only when it's too big, and keeps a head and a tail so the model still sees something useful.
• An image spills always, whole — because half a PNG isn't a smaller PNG, it's a broken one.

That split is the one judgement call worth arguing about on this page. Making the existing spill function take a flag would have crammed both rules into one function body, and the flag would have had to be threaded through every caller that already existed.

Then read SIGNATURES, which is how a file of bytes becomes a picture with known dimensions. There's no image library in edgar, and no new dependency was added for this: PNG, JPEG, GIF and WebP each announce themselves in their first few bytes, and each one carries its width and height a fixed distance into the file. Fifteen lines of header-reading, against a dependency that would have doubled the install size. A file whose header can't be read is still treated as an image — it just gets width and height of zero, and stop 5 explains what that costs.

Take with you: "reuse it" and "extract the shared part" are different answers. Share the mechanism, not the policy.

3The model that cannot see

providers/quirks.py · providers/routing.py

Look for: images check_images check_capabilities

images is just one more column in a table that already had several. That's the whole point of quirks-as-data: a capability is a row, never a branch of if statements buried in the loop. Read the table and you can say which model can see pictures, without reading a line of code.

check_images is deliberately built the same shape as check_capabilities, which M9 wrote for ROUTE-6, and it refuses at the same moment: where the model is chosen, before a request is even built and before a cent is spent. The alternative — send the image anyway and let the provider reject it with a 400 — costs a round trip and hands you an error in the provider's words instead of edgar's.

There are two different ways a picture can arrive, so there are two refusals:

flowchart TD
    a["you attach @photo.png"] --> b{"can this model see?"}
    b -- "no" --> c["refused up front
before the turn starts"]
    b -- "yes" --> d["turn proceeds normally"]
    t["a tool produces a picture
mid-turn"] --> e{"can this model see?"}
    e -- "no" --> f["read answers with a sentence:
what the file is, that it can't look"]
    e -- "yes" --> g["ImageBlock reaches the model"]

An attachment is checked before the turn even starts — follow the caller into cli/oneshot.py and cli/repl.py and you'll see it fires on the attachment, not on the whole turn. But a picture a tool produces mid-turn can't be refused that way, because the model is already chosen by then. So ToolContext carries an images flag instead, and read on a picture answers a blind model with a plain sentence: here's what the file is, and I can't look at it. That's not an error — the tool call still worked, and the model gets to decide what to do next.

Take with you: when a new capability needs a refusal, find the refusal the last one used and copy its shape and its timing.

4Three ways to say the same picture

providers/openai_compat.py · providers/anthropic.py · providers/fake.py

Look for: _parts _image encoded

This is where the bytes finally come back off disk. encoded lives in providers/http.py because both real adapters need it and neither should own it. It reads the blob, base64-encodes it, and raises a ProviderError if the file is gone — rather than quietly sending an empty image. The base64 exists only for the length of one HTTP request; it's never written to disk or stored.

Read the two provider shapes side by side and they turn out to want almost the same thing, said differently:

• Chat Completions wants a content array with an image_url part holding a data: URL.
• Anthropic wants a block with a source table that names the media type and the data separately.

Both adapters just translate. Neither one decides anything — the same rule as every other line in these files.

The interesting case is a picture a tool produced. Anthropic can put an image straight inside a tool_result block. The OpenAI shape has no room for that, so openai_compat follows the tool results with one extra user message carrying the pictures instead. Different bytes go out on the wire, but it's the same conversation either way.

Finish on the fake provider — the one the whole test suite leans on. It can't actually look at anything, so it just says what arrived: media type, dimensions and reference. That one honest line is what lets a test assert an image survived a whole turn, and a --resume, without any real provider ever being reachable.

Take with you: a fake that reports what it received is worth more than a fake that pretends to understand it.

5What a picture costs

context/tokens.py

Look for: image_tokens _fit approx_message_tokens PIXELS_PER_TOKEN

Forty lines. No I/O, no clock, no config — just pure arithmetic over four integers. That purity is the only reason a number this load-bearing can be trusted: compaction decides when to run based on it, and a budget that undercounts images will silently send requests that then get rejected for being too long.

The key idea: a picture costs by its geometry, never by its file size. A 4 MB photograph and a 40 KB screenshot of the same pixel dimensions cost exactly the same, because the model resizes the image before it ever looks at it. Two rules cover every provider edgar talks to:

• Anthropic — shrink to a 1568-pixel longest side, then charge for the area over 750.
• The Chat Completions family — shrink to fit inside 2048, then shrink the short side to 768, then charge a base price plus one price per 512-pixel tile.

_fit is the shrinking step both rules share, and it only ever shrinks — it never enlarges a small image to make it cost more.

Read the first two lines of image_tokens closely. If a file's header couldn't be parsed, its size was already recorded as zero back in stop 2. Rather than guess at a picture nobody ever actually measured, the function charges for one tile's worth. That's wrong, certainly — but wrong by a bounded amount, in a known direction. That's exactly what you want from an estimate that gates something as important as compaction.

Take with you: when the input might be unmeasurable, decide what you charge for "unknown" before you actually need the answer.

6Eliding an old picture

context/compact.py

Look for: elide _stub ELIDED

This is the most dangerous file in the milestone, because of invariant 1: every assistant message with tool calls must be followed by exactly one tool message whose result ids match. Break that and every provider answers with an HTTP 400. Compaction keeps that promise by only ever removing, stubbing or summarising a whole unit — never trimming inside one.

So the change for images lives inside _stub, the same per-block helper elide already runs over everything before the cut — not in a second pass built just for pictures. That was the real judgement call. A separate _elide_images function would have been easier to read on its own, but it would also have been a second function that knows where a turn is allowed to be cut. The whole value of the invariant is that exactly one function knows that.

def elide(block):
    if block is old enough to cut:
        if block is ImageBlock:
            return TextBlock("[elided image: {media_type}, {width}x{height}, {ref}]")
        if block is already a stub:
            return block                    # idempotent: no-op on a second pass
        return _stub(block)                 # text: summarised the usual way
    return block                            # too recent, leave alone

An old picture becomes one line naming it — media type, dimensions and the reference — so a human can re-attach it later, and the model stops paying to see it on every single request. Nothing is actually lost: the bytes were never in the transcript in the first place, only in the blobs folder, which elision never touches.

Two details keep this safe:

• A tool result carrying images is stubbed together with its text — the whole unit is elided at once, never a stub still dragging its screenshots along behind it.
• The operation is idempotent. An ImageBlock that becomes a TextBlock stays that way on a second pass, and a result that's already a stub returns early instead of stubbing again.

The property suite runs elision twice over transcripts with an image on every other tool result, and checks the pairing invariant still holds after both passes.

Take with you: when one function is the only thing that knows an invariant, extending it is almost always cheaper than writing a second function that has to know it too.

7Two ways in

context/attach.py · tools/builtin/fs.py

Look for: attach _refuse Read ToolFinished

A picture can arrive two ways — a human attaches it, or a tool produces it — and both end up at the exact same ImageBlock.

@photo.png extends M19's existing path instead of growing a second one. Same token scan, same refusals in the same words (missing, a directory, a credential file, outside the working directory) — and then exactly one new fork at the end. Bytes that begin with a known image signature spill to disk and come back as an ImageBlock; anything else is decoded as text, same as before. Compare _refuse here with the version on the working state page — it's unchanged. A file that's neither text nor a recognised image is still refused, by name.

On the tool side, Read now answers a picture with the picture itself, instead of the old "looks binary; not shown". And ToolFinished gained one optional field carrying the reference, so something watching --events can find the file on disk — while the actual bytes never pass through the event stream at all. Same reason blob is already on that event for spilled text.

Take with you: a second way in should join the first one as early as possible. Here that's one if, and everything downstream never even learns there were two roads in.

8Web search, as files

examples/tools/web_search.toml · examples/skills/web-research/SKILL.md

Look for: the three commented variants, and the host that is not there.

Searching the web is really just an HTTP call with a key in a header — and M3's HTTP-tool format already describes exactly that: a request template with the host fixed, arguments filling only the path and query, and ${env:NAME} filling headers at call time (then scrubbed from whatever comes back). So web search doesn't need any new code at all. It's a TOML file.

The file ships three variants: Brave is active, Tavily and a self-hosted SearXNG sit commented out right beside it. There's no default host — that's PRV-15 doing its job. Who sees your search queries is exactly the kind of choice a harness must never make quietly for you, and here the whole answer is in a file you can read in ten seconds.

The skill is the half that actually matters. A search tool on its own just produces a model that answers from snippets — and a snippet is an advertisement for a page, not real evidence. The skill tells the model: search, then fetch the best two or three pages, then answer citing the URLs. It also restates a rule the harness already enforces elsewhere: a search result is untrusted output, so in auto mode any shell or network call that follows it has to ask first.

This page's claim was actually tested, not just asserted: a subagent with none of this milestone's context, given only the Cookbook and the extension guide and forbidden from opening src/, produced the same host, header and environment variable completely unaided.

Take with you: the tool is the easy half. The instructions for using it well are the part that makes it worth having.

9git, as files

examples/tools/git-commit.toml · examples/skills/git/SKILL.md · examples/extensions/git-trail/hooks.toml

Look for: the message as one argument, and the two verbs that are missing.

git is just a CLI, and M3's command-tool format runs an argv template directly, never through a shell. Four verbs, one file each: look at the tree, read the diff, read the log, commit what's staged. Three are read_only; the fourth writes, and goes through the permission guard like any other write.

Open git-commit.toml and look at one line: the whole commit message is a single argv element. That matters because no shell ever sees it — so a message containing newlines, semicolons and backticks is just message text, nothing more. Every agent harness that shells out to git commit -m "..." the naive way has this bug sitting latent inside it.

Then notice what's missing, because the absences are the design:

• No git-add. Staging is where a human decides what actually goes into the commit.
• No push, merge or tag. Those are calls only the user gets to make.
• git-log takes a required count, not an optional one. An argv element naming an absent argument gets dropped entirely — so an optional --max-count would silently mean "the entire history", which is the opposite of a sensible default.

The extension is the post_tool example, and it's observation only: fired and forgotten, its output never reaching the model, matched to the one tool worth recording. Its script has an interesting constraint — the event carries the call's outcome, never the tool's actual output, so it has to read the finished commit back out of git itself. That's the shape of every hook that isn't a pre_tool hook.

Take with you: the argument for declarative extension formats isn't that they're elegant. It's that this whole stop is a diff with no code in it.