Iurii Explains: Typography Pipeline Architecture: Order, Spans, Idempotency

Here’s a bug that only shows up if you go looking for it: take the string .--. in German locale conventions, run it through a typography pass once, and you get . – . – the bare hyphen pair correctly promoted to a spaced en dash. Run the same output through the same pass a second time, and it becomes . –. – the space before the full stop quietly stripped, leaving the dash asymmetrically spaced. Nothing in either rule was wrong on its own. The spacing rule was right to strip a space before a full stop; the dash rule was right to promote -- to a spaced dash. Put them in a pipeline together and run the pipeline twice, and the composition breaks in a way neither rule’s own tests could ever catch.

I built polytypo – a typography-normalization engine – and this week wired it into this site’s own build: a git pre-commit hook that normalizes staged content, and a CI gate meant to run the same check against the live site. The hook is written and staged; the CI gate is written too, but hasn’t run anywhere yet – there’s no green run to point to, and I’m not going to pretend otherwise. It also exists in five separately published runtimes (JavaScript, Python, Go, Ruby, PHP), each independently conformant against the same versioned spec – that part gets its own short section below, because it’s genuinely true and worth stating, but it’s not the interesting engineering problem. The interesting problem is the one above: what does it actually take to run nine text-rewriting rules over arbitrary input, guarantee the order they run in never changes the meaning of ‘correct,’ and prove the whole composition is a fixed point – not just each rule in isolation?

Nine Rules, One Fixed Order

polytypo’s rule pipeline is nine rules, and spec/rules/order.json is the single source of truth for what order they run in – never registration order, never map-iteration order (Go randomizes map iteration; a pipeline built on a map behaves differently in Go than in JavaScript for that reason alone). Rule ids are public API: they show up in the rules option a caller can pass to disable one, and renaming an id is a breaking change, full stop.

ranges is worth a note on its own: it defaults off, but it’s still a full rule id with the same public-API stability as the other eight. An implementation that skips it isn’t ‘conformant with one optional feature missing’ – it’s non-conforming, exactly as skipping any other rule would be. ‘Off by default’ is a runtime option a caller sets; it is not license for a port to leave the rule unimplemented.

The order column isn’t arbitrary, and the spec says so in its own comments rather than leaving it to be inferred. quotes runs before apostrophe specifically ‘so that a straight apostrophe is still available as quote evidence’ – quotes needs to look at an unconverted mark to decide whether it’s plausibly an opening quotation before handing off whatever it declines to apostrophe. hyphen runs after dashes, ‘which has already declined every letter-adjacent hyphen’ – by the time hyphen sees a token, dashes has already ruled out every case where a letter sits on either side. nbsp runs last ‘so that every other rule has already settled the surrounding characters’ – it’s the rule that decides whether a space should become non-breaking, and that decision only makes sense once nothing upstream is still going to move the space.

The spec states the general principle for why any of this is load-bearing rather than cosmetic, in its idempotency document: a rule may freely create work for a rule ordered after it, ‘because the later rule has not run yet and will clean it up in the same pass.’ What it must never do is create work for a rule ordered before it – there’s no second chance to go back. That asymmetry, in the spec’s own phrasing, is the whole content of ‘the rules run in a fixed order’.

You can see the quotes-before-apostrophe ordering do real work on an ordinary sentence. quotes‘own fixture set walks Don't touch it — it's the '90s. directly: the two apostrophes inside Don't and it's get vetoed as quote candidates by a medial-apostrophe heuristic, and the leading mark in '90s is recognized as a plausible quote-opener (canOpen) but never finds a matching close anywhere in the sentence – so quotes declines it and hands the untouched character to apostrophe, which renders it correctly as the decade-elision mark on its own turn. None of that reasoning is available after the fact: apostrophe doesn’t scan the rest of the document looking for a matching quote mark, and it isn’t supposed to – that’s specifically quotes’ job, and it only works because quotes gets first look at the raw, unconverted character.

The Parser Locates Text; It Never Produces Output

html and markdown mode hand the rules a document where most of the bytes must never be touched – tags, code, URLs – and the actual prose is scattered across dozens of disconnected fragments. The spec’s mode-adapter document (spec/rules/modes.md) settles, in one sentence, what the rewritten output is allowed to be: ‘the output is the input with a set of disjoint substring replacements applied, and nothing else.’ The parser’s only job is to locate which spans of text are processable and which are markup. It is never used to produce output. That’s a deliberately stronger claim than ‘byte-identical when no changes are needed’ – it’s a prohibition, stated that way because it’s the only form five separate parsers (parse5 in JS, nokogiri in Ruby, Go’s x/net/html, PHP’s DOM, lxml in Python) can’t quietly drift apart on. Reconstruct output from a parsed tree instead of patching the original bytes at recorded offsets, and you’re non-conforming even if your output happens to look right, because attribute quoting, self-closing tag style, character-reference spelling and comment formatting are all things a re-serializing printer is free to normalize away – and a typography tool that reformats markup nobody asked it to touch has broken something worse than what it fixed.

The spec argues for this design rather than just asserting it, by showing what breaks under the two more obvious alternatives. Take the ordinary sentence He said <em>'hi'</em> loudly. Run the pipeline on each of the three text fragments (He said , 'hi', loudly) completely independently – call it Model A – and 'hi' gets processed in total isolation: its quote pair looks like a top-level quotation with nothing around it, so in en-US it comes out as the primary double-quote pair, “hi”, when the correct answer is the secondary pair ‘hi’, nested one level inside the outer sentence’s own (absent, in this example, but generally present) quotation. That’s not an edge case – it’s wrong on a construction that shows up in ordinary prose constantly. The other obvious fix, Model B, is to concatenate every fragment into one long string, run the pipeline once, then redistribute the edits back by offset. That fixes the nesting problem and introduces two new ones, both from adjacency that only exists because of the concatenation: "a"<code>x</code>"b" concatenates to "a""b", which a quote-pairing rule reads as two identical adjacent marks that are not actually adjacent in the real document, producing “a""b” – visibly wrong. And He said <code>x</code> "hi" concatenates to He said "hi" with two spaces that render as one in the real document but read as a genuine double-space run to the concatenated string, which a spacing rule then collapses – deleting a character from a text node that only ever had one space in it.

polytypo’s actual answer, Model C, concatenates the fragments with an explicit boundary marker between each pair – a negative integer that can’t collide with real text, present in the array the rules see but invisible to every character-class test except the one or two that specifically need to know a boundary is there. He said ⟦'hi'⟧ loudly (writing ⟦ for the marker) gives the pipeline exactly what Model A denied it – that 'hi' sits inside a larger sentence – without giving it what Model B wrongly granted, the belief that the marker’s neighbors on either side are really touching. The nesting resolves correctly, the two spurious adjacency bugs disappear, and every rule other than quotes gets the right behaviour across a boundary ‘for free’ – the spec’s own heading for that section – because the marker is simply absent from every character class a rule checks against, except the one exemption written to make quote-pairing work across an element boundary at all.

The same document has to answer a second, less obvious question: is it safe for the pipeline to emit a new character right at one of these span boundaries? The obvious case – inserting a new character exactly at the edge – is easy to forbid. The harder case, and the one that actually shipped as a real defect during development, is a replacement that grows: dashes converts -- to a spaced en dash in some locales, which is one code point becoming three. When that dash sits at the very edge of an inline element, the two new spaces land outside where the original dash was, and a<em>--</em>b becomes a<em> – </em>b – an element that now begins and ends with a space it never contained. In Markdown the same growth de-flanks emphasis delimiters, turning *–* into * – * and silently demoting what used to be italics into literal asterisks on the next pass. And in the worst case, the emitted space migrates outside every span entirely and the document grows a little on every single save – which, for a pre-commit hook that runs on every commit, is not a cosmetic bug, it’s an unbounded diff. The fix is a single mechanical rule, checkable from nothing but the edit’s position and length: any edit that would place new characters at the very edge of its span is discarded rather than applied. It costs some conversions – the tight em dash in en-US still converts at an edge because shrinking never grows a span, but a spaced form generally doesn’t – and it’s the price of a design that never has to know what markup or Markdown syntax actually looks like to stay safe next to it.

I’d hit a version of this problem from the other direction before, building this site’s own JSX-attribute normalization pass (below). An earlier, separate .tsx scanner – built for a one-time cleanup pass across the site’s TSX files, never persisted to this repo – used ts-morph to parse a JSX expression, edit the string literals it found, and call the library’s own printer to write the result back out, and, as I remember it, the printer reformatted more of the file than the literals that had actually changed. The JSX-attribute pass below never calls the printer at all: it parses with ts-morph, collects the exact start/end offsets of the nodes to change, and splices the replacement text into the original source string by hand – and its own comment says why, crediting that earlier scanner: ‘same byte-splice-not-printer technique as the (unpersisted).tsx scanner, to avoid ts-morph’s save()/printer corrupting formatting.’ That’s the identical decision polytypo’s spec makes at the architecture level, arrived at independently and the hard way: locate what needs to change, and touch only that – never hand the whole document to something that reconstructs it from a parsed representation, because a reconstruction is free to ‘improve’ things nobody asked it to touch.

Idempotency Is a Property of the Composition, Not of Any One Rule

pipeline-idempotency.md states the invariant transform(transform(x)) == transform(x) as a promise about the whole public function, not about any individual rule – and then proves, correctly, that per-rule idempotency doesn’t get you there. Write the pipeline as eight always-on rules chained in order (the document’s own table predates ranges later split out of dashes, and – tellingly, for a project this insistent on catching its own drift – hasn’t been renumbered since; the composition argument still holds for the eight rules it covers). If each rule is individually a fixed point on its own output, the obvious hope is that the whole chain is too. It isn’t, automatically, and the spec names the missing piece the composition obligation: for any two rules where one runs before the other in the pipeline, if the output already satisfies the earlier rule, the later rule running on it must not undo that. In the spec’s own words: ‘a rule must never create work for an earlier-ordered rule.’

The tempting shortcut – just keep re-running the whole pipeline in a loop until nothing changes anymore – is explicitly forbidden, and for reasons worth stating rather than assuming: it would make the invariant true by construction and hide exactly the class of bug this document exists to find; it downgrades a rule being provably a fixed point in isolation – a testable, portable claim – to ‘the loop converges eventually,’ which is not; and any difference in how many iterations two of the five runtimes need to converge on the same input becomes a silent conformance divergence rather than a bug anyone would notice. The rules have to compose correctly on a single pass, or not at all.

Two real defects were found this way, by an exhaustive sweep over every short string built from a small alphabet across all locales – not by anyone’s intuition about what might go wrong. The first is the bug from the top of this article: a German- or Finnish-style spaced dash landing directly before a full stop gets its trailing space stripped on the second pass, because spaces (order 10) doesn’t know that space came from dashes (order 30) a moment ago rather than from the author. The fix lives in dashes, not spaces: a token is no longer given a spaced form at all when the space it would emit lands somewhere spaces would delete it – dashes declines the promotion up front rather than emitting something a later pass would have to unpick. The second is subtler and French-specific: a bare hyphen inside straight double quotes, "-", first becomes guillemets with no-break spacing inserted around them, «⍽-⍽», and on the next pass that same hyphen – now flanked by space-like characters on both sides – looks structurally identical to a spaced parenthetical dash and gets promoted into one, «⍽–⍽», which was never the intent.

That second one is the more interesting failure, because the actual fix had to answer a question about layering, not just about characters: which rule gets to change? nbsp is what inserted the no-break spaces in the first place, so the naive fix is to teach nbsp not to insert next to a hyphen. That’s wrong, and it’s wrong for a mechanical reason baked into order.json: dashes is declared with access only to its own dash locale data, and literally cannot read the quotes locale data to recognize that the character next to it is a guillemet rather than ordinary punctuation. Teaching nbsp to encode dashes’ own admissibility rules would mean duplicating a subtle guard in a second place with no shared source of truth for it. The fix that actually shipped lives in dashes alone: neither of the two no-break-space characters counts as dash-spacing, on either side, ever – not ‘sometimes,’ not ‘unless it came from a quote rule.’ That’s the sufficient condition the spec converges on for proving one rule can’t create work for another: rather than enumerate every situation where a later rule’s output might confuse an earlier one, make the set of characters the later rule can possibly emit structurally inert to the earlier rule – a rule the earlier one declines to touch adjacent to, unconditionally. Two earlier attempts at this exact fix tried narrower formulations (a no-break space counts as dash-spacing; a no-break space counts as dash-spacing only on the left) and each one closed one bug while opening a different one. Only the unconditional version holds, and the spec notes it’s also the shortest of the three to state – a real sign, in its own words, of having found the right invariant rather than another special case.

Locale Correctness Is a Data-Governance Problem

One design choice is easy to miss because it looks like the boring part: how the caller’s locale string turns into a specific locale file. polytypo resolves en to en-US and de to de-DE through a small, explicit alias table – never through a platform’s own locale-negotiation library, because ICU, Go’s language matcher, PHP’s Locale::lookup and Python’s babel.negotiate_locale all implement different, sometimes probabilistic, fallback policies, and five runtimes agreeing on locale resolution matters at least as much as five runtimes agreeing on what a curly quote looks like. An unrecognized locale – a typo, a region the spec doesn’t cover, nb hopefully falling back to the similar sv – throws. There is no default, no ‘closest match,’ and no silent substitution of one country’s typographic convention for another’s. Getting a locale wrong is a mistake the caller finds immediately, in an error, rather than a mistake a reader finds later in oddly punctuated copy.

Every locale file’s typographic rules are also, deliberately, data rather than code – and each rule in that data carries a mandatory citation. spec/locales/de-DE.json’s entries cite Duden’s own rules for German quotation marks and parenthetical dashes by section number, and one entry is candid about a gap in its own sourcing: the file needed a rule for how German handles a dash meaning ‘to’ (1939–1945), couldn’t find an equally precise Duden citation for it, and cites the Swiss federal chancellery’s style guide instead – naming that substitution explicitly rather than quietly presenting it as an equivalent German-specific source. A dedicated check fails the build if a locale carries data for a rule with no matching citation. The Greek locale file makes a related point from a messier angle: el.json declares its parenthetical-dash setting as ‘none’ – not because Greek convention has no opinion, but because its own citation says Greek convention prescribes an em dash pair with ordinary spaces outside and none on the inner edges, a shape the schema’s dash enum currently has no value to express. So in a generated sentence like Περπατήσαμε -- σχεδόν 3 χλμ --, the plain -- is left exactly as typed – not because the locale is correctly declining a feature it doesn’t need, but because the spec is honest about a gap between what Greek typography wants and what the schema can currently say, and tracks that gap as data instead of quietly picking a value. The same citation separately flags an unresolved ‘SOURCE CONFLICT, recorded and deliberately not settled’: one guide prescribes that em dash for the parenthetical case, a second, corroborating source elsewhere in the same file appears to use a narrower dash in the same role, and nothing has ranked the two – a second, unrelated honesty artifact in a file that’s already declining to paper over one gap.

Five Runtimes, One Conformance Suite

All of this pipeline machinery exists once, in the spec, and gets exercised by five independent published packages – JavaScript, Python, Go, Ruby and PHP – each graded against the same versioned fixture suite rather than five separately maintained opinions about what typography normalization – turning straight quotes, three dots and a bare hyphen into the locale-correct forms each language actually uses – should mean. The rule the spec states for what ‘conformant’ is allowed to mean is unusually blunt: an implementation is polytypo if and only if it passes the conformance suite for the spec version it claims. As of this week, all five runtimes pass – with two honesty notes kept rather than smoothed over: ‘last verified’ means an operator observed that runtime’s own CI green, not an automated cross-runtime signal, and these repos are days old at the time of writing, zero stars each. The interesting fact is that it shipped conformant across five languages from day one, not that it has any track record yet.

The clearest example of the suite forcing an honest answer rather than a fudged one is PHP’s markdown mode, which doesn’t exist in either dialect – commonmark or mdx – while every other runtime implements at least commonmark. The reason traces straight back to the span-and-offset architecture above: league/commonmark, the standard PHP library, reports only line numbers for block-level nodes and gives no position data at all for inline text – and polytypo’s markdown mode needs exact character offsets to know which raw substrings are prose. No maintained alternative library was found with that property, so the conformance document records PHP’s gap honestly rather than shipping a mode that would silently misfire. PHP is still conformant – for a scope that’s narrower than the other four runtimes, stated in the same breath as the checkmark, not hidden behind it.

Getting It Onto This Site

I didn’t just read the spec – I put it to work. This site runs polytypo/markdown (dialect: "mdx", since this is a Next.js/MDX blog) and polytypo/text on every commit on my machine – a lint-staged hook normalizes staged blog and project MDX, JSON data fields, and review quotes before they land. The hook script and the lint-staged config are written and staged in git, not committed yet, so this is a local, working setup rather than something live for anyone who clones the repo. One small, honest decision up front: the site uses locale en-GB, not en-US, specifically because en-GB’s spaced en dash matches the spacing convention this site’s prose already used, where en-US’s tight em dash would have collapsed it into something different. Locale choice here wasn’t about where the site’s readers are – it’s about which locale’s output matched an existing house style.

One real problem showed up building that hook, and it isn’t a polytypo bug – it’s a boundary the spec draws on purpose, which I had to work around rather than report. JSX component attributes are opaque to markdown mode, by design. polytypo/markdown skips code spans and fenced code blocks – that’s the whole point of a mode that isn’t supposed to touch code – and it applies the identical logic to a JSX attribute like the text prop on <Service text="..."> inside an .mdx file: an attribute string is source code from the parser’s point of view, not phrasing content. Correct behaviour, and it meant a chunk of this site’s actual reader-facing prose, sitting inside component props rather than plain Markdown, was invisible to the markdown pass entirely. Finding that gap wasn’t guesswork: a full-site render audit (below) flagged un-normalized text sitting inside rendered JSX props. Fixing it took a second, dedicated pass over the MDX source – for plain string attributes, a regex scan against a curated set of prose-bearing prop names; for attributes holding a JS expression, like an array of FAQ items, the offset-and-splice technique described above, ts-morph used only to find the string literals, never to write anything back through its own printer.

apply-typography.mjs calls the markdown transform once over an entire file’s contents – never on a fragment carved out of it – so there’s no fragment boundary for it to trip over the way the spec’s Model A and B failures describe. That’s a property of how this specific script is wired, not something the spec guarantees automatically for every integration: anyone building a DOM-fragment-level integration instead of a whole-document one should flatten a whole element’s inner HTML into one string before transforming it, not transform each rendered text node in isolation – the moment you split a document into isolated fragments, you’re back to Model A or Model B’s failure modes, and the spec’s answer to both is the same: don’t split it that way in the first place.

The Checker, and Its Own Honest Scope

Once the hook was writing normalized content, the actual question became: does every page on the live site actually render normalized text – not just the MDX source, but the JSON-driven copy, the hardcoded strings in components, everything a browser shows a real visitor? Grepping source files for straight quotes doesn’t answer that. A JSX prop, a template literal, a YAML scalar in single quotes instead of double – all invisible to a naive source grep, for the same reason the markdown pass needed its own dedicated code path for attributes.

So I wrote a crawler instead: it walks the live sitemap, fetches every public route, parses the actual rendered HTML with cheerio, pulls out every text node a visitor would see (skipping script, style, pre, code and a few excluded internal paths), and checks transform(text) === text for each one. If a node isn’t already a fixed point of the transform, it’s a violation. That crawl is what actually found the JSX-attribute gap above during development. What I haven’t done is re-run it for this article: it needs a live dev server up, and I’m not going to report a number I didn’t watch come out. So the honest status is narrower than ‘it came back clean this week’ – the checker exists, it already found one real class of gap once, it fails loudly on any violation or on a suspiciously small crawl, and I don’t have a current result to quote.

I want to be precise about what this checker actually verifies, independent of any one run. It confirms each rendered text node is idempotent on its own – it does not reconstruct cross-node adjacency the way the spec’s span model does for a single document. A dash split across a tag boundary in two sibling nodes would be exactly as invisible to this checker as it would be to a fragment-based integration, for the identical structural reason described above. That’s precisely why the real normalization work happens earlier and at document granularity – the git hook runs on whole MDX files and JSON strings, never on text split out of rendered DOM – and this checker’s job is narrower: confirm the rendered result is stable, not verify the transformation logic itself. Scoped honestly, that’s still a useful, mechanically run check that a source-level grep can’t give you, and it runs the same discipline the spec’s own conformance suite runs: check the actual output, not the process that’s supposed to have produced it.

Where I Was Wrong This Week

One more thing worth saying plainly, because it’s the same honesty this article keeps asking of the spec. While chasing down the fragment-boundary issue, I went looking for a previously noted internal error – a mode extractor reportedly producing ‘overlapping spans’ – to use as another concrete example here. I couldn’t reproduce it against the actual source that supposedly triggered it. Rather than describe a bug I can’t currently demonstrate, I’m leaving it out and saying so: not every lead I chased this week held up on a second look, and that’s a better outcome than writing it into an article about honest engineering and hoping nobody checked.

None of this is about typography being hard in the abstract – regex-and-a-prayer ‘smart quotes’ scripts have existed forever, and mostly work well enough that nobody questions them until a client’s German copy reads with English quote marks. The actual engineering decision here was narrower and more specific: run nine text-rewriting rules over arbitrary input, in a fixed order that’s part of the public contract, without ever building a document tree to mutate and reprint, and prove the whole nine-rule chain is a fixed point rather than assuming it because each rule looked fine on its own. The answer, this week, was a composition obligation that caught two real bugs unit tests missed, a span model that locates text instead of reparsing it, and a checker that knows exactly what it doesn’t verify. If you’re building something that has to rewrite text correctly at scale – or just needs its own build pipeline treated with that level of rigor – get in touch; that’s the kind of foundation work I do before feature work starts.

Further reading:

• polytypo.dev – the live playground, running the real published npm package in the browser
• polytypo – Locale-Correct Typography Engine and Spec – the full project case study
• The Typography Details Your Visitors Notice Without Knowing Why – the non-technical side of the same project: what a visitor actually senses, without being able to name it
• A Typed MDX Content Pipeline with Velite – this site’s other build-time content pipeline, same ‘fail at build time, not in production’ instinct
• Idempotency Keys: Building Retries That Don’t Double-Charge – a different domain, the same underlying question: what does it take to actually prove a function is a fixed point, not just claim one