logo svelte /markdown v2.0.0

Headless Parser

@humanspeak/svelte-markdown exports the streaming engine that powers streaming={true} as a standalone class: IncrementalParser. The same instance is used internally by the Svelte component, so the diffing behavior and token shape are identical — the export is for cases where the Svelte component itself is the wrong layer.

Reach for IncrementalParser when:

  • You’re rendering markdown outside Svelte — a custom DOM library, a CLI, a React island in the same app — and want the streaming-aware token diff without re-implementing it.
  • You need the parsed tokens on the server before the page reaches the client (e.g. precomputing token arrays for analytics, building a search index of LLM output, gating content via policy before HTML is generated).
  • You’re writing a higher-level abstraction over @humanspeak/svelte-markdown and want to reuse the diff index without owning the renderer.
  • You’re running benchmarks or property tests that need direct token access.

For typical Svelte apps, <SvelteMarkdown source={...} streaming={true} /> is what you want — it wraps IncrementalParser and handles rendering for you.

Basic Usage

import { IncrementalParser } from '@humanspeak/svelte-markdown'

const parser = new IncrementalParser({ gfm: true })

// First update — all tokens are "new"
const r1 = parser.update('# Hello')
// r1.tokens   → [{ type: 'heading', depth: 1, text: 'Hello', ... }]
// r1.divergeAt → 0

// Second update — heading unchanged, paragraph appended
const r2 = parser.update('# Hello\n\nWorld')
// r2.tokens   → [headingToken, paragraphToken]
// r2.divergeAt → 1   (token at index 0 is reference-equal to the previous one)
import { IncrementalParser } from '@humanspeak/svelte-markdown'

const parser = new IncrementalParser({ gfm: true })

// First update — all tokens are "new"
const r1 = parser.update('# Hello')
// r1.tokens   → [{ type: 'heading', depth: 1, text: 'Hello', ... }]
// r1.divergeAt → 0

// Second update — heading unchanged, paragraph appended
const r2 = parser.update('# Hello\n\nWorld')
// r2.tokens   → [headingToken, paragraphToken]
// r2.divergeAt → 1   (token at index 0 is reference-equal to the previous one)

The parser holds the previous parse internally and diffs against it on every update(), so consumers don’t need to track the prior state themselves.

API

new IncrementalParser(options)

Constructor. Takes SvelteMarkdownOptions — the same options accepted by the Svelte component’s options prop.

import { IncrementalParser, type SvelteMarkdownOptions } from '@humanspeak/svelte-markdown'

const options: SvelteMarkdownOptions = {
    gfm: true,       // GitHub-Flavored Markdown (tables, strikethrough, task lists)
    breaks: false,   // Convert single newlines into <br>
    pedantic: false
}

const parser = new IncrementalParser(options)
import { IncrementalParser, type SvelteMarkdownOptions } from '@humanspeak/svelte-markdown'

const options: SvelteMarkdownOptions = {
    gfm: true,       // GitHub-Flavored Markdown (tables, strikethrough, task lists)
    breaks: false,   // Convert single newlines into <br>
    pedantic: false
}

const parser = new IncrementalParser(options)

The full set of options matches Marked’s MarkedOptions, including extension hooks (walkTokens, tokenizer). When any custom tokenizer or extension is registered, the internal tail-window optimization is disabled (each update() becomes a full re-parse), since user-provided tokenizers can mutate tokens in ways the diff can’t detect.

parser.update(source, appendsTo?)

Parses the full source string and returns the diff against the previous parse.

const result = parser.update(currentSource)

// When you already know the new source extends the previous one:
const next = parser.update(currentSource, previousSource)
const result = parser.update(currentSource)

// When you already know the new source extends the previous one:
const next = parser.update(currentSource, previousSource)

Always pass the full accumulated source — not just the delta. Append-only streaming (typical LLM use case) is the optimized path, but the parser correctly handles arbitrary edits (insertions, replacements, full rewrites) too.

The optional appendsTo argument is a string you have already verified source starts with (typically your buffer before the chunk was appended). When it equals the previously parsed source, the parser skips its own full-length append check. Passing a string that source does not start with breaks parsing, so omit it when unsure.

IncrementalUpdateResult

interface IncrementalUpdateResult {
    /** The full new token array */
    tokens: Token[]
    /** Index of the first token that differs from the previous parse */
    divergeAt: number
    /** Source offset of the first differing token, when known without scanning the prefix */
    divergeOffset?: number
    /** Shorthand for `reuseMode === 'prefix'` */
    canReuse: boolean
    /** How token objects from the previous parse may be reused: 'prefix' | 'tree' | 'none' */
    reuseMode: 'prefix' | 'tree' | 'none'
    /** Leading roots that are the same objects, at the same indices, as in the previous result */
    reusedPrefixCount: number
    /** Whether only the appended tail was re-lexed (vs the whole source) */
    usedTailWindow: boolean
}
interface IncrementalUpdateResult {
    /** The full new token array */
    tokens: Token[]
    /** Index of the first token that differs from the previous parse */
    divergeAt: number
    /** Source offset of the first differing token, when known without scanning the prefix */
    divergeOffset?: number
    /** Shorthand for `reuseMode === 'prefix'` */
    canReuse: boolean
    /** How token objects from the previous parse may be reused: 'prefix' | 'tree' | 'none' */
    reuseMode: 'prefix' | 'tree' | 'none'
    /** Leading roots that are the same objects, at the same indices, as in the previous result */
    reusedPrefixCount: number
    /** Whether only the appended tail was re-lexed (vs the whole source) */
    usedTailWindow: boolean
}

divergeAt is the actionable field for the common case: every token at index < divergeAt is reference-equal to the corresponding token from the previous update() call, so a consumer can skip re-rendering them. On the first call (or after a structural change that invalidates the prefix), divergeAt is 0.

reuseMode tells you which reuse strategy is safe:

  • 'prefix' — the first divergeAt roots are unchanged; reuse them positionally.
  • 'tree' — the update was append-only but a reference definition ([label]: url) arrived, so inline children may have changed inside roots whose raw did not. Compare the previous and new arrays index by index with a semantic comparator (every render-affecting field, not just raw) and reuse whatever is equal; the Svelte component does this internally. divergeAt is 0 in this mode.
  • 'none' — the source was edited rather than appended; replace the token array.

reusedPrefixCount is a further shortcut for renderers that keep the previous array: the first reusedPrefixCount entries of tokens are the very same objects at the same indices, so they need no comparison at all.

Streaming Pattern

The common loop for LLM output:

import { IncrementalParser } from '@humanspeak/svelte-markdown'

const parser = new IncrementalParser({ gfm: true })
let source = ''

async function consumeStream(response: Response) {
    const reader = response.body!.getReader()
    const decoder = new TextDecoder()

    while (true) {
        const { done, value } = await reader.read()
        if (done) break

        source += decoder.decode(value, { stream: true })
        const { tokens, divergeAt } = parser.update(source)

        // Render only what changed. The first `divergeAt` tokens
        // are reference-equal to the previous frame, so a virtual
        // DOM or manual reconciler can short-circuit them.
        renderFrom(tokens, divergeAt)
    }
}
import { IncrementalParser } from '@humanspeak/svelte-markdown'

const parser = new IncrementalParser({ gfm: true })
let source = ''

async function consumeStream(response: Response) {
    const reader = response.body!.getReader()
    const decoder = new TextDecoder()

    while (true) {
        const { done, value } = await reader.read()
        if (done) break

        source += decoder.decode(value, { stream: true })
        const { tokens, divergeAt } = parser.update(source)

        // Render only what changed. The first `divergeAt` tokens
        // are reference-equal to the previous frame, so a virtual
        // DOM or manual reconciler can short-circuit them.
        renderFrom(tokens, divergeAt)
    }
}

Per-update parser cost is proportional to the open block at the end of the source, not to the document, because the tail-window optimization re-lexes only the suffix beyond the last closed block. In the repository’s paired benchmark (2026-09-28, headless Chromium, ~24 KB corpora streamed at 32 characters per frame) the parser’s share of a frame was 0.2—0.3 ms on prose and citation-heavy prose, and about 2.5 ms on a single open ~200-item list, where the whole open list is re-lexed each frame by design. The streamed token tree is checked against a fresh one-shot parse at every sampled frame; the check reported zero mismatches in every run, including while reference-definition URLs stream in character by character.

Reconciling With Sanitization

IncrementalParser returns tokens with sanitization not yet applied — href allow-listing, on* stripping, and other guards run inside the Svelte component layer. If you’re using the headless parser to render to a non-Svelte target, you’ll want to apply the same defenses before mounting the result.

The defaults are exported as plain functions so you can plug them in:

import {
    defaultSanitizeUrl,
    defaultSanitizeAttributes
} from '@humanspeak/svelte-markdown'

// Apply per token after parsing, before handing off to your renderer.
for (const token of tokens) {
    if (token.type === 'link' || token.type === 'image') {
        token.href = defaultSanitizeUrl(token.href, {
            type: token.type,
            tag: token.type === 'image' ? 'img' : 'a'
        })
    }
    // Walk html tokens, etc.
}
import {
    defaultSanitizeUrl,
    defaultSanitizeAttributes
} from '@humanspeak/svelte-markdown'

// Apply per token after parsing, before handing off to your renderer.
for (const token of tokens) {
    if (token.type === 'link' || token.type === 'image') {
        token.href = defaultSanitizeUrl(token.href, {
            type: token.type,
            tag: token.type === 'image' ? 'img' : 'a'
        })
    }
    // Walk html tokens, etc.
}

See Security for the full sanitizer contract, including the context-aware override pattern.

Caveats

  • Sanitization is your responsibility. As above — the headless parser hands you raw Marked tokens.
  • Tail-window optimization is disabled when extensions are registered. If you pass walkTokens, tokenizer, or block/inline extensions, every update() becomes a full re-parse. Per-update cost stays bounded, but the constant factor goes up.
  • Reference-syntax-aware reparse. When the source contains markdown reference links ([text][ref]) or reference definitions ([ref]: url), the parser falls back to a full re-parse, since later definitions can retroactively change earlier inline children without changing their raw strings.
  • Not thread-safe. One IncrementalParser instance per stream. Don’t share instances across concurrent inputs.

Related

  • LLM Streaming — the Svelte component wrapper around this parser, with writeChunk() / resetStream() ergonomics.
  • Token Caching — the LRU cache layer used by the parser. Exported as TokenCache and MemoryCache for direct use.
  • Security — sanitization contract you’ll want to layer on top when rendering to a non-Svelte target.
  • Types & Exports — full type reference for IncrementalUpdateResult, Token, SvelteMarkdownOptions.