Markov Text Generator

Basic explanation

A Markov text generator learns which words tend to follow which short contexts in a corpus, then samples new text from those conditional distributions. The process is memoryless beyond a fixed window: the next token depends on the current $n$-gram, not on a planned sentence structure.

Empirical transitions from a shared context; generation samples one successor, then advances the window.

Model

Tokenise a source into words (or characters). For order $k$, each observed sequence of prior tokens updates a multiset of successors. The predictive distribution is the normalised count of each next token given that context. Generation starts from a seed context and repeatedly draws the next token by weighted random sampling, sliding the context forward by one symbol.

Context-length trade-off

Short contexts yield a denser transition table and more varied output, but they forget long-range grammar. Longer contexts preserve local phrasing from the author and can regurgitate corpus fragments, yet most $k$-grams appear only once, so the model becomes sparse and brittle. Exploring that bias-variance style trade-off was the main experimental interest of the project.

Implementation notes

The Python version builds an explicit map from contexts to successor weights. The browser demonstration reconstructs the same idea in TypeScript: paste a corpus, choose order, and generate locally - nothing is uploaded. That keeps the demo honest as an illustration of the stochastic mechanism rather than a hosted NLP service.