Module text 

ANSI-aware byte-stream tokenizer for text utilities.

Category

The tokenizer classifies an input byte slice as visible grapheme clusters, complete ANSI escape/string sequences, or standalone control bytes. Width, stripping, truncation, and wrapping utilities all build on this stream.

7-bit and 8-bit controls

Both 7-bit forms (ESC [, ESC ], ESC P, ESC X, ESC ^, ESC _) and their 8-bit C1 bytes (0x9B, 0x9D, 0x90, 0x98, 0x9E, 0x9F) open a sequence.

A control string ends at ST, in either form - the byte 0x9C, or ESC \. BEL ends an OSC and nothing else: OSC Ps ; Pt BEL is an xterm convention rather than a rule about control strings, and ECMA-48 gives them all one terminator. A 0x07 inside a DCS, SOS, PM or APC is payload, which matters because DCS carries arbitrary data. A lone ESC also ends a string and is re-parsed as the start of the next sequence.

C1 bytes depend on decoder state

A byte in 0x80..=0x9F is a C1 control between characters and a UTF-8 continuation byte inside one. Which it is depends on where the decoder is and on nothing else, so:

• the byte 0x9C is 8-bit ST, and the byte 0x9D opens an OSC;
• C2 9C is the character U+009C and is text, as is C2 9D;
• the 0x9C inside E2 9C 85 (“✅”) is neither, and is never examined.

That last case is why every code point in U+2700..U+273F survives inside an OSC title. A &str cannot hold a raw C1 byte at all, which is what the 7-bit forms are for.

Malformed input

tokenize takes bytes, so its input need not be valid UTF-8. A byte that begins no well-formed character is emitted as Token::Control to keep forward progress, and contributes no width. Tokens always concatenate back to the input exactly.

Mode interaction

This module does not interpret terminal modes or sequence semantics. Escape bytes are passed through as zero-width tokens so callers can preserve or drop them according to their own policy.

Re-exports

pub use crate::text::WidthMode;

Structs

Tokenizer

Iterator returned by tokenize.

Enums

Token

A single token produced by tokenize.

Functions

string_width

Return the display width of bytes ignoring ANSI escapes.

tokenize

Tokenize an input byte slice into ANSI-aware tokens.