Enum Token 

pub enum Token<'a> {
    Text {
        text: &'a [u8],
        width: u16,
    },
    Escape(&'a [u8]),
    Control(u8),
}

A single token produced by tokenize.

Variants

Text

One grapheme cluster and its display width.

Always whole, well-formed UTF-8. The width may be zero - a combining mark, a variation selector, or a C1 character such as U+009C written as C2 9C - so “visible” here means “not a control byte”, not “occupies a cell”.

Fields

text: &'a [u8]

Grapheme bytes.

width: u16

Display width in terminal cells.

Escape(&'a [u8])

An ANSI escape sequence (passed through verbatim, no width).

Control(u8)

A single byte that stands on its own: a C0 control, DEL, a C1 byte that introduces nothing, or a byte that begins no well-formed UTF-8 character. All are zero width.

Trait Implementations

impl<'a> Clone for Token<'a>

fn clone(&self) -> Token<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Token<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> Eq for Token<'a>

impl<'a> PartialEq for Token<'a>

fn eq(&self, other: &Token<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> StructuralPartialEq for Token<'a>