aix.lexer.core

`aix.lexer.core`

Source: packages/aix/src/aix/lexer/core.py

title: AIX Unicode lexer.

Classes

`TokenKind(Enum)`

title: Token kinds emitted by the AIX lexer.

`Token`

title: Store one token kind, value, and source location.
attributes:
  kind:
    type: TokenKind
  value:
    type: Any
  location:
    type: SourceLocation

Methods

`get_name(self) -> str`

title: Return one user-facing token name.
returns:
  type: str

`get_display_value(self) -> str`

title: Return a compact display value for token printing.
returns:
  type: str

`TokenList`

title: Simple token stream consumed by the parser.
attributes:
  tokens:
    type: list[Token]
  position:
    type: int
  cur_tok:
    type: Token

Methods

`get_token(self) -> Token`

title: Return the next token and advance.
returns:
  type: Token

`get_next_token(self) -> Token`

title: Advance parser cursor and return the current token.
returns:
  type: Token

`LexerError(Exception)`

title: AIX-specific lexer error with source location.
attributes:
  location:
    type: SourceLocation

`Lexer`

title: Tokenize AIX source text.
attributes:
  source:
    type: str | None

Methods

`clean(self) -> None`

title: Keep API compatibility with the Arx-derived lexer.

`lex(self) -> TokenList`

title: Tokenize configured source and return a token stream.
returns:
  type: TokenList

`tokenize(self) -> TokenList`

title: Alias for lex used by tests and callers.
returns:
  type: TokenList