abstract class AbstractLexer (View source)

Base class for writing simple lexers, i.e. for creating small DSLs.

Properties

array $lookahead The next token in the input.
array $token The last matched/seen token.

Methods

void
setInput(string $input)

Sets the input data to be tokenized.

void
reset()

Resets the lexer.

void
resetPeek()

Resets the peek pointer to 0.

void
resetPosition(integer $position = 0)

Resets the lexer position on the input to the given position.

string
getInputUntilPosition(integer $position)

Retrieve the original lexer's input until a given position.

boolean
isNextToken(integer|string $token)

Checks whether a given token matches the current lookahead.

boolean
isNextTokenAny(array $tokens)

Checks whether any of the given tokens matches the current lookahead.

boolean
moveNext()

Moves to the next token in the input string.

void
skipUntil(string $type)

Tells the lexer to skip input tokens until it sees a token with the given value.

boolean
isA(mixed $value, integer $token)

Checks if given value is identical to the given token.

array|null
peek()

Moves the lookahead token forward.

array|null
glimpse()

Peeks at the next token, returns it and immediately resets the peek.

void
scan(string $input)

Scans the input string for tokens.

string
getLiteral(integer $token)

Gets the literal for a given token.

string
getModifiers()

Regex modifiers

array
getCatchablePatterns()

Lexical catchable patterns.

array
getNonCatchablePatterns()

Lexical non-catchable patterns.

integer
getType(string $value)

Retrieve token type. Also processes the token value if necessary.

Details

void setInput(string $input)

Sets the input data to be tokenized.

The Lexer is immediately reset and the new input tokenized. Any unprocessed tokens from any previous input are lost.

Parameters

string $input The input to be tokenized.

Return Value

void

void reset()

Resets the lexer.

Return Value

void

void resetPeek()

Resets the peek pointer to 0.

Return Value

void

void resetPosition(integer $position = 0)

Resets the lexer position on the input to the given position.

Parameters

integer $position Position to place the lexical scanner.

Return Value

void

string getInputUntilPosition(integer $position)

Retrieve the original lexer's input until a given position.

Parameters

integer $position

Return Value

string

boolean isNextToken(integer|string $token)

Checks whether a given token matches the current lookahead.

Parameters

integer|string $token

Return Value

boolean

boolean isNextTokenAny(array $tokens)

Checks whether any of the given tokens matches the current lookahead.

Parameters

array $tokens

Return Value

boolean

boolean moveNext()

Moves to the next token in the input string.

Return Value

boolean

void skipUntil(string $type)

Tells the lexer to skip input tokens until it sees a token with the given value.

Parameters

string $type The token type to skip until.

Return Value

void

boolean isA(mixed $value, integer $token)

Checks if given value is identical to the given token.

Parameters

mixed $value
integer $token

Return Value

boolean

array|null peek()

Moves the lookahead token forward.

Return Value

array|null The next token or NULL if there are no more tokens ahead.

array|null glimpse()

Peeks at the next token, returns it and immediately resets the peek.

Return Value

array|null The next token or NULL if there are no more tokens ahead.

protected void scan(string $input)

Scans the input string for tokens.

Parameters

string $input A query string.

Return Value

void

string getLiteral(integer $token)

Gets the literal for a given token.

Parameters

integer $token

Return Value

string

protected string getModifiers()

Regex modifiers

Return Value

string

abstract protected array getCatchablePatterns()

Lexical catchable patterns.

Return Value

array

abstract protected array getNonCatchablePatterns()

Lexical non-catchable patterns.

Return Value

array

abstract protected integer getType(string $value)

Retrieve token type. Also processes the token value if necessary.

Parameters

string $value

Return Value

integer