abstract class Lexer (View source)

Base class for writing simple lexers, i.e. for creating small DSLs.

Properties

array $lookahead
array $token

Methods

setInput(string $input)

Sets the input data to be tokenized.

reset()

Resets the lexer.

resetPeek()

Resets the peek pointer to 0.

resetPosition(integer $position = 0)

Resets the lexer position on the input to the given position.

boolean
isNextToken(integer|string $token)

Checks whether a given token matches the current lookahead.

boolean
isNextTokenAny(array $tokens)

Checks whether any of the given tokens matches the current lookahead

array|null
moveNext()

Moves to the next token in the input string.

skipUntil(string $type)

Tells the lexer to skip input tokens until it sees a token with the given value.

boolean
isA(mixed $value, integer $token)

Checks if given value is identical to the given token

array
peek()

Moves the lookahead token forward.

array|null
glimpse()

Peeks at the next token, returns it and immediately resets the peek.

scan(string $input)

Scans the input string for tokens.

string
getLiteral(integer $token)

Gets the literal for a given token.

array
getCatchablePatterns()

Lexical catchable patterns.

array
getNonCatchablePatterns()

Lexical non-catchable patterns.

integer
getType(string $value)

Retrieve token type. Also processes the token value if necessary.

Details

setInput(string $input)

Sets the input data to be tokenized.

The Lexer is immediately reset and the new input tokenized. Any unprocessed tokens from any previous input are lost.

Parameters

string $input The input to be tokenized.

reset()

Resets the lexer.

resetPeek()

Resets the peek pointer to 0.

resetPosition(integer $position = 0)

Resets the lexer position on the input to the given position.

Parameters

integer $position Position to place the lexical scanner

boolean isNextToken(integer|string $token)

Checks whether a given token matches the current lookahead.

Parameters

integer|string $token

Return Value

boolean

boolean isNextTokenAny(array $tokens)

Checks whether any of the given tokens matches the current lookahead

Parameters

array $tokens

Return Value

boolean

array|null moveNext()

Moves to the next token in the input string.

A token is an associative array containing three items: - 'value' : the string value of the token in the input string - 'type' : the type of the token (identifier, numeric, string, input parameter, none) - 'position' : the position of the token in the input string

Return Value

array|null the next token; null if there is no more tokens left

skipUntil(string $type)

Tells the lexer to skip input tokens until it sees a token with the given value.

Parameters

string $type The token type to skip until.

boolean isA(mixed $value, integer $token)

Checks if given value is identical to the given token

Parameters

mixed $value
integer $token

Return Value

boolean

array peek()

Moves the lookahead token forward.

Return Value

array | null The next token or NULL if there are no more tokens ahead.

array|null glimpse()

Peeks at the next token, returns it and immediately resets the peek.

Return Value

array|null The next token or NULL if there are no more tokens ahead.

protected scan(string $input)

Scans the input string for tokens.

Parameters

string $input a query string

string getLiteral(integer $token)

Gets the literal for a given token.

Parameters

integer $token

Return Value

string

abstract protected array getCatchablePatterns()

Lexical catchable patterns.

Return Value

array

abstract protected array getNonCatchablePatterns()

Lexical non-catchable patterns.

Return Value

array

abstract protected integer getType(string $value)

Retrieve token type. Also processes the token value if necessary.

Parameters

string $value

Return Value

integer