Interface Tokeniser
access this type via: parsing.Tokeniser (provides, requires or uses)
This interface is used to tokenise a text document, using a given set of tokens, and supporting both line and block comments.
Functions
Tokeniser(
storeString tokens[])
void setLineComment(char lineComment[])
void setBlockComment(char start[], char end[])
Tokeniser(
storeString tokens[])
Initialise a tokeniser instance with the given set of tokens.
tokens The tokens that will be used by the tokeniser to separate text.
void setLineComment(char lineComment[])
Set the string that indicates the remainder of a line is a comment.
lineComment A string that indicates the remainder of a line is a comment.
void setBlockComment(char start[], char end[])
Set the starting and ending string that indicate the enclosed text is a comment.
start A string that indicates the start of a block comment.
end A string that indicates the end of a block comment.
Run the tokeniser, returning an array of all tokens.
content The text to split into tokens.
returns: An array of tokens.