Skip to content

toml.scanner #

Constants #

const (
	digit_extras = [`_`, `.`, `x`, `o`, `b`, `e`, `E`]
	end_of_text  = -1
)

fn new_scanner #

fn new_scanner(config Config) ?&Scanner

new_scanner returns a new heap allocated Scanner instance.

fn new_simple #

fn new_simple(toml_input string) ?Scanner

returns a new stack allocated Scanner instance.

struct Config #

struct Config {
pub:
	input               input.Config
	tokenize_formatting bool = true
}

Config is used to configure a Scanner instance.
Only one of the fields text and file_path is allowed to be set at time of configuration.

struct Scanner #

struct Scanner {
pub:
	config Config
	text   string
mut:
	col        int
	line_nr    int = 1
	pos        int
	header_len int

	is_left_of_assign bool = true
}

Scanner contains the necessary fields for the state of the scan process.
the task the scanner does is also refered to as "lexing" or "tokenizing".
The Scanner methods are based on much of the work in vlib/strings/textscanner.

fn (Scanner) scan #

fn (mut s Scanner) scan() ?token.Token

scan returns the next token from the input.

fn (Scanner) free #

unsafe
fn (mut s Scanner) free()

free frees all allocated resources.

fn (Scanner) remaining #

fn (s &Scanner) remaining() int

remaining returns how many characters remain in the text input.

fn (Scanner) next #

fn (mut s Scanner) next() int

next returns the next character code from the input text.
next returns end_of_text if it can't reach the next character.

fn (Scanner) skip #

fn (mut s Scanner) skip()

skip skips one character ahead.

fn (Scanner) skip_n #

fn (mut s Scanner) skip_n(n int)

skip_n skips ahead n characters.
If the skip goes out of bounds from the length of Scanner.text, the scanner position will be sat to the last character possible.

fn (Scanner) at #

fn (s &Scanner) at() int

at returns the current character code from the input text.
at returns end_of_text if it can't get the current character.
unlike next(), at() does not change the state of the scanner.

fn (Scanner) peek #

fn (s &Scanner) peek(n int) int

peek returns the character code from the input text at position + n.
peek returns end_of_text if it can't peek n characters ahead.

fn (Scanner) reset #

fn (mut s Scanner) reset()

reset resets the internal state of the scanner.

fn (Scanner) excerpt #

fn (s Scanner) excerpt(pos int, margin int) string

excerpt returns a string excerpt of the input text centered at pos. The margin argument defines how many chacters on each side of pos is returned

fn (Scanner) state #

fn (s Scanner) state() State

state returns a read-only view of the scanner's internal state.

struct State #

struct State {
pub:
	col     int
	line_nr int = 1
	pos     int
}

State is a read-only copy of the scanner's internal state.
See also Scanner.state().