UTF-8 utilities

Reversing a string

reverse reverses decoded Unicode code points. Each invalid UTF-8 byte becomes the replacement character U+FFFD, including when the input contains only one byte.

Decoding a rune

get_rune(s, index) returns the rune that starts at the byte offset index of s. decode_rune_at(s, index) returns that rune and the number of bytes that encode it, like Go's utf8.DecodeRuneInString(s[index:]):

A U+FFFD that is really in s has a size of 3, and a NUL byte decodes to rune(0) with a size of 1, so the size tells an out of range read from every character of the string.

import encoding.utf8

s := 'a★'
mut i := 0
for {
    r, size := utf8.decode_rune_at(s, i)
    if size == 0 {
        break
    }
    println('${r} is ${size} byte(s) long')
    i += size
}

Unicode categories

The letter, number and punctuation predicates use Unicode 15.0.0 general categories: is_letter recognizes category L, is_number recognizes N, and is_punct and is_rune_punct recognize all P categories.

is_punct takes a UTF-8 string and a byte offset; is_rune_punct takes a rune. Both recognize punctuation from every script, including ASCII hyphen-minus (-), Arabic comma and ideographic full stop. Their membership matches is_global_punct and is_rune_global_punct.

The category tables match the Unicode 15.0.0 character database and Go's Unicode 15.0.0 tables.

fn decode_rune_at(s string, index int) (rune, int)

decode_rune_at decodes the UTF-8 sequence that starts at the byte offset index of s. It returns the rune and the number of bytes that encode it, like Go's utf8.DecodeRuneInString(s[index:]): a valid sequence returns its rune and a size from 1 to 4; an invalid or truncated sequence returns U+FFFD and a size of 1; an index outside of s, which is any index when s is empty, returns U+FFFD and a size of 0. A U+FFFD that is really in s has a size of 3, so the size tells the three cases apart.

fn get_rune(s string, index int) rune

get_rune convert a UTF-8 unicode codepoint in string[index] into a UTF-32 encoded rune. It returns U+FFFD, the Unicode replacement character, for an invalid sequence and for an index outside of s, which is any index when s is empty. A NUL byte in s returns rune(0). Use decode_rune_at to also get the size, which tells these cases apart.

fn is_control(r rune) bool

is_control return true if the rune is control code

fn is_global_punct(s string, index int) bool

is_global_punct return true if the string[index] byte of is the start of a global unicode punctuation

fn is_letter(r rune) bool

is_letter returns true if the rune is unicode letter or in unicode category L

fn is_number(r rune) bool

is_number returns true if the rune is unicode number or in unicode category N

fn is_punct(s string, index int) bool

is_punct returns true if the rune starting at the byte offset index belongs to a Unicode punctuation category. It recognizes punctuation from every script, with the same Unicode 15.0.0 membership as is_global_punct.

fn is_rune_global_punct(r rune) bool

is_rune_global_punct return true if the input unicode is a global unicode punctuation

fn is_rune_punct(r rune) bool

is_rune_punct return true if the input unicode is a unicode punctuation

fn is_space(r rune) bool

is_space returns true if the rune is character in unicode category Z with property white space or the following character set:

`\t`, `\n`, `\v`, `\f`, `\r`, ` `, 0x85 (NEL), 0xA0 (NBSP)

fn len(s string) int

len return the length as number of unicode chars from a string

fn raw_index(s string, index int) string

raw_index - get the raw unicode character from the UTF-8 string by the given index value as UTF-8 string. example: utf8.raw_index('我是V Lang', 1) => '是'

fn reverse(s string) string

reverse - returns a reversed string. A byte that is not part of a valid UTF-8 sequence counts as one rune, as in len, and is returned as U+FFFD, whatever the length of the string. example: utf8.reverse('你好世界hello world') => 'dlrow olleh界世好你'.

fn to_lower(s string) string

to_lower return an lowercase string from a string

fn to_upper(s string) string

to_upper return an uppercase string from a string

fn validate(data &u8, len int) bool

validate reports if data consists of valid UTF-8 runes

fn validate_str(str string) bool

validate_str reports if str consists of valid UTF-8 runes