net.urllib #

Description

net.urllib parses and assembles URLs and provides path and query escaping.

URL.str(), URL.escaped_path() and URL.request_uri() escape characters that are invalid in a URL path, including backslashes. Parsing accepts a raw backslash in the path; serializing that URL emits %5C.

import net.urllib

u := urllib.parse(r'http://example.com/\/x')!
assert u.path == r'/\/x'
assert u.str() == 'http://example.com/%5C/x'

Valid existing percent encodings retain their spelling, such as %5c.

Parsing preserves the distinction between an omitted password and an explicitly empty password. For http://user:@example.com/, URL.user.password_set is true and URL.str() retains user:@. The same applies to an empty username: http://:@example.com/ retains :@ when serialized. A percent-encoded colon in a username, such as user%3Aname, does not set a password unless a literal colon follows it.

A URL with a single trailing ? retains its empty query marker. Additional question marks inside a nonempty query remain part of raw_query, including a question mark at its end.

Joining hosts and ports

urllib.join_host_port(host, port) constructs a host:port address. IPv6 literals and scoped IPv6 literals are bracketed; existing brackets are preserved. The separator remains when the port is empty, so an empty host and port produce :. Host spelling and zone identifiers are unchanged.

import net.urllib

assert urllib.join_host_port('example.com', '443') == 'example.com:443'
assert urllib.join_host_port('::1', '8080') == '[::1]:8080'
assert urllib.join_host_port('fe80::1%eth0', '') == '[fe80::1%eth0]:'
assert urllib.join_host_port('', '80') == ':80'

The helper only joins strings: it does not validate the host or resolve a service name such as https. split_host_port can split the result when the port is numeric or empty.

fn ishex(c u8) bool

fn join_host_port(host string, port string) string

join_host_port joins host and port into a network address of the form host:port. The colon is retained when port is empty. IPv6 addresses are bracketed, and an already bracketed host is preserved. No validation or service-name resolution is performed.

fn new_values() Values

new_values returns a new Values struct for creating urlencoded query string parameters. it can also be to post form data with application/x-www-form-urlencoded. values.encode() will return the encoded data

fn parse(rawurl string) !URL

parse parses rawurl into a URL structure. IPv6 zone identifiers permit escaped percent signs (%25) and spaces (%20), including at the end of the zone. Invalid or prohibited escapes return an error.

The rawurl may be relative (a path, without a host) or absolute (starting with a scheme). Trying to parse a hostname and path without a scheme is invalid but may not necessarily return an error, due to parsing ambiguities. A literal colon in userinfo sets the password, even when it is empty. Percent-encoded colons do not act as userinfo separators.

fn parse_query(query string) !Values

Values maps a string key to a list of values. It is typically used for query parameters and form values. Unlike in the http.Header map, the keys in a Values map are case-sensitive. parseQuery parses the URL-encoded query string and returns a map listing the values specified for each key. parseQuery always returns a non-nil map containing all the valid query parameters found; err describes the first decoding error encountered, if any.

Query is expected to be a list of key=value settings separated by ampersands or semicolons. A setting without an equals sign is interpreted as a key set to an empty value.

fn parse_request_uri(rawurl string) !URL

parse_request_uri parses rawurl into a URL structure for an HTTP request. It accepts only absolute URIs or absolute paths and preserves leading // sequences as part of the path for request targets. The string rawurl is assumed not to have a #fragment suffix. (Web browsers strip #fragment before sending the URL to a web server.)

fn path_escape(s string) string

path_escape escapes the string so it can be safely placed inside a URL path segment, replacing special characters (including /) with %XX sequences as needed.

fn path_unescape(s string) !string

path_unescape does the inverse transformation of path_escape, converting each 3-byte encoded substring of the form '%AB' into the hex-decoded byte 0xAB. It returns an error if any % is not followed by two hexadecimal digits.

path_unescape is identical to query_unescape except that it does not unescape '+' to ' ' (space).

fn query_escape(s string) string

query_escape escapes the string so it can be safely placed inside a URL query.

fn query_unescape(s string) !string

query_unescape does the inverse transformation of query_escape, converting each 3-byte encoded substring of the form '%AB' into the hex-decoded byte 0xAB. It returns an error if any % is not followed by two hexadecimal digits.

fn split_host_port(hostport string) (string, string)

split_host_port separates host and port. If the port is not valid, it returns the entire input as host, and it doesn't check the validity of the host. Per RFC 3986, it requires ports to be numeric.

fn user(username string) Userinfo

user returns a Userinfo containing the provided username and no password set.

fn valid_userinfo(s string) bool

valid_userinfo reports whether s is a valid userinfo string per RFC 3986 Section 3.2.1:

userinfo    = *( unreserved / pct-encoded / sub-delims / ':' )
unreserved  = ALPHA / DIGIT / '-' / '.' / '_' / '~'
sub-delims  = '!' / '$' / '&' / ''' / '(' / ')'
            / '*' / '+' / ',' / ';' / '='

It doesn't validate pct-encoded. The caller does that via fn unescape.

struct QueryValue {
pub mut:
	key   string
	value string
}

struct URL {
pub mut:
	scheme      string
	opaque      string    // encoded opaque data
	user        ?Userinfo // username and password information
	host        string    // host or host:port
	path        string    // path (relative paths may omit leading slash)
	raw_path    string    // encoded path hint (see escaped_path method)
	force_query bool      // append a query ('?') even if raw_query is empty
	raw_query   string    // encoded query values, without '?'
	fragment    string    // fragment for references, without '#'
}

A URL represents a parsed URL (technically, a URI reference). The general form represented is: [scheme:][//[userinfo@]host][/]path[?query][#fragment] URLs that do not start with a slash after the scheme are interpreted as: scheme:opaque[?query][#fragment]

Note that the path field is stored in decoded form: /%47%6f%2f becomes /Go/. A consequence is that it is impossible to tell which slashes in the path were slashes in the raw URL and which were %2f. This distinction is rarely important, but when it is, the code should use raw_path, an optional field which only gets set if the default encoding is different from path.

URL's String method uses the escaped_path method to obtain the path. See the escaped_path method for more details.

fn (url &URL) debug() string

debug returns a string representation of ALL the fields of the given URL

fn (mut u URL) set_path(p string) !bool

set_path sets the path and raw_path fields of the URL based on the provided escaped path p. It maintains the invariant that raw_path is only specified when it differs from the default encoding of the path. For example:

• set_path('/foo/bar') will set path='/foo/bar' and raw_path=''
• set_path('/foo%2fbar') will set path='/foo/bar' and raw_path='/foo%2fbar' set_path will return an error only if the provided path contains an invalid escaping.

fn (u &URL) escaped_path() string

escaped_path returns the escaped form of u.path. In general there are multiple possible escaped forms of any path. escaped_path returns u.raw_path when it is a valid escaping of u.path. Otherwise escaped_path ignores u.raw_path and computes an escaped form on its own. The String and request_uri methods use escaped_path to construct their results. In general, code should call escaped_path instead of reading u.raw_path directly.

fn (u URL) str() string

str reassembles the URL into a valid URL string. The general form of the result is one of:

scheme:opaque?query#fragment scheme://userinfo@host/path?query#fragment

If u.opaque is non-empty, String uses the first form; otherwise it uses the second form. Any non-ASCII characters in host are escaped. To obtain the path, String uses u.escaped_path().

In the second form, the following rules apply:

• if u.scheme is empty, scheme: is omitted.
• if u.user is none, userinfo@ is omitted.
• if u.host is empty, host/ is omitted.
• if u.scheme and u.host are empty and u.user is nil, the entire scheme://userinfo@host/ is omitted.
• if u.host is non-empty and u.path begins with a /, the form host/path does not add its own /.
• if u.raw_query is empty, ?query is omitted.
• if u.fragment is empty, #fragment is omitted.

fn (u &URL) is_abs() bool

is_abs reports whether the URL is absolute. Absolute means that it has a non-empty scheme.

fn (u &URL) parse(ref string) !URL

parse parses a URL in the context of the receiver. The provided URL may be relative or absolute. parse returns nil, err on parse failure, otherwise its return value is the same as resolve_reference.

fn (u &URL) resolve_reference(ref &URL) !URL

resolve_reference resolves a URI reference to an absolute URI from an absolute base URI u, per RFC 3986 Section 5.2. The URI reference may be relative or absolute. resolve_reference always returns a new URL instance, even if the returned URL is identical to either the base or reference. If ref is an absolute URL, then resolve_reference ignores base and returns a copy of ref.

fn (u &URL) query() Values

query parses raw_query and returns the corresponding values. It silently discards malformed value pairs. To check errors use parseQuery.

fn (u &URL) request_uri() string

request_uri returns the encoded path?query or opaque?query string that would be used in an HTTP request for u.

fn (u &URL) hostname() string

hostname returns u.host, stripping any valid port number if present.

If the result is enclosed in square brackets, as literal IPv6 addresses are, the square brackets are removed from the result.

fn (u &URL) port() string

port returns the port part of u.host, without the leading colon. If u.host doesn't contain a port, port returns an empty string.

struct Userinfo {
pub:
	username     string
	password     string
	password_set bool
}

The Userinfo type is an immutable encapsulation of username and password details for a URL. An existing Userinfo value is guaranteed to have a username set (potentially empty, as allowed by RFC 2396), and optionally a password.

struct Values {
pub mut:
	data []QueryValue
	len  int
}

fn (v Values) encode() string

encode encodes the values into ``URL encoded'' form ('bar=baz&foo=quux'). The syntx of the query string is specified in the RFC173 https://datatracker.ietf.org/doc/html/rfc1738

HTTP grammar

httpurl = "http://" hostport [ "/" hpath [ "?" search ]] hpath = hsegment *[ "/" hsegment ] hsegment = *[ uchar | ";" | ":" | "@" | "&" | "=" ] search = *[ uchar | ";" | ":" | "@" | "&" | "=" ]

fn (v &Values) get(key string) ?string

get gets the first value associated with the given key. If there are no values associated with the key, get returns none.

fn (v &Values) get_all(key string) []string

get_all gets the all the values associated with the given key. If there are no values associated with the key, get returns a empty []string.

fn (mut v Values) set(key string, value string)

set sets the key to value. It replaces any existing values, or create a new bucket with the new key if it is missed.

fn (mut v Values) add(key string, value string)

add adds the value to key. It appends to any existing values associated with key.

fn (mut v Values) del(key string)

del deletes the values associated with key.

fn (v Values) values() []string

return the list of values in the query string

fn (v Values) to_map() map[string][]string

return a map <key []value> of the query string