Binaries, strings, and charlists

Source: Elixir guide, Binaries, strings, and charlists.

A string is a binary, a binary is a bitstring, and a charlist is a list. This chapter sorts out which is which.

Every string is a binary and every binary is a bitstring. Not every binary is a valid string.

Code points and UTF-8

Unicode gives every character a number, its code point. ?a is 97, ?ł is 322. "a" writes a character by its hex code point.

A code point is what is stored. An encoding is how. Elixir strings are UTF-8, a variable-width encoding that uses 1 to 4 bytes per code point.

"hełło": five characters, five code points, seven bytes. ł is two bytes.

"héllo" |> String.length()   #=> 5
"héllo" |> byte_size()       #=> 6

"hełło" <> <<0>>
#=> <<104, 101, 197, 130, 197, 130, 111, 0>>   (the exact bytes)

A grapheme is what a reader sees as one character. It can be several code points: the woman firefighter emoji is woman + zero-width joiner + fire engine. String.codepoints/1 returns three, String.graphemes/1 and String.length/1 see one.

Bitstrings and binaries

<<>> builds a bitstring, a contiguous run of bits. Each entry is 8 bits unless you say otherwise with ::n.

<<42>> == <<42::8>>                    #=> true
<<0::1, 0::1, 1::1, 1::1>> == <<3::4>> #=> true
<<1>> == <<257>>                       #=> true   (257 doesn't fit in 8 bits, it's truncated)

A binary is a bitstring whose bit count is divisible by 8.

is_bitstring(<<3::4>>)   #=> true
is_binary(<<3::4>>)      #=> false
is_binary(<<0, 255, 42>>) #=> true

You can pattern match on binaries. Each entry matches exactly one byte unless you add a modifier:

<<0, 1, x>> = <<0, 1, 2>>              # x = 2
<<0, 1, x>> = <<0, 1, 2, 3>>           #=> ** (MatchError)
<<0, 1, x::binary>> = <<0, 1, 2, 3>>   # x = <<2, 3>>
<<head::binary-size(2), rest::binary>> = <<0, 1, 2, 3>>

Strings are binaries

A string is a UTF-8 binary. <> is really binary concatenation, so <<0, 1>> <> <<2, 3>> works. Not every binary is a valid string: String.valid?(<<239, 191, 19>>) is false.

Matching a string works on bytes, not characters. Use the utf8 modifier to match a whole character:

<<x, rest::binary>> = "über"
x == ?ü            #=> false   (x is the first byte of ü)

<<x::utf8, rest::binary>> = "über"
x == ?ü            #=> true    rest is "ber"

Without utf8 the pattern takes one byte. With utf8 it takes as many as the character needs.

Charlists

A charlist is a list of integers that are valid code points. You mostly meet them when calling older Erlang libraries that don't take binaries.

~c"hello"            #=> ~c"hello"
[?h, ?e, ?l, ?l, ?o] #=> ~c"hello"
~c"hełło"            #=> [104, 101, 322, 322, 111]

IEx prints a list as ~c"…" only when every number is in the ASCII range, so [99, 97, 116] shows as ~c"cat". inspect(list, charlists: :as_list) forces the list form. to_string/1 and to_charlist/1 convert, and to_string/1 also takes atoms and numbers.

Strings use <>, charlists (being lists) use ++. Mixing them fails: ~c"this " <> ~c"fails" raises ArgumentError.