Encoding

• Maps characters (text) to byte sequences and vice versa. Used for text fields in serialization.

Naming Confusion

• When someone says "custom package encoding" , they usually mean:

  • A framing protocol  (how message start/end is delimited).

  • A custom serialization/deserialization  strategy.

  • A binary or textual format for transmitting structures over the network.

• Using "encoding" for package framing strategies is technically valid but potentially ambiguous.

• In networking, it’s better to use more specific terms.

• The word "encoding" itself isn’t wrong but should be interpreted in the technical context.

• In Odin, JSON and CBOR are considered "encoding" .

Text

Each solves a different constraint:

ASCII → simplicity (pre-Unicode world)
UTF-8 → compatibility + compactness
UTF-16 → compromise for legacy 16-bit systems
UTF-32 → simplicity of indexing

if it weren't for grapheme, array of unicode would be a viable choice?
Yes—if grapheme clusters did not exist, then “array of Unicode code points” ([]u32) would be a much cleaner and more viable model. In fact, it would be close to an idealized design.
but: every thing is 4 bytes; which is very bad.

Unicode

• Unicode vs Encoding :

  • Unicode defines characters and assigns them numbers.

  • Encodings define how those numbers are stored as bytes.

  • Computers store bytes, not code points.

  • An encoding converts: Code Point <-> Bytes.

  • The major Unicode encodings are:

    • ASCII

    • UTF-8

    • UTF-16

    • UTF-32

• A code point is just an integer.

• It uses values from U+0000 (0) -> U+10FFFF (1_114_111) , which fits in 21 bits.
  | Character | Code Point | Decimal |
  |-----------|------------|---------|
  | A        | U+0041     | 65      |
  | é        | U+00E9     | 233     |
  | 中       | U+4E2D     | 20013   |
  | 😀       | U+1F600    | 128512  |

ASCII (American Standard Code for Information Interchange)

• ASCII is simpler because it's both a character set and an encoding.

• Uses exactly 1 byte (8 bits) per character.

• Only defines 0x00 (0) -> 0x7F (127)
  | Character | Code Point | Hex Byte          |
  |-----------|------------|-------------------|
  | A        | U+0041     | 41                |
  | é        | U+00E9     | not representable |
  | 中       | U+4E2D     | not representable |
  | 😀       | U+1F600    | not representable |

• The problem :

  • Too few representations.

UTF-8

• Variable-width encoding.

  • 1 byte for ASCII

  • 2 bytes for some European characters

  • 3 bytes for most world scripts

  • 4 bytes for emoji and other supplementary characters
    | Character | Code Point | UTF-8 Bytes |
    |-----------|------------|-------------|
    | A        | U+0041     | 41          |
    | é        | U+00E9     | C3 A9       |
    | 中       | U+4E2D     | E4 B8 AD    |
    | 😀       | U+1F600    | F0 9F 98 80 |

• The problem :

  • For chars: UTF-8 only answers "How do I store text as bytes?", but it does not answer "What is a character?".

    • If your language says type char = u8 , then what is '😀'? It cannot fit.

  • For strings: they are no longer indexable in O(1) by character

UTF-16

• Stores text in 16-bit units.

• Size :

  • BMP characters (Basic Multilingual Plane, U+0000 to U+FFFF) use 2 bytes

  • Characters outside BMP (e.g., emojis, historical scripts) use 4 bytes (surrogate pairs)

  • More efficient for languages with many BMP characters (e.g., many Asian languages)
    | Character | Code Point | UTF-16 Bytes |
    |-----------|------------|--------------|
    | A        | U+0041     | 0041         |
    | é        | U+00E9     | 00E9         |
    | 中       | U+4E2D     | 4E2D         |
    | 😀       | U+1F600    | D83D DE00    |

• The emoji requires two 16-bit units; Those paired values are called a surrogate pair.

UTF-32

• Stores every code point as a 32-bit integer.

• Very simple.

• Very memory-heavy.
  | Character | Code Point | UTF-32 Bytes |
  |-----------|------------|--------------|
  | A        | U+0041     | 00000041     |
  | é        | U+00E9     | 000000E9     |
  | 中       | U+4E2D     | 00004E2D     |
  | 😀       | U+1F600    | 0001F600     |

Example

s := "hello 😃 world こんにちは"

• Total of bytes: 32 .

  • hello : 5

  •  (space): 1

  • 😃 : 4

  •  (space): 1

  • world : 5

  •  (space): 1

  • こんにちは : 15 (5 × 3)

• Total of codepoints: 19 .

  • hello : 5

  •  (space): 1

  • 😃  → 1

  •  (space): 1

  • world  → 5

  •  (space): 1

  • こんにちは  → 5

• Overview:
  | Character | UTF-8 bytes (hex) | Decimal            |
  | --------- | ----------------- | ------------------ |
  | h        | 68               | 104                |
  | e        | 65               | 101                |
  | l        | 6C               | 108                |
  | l        | 6C               | 108                |
  | o        | 6F               | 111                |
  | space     | 20               | 32                 |
  | 😃       | F0 9F 98 83      | 240, 159, 152, 131 |
  | space     | 20               | 32                 |
  | w        | 77               | 119                |
  | o        | 6F               | 111                |
  | r        | 72               | 114                |
  | l        | 6C               | 108                |
  | d        | 64               | 100                |
  | space     | 20               | 32                 |
  | こ        | E3 81 93         | 227, 129, 147      |
  | ん        | E3 82 93         | 227, 130, 147      |
  | に        | E3 81 AB         | 227, 129, 171      |
  | ち        | E3 81 A1         | 227, 129, 161      |
  | は        | E3 81 AF         | 227, 129, 175      |

Base64

• Is a way to represent arbitrary binary data using only printable ASCII characters.

• It is not encryption or compression—just an encoding so binary data can be stored safely in text formats.

• It is called Base64 because the encoding uses a numeral system with 64 distinct symbols to represent data.

  • Each Base64 character encodes 6 bits.

  • So you need exactly 64 symbols ( 2^6 = 64 ) to represent every possible 6-bit value.

• Base64 exists as many systems historically handled text only.

• Raw binary can contain:

  • null bytes (0x00)

  • control characters

  • non-printable bytes

• Converts binary → safe text using only:

  A–Z a–z 0–9 + /
  

• Padding uses =  but it is not part of the base.

Size

• Base64 increases size by about 33%.

• 4 output bytes per 3 input bytes.

encoded_size ≈ ceil(input_size / 3) * 4

• "Why on earth would you use an encoding that increases the size of the thing?"

  • Because Base64 solves transport and compatibility problems, not size efficiency. It is used when binary must safely travel through systems that are text-only or text-fragile.

  • Raw binary can break many pipelines due to:

    • null bytes (0x00)

    • control characters

    • encoding assumptions (UTF-8/UTF-16)

    • line-ending conversions

    • legacy text parsers

  • Historically (and still today), many formats and tools expect text, not arbitrary bytes.

  • Base64 guarantees the data contains only safe printable ASCII.

Core idea

• Base64 works in 6-bit chunks.

• Binary bytes are 8 bits each

• Base64 symbols encode 6 bits each

• So it repacks data

3 bytes (24 bits) → 4 Base64 characters

• as

3 × 8 = 24 bits
4 × 6 = 24 bits

• Example :

  • Input:

    "Man"
    

  • Write bytes:

    M = 01001101
    a = 01100001
    n = 01101110
    
    010011010110000101101110
    

  • Split into 6-bit groups:

    010011 010110 000101 101110
    

  • In decimals:

    19 22 5 46
    

  • Map to Base64 alphabet:

    0–25  → A–Z
    26–51 → a–z
    52–61 → 0–9
    62    → +
    63    → /
    

  • Output:

    19 → T
    22 → W
    5  → F
    46 → u
    

    • Concatenated:

    TWFu
    

Padding rules

• If input length is not divisible by 3, Base64 pads with = .

• Example :

  • Input:

    "Ma"
    

  • Write binary:

    01001101 01100001
    

  • Split into 6-bit groups:

    010011 010110 000100 000000
    

  • In decimals -> Map to Base64 alphabet.

  • Output:

    TWE=