Overview

• What is Unicode?
• Why Unicode?
• Terminology
• Character Sets and Encodings
• UTF-8
• UTF-16
• UTF-32
• Unicode in C/C++
• Mojibake
• Miscellaneous
• Q & A

What is Unicode?

• Unicode is an international standard, published by the Unicode Consortium and coordinated with the ISO.
• It is a single character set that encompasses all major living scripts.
• It includes a character set, several encodings, a database of character properties, and numerous associated documents coverings such topics as bidirectional text, collation, normalization, and more.
• As of version 4.1 encodes almost 100,000 characters.

Unicode provides a unique number for every character,
no matter what the platform,
no matter what the program,
no matter what the language.

Why Unicode?

• Unicode is a single character set that supports all major living scripts
• Unicode supports multilingual applications
• Unicode is widely supported in modern operating systems and software
• Unicode is an international standard
• Unicode is the foundation for other standards

Terminology

Characters

Abstract units of a writing system

• Appearance may vary due to font differences, e.g. lowercase 'a' and 'g' in Latin script.
• Appearance may vary due to context, e.g. Arabic '‌ه‌', '‍ه‌', '‌ه‍' '‍ه‍'
• Examples: 'A', 'Å', 'ij', 'א', '飽'

Glyph

The graphical representation of a character

Character Set

A mapping between abstract characters and abstract numbers

Encoding

A concrete representation of the numbers in a character set

Script

A collection of written symbols used together

Code Point

A single character in a character set

Code Unit

A portion of an encoded character

Character Sets and Encodings

• Single byte character sets such as US-ASCII, ISO-8859-1, and Windows-1252 have a trivial encoding.
• Some character sets can have multiple encodings, e.g. JIS may be encoded in Shift-JIS, EUC, or ISO-2022
• Some encodings may be used with multiple character sets, eg. ISO-2022 and EUC.
• Unicode has three standard encodings: UTF-8, UTF-16, and UTF-32.
• "Charset" usually refers to a combination of a character set and an encoding.

UTF-8

• Variable length encoding
• Compatible with byte-oriented processing
• US-ASCII compatible
• Often used for data interchange
• Usually larger size relative to legacy character set encodings
• All bytes are unabiguously single byte, leading byte, or trailing byte

Unicode Range	UTF-8 Encoded Bytes
U+0000-U+007F	0xxxxxxx
U+0080-U+07FF	110xxxxx 10xxxxxx
U+0800-U+FFFF	1110xxxx 10xxxxxx 10xxxxxx
U+10000-U+10FFFF	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

UTF-16

• Variable length encoding
• Representation of 16-bit code units is based machine endianness
• Can be labelled as BE, LE, or use a BOM
• All characters in the BMP are one code unit long
• Commonly used for internal representations
• Same storage as DBCS, double storage of SBCS

Unicode Range	Scalar Value	UTF-16 Enocded Data
U+0000-U+D7FF	xxxxxxxxxxxxxxxx	xxxxxxxxxxxxxxxx
U+D800-U+DFFF	N/A	N/A
U+E000-U+FFFF	xxxxxxxxxxxxxxxx	xxxxxxxxxxxxxxxx
U+10000-U+10FFFF	000uuuuu xxxxxxxx xxxxxxxx	110110wwwwxxxxxx 110111xxxxxxxxxx

where wwww = uuuu - 1

UTF-32

• Fixed length encoding
• Unicode only uses 21 bits, so always wastes space
• Often used in APIs to hold a single Unicode code point in a single code unit

Normalization

• Characters can have multiple representations, e.g. Ä could be a single precomposed character or it could be A +  ̈
• Some characters are duplicated for compatibility, e.g. U+00C5 Latin Capital Letter A With Ring Above Å and U+212B Angstrom Sign Å
• Unicode has rules for normalizing to standard representations
  • Normalization Form C: precompose
  • Normalization Form D: decompose
  • Normalization Form KC: collapse compatibility characters, precompose
  • Normalization Form KD: collapse compatibility characters, decompose
• NFC is the preferred normalization form on the Internet.

Script Complexities

Unicode supports scripts which may violate your assumptions. For example:

• Very large number of characters (Han, Hangul)
• Bidirectional text (Hebrew, Arabic)
• Contextual rendering (Arabic, Indic scripts)

Unicode in C/C++

• UTF-16 in wchar_t*
  • Pros:
  • Standard type
  • Cons:
  • size of wchar_t is implmementation dependant
  • semantics of wchar_t is implementation dependant
• UTF-8 in char*
  • Pros:
  • Standard type
  • Cons:
  • semantics of char* is ambiguous
• Custom Type
  • Pros:
  • Well defined semantic
  • Cons:
  • Not standardized (yet)
  • Many to choose from depending on library: OLECHAR, PRUnichar, UChar16, FcChar16, UniChar, etc.

Mojibake

• Mokjibake is a Japanese term for garbled text
• Happens when text in one encoding is interpreted as if it were in a different encoding

Q & A

Any questions?