Why Learn Character Encoding? Every email you send, website you visit, and text message you write relies on character encoding to convert letters into binary. Understanding ASCII and Unicode is essential for text processing, internationalization, and avoiding encoding bugs that corrupt data.
8 min read | Beginner Level
What You'll Learn
- Understand ASCII encoding and its 128-character limitation
- Learn how Unicode supports all world languages
- Compare different encoding systems (ASCII, UTF-8, UTF-16)
- Apply encoding knowledge to real-world text processing
What is ASCII Code?
ASCII (American Standard Code for Information Interchange) is a character encoding standard that represents text characters as 7-bit numeric codes in digital systems. Each character has a unique code from 0 to 127, allowing for 128 possible characters.
Example: The letter "A" is represented in ASCII as decimal 65 (binary 1000001), "B" is 66, and so on.
ASCII Code Examples
| Character | ASCII Code (Decimal) | ASCII Code (Binary) | Category |
|---|---|---|---|
| A | 65 | 1000001 | Uppercase letter |
| a | 97 | 1100001 | Lowercase letter |
| 0 | 48 | 0110000 | Digit |
| ! | 33 | 0100001 | Punctuation |
| % | 37 | 0100101 | Symbol |
| Space | 32 | 0100000 | Whitespace |
ASCII uses 7 bits, but is often stored in 8-bit bytes with the most significant bit set to 0. This leaves room for extended ASCII variants that use all 8 bits for additional characters.
ASCII Character Ranges
ASCII codes are organized into specific ranges:
- 0-31: Control characters (non-printable: NULL, TAB, CR, LF, etc.)
- 32-47: Special characters and punctuation (space, !, ", #, $, %, etc.)
- 48-57: Digits 0-9
- 58-64: More punctuation (: ; < = > ? @)
- 65-90: Uppercase letters A-Z
- 91-96: Brackets and symbols ([ \ ] ^ _ `)
- 97-122: Lowercase letters a-z
- 123-127: Additional symbols ({ | } ~ DEL)
Converting between uppercase and lowercase in ASCII is simple: just flip bit 5. 'A' (65 = 01000001) becomes 'a' (97 = 01100001). The difference is exactly 32.
ASCII in Hardware ASCII is widely used in embedded systems, serial communication protocols (UART, RS-232), LCD displays, and keyboard interfaces. Many microcontrollers have built-in ASCII conversion functions for displaying numbers as text on screens.
Limitations of ASCII
ASCII has significant limitations:
- Only 128 characters: Cannot represent accented letters (é, ñ, ü)
- English-centric: No support for non-Latin scripts (Arabic, Chinese, Japanese)
- No emoji or symbols: Cannot represent modern communication needs
- Multiple incompatible extensions: Different "extended ASCII" variants (ISO-8859-1, Windows-1252) caused compatibility issues
What is Unicode?
Unicode is a universal character encoding standard designed to represent every character from every writing system in the world. It solves ASCII's limitations by providing a unique code point for over 1 million potential characters.
Unicode Structure
Unicode uses code points written as U+ followed by hexadecimal digits:
| Character | Unicode Code Point | Description |
|---|---|---|
| A | U+0041 | Latin capital letter A |
| α | U+03B1 | Greek small letter alpha |
| ★ | U+2605 | Black star |
| 👍 | U+1F44D | Thumbs up emoji |
| ا | U+0627 | Arabic letter alef |
| 中 | U+4E2D | CJK unified ideograph (Chinese) |
Unicode Encoding Formats
Unicode code points can be encoded in different formats:
| Encoding | Bytes per Character | Range | Use Case |
|---|---|---|---|
| UTF-8 | 1-4 bytes (variable) | All Unicode | Web, files, most common |
| UTF-16 | 2 or 4 bytes | All Unicode | Windows, Java, JavaScript |
| UTF-32 | 4 bytes (fixed) | All Unicode | Internal processing (rare) |
UTF-8: The Universal Standard
UTF-8 is the most widely used Unicode encoding because it's backward-compatible with ASCII and space-efficient:
- ASCII characters (U+0000 to U+007F): Encoded in 1 byte (same as ASCII)
- Latin extended, Greek, Cyrillic, etc.: 2 bytes
- Most common Chinese/Japanese/Korean: 3 bytes
- Emoji and rare characters: 4 bytes
Example: The string "Hello 世界" in UTF-8:
H e l l o Space 世 界 48 65 6C 6C 6F 20 E4B896 E7958C 1byte 1byte 1byte 1byte 1byte 1byte 3bytes 3bytes
UTF-8's variable-length encoding means ASCII text (like English) uses the same space as before, while other languages use more bytes as needed. This makes it ideal for web content where most text is ASCII.
Unicode in Modern Systems Over 95% of websites use UTF-8 encoding. All modern operating systems (Windows, macOS, Linux), programming languages (Python 3, JavaScript, Go), and databases (MySQL, PostgreSQL) support Unicode natively. This ensures text displays correctly across different devices and languages worldwide.
Quick Check Q: Why can't ASCII represent the character "é" (e with acute accent)?
Show Answer
A: ASCII only has 128 code points (0-127), which are all assigned to basic English characters, digits, and control codes. Accented characters like "é" are not included in the standard ASCII table.
Solution: Use Unicode (UTF-8), where "é" is U+00E9, or extended ASCII variants like ISO-8859-1 (though Unicode is preferred for universal compatibility).
Common Beginner Mistakes
Mistake #1: Assuming all characters are 1 byte
Wrong: Using strlen() in C to count characters in UTF-8 text
Right: strlen() counts bytes, not characters. "世界" is 6 bytes but 2 characters in UTF-8
Why? Variable-length encoding means one character can be multiple bytes.
Mistake #2: Mixing encodings without conversion
Wrong: Reading UTF-8 file as ASCII/Latin-1 → garbled characters (é becomes é)
Right: Always specify encoding when reading/writing text files
Why? Different encodings interpret the same bytes differently.
Mistake #3: Forgetting the difference between code point and encoding
Wrong: Thinking U+1F44D (👍) is stored as those 6 characters
Right: U+1F44D is the code point; it's encoded as 4 bytes (F0 9F 91 8D) in UTF-8
Why? Code points are abstract identifiers; encodings are the actual byte representation.
ASCII vs Unicode Comparison
| Feature | ASCII | Unicode (UTF-8) |
|---|---|---|
| Character Limit | 128 characters | Over 140,000 (1.1 million possible) |
| Languages Supported | English only | All world languages |
| Bytes per Character | 1 byte (7 bits used) | 1-4 bytes (variable) |
| Backward Compatibility | N/A | ASCII is subset of UTF-8 |
| Emoji Support | No | Yes |
| Common Use | Legacy systems, embedded | Web, modern applications |
Summary
In this tutorial, you learned:
- ASCII uses 7-bit codes (0-127) to represent 128 basic English characters and symbols
- ASCII is limited to English and cannot represent accented letters or non-Latin scripts
- Unicode provides unique code points for over 140,000 characters from all writing systems
- UTF-8 is the most common Unicode encoding, using 1-4 bytes per character
- UTF-8 is backward-compatible with ASCII (ASCII text is valid UTF-8)
- Modern systems use Unicode/UTF-8 to support international text correctly
