Chapter 1: A brief history of world knowledge

The need for communication and passing on knowledge is one of the most essential human traits. We have evolved from sign languages and cave paintings to using leaves and stones for the written word. We sped through the whole paper era (in a short couple of centuries ) and landed in the information age where everything went digital.

In the beginning, computers had a small vocabulary. We were so busy getting the word out about the internet, we had little time to consider if the word was understood. As the internet grew from a mostly English-speaking network into a place for the whole world, text had to evolve with it. Every step made online communication richer, but it also quietly broke another assumption programmers had made about what a “character” really is.

🧐 Are emojis modern day cave paintings?? 😯

ASCII (The Simple Days)

A = 65
B = 66
...
Z = 90

128 characters total. All English letters, numbers, basic punctuation. 7 bits. Life was simple.

🌎 Then the internet happened.

Unicode (The Complicated Now)

A = U+0041 (Latin Capital Letter A)
Á = U+00C1 (Latin Capital Letter A with Acute)
А = U+0410 (Cyrillic Capital Letter A)
Α = U+0391 (Greek Capital Letter Alpha)
𝐀 = U+1D400 (Mathematical Bold Capital A)

140,000+ characters. Multiple ways to represent the same thing. Combining characters. Emoji. Emoji with skin tones. Emoji combined with other emoji.

Unicode went from “character set” to “every written symbol in human history plus emoji”.

Chapter 2: The Many Ways Characters Lie About Their Size

Traditional knowledge tells us that the size of a string should be equal to the number of character you see in it. And traditionally all characters used to take a fixed 1 byte memory. So if i wanted to store a Name of 100 characters, I would allocate 100 length to my database column.
But unfortunately modern unicode breaks these assumption in many ways, most commonly due to the way the size of a character is calculated.

Size Lie #1: UTF-8 Variable Length

Single characters in print can often need more space to store.

'A'.length;        // 1 UTF-16 code unit (1 byte in UTF-8)
'€'.length;        // 1 UTF-16 code unit (3 bytes in UTF-8)
'💩'.length;       // 2 UTF-16 code units (4 bytes in UTF-8)

If you had a 100-byte limit thinking it should store 100 characters, what fits is actually:

  • 100 ASCII characters ✅
  • 33 Euro signs (3 bytes each)
  • 25 emoji (4 bytes each)

Size Lie #2: Combined Characters

Sometime single characters may not be single characters at all. Depending on how you process them they can be of different lengths,

'é'.length;  // Could be 1 or 2!

// Option 1: Single character (NFC - composed)
'é' = U+00E9 (1 character)

// Option 2: Base + combining accent (NFD - decomposed)  
'e' + '́' = U+0065 + U+0301 (2 characters)

The characters look identical. But they’re different in memory. This also leads to an interesting comparison problem.
‘é’ === ‘é’ can sometimes be false! Which means your database indexes them as different values and your lookups fail mysteriously.

Usually most languages have an equivalent of normalise() function for strings. In that case ‘é’.normalize() === ‘é’.normalize() gives a True result.

Size Lie #3: Emoji Combiners

Emojis revolutionised the way humans communicate. However emojis follow an interesting composition pattern.

'👨‍👩‍👧‍👦'.length;  // 11

// Actually:
'👨' + ZWJ + '👩' + ZWJ + '👧' + ZWJ + '👦'
// Man + Zero Width Joiner + Woman + ZWJ + Girl + ZWJ + Boy
// = Family emoji (displays as 1)

Your JavaScript counter reports “This is 11 UTF-16 code units,” while the user sees just 1 emoji. Your validation thinks “That’s fine,” but a byte-limited field can reject it: this sequence is 25 UTF-8 bytes.

Chapter 3: Real-World Horror Stories

All this information is not just theoritical. It happens in production system day-in day-out. Below are some ways in which they can happen trivially.

Story 1: The Name That Broke Authentication

A user named José creates an account. Your system normalizes and stores it as “José” using NFC normalization. When José logs in the next day and types their name, their keyboard or input method produces the NFD normalized version: “José”. Your system tries to look up “José” in the database but finds nothing, because it’s searching for the NFD form against the stored NFC form. The user sees “Who’s José? I don’t know that user.”

NFD - Normalization Form Decomposed. Breaks characters into base character + combining marks. Example: é = e + ´ (separate bytes) NFC - Normalization Form Composed. Combines base character + marks into single precomposed character. Example: é = é (single byte)

Fix: Normalize all input before comparison

function normalizeText(str) {
  return str.normalize('NFC');
}

username === savedUsername  // ❌ Can fail
normalizeText(username) === normalizeText(savedUsername)  // ✅ Works

Story 2: The Email That Traveled Through Time

A User in Japan sends email with kanji characters.
Email server 1 (UTF-8): Passes it along fine
Email server 2 (ISO-8859-1): Can’t handle kanji, converts to ?
Email server 3 (UTF-8): Sees ? and assumes that’s the data
Recipient: Gets email full of ? characters

This is mojibake. Text corrupted by charset mismatches.

The fix: Always specify charset in Content-Type headers

Content-Type: text/html; charset=UTF-8

This is also one of the reasons why old smartphones could not render emojis in SMS and showed a string of ? or black boxed question marks.

Story 3: The Tweet That Was Too Long

On Twitter (I refuse to call it X! ), tweets are limited to 280 characters.

User tweets: “🏴󠁧󠁢󠁥󠁮󠁧󠁿🏴󠁧󠁢󠁳󠁣󠁴󠁿🏴󠁧󠁢󠁷󠁬󠁳󠁿 (3 flag emoji).
Intuition: “This is 3 characters”
Reality: This is 42 code points (14 per flag)

Result: The twitter character counter disagrees with you and shows you are using much more than 3 characters.

Platform reality: X uses its own weighted counting rules, not JavaScript’s built-in string length.

You can read about there actual computation on their developer site.

Grapheme clusters are the user-perceived characters. A single emoji flag (like 🏴󠁧󠁢󠁥󠁮󠁧󠁿) is one grapheme cluster even though it’s 14 code points. A code point is Unicode’s assigned number for a unit of text, such as a letter, emoji component, or invisible joiner.

Story 4: The Database Migration That Deleted Data

When migrating database, your primary focus is that tables should not get dropped, there should be no downtime, the columns should get mapped correctly. However managing migrations of character set has its own set of challengers.

Scenario: Migrating from MySQL with latin1 to utf8mb4

-- Old table
CREATE TABLE users (
  name VARCHAR(100) CHARACTER SET latin1
);

-- New table
CREATE TABLE users (
  name VARCHAR(100) CHARACTER SET utf8mb4
);

The trap:

  • latin1: VARCHAR(100) stores up to 100 characters, using up to 100 bytes for the value
  • utf8mb4: VARCHAR(100) still stores up to 100 characters, using up to 400 bytes for the value

But MySQL also has row and index-size limits, depending on the storage engine, row format, and indexes.

If you have 200 VARCHAR(100) columns:

  • latin1: 200 × 100 = 20,000 bytes ✅
  • utf8mb4: 200 × 400 = 80,000 bytes at the theoretical maximum

your migration could hit an engine-specific limit, or your data could get truncated if the migration is not planned carefully. The fix would usually be to reduce VARCHAR sizes or split across tables.

Story 5: The Search That Found Nothing

Since we are talking databses, searching for an existing value is often the most popular use case. Lets take an example.
User searches for: “café”
Database has: “cafe”, “café”, “café” (composed), “café” (decomposed)

Your search:

SELECT * FROM places WHERE name = 'café';

Results: DB would usually return only exact matches and misses variations which in this case you obviously wanted.

The fix: Normalize before storing, normalize before searching

-- PostgreSQL example
SELECT * FROM places WHERE unaccent(name) = unaccent('café');

Story 6: The URL That Broke Routing

Unicode also turns up in places where you assume some default behaviour. One of the common ways sites create persistent readable links is to have text in the URL path. However browsers have been built to handle unicode by default when routing.

User creates profile: https://site.com/users/José.
Browser URL-encodes: https://site.com/users/Jos%C3%A9

Your router:

app.get('/users/:name', (req, res) => {
  const user = getUser(req.params.name);
  // req.params.name is "Jos%C3%A9" or "José" depending on framework
});

Some frameworks would decode automatically but it is always good to check. Anyway, a better way to create permalinks is not to have user input in the URL.

The fix: Always use IDs in URLs, not names

// ❌  Bad
/users/José

// ✅ Good
/users/123

Chapter 4: The Emoji Special Cases

We talk about compositional emojis. However the composition itself has many different variation.

Skin Tone Modifiers

Emojis have additive modifiers. You take a base emoji and add an inflextion on it. This is how you get a hand wave sign in your own skin tone.

'👋'.length;      // 2 (base emoji)
'👋🏻'.length;     // 4 (base emoji + skin tone modifier)

Your emoji picker now shows 5 skin tone options. But in your database each takes different space. So beware of these additions.

Flags

Flags are a fun variation of composition. For example, If you look at the US flag emoji.

'🇺🇸'.length;     // 4

// Actually two "Regional Indicator" characters:
// U+1F1FA (Regional Indicator U) + U+1F1F8 (Regional Indicator S)
// = US flag

Your database may not recognize this as one flag
Your rendering might show “U” “S” instead of 🇺🇸

Gender and Profession Variations

Similar variations arise for gender and profession emojis.

'👨'              // Man
'👨‍⚕️'             // Man + ZWJ + Medical symbol = Male doctor
'👩‍⚕️'             // Woman + ZWJ + Medical symbol = Female doctor
'🧑‍⚕️'             // Person + ZWJ + Medical symbol = Doctor (gender neutral)

Each variation has a different byte count

Chapter 5: The Database Encoding Trap

I have spent years working with databases without caring about caracter sets. In most case, databases come with good defaults, or their onboarding guides give examples of most defensive settings. However it is good to know how they affect storage in the context of what we discussed thus far.

MySQL Character Sets (A History of Mistakes)

latin1 (default for ancient MySQL)

  • 1 byte per character
  • Only Western European languages
  • Can’t store emoji

utf8 (MySQL’s version, not real UTF-8)

  • Max 3 bytes per character
  • Can store most characters
  • Cannot store emoji (emoji need 4 bytes)

utf8mb4 (actual UTF-8)

  • Max 4 bytes per character
  • Can store emoji
  • This is what you want

The trap:

-- Your table uses default charset (latin1)
CREATE TABLE users (name VARCHAR(100));

-- User tries to save emoji
INSERT INTO users (name) VALUES ('Alice 😀');

-- MySQL:
-- ❌ Error: Incorrect string value
-- OR
-- ⚠️ Silently truncates emoji

The fix:

-- Specify utf8mb4 explicitly
CREATE TABLE users (
  name VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
);

-- Or set database default
ALTER DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

PostgreSQL (Mostly Gets It Right)

PostgreSQL uses UTF-8 by default. Emoji just work. But CHAR(n) is still usually the wrong type for user input:

CREATE TABLE users (name CHAR(10));
INSERT INTO users (name) VALUES ('Hello 👋');

-- This fits, but CHAR pads values with spaces.

The fix: Use VARCHAR(n) when you need a character limit, or TEXT when you do not. Avoid CHAR(n) for variable-length user input.

Chapter 6: The Application Layer Traps

We talked a lot about database level storage till now. However at some point you also have to show the stored text to the users, or atlest process it somehow in your application code. There are many traps there as well.

Trap 1: String Length Validation

// User enters: "Hello 👨‍👩‍👧‍👦"
const input = "Hello 👨‍👩‍👧‍👦";

// Wrong
if (input.length > 20) {
  throw new Error('Too long');
}
// input.length = 17 (11 for emoji + 6 for "Hello ")

// Right
const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
const graphemes = Array.from(segmenter.segment(input));
if (graphemes.length > 20) {
  throw new Error('Too long');
}
// graphemes.length = 7 (1 for emoji + 6 for "Hello ")

The choice of using graphemes or actual char count depends on how you want to impose limits on user input.

Trap 2: Substring Operations

If we are having trouble deterministically finding how long a string is, it falls to reason that splitting it would also be hard.

const text = "Hello 👨‍👩‍👧‍👦 World";

// Wrong
text.substring(0, 10);  // "Hello 👨‍�" (cuts emoji in half!)

The solution usually is to use grapheme-aware libraries or avoid substring with emoji 😜

Trap 3: Regex Matching

Regex match is string processing final boss. If you thought splitting string was hard, matching it against a pattern can be a whole new box of wonders. s

// Match any single character
const regex = /^.$/;

'A'.match(regex);           // ✅ Match
'💩'.match(regex);          // ❌ No match (2 UTF-16 units)
'👨‍👩‍👧‍👦'.match(regex);      // ❌ No match (11 UTF-16 units)

// Better
const regex = /^.$/u;  // Unicode flag
'💩'.match(regex);          // ✅ Match

// But still won't match combined emoji
'👨‍👩‍👧‍👦'.match(regex);      // ❌ Still no match

Chapter 7: The Fixes

Despite all the above, the world continues to communicate in emojis, accented characters, regional languages; So there must be some way all of it continues to work. Here are some tips.

Fix 1: Use UTF-8 Everywhere

Database: utf8mb4.
API responses: Content-Type: application/json; charset=UTF-8.
HTML: <meta charset="UTF-8">
Files: Save as UTF-8.
Environment: Set LANG=en_US.UTF-8.

Fix 2: Normalize User Input

function normalizeInput(str) {
  return str
    .normalize('NFC') // Normalize to composed form
    .trim(); // Remove whitespace
}

Fix 3: Count Graphemes, Not Code Points

function countGraphemes(str) {
  const segmenter = new Intl.Segmenter('en', { granularity: 'grapheme' });
  return Array.from(segmenter.segment(str)).length;
}

Fix 4: Validate Before Storing

function validateText(str, maxGraphemes, maxBytes) {
  const graphemes = countGraphemes(str);
  const bytes = new TextEncoder().encode(str).length;
  
  if (graphemes > maxGraphemes) {
    throw new Error(`Too many characters (max ${maxGraphemes})`);
  }
  
  if (bytes > maxBytes) {
    throw new Error('Text too long in bytes');
  }
}

Fix 5: Use TEXT Columns, Not VARCHAR

-- Instead of guessing VARCHAR size
CREATE TABLE posts (
  content VARCHAR(10000)  -- Might not be enough for emoji-heavy text
);

-- Use TEXT when there is no application-level character limit
CREATE TABLE posts (
  content TEXT
);

In PostgreSQL, TEXT and VARCHAR have the same storage characteristics; VARCHAR(n) adds a character limit.

Fix 6: Test With Emoji

// Add these to your test suite
const testStrings = [
  'Simple ASCII',
  'Café',                    // Accented characters
  'Hello 世界',              // CJK characters
  'مرحبا',                   // RTL text
  'Hello 👋',               // Basic emoji
  '👨‍👩‍👧‍👦',                    // Combined emoji
  '🏴󠁧󠁢󠁥󠁮󠁧󠁿',                        // Flag emoji
  '👋🏻👋🏿',                   // Emoji with skin tones
  'ℌ𝔢𝔩𝔩𝔬',                // Mathematical alphanumeric symbols
];

Key Takeaways

  1. Use UTF-8 (utf8mb4 in MySQL) everywhere - No exceptions
  2. Normalize text on input - NFC is usually the right choice
  3. Count graphemes, not code points - Use Intl.Segmenter
  4. Test with emoji - They will break your assumptions
  5. Never use CHAR for user input - Use VARCHAR or TEXT
  6. Validate both length and byte size - They’re different
  7. Normalize before comparing - ‘é’ has multiple representations
  8. Don’t use names in URLs - Use IDs instead

Conclusion

Unicode is humanity’s attempt to represent every written symbol in a computer. Emoji are humanity’s attempt to communicate without words. Together, they create a perfect storm of edge cases.

Remember: If your system can’t handle 💩, it’s not ready for production.