2. Characters and Encoding

Reading asks that you bring your whole life experience and your ability to decode the written word and your creative imagination to the page and be a co-author with the writer, because the story is just squiggles on the page unless you have a reader.

Katherine Paterson

Characters. Letters. Symbols. Items used on a printed page and on a multitude of displays we use today. When used in an organized sequence, they are interpreted to give meaning or, in other words, create words. Languages use different characters with accent marks and pronunciation marks to accentuate or provide meaning. We’ll talk about what’s involved in creating characters, things like diacritics and surrogate pairs and ligatures, and storing those characters (encoding and code points).

Chapter topics include the following:

What’s behind the scenes with characters

How characters are stored and accessed by the OS

How the OS determines what character to use based on its language setting

How glyphs allow us to have different renditions or renderings of the same character

What causes “garbage” characters or empty box characters to display

We’ll hit the essentials about characters, strings, encoding, Unicode, and glyphs, and wrap up with fonts.

The main use of accents is to change the accented character’s sound value. English examples include naïve and Noël. The accented characters (diereses in this case) show that these characters (vowels) are pronounced separately from the preceding vowel.

Acute and grave accents can indicate that a final vowel is to be pronounced, such as the French works résumé or été.

Accents can perform other functionality with different alphabetic systems. The Arabic harakat and the Hebrew niqqud systems are used for indicated vowel and tone sounds that are not conveyed through the basic alphabet. The Arabic sukūn and the Indic virama both designate the absence of a vowel. Special characters exist to mark for abbreviations or acronyms—as in the Cyrillic titlo and the Hebrew gershayim. The Greek language includes accents to indicate that letters of the alphabet are being used as numerals. In the Chinese Hanyu Pinyin system, accents are used to mark syllable tones in which the marked vowels occur.

Some Chinese characters have been adopted as part of the writing systems of other East Asian languages, such as Japanese and Korean. International software support for the Chinese, Japanese, and Korean languages is often shortcut as CJK. Table 2.1 shows examples of Asian characters.

Characters for the Japanese language are usually a mixture of Chinese characters, or kanji, plus two syllabic scripts. At times the English (U.S.) alphabet is used as well. Having a working knowledge of 2,000 kanji characters is sufficient to read and comprehend most Japanese text.

Korean characters come mainly from an alphabetic script, Hangeul. Some hanja Chinese characters are used, but to a much lesser extent than with Japanese. When reading older Korean texts, an understanding of about 2,000 hanja characters is essential.

Table 2.2 contains different types of characters that are not necessarily found in any language’s alphabet but are interesting nonetheless. We’ll go over these types of characters and how to use and access them in the section, “Unicode and Encoding.” Punctuation characters have the capability to accentuate meaning, context, and understanding of text, but they do need to be associated with characters and words to accomplish that task. They cannot add meaning or understanding by themselves.

In general computer science terms, a string is traditionally a sequence of characters. This sequence of characters can be stored as a variable. Strings are generally treated as data types and are often implemented as an array that stores the characters using a manner of character encoding. The term encoding here refers to converting a character to its internal code point representation. Encoding is covered in detail in an upcoming section of this chapter.

In Objective-C, the string class is NSString and is the basic tool for representing text within your application. The NSString class provides powerful and flexible methods for manipulating its contents, as well as searching. And to add to its coolness, this class has native Unicode support.

But where did this complexity originate? If we’re talking about characters—which are typically stored in one or two bytes—and numeric codes assigned to these characters, then why is there not a one-to-one correspondence? Let’s all sit back, relax, and enjoy a small history lesson on encoding.

After computers were purchased outside of the U.S., all manner of different OEM character sets appeared. All of these used the spare 128 characters for their own designs. Now what would happen in some circumstances was that a character that was encoded based on a different character set would appear as a completely different character on a computer using its own extended character set. For example, on some computers, the character code 130 would display as “é” but on computers sold in Israel the character would display as the Hebrew letter gimel (). When Americans would send their “résumés” to Israel, they would arrive as “.” In many cases, such as with Russian, there were many divergent ideas related to the upper 128 characters, which resulted in not being able to reliably interchange Russian documents.

The different ideas were codified into what are known as code pages. A code page is a table of values that describes a language’s encoding for a particular character set. Each of these code pages had a value associated with it. Greek speakers would use code page 737, Cyrillic speakers code page 855, and so on. All of these code pages were the same from codes 0 to 128, but different from codes 129 and up.

Each Asian character is represented by a pair of code points (hence the term double-byte), which allows for representing up to 65,536 characters. For programming awareness, a set of points are set aside to represent the first byte of the set and are not valued unless they are immediately followed by a defined second byte. DBCS meant that you had to write code that would treat these pairs of code points as one, and this still disallowed the combining of, say, Japanese and Chinese in the same data stream because depending on the code page, the same double-byte code points represent different characters for the different languages.

In simplest terms, Unicode provides a code point for every character or symbol in nearly all the world’s writing systems. Unicode code points are written in the form “U+####” in which “####” is made up of four to six hexadecimal digits. To give three quick examples, the code point U+0062 (decimal 98) represents a lowercase “b”—the same character and same value in the Latin ASCII table. The Cyrillic capital letter “de” or “” has the Unicode code point of U+0414 (decimal 1044), and the fleur-de-lis symbol “” is U+269C (decimal 9884).

To provide room for 65,536 characters, Unicode was originally conceived as a 16-bit encoding. This provided enough space to encode all modern scripts around the world. Private Use areas were designated to hold rare or obsolete characters. Unicode has approximately 10% of its total available code points in use, leaving ample room to grow.

As far as Unicode is concerned, the two characters are not equivalent—because they contain different code points—but they do have canonical equivalency. In other words, they have the same appearance and meaning.  See the “Diacritics” section later in this chapter for examples of this combination in action.

Of course, there is the contrary scenario in which there truly are duplicate characters, both in display and in the character’s meaning. An example here would be the Angstrom sign, “Å.” This character owns two listings; the first has the character info “Latin Capital Letter A with Ring Above” (U+00C5), and the second has “Angstrom Sign” (U+212B).

Other characters that fall into this category have a property known as compatibility equivalence. Compatibility represents essentially the character but a slightly different visual appearance and a different behavior for this character. Concrete examples include Greek letters, which can be mathematical and technical symbols, Roman numerals, or actually Greek text.

Other examples of compatibility equivalence include ligatures. The single character “ff”—having character info of “Latin Small Ligature FF” (U+FB00) is compatible with the sequence of two characters “ff” having character info of “Latin Small Letter F” (U+0066) + “Latin Small Letter F” (U+0066). They might render and display similarly, but that similarity is dependent on context, typeface, and the text renderer.

The UTF-8 format uses variable-width encoding and is capable of storing and representing every character in the Unicode character set. Its design was based on avoiding endianness complications and byte order marks found in the UTF-16 and UTF-32 encoding formats and, even more important, backward compatibility with the ASCII format (see the “Endianness” note later in the chapter for more detail about that and byte order). This encoding format accounts for more than half of all web pages, and the Internet Mail Consortium recommends that all email programs create and display messages using UTF-8. It’s increasingly becoming the default character encoding in software applications, operating systems, and programming languages. Xcode is a prime example of this, as shown in Figure 2.1.

Text comes down to the wire as an NSData instance in which the “wire” could be a network condition or a file I/O action. To encode the text you are accessing, you can allocate an NSString and initialize it via the initWithData:encoding method. Notice the second parameter: encoding! So you need to know how the text was encoded.

Listing 2.1 Encoding Text to ASCII

Click here to view code image

---

// Fußgängerübergänge - "sea voyage"
NSString *uberUmlats = @"Fußgängerübergänge";
NSData *ASCIIData = [uberUmlats dataUsingEncoding:NSASCIIStringEncoding allowLossyConversion:YES];
NSString *encodeToASCII = [[NSString alloc] initWithData:ASCIIData encoding:NSASCIIStringEncoding];
NSLog(@"Encoded to ASCII - %@", encodeToASCII);

---

The encoding that is in place is ASCII via the NSASCIIEncoding specifier. The reason we get Fusgangerubergange as our return value is two-fold. First, by specifying ASCII we are limited to characters with code point values under 128, so essentially no accented characters, of which our string has three, “ß,” “ä,” and “ü.” Second, by using NSString’s dataUsingEncoding:allowLossyConversion: instance method, we can specify, in a sense, “Handle all the characters I give you, and if it’s an accented character, I’m okay with your losing that accent.” Although the result is not a correct German word, its display is very close, and its meaning can reasonably be interpreted. If we change the encoding type to NSMacOSRomanStringEncoding, there’s no guarantee how the characters will be converted and encoded. In fact, with this encoding, the result is an unintelligible Fu§g?¿nger¿berg¿nge.

Listing 2.2 Returning the ASCII Value for a Character

Click here to view code image

---

NSString *encodingFun = @"a";
if ([encodingFun length] > 0) {
    unichar ASCIIValue = [encodingFun characterAtIndex:0];
NSLog(@"ASCII value is %d", ASCIIValue);
}

---

The returned ASCII value for the character a is 97. Note that the type unichar is used because it is a typedef for an unsigned short. The value returned by the characterAtIndex method is the Unicode decimal representation for the code point. An NSString object is usually represented by an array of unichars internally, hence the reason we are using unichar as a return type.

Listing 2.3 Encoding an ASCII String to UTF-8

Click here to view code image

---

NSString *notUTF = @"Nürnberg";
NSString *nowUTF = [NSString stringWithUTF8String:[notUTF cStringUsingEncoding: NSMacOSRomanStringEncoding]];
NSLog(@"Now a UTF8 string: %@", nowUTF);

---

The returned value is Nürnberg.

Listing 2.4 Returning a String from an Encoding URL

Click here to view code image

---

NSString *curentEncodedString = @"%3CTom%26Jerry%3E";
NSString *currentDecodedString = [curentEncodedString stringByReplacingPercentEscapesUsingEncoding:NSUTF8StringEncoding];
NSLog(@"Decoded string: %@", currentDecodedString);

---

The code returns <Tom&Jerry>.

Diacritics can be treated in several ways:

A diacritic can be a Roman base character plus a diacritic character. A combination such as “á” might be encoded either as a single character with character info of “Small A With Acute,” or as a sequence of characters, “Small A” + “Combining Acute.” Another example is the character “Ä” represented as the Unicode code point U+00C4, or as a pair of code points, U+0041 and U+0308.

The “Small E with Grave,” or “è” character, might be rendered via glyphs—either a single composite glyph, or using two separate glyphs, one for “e” and another for an overstriking grave accent.

In some orthographies (the relationship between sounds and letters), the combination “è” would be considered a grapheme (a letter or a number of letters that represent a phoneme, or speech sound that distinguishes one word from another), whereas in other orthographies “e” and the grave accent would each be considered a grapheme in a word. In other words, if an orthography has an “è” as a grapheme, then it should be encoded as a single character and an associated single glyph. On the flip side, if an orthography has separate graphemes for the “e” and the “`” (the grave accent), they should be encoded as separate characters and rendered as separate glyphs.

In looking at our “Small Letter N with Tilde” example, we could potentially be dealing with one single character or two individual, separate characters. If our code is doing any kind of character or string comparison, it is possible to have a test fail. To ensure that the expected single characters are used in the comparison, Unicode normalization is required. This can be accomplished via the precomposedStringWithCanonicalMapping method. Let’s look at some code. In Listing 2.5, we’ll work with our “Latin Small Letter N with Tilde” as a combined character and compare it to the precomposed character.

Listing 2.5 Displaying Combined and Precomposed Characters

Click here to view code image

---

NSString *combinedCharacter = @"nu0303";
NSString *precomposedCharacter = @"ñ";
BOOL isEqual = [combinedCharacter isEqualToString:precomposedCharacter];
NSLog(@"The 'combined' character, '%@', is %@ to 'precomposed' character, '%@'", combinedCharacter, isEqual ? @"equal" : @"not equal", precomposedCharacter);

---

This returns “ñ is not equal to ñ.”

Now applying the same test, but first normalizing the characters, we use this:

Click here to view code image

NSString *combinedNormalized = [combinedCharacter precomposedStringWithCanonicalMapping];
NSString *precomposedNormalized = [precomposedCharacter precomposedStringWithCanonicalMapping];
BOOL isEqualNorm = [combinedNormalized isEqualToString:precomposedNormalized];
NSLog(@"The 'combined-normalized' character, '%@', is %@ to 'precomposed-normalized' character, '%@'", combinedCharacter, isEqualNorm ? @"equal" : @"not equal", precomposedCharacter);

This returns “ñ is equal to ñ.”

This reserved range is composed of two parts:

High surrogates—U+D800 to U+DBFF (total of 1,024 code points)

Low surrogates—U+DC00 to U+DFFF (total of 1,024 code points)

A lone surrogate is invalid in UTF-16; surrogates are always written in pairs, with the high surrogate followed by the low.

With UTF-16 encoding, characters with code points in ranges U+0000 through U+D7FF and U+E000 through U+FFFD are stored as single 16-bit units.

Table 2.6 contains examples of surrogate pairs.

The following code snippet shows you how to get a printout of a surrogate pair when you are given its code point:

Click here to view code image

uniChar characterArray[2];
CFStringGetSurrogatePairForLongCharacter(0x10FFFF, characterArray);
NSString *surrogate = [[NSString alloc] initWithCharacters:characterArray length:2];
NSLog(@"Surrogate: %@", surrogate);

Note that this is taking advantage of the CFStringGetSurrogatePairForLongCharacter function, which maps a UTF-32 character to a pair of UTF-16 surrogate characters. We need an array to plug the resulting UTF-16 pair into—that’s what the characterArray is for—and then the initWithCharacters:length: method of NSString does the rest.

Introduced in the late 1990s from a Japanese mobile phone provider, Emoji is the Japanese term for picture characters. Created by Shigetaka Kurita as an effort to retain his company’s customer base, the smiley-faced icons gave their text messages more cuteness. The other supporting factor of the Emoji characters was the ability to give contextual information with a single character. What’s the weather going to be like today? That’s easily presented with a sun or umbrella or cloud Emoji character.

Figure 2.2 shows the first page of the Emoji keyboard.

Apple Color Emoji is a font available on both iOS and OS X to provide support for the Unicode Emoji characters. Instead of this font having glyphs with black and white outlines, it has full-color, higher-resolution images for each of the nearly 900 glyphs it supports.

Strong support of Emoji has been a hard target to hit because it has historically occupied a private use area of Unicode with a range of code points from U+1F604 to U+1F539.

Here, I simply entered “414” for the Cyrillic capital character “de” (“”), mentioned in an early section. The binary representation of the code point value is displayed as well. This is extremely useful if you have the code point number, but what if you do not?

The workflow of my project is this:

You see a character from a text or Web site or even a file you’ve opened on your device.

You select the character and then copy it.

You switch to my project and paste it into the “String” field.

The GitHub project link is https://github.com/ShawnLaAppleDev/UnicodeValueGrabber.

The code takes advantage of both the NSString class method initWithString and the format specifier %04x to return the Hexadecimal value of supplied character. Note that the format specifier is set up to have four placeholders that include leading zeros (see Listing 2.6).

Listing 2.6 Returning the Unicode Code Points

Click here to view code image

---

unichar ch = [unicodeString characterAtIndex:0];
NSString *unicodeHex = [[NSString stringWithFormat:@"U+%04x", ch] uppercaseString];   
return unicodeHex;

---

Two points need to be called out:

A character conveys differences in meaning or sound. No appearance property is associated with it.

A glyph conveys differences in appearance. The key thing is appearance. A glyph has no intrinsic meaning.

Figure 2.4 shows a sampling of some glyphs for the Latin character “c.” Note that they are all of the same character, lowercase “c,” and therefore have the same meaning, but they are displayed, shaped, and outlined differently.

There are also cases in which a character will have a glyph assigned to it based on the font set so that the shape it displays is nothing like the traditional shape of the character. “Wingdings” and assorted symbol fonts like that are prime examples in which the lowercase “c” character actually displays as .

Also, variations in a character’s glyph can be associated with things like cursive connectors. In this scenario, the character is displayed traditionally, but it has slight changes depending on what character precedes it and what character follows it. Again, no meaning is lost; we have just gained flourishes.

Arabic is a right-to-left language, so the first/initial character is the rightmost, the second is to the left of that, the third is to the left of the second, and so on.

Another example is the Greek character sigma (σ). When it is used at the end of a word and the characters of that word are not all uppercase, the final form of the character “ς” is used, for example, “” (Odysseus). Note the two sigmas in the center of the name that remain the same and the word-final sigma at the end. Same character, different glyphs. This example also demonstrates that uppercase and lowercase are handled as separate characters and not as the same character—same character value, only displayed differently.

Now that we’ve covered glyphs, let’s move on to fonts.

Font files are your storage depot for the glyphs that are associated with the characters. Well-crafted fonts won’t fake bold, italic, and bold-italic variations but have built-in, designed glyphs for these variations. After your application has worked out what characters it is dealing with, it will look in the font for glyphs in order to display or print those characters. Of course, if the encoding information was wrong, it will be looking up glyphs for the wrong characters.

A given font will usually cover a single character set. In the case of a large character set, like Unicode, just a subset of all the available characters will be available. This is one of the many reasons you will see specific fonts for CJK characters. It is more practical to have specific fonts hold specific character sets for both performance and file-size benefits.

If your font doesn’t have a glyph for a particular character, some applications as well as the OS will look for the missing glyph in other fonts on your system. Although this eliminates a missing glyph from displaying as an empty box or a box containing a question mark, it does have the potential of having the glyph look different from the surrounding text, like a ransom note.

Some ligatures are two separate characters displayed with a connected glyph, whereas the glyph is one character with its own code point.

Standard ligatures include might include fi, fl, ff, ffi, ffl, and ft. The purpose of these ligatures is to make certain letter parts that tend to knock up against each other more attractive.

Here are some individual Unicode ligature characters:

æ—CYRILLIC SMALL LIGATURE A IE; Unicode: U+04D5; UTF-8: D3 95

fl—LATIN SMALL LIGATURE FL; Unicode: U+FB02; UTF-8: EF AC 82

fi—LATIN SMALL LIGATURE FI; Unicode: U+FB01; UTF-8: EF AC 81

Listing 2.7 Using Localized Compare to Determine Whether Characters Are Equal

Click here to view code image

---

NSString *characters = @"ff"; // Two "f" characters
NSString *ligature = @"uFB00"; // Single character - "ff" ligature
NSComparisonResult result = [characters localizedCompare: ligature];
if (result == NSOrderedSame){
    NSLog(@"%@ is equal to %@", characters, ligature);
} else{
    NSLog(@"Characters are not equal.");
};

---

The code returns “ff is equal to ff.”