Use a regular expression to find the trademark sign (™) by specifying its Unicode code point rather than copying and pasting an actual trademark sign. If you like copy and paste, the trademark sign is just another literal character, even though you cannot type it directly on your keyboard. Literal characters are discussed in Recipe 2.1.
Create a regular expression that matches any character is in the “Currency Symbol” Unicode category.
Create a regular expression that matches any character in the “Greek Extended” Unicode block.
Create a regular expression that matches any character that, according to the Unicode standard, is part of the Greek script.
Create a regular expression that matches a grapheme, or what is commonly thought of as a character: a base character with all its combining marks.
u2122
Regex options: None |
Regex flavors: .NET, Java, JavaScript, Python, Ruby 1.9 |
U00002122
Regex options: None |
Regex flavors: Python |
These regexes work in Python 2.x only when quoted as
Unicode strings: u"u2122"
or u"U00002122"
.
x{2122}
Regex options: None |
Regex flavors: Java 7, PCRE, Perl |
PCRE must be compiled with UTF-8 support; in PHP, turn
on UTF-8 support with the /u
pattern modifier.
u{2122}
Regex options: None |
Regex flavors: Ruby 1.9 |
p{Sc}
Regex options: None |
Regex flavors: .NET, Java, XRegExp, PCRE, Perl, Ruby 1.9 |
PCRE must be compiled with UTF-8 support; in PHP, turn
on UTF-8 support with the /u
pattern modifier. JavaScript and Python do not support Unicode
properties. XRegExp adds support for Unicode properties to JavaScript.
Ruby 1.8 does not support Unicode regular expressions.
p{IsGreekExtended}
Regex options: None |
Regex flavors: .NET, Perl |
p{InGreekExtended}
Regex options: None |
Regex flavors: Java, XRegExp, Perl |
JavaScript, PCRE, Python, and Ruby 1.9 do not support Unicode blocks. They do support Unicode code points, which you can use to match blocks as shown in the section in this recipe. XRegExp adds support for Unicode blocks to JavaScript.
p{Greek}
Regex options: None |
Regex flavors: XRegExp, PCRE, Perl, Ruby 1.9 |
p{IsGreek}
Regex options: None |
Regex flavors: Java 7, Perl |
Unicode script support requires PCRE 6.5 or later, and
PCRE must be compiled with UTF-8 support. In PHP, turn on UTF-8
support with the /u
pattern
modifier. .NET, JavaScript, and Python do not support Unicode
properties. XRegExp adds support for Unicode properties to JavaScript.
Ruby 1.8 does not support Unicode regular expressions.
X
Regex options: None |
Regex flavors: PCRE, Perl |
PCRE and Perl have a dedicated token for matching
graphemes. PCRE must be compiled with UTF-8 support; in PHP, turn on
UTF-8 support with the /u
pattern
modifier.
(?>P{M}p{M}*)
Regex options: None |
Regex flavors: .NET, Java, Ruby 1.9 |
(?:P{M}p{M}*)
Regex options: None |
Regex flavors: XRegExp |
.NET, Java, XRegExp, and Ruby 1.9 do not have a token for matching graphemes. But they do support Unicode categories, which we can use to emulate matching graphemes.
JavaScript (without XRegExp) and Python do not support Unicode properties. Ruby 1.8 does not support Unicode regular expressions.
A code point is one entry in the Unicode character database. A code point is not the same as a character, depending on the meaning you give to “character.” What appears as a character on screen is called a grapheme in Unicode.
The Unicode code point U+2122
represents the “trademark sign” character. You can match this with
‹u2122
›, ‹u{2122}
›, or ‹x{2122}
›, depending on the
regex flavor you’re working with.
The ‹u
›
syntax requires exactly four hexadecimal digits. This means you can
only use it for Unicode code points U+0000 through U+FFFF.
‹u{⋯}
› and ‹x{⋯}
› allow between one and six
hexadecimal digits between the braces, supporting all code points
U+000000 through U+10FFFF. You can match U+00E0 with ‹x{E0}
› or ‹x{00E0}
›. Code points U+100000
and above are used very infrequently. They are poorly supported by
fonts and operating systems.
Python’s regular expression engine has no support for Unicode
code points. Literal Unicode strings in Python 2.x and literal text
strings in Python 3.x do have escapes for Unicode code points.
u0000
through uFFFF
represent Unicode code points U+0000
through U+FFFF. U00000000
through
U0010FFFF
represent all Unicode
code points. You have to specify eight hexadecimal numbers after
U
, even though there
are no Unicode code points beyond U+10FFFF.
When hard-coding regular expressions as literal strings in your
Python code, you can directly use ‹u2122
› and ‹U00002122
› in your regexes. When reading
regexes from a file or receiving them from user input, these Unicode
escapes will not work if you pass the string you read or received
directly to re.compile()
. In Python 2.x, you can decode the
Unicode escapes by calling string.decode('unicode-escape')
. In Python 3.x
you can call string.encode('utf-8').decode('unicode-escape')
.
Code points can be used inside and outside character classes.
Each Unicode code point fits into a single Unicode category. There are 30 Unicode categories, specified with a code consisting of two letters. These are grouped into 7 super-categories that are specified with a single letter.
‹p{L} ›: Any
kind of letter from any language |
‹p{Ll} ›: A
lowercase letter that has an uppercase variant |
‹p{Lu} ›: An
uppercase letter that has a lowercase variant |
‹p{Lt} ›: A
letter that appears at the start of a word when only the first
letter of the word is capitalized |
‹p{Lm} ›: A
special character that is used like a letter |
‹p{Lo} ›: A
letter or ideograph that does not have lowercase and uppercase
variants |
‹p{M} ›: A
character intended to be combined with another character (accents,
umlauts, enclosing boxes, etc.) |
‹p{Mn} ›: A
character intended to be combined with another character that does
not take up extra space (e.g., accents, umlauts, etc.) |
‹p{Mc} ›: A
character intended to be combined with another character that does
take up extra space (e.g., vowel signs in many Eastern
languages) |
‹p{Me} ›: A
character that encloses another character (circle, square, keycap,
etc.) |
‹p{Z} ›: Any
kind of whitespace or invisible separator |
‹p{Zs} ›: A
whitespace character that is invisible, but does take up
space |
‹p{Zl} ›: The
line separator character U+2028 |
‹p{Zp} ›: The
paragraph separator character U+2029 |
‹p{S} ›: Math
symbols, currency signs, dingbats, box-drawing characters,
etc. |
‹p{Sm} ›: Any
mathematical symbol |
‹p{Sc} ›: Any
currency sign |
‹p{Sk} ›: A
combining character (mark) as a full character on its own |
‹p{So} ›:
Various symbols that are not math symbols, currency signs, or
combining characters |
‹p{N} ›: Any
kind of numeric character in any script |
‹p{Nd} ›: A
digit 0 through 9 in any script except ideographic scripts |
‹p{Nl} ›: A
number that looks like a letter, such as a Roman numeral |
‹p{No} ›: A
superscript or subscript digit, or a number that is not a digit 0…9
(excluding numbers from ideographic scripts) |
‹p{P} ›: Any
kind of punctuation character |
‹p{Pd} ›: Any
kind of hyphen or dash |
‹p{Ps} ›: Any
kind of opening bracket |
‹p{Pe} ›: Any
kind of closing bracket |
‹p{Pi} ›: Any
kind of opening quote |
‹p{Pf} ›: Any
kind of closing quote |
‹p{Pc} ›: A
punctuation character such as an underscore that connects
words |
‹p{Po} ›: Any
kind of punctuation character that is not a dash, bracket, quote or
connector |
‹p{C} ›:
Invisible control characters and unused code points |
‹p{Cc} ›: An
ASCII or Latin-1 control character 0x00…0x1F and 0x7F…0x9F |
‹p{Cf} ›: An
invisible formatting indicator |
‹p{Co} ›: Any
code point reserved for private use |
‹p{Cs} ›: One
half of a surrogate pair in UTF-16 encoding |
‹p{Cn} ›: Any
code point to which no character has been assigned |
‹p{Ll}
›
matches a single code point that is in the Ll
, or “lowercase letter,” category.
‹p{L}
› is
a quick way of writing ‹[p{Ll}p{Lu}p{Lt}p{Lm}p{Lo}]
› that matches a
single code point in any of the “letter” categories.
‹P
› is
the negated version of ‹p
›.
‹P{Ll}
›
matches a single code point that is not in the Ll
category. ‹P{L}
›
matches a single code point that does not have any of the “letter”
properties. This is not the same as ‹[P{Ll}P{Lu}P{Lt}P{Lm}P{Lo}]
›, which matches
all code points. ‹P{Ll}
›
matches the code points in the Lu
category (and every other category except Ll
), whereas ‹P{Lu}
›
includes the Ll
code points.
Combining just these two in a code point class already matches all
possible code points.
The Unicode character database divides all the code points into blocks. Each block consists of a single range of code points. The code points U+0000 through U+FFFF are divided into 156 blocks in version 6.1 of the Unicode standard:
‹U+0000…U+007F
p{InBasicLatin} › |
‹U+0080…U+00FF
p{InLatin-1Supplement} › |
‹U+0100…U+017F
p{InLatinExtended-A} › |
‹U+0180…U+024F
p{InLatinExtended-B} › |
‹U+0250…U+02AF
p{InIPAExtensions} › |
‹U+02B0…U+02FF
p{InSpacingModifierLetters} › |
‹U+0300…U+036F
p{InCombiningDiacriticalMarks} › |
‹U+0370…U+03FF
p{InGreekandCoptic} › |
‹U+0400…U+04FF
p{InCyrillic} › |
‹U+0500…U+052F
p{InCyrillicSupplement} › |
‹U+0530…U+058F
p{InArmenian} › |
‹U+0590…U+05FF
p{InHebrew} › |
‹U+0600…U+06FF
p{InArabic} › |
‹U+0700…U+074F
p{InSyriac} › |
‹U+0750…U+077F
p{InArabicSupplement} › |
‹U+0780…U+07BF
p{InThaana} › |
‹U+07C0…U+07FF
p{InNKo} › |
‹U+0800…U+083F
p{InSamaritan} › |
‹U+0840…U+085F
p{InMandaic} › |
‹U+08A0…U+08FF
p{InArabicExtended-A} › |
‹U+0900…U+097F
p{InDevanagari} › |
‹U+0980…U+09FF
p{InBengali} › |
‹U+0A00…U+0A7F
p{InGurmukhi} › |
‹U+0A80…U+0AFF
p{InGujarati} › |
‹U+0B00…U+0B7F
p{InOriya} › |
‹U+0B80…U+0BFF
p{InTamil} › |
‹U+0C00…U+0C7F
p{InTelugu} › |
‹U+0C80…U+0CFF
p{InKannada} › |
‹U+0D00…U+0D7F
p{InMalayalam} › |
‹U+0D80…U+0DFF
p{InSinhala} › |
‹U+0E00…U+0E7F
p{InThai} › |
‹U+0E80…U+0EFF
p{InLao} › |
‹U+0F00…U+0FFF
p{InTibetan} › |
‹U+1000…U+109F
p{InMyanmar} › |
‹U+10A0…U+10FF
p{InGeorgian} › |
‹U+1100…U+11FF
p{InHangulJamo} › |
‹U+1200…U+137F
p{InEthiopic} › |
‹U+1380…U+139F
p{InEthiopicSupplement} › |
‹U+13A0…U+13FF
p{InCherokee} › |
‹U+1400…U+167F
p{InUnifiedCanadianAboriginalSyllabics} › |
‹U+1680…U+169F
p{InOgham} › |
‹U+16A0…U+16FF
p{InRunic} › |
‹U+1700…U+171F
p{InTagalog} › |
‹U+1720…U+173F
p{InHanunoo} › |
‹U+1740…U+175F
p{InBuhid} › |
‹U+1760…U+177F
p{InTagbanwa} › |
‹U+1780…U+17FF
p{InKhmer} › |
‹U+1800…U+18AF
p{InMongolian} › |
‹U+18B0…U+18FF
p{InUnifiedCanadianAboriginalSyllabicsExtended} › |
‹U+1900…U+194F
p{InLimbu} › |
‹U+1950…U+197F
p{InTaiLe} › |
‹U+1980…U+19DF
p{InNewTaiLue} › |
‹U+19E0…U+19FF
p{InKhmerSymbols} › |
‹U+1A00…U+1A1F
p{InBuginese} › |
‹U+1A20…U+1AAF
p{InTaiTham} › |
‹U+1B00…U+1B7F
p{InBalinese} › |
‹U+1B80…U+1BBF
p{InSundanese} › |
‹U+1BC0…U+1BFF
p{InBatak} › |
‹U+1C00…U+1C4F
p{InLepcha} › |
‹U+1C50…U+1C7F
p{InOlChiki} › |
‹U+1CC0…U+1CCF
p{InSundaneseSupplement} › |
‹U+1CD0…U+1CFF
p{InVedicExtensions} › |
‹U+1D00…U+1D7F
p{InPhoneticExtensions} › |
‹U+1D80…U+1DBF
p{InPhoneticExtensionsSupplement} › |
‹U+1DC0…U+1DFF
p{InCombiningDiacriticalMarksSupplement} › |
‹U+1E00…U+1EFF
p{InLatinExtendedAdditional} › |
‹U+1F00…U+1FFF
p{InGreekExtended} › |
‹U+2000…U+206F
p{InGeneralPunctuation} › |
‹U+2070…U+209F
p{InSuperscriptsandSubscripts} › |
‹U+20A0…U+20CF
p{InCurrencySymbols} › |
‹U+20D0…U+20FF
p{InCombiningDiacriticalMarksforSymbols} › |
‹U+2100…U+214F
p{InLetterlikeSymbols} › |
‹U+2150…U+218F
p{InNumberForms} › |
‹U+2190…U+21FF
p{InArrows} › |
‹U+2200…U+22FF
p{InMathematicalOperators} › |
‹U+2300…U+23FF
p{InMiscellaneousTechnical} › |
‹U+2400…U+243F
p{InControlPictures} › |
‹U+2440…U+245F
p{InOpticalCharacterRecognition} › |
‹U+2460…U+24FF
p{InEnclosedAlphanumerics} › |
‹U+2500…U+257F
p{InBoxDrawing} › |
‹U+2580…U+259F
p{InBlockElements} › |
‹U+25A0…U+25FF
p{InGeometricShapes} › |
‹U+2600…U+26FF
p{InMiscellaneousSymbols} › |
‹U+2700…U+27BF
p{InDingbats} › |
‹U+27C0…U+27EF
p{InMiscellaneousMathematicalSymbols-A} › |
‹U+27F0…U+27FF
p{InSupplementalArrows-A} › |
‹U+2800…U+28FF
p{InBraillePatterns} › |
‹U+2900…U+297F
p{InSupplementalArrows-B} › |
‹U+2980…U+29FF
p{InMiscellaneousMathematicalSymbols-B} › |
‹U+2A00…U+2AFF
p{InSupplementalMathematicalOperators} › |
‹U+2B00…U+2BFF
p{InMiscellaneousSymbolsandArrows} › |
‹U+2C00…U+2C5F
p{InGlagolitic} › |
‹U+2C60…U+2C7F
p{InLatinExtended-C} › |
‹U+2C80…U+2CFF
p{InCoptic} › |
‹U+2D00…U+2D2F
p{InGeorgianSupplement} › |
‹U+2D30…U+2D7F
p{InTifinagh} › |
‹U+2D80…U+2DDF
p{InEthiopicExtended} › |
‹U+2DE0…U+2DFF
p{InCyrillicExtended-A} › |
‹U+2E00…U+2E7F
p{InSupplementalPunctuation} › |
‹U+2E80…U+2EFF
p{InCJKRadicalsSupplement} › |
‹U+2F00…U+2FDF
p{InKangxiRadicals} › |
‹U+2FF0…U+2FFF
p{InIdeographicDescriptionCharacters} › |
‹U+3000…U+303F
p{InCJKSymbolsandPunctuation} › |
‹U+3040…U+309F
p{InHiragana} › |
‹U+30A0…U+30FF
p{InKatakana} › |
‹U+3100…U+312F
p{InBopomofo} › |
‹U+3130…U+318F
p{InHangulCompatibilityJamo} › |
‹U+3190…U+319F
p{InKanbun} › |
‹U+31A0…U+31BF
p{InBopomofoExtended} › |
‹U+31C0…U+31EF
p{InCJKStrokes} › |
‹U+31F0…U+31FF
p{InKatakanaPhoneticExtensions} › |
‹U+3200…U+32FF
p{InEnclosedCJKLettersandMonths} › |
‹U+3300…U+33FF
p{InCJKCompatibility} › |
‹U+3400…U+4DBF
p{InCJKUnifiedIdeographsExtensionA} › |
‹U+4DC0…U+4DFF
p{InYijingHexagramSymbols} › |
‹U+4E00…U+9FFF
p{InCJKUnifiedIdeographs} › |
‹U+A000…U+A48F
p{InYiSyllables} › |
‹U+A490…U+A4CF
p{InYiRadicals} › |
‹U+A4D0…U+A4FF
p{InLisu} › |
‹U+A500…U+A63F
p{InVai} › |
‹U+A640…U+A69F
p{InCyrillicExtended-B} › |
‹U+A6A0…U+A6FF
p{InBamum} › |
‹U+A700…U+A71F
p{InModifierToneLetters} › |
‹U+A720…U+A7FF
p{InLatinExtended-D} › |
‹U+A800…U+A82F
p{InSylotiNagri} › |
‹U+A830…U+A83F
p{InCommonIndicNumberForms} › |
‹U+A840…U+A87F
p{InPhags-pa} › |
‹U+A880…U+A8DF
p{InSaurashtra} › |
‹U+A8E0…U+A8FF
p{InDevanagariExtended} › |
‹U+A900…U+A92F
p{InKayahLi} › |
‹U+A930…U+A95F
p{InRejang} › |
‹U+A960…U+A97F
p{InHangulJamoExtended-A} › |
‹U+A980…U+A9DF
p{InJavanese} › |
‹U+AA00…U+AA5F
p{InCham} › |
‹U+AA60…U+AA7F
p{InMyanmarExtended-A} › |
‹U+AA80…U+AADF
p{InTaiViet} › |
‹U+AAE0…U+AAFF
p{InMeeteiMayekExtensions} › |
‹U+AB00…U+AB2F
p{InEthiopicExtended-A} › |
‹U+ABC0…U+ABFF
p{InMeeteiMayek} › |
‹U+AC00…U+D7AF
p{InHangulSyllables} › |
‹U+D7B0…U+D7FF
p{InHangulJamoExtended-B} › |
‹U+D800…U+DB7F
p{InHighSurrogates} › |
‹U+DB80…U+DBFF
p{InHighPrivateUseSurrogates} › |
‹U+DC00…U+DFFF
p{InLowSurrogates} › |
‹U+E000…U+F8FF
p{InPrivateUseArea} › |
‹U+F900…U+FAFF
p{InCJKCompatibilityIdeographs} › |
‹U+FB00…U+FB4F
p{InAlphabeticPresentationForms} › |
‹U+FB50…U+FDFF
p{InArabicPresentationForms-A} › |
‹U+FE00…U+FE0F
p{InVariationSelectors} › |
‹U+FE10…U+FE1F
p{InVerticalForms} › |
‹U+FE20…U+FE2F
p{InCombiningHalfMarks} › |
‹U+FE30…U+FE4F
p{InCJKCompatibilityForms} › |
‹U+FE50…U+FE6F
p{InSmallFormVariants} › |
‹U+FE70…U+FEFF
p{InArabicPresentationForms-B} › |
‹U+FF00…U+FFEF
p{InHalfwidthandFullwidthForms} › |
‹U+FFF0…U+FFFF
p{InSpecials} › |
A Unicode block is a single, contiguous range of code points. Although many blocks have the names of Unicode scripts and Unicode categories, they do not correspond 100% with them. The name of a block only indicates its primary use.
The Currency
block does not
include the dollar and yen symbols. Those are found in the BasicLatin
and Latin-1Supplement
blocks, for historical
reasons. Both are in the Currency
Symbol
category. To match any currency symbol, use ‹p{Sc}
› instead of ‹p{InCurrency}
›.
Most blocks include unassigned code points, which are in the
category ‹p{Cn}
›. None
of the other Unicode categories, and none of the Unicode scripts,
include unassigned code points.
The ‹p{InBlockName}
› syntax works with .NET,
XRegExp, and Perl. Java uses the ‹p{IsBlockName}
›
syntax.
Perl also supports the Is
variant, but we recommend you stick with the In
syntax, to avoid confusion with Unicode
scripts. For scripts, Perl supports ‹p{Script}
› and ‹p{IsScript}
›, but not ‹p{InScript}
›.
The Unicode standard stipulates that block names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. All versions of .NET and Java 4 require the block names to be capitalized as shown in the preceding list. Perl 5.8 and later and Java 5 and later allow any mixture of case. Perl, Java, and .NET all support the notation with hyphens and without spaces used in the preceding list. We recommend you use this notation. Of the flavors discussed in this book, only XRegExp and Perl 5.12 and later are fully flexible with regard to spaces, hyphens, and underscores in Unicode block names.
Each Unicode code point, except unassigned ones, is part of exactly one Unicode script. Unassigned code points are not part of any script. The assigned code points up to U+FFFF are assigned to these 72 scripts in version 6.1 of the Unicode standard:
‹p{Common} › | ‹p{Lepcha} › |
‹p{Arabic} › | ‹p{Limbu} › |
‹p{Armenian} › | ‹p{Lisu} › |
‹p{Balinese} › | ‹p{Malayalam} › |
‹p{Bamum} › | ‹p{Mandaic} › |
‹p{Batak} › | ‹p{Meetei_Mayek} › |
‹p{Bengali} › | ‹p{Mongolian} › |
‹p{Bopomofo} › | ‹p{Myanmar} › |
‹p{Braille} › | ‹p{New_Tai_Lue} › |
‹p{Buginese} › | ‹p{Nko} › |
‹p{Buhid} › | ‹p{Ogham} › |
‹p{Canadian_Aboriginal} › | ‹p{Ol_Chiki} › |
‹p{Cham} › | ‹p{Oriya} › |
‹p{Cherokee} › | ‹p{Phags_Pa} › |
‹p{Coptic} › | ‹p{Rejang} › |
‹p{Cyrillic} › | ‹p{Runic} › |
‹p{Devanagari} › | ‹p{Samaritan} › |
‹p{Ethiopic} › | ‹p{Saurashtra} › |
‹p{Georgian} › | ‹p{Sinhala} › |
‹p{Glagolitic} › | ‹p{Sundanese} › |
‹p{Greek} › | ‹p{Syloti_Nagri} › |
‹p{Gujarati} › | ‹p{Syriac} › |
‹p{Gurmukhi} › | ‹p{Tagalog} › |
‹p{Han} › | ‹p{Tagbanwa} › |
‹p{Hangul} › | ‹p{Tai_Le} › |
‹p{Hanunoo} › | ‹p{Tai_Tham} › |
‹p{Hebrew} › | ‹p{Tai_Viet} › |
‹p{Hiragana} › | ‹p{Tamil} › |
‹p{Inherited} › | ‹p{Telugu} › |
‹p{Javanese} › | ‹p{Thaana} › |
‹p{Kannada} › | ‹p{Thai} › |
‹p{Katakana} › | ‹p{Tibetan} › |
‹p{Kayah_Li} › | ‹p{Tifinagh} › |
‹p{Khmer} › | ‹p{Vai} › |
‹p{Lao} › | ‹p{Yi} › |
‹p{Latin} › |
A script is a group of code points used by a particular
human writing system. Some scripts, such as Thai
, correspond with a single human
language. Other scripts, such as Latin
, span multiple languages. Some
languages are composed of multiple scripts. For instance, there is no
Japanese Unicode script; instead, Unicode offers the Hiragana
, Katakana
, Han
, and Latin
scripts that Japanese documents are
usually composed of.
We listed the Common
script
first, out of alphabetical order. This script contains all sorts of
characters that are common to a wide range of scripts, such as
punctuation, whitespace, and miscellaneous symbols.
Java requires the name of the script to be prefixed with
Is
, as in ‹p{IsYi}
›. Perl allows the Is
prefix, but doesn’t require it. XRegExp,
PCRE, and Ruby do not allow the Is
prefix.
The Unicode standard stipulates that script names should be case insensitive, and that any differences in spaces, hyphens, or underscores should be ignored. Most regex flavors are not this flexible, unfortunately. The notation with the words in the script names capitalized and with underscores between the words works with all flavors in this book that support Unicode scripts.
The difference between code points and characters comes into play when there are combining marks. The Unicode code point U+0061 is “Latin small letter a,” whereas U+00E0 is “Latin small letter a with grave accent.” Both represent what most people would describe as a character.
U+0300 is the “combining grave accent” combining mark. It can be
used sensibly only after a letter. A string consisting of the Unicode
code points U+0061 U+0300 will be displayed as à
, just like U+00E0. The combining mark
U+0300 is displayed on top of the character U+0061.
The reason for these two different ways of displaying an accented letter is that many historical character sets encode “a with grave accent” as a single character. Unicode’s designers thought it would be useful to have a one-on-one mapping with popular legacy character sets, in addition to the Unicode way of separating marks and base letters, which makes arbitrary combinations not supported by legacy character sets possible.
What matters to you as a regex user is that all regex flavors
discussed in this book operate on code points rather than graphical
characters. When we say that the regular expression ‹.
› matches a single character,
it really matches just a single code point. If your subject text
consists of the two code points U+0061 U+0300, which can be
represented as the string literal "u0061u0300"
in a programming language such as
Java, the dot will match only the code point U+0061, or a
, without the accent U+0300. The regex
‹..
› will match
both.
Perl and PCRE offer a special regex token ‹X
›,
which matches any single Unicode grapheme. Essentially, it is the
Unicode version of the venerable dot. ‹X
› will
find two matches in the text àà
, regardless of how it is encoded. If
it is encoded as u00E0u0061u0300
the first match is u00E0
, and the second u0061u0300
. The dot, which
matches any single Unicode code point, would find three matches as it
matches u00E0
,
u0061
, and u0300
separately.
The rules for exactly which combinations of Unicode code points
are considered graphemes are quite complicated.[2] Generally speaking, to match a grapheme we need to match
any character that is not a mark and all the marks that follow it, if
any. We can match this with the regex ‹(?>P{M}p{M}*)
› in all regex flavors that
support Unicode but not the ‹X
› token
for graphemes. ‹P{M}
›
matches any character that is not in the Mark
category. ‹p{M}*
› matches all the marks, if any, that
follow it.
We put these two regex tokens in an atomic group to make sure
the ‹p{M}*
› won’t
backtrack if any following regex tokens fail to match. ‹X{2}.
› does not match
àà
, because
there is nothing left for the dot to match after ‹X{2}
› has matched the two
accented letters. ‹(?>P{M}p{M}*){2}.
› does not match
àà
for the
same reason. But ‹(?:P{M}p{M}*){2}.
› with an non-capturing group
does match àà
if it is encoded as u00E0u0061u0300
. Upon the
second iteration of the group, ‹p{M}*
› will match u0300
. The dot will then fail to match. This
causes the regex to backtrack, forcing ‹p{M}*
› to give up its match, allowing the dot
to match u0300
.
JavaScript’s regex engine does not support atomic
grouping. This is not a feature that could be added by XRegExp,
because XRegExp still relies on JavaScript’s regex engine for the
actual pattern matching. So when using XRegExp, ‹(?:P{M}p{M}*)
› is the closest
we can get to emulating ‹X
›.
Without the atomic group, you’ll have to keep in mind that ‹p{M}*
› may backtrack if
whatever follows ‹(?:P{M}p{M}*)
› in your regex can match
characters in the Mark category.
The uppercase ‹P
› is
the negated variant of the lowercase ‹p
›. For
instance, ‹P{Sc}
›
matches any character that does not have the “Currency Symbol” Unicode
property. ‹P
› is
supported by all flavors that support ‹p
›, and
for all the properties, block, and scripts that they
support.
All flavors allow all the ‹u
›,
‹x
›,
‹p
›, and
‹P
›
tokens they support to be used inside character classes. The character
represented by the code point, or the characters in the category,
block, or script, are then added to the character class. For instance,
you could match a character that is either an opening quote (initial
punctuation property), a closing quote (final punctuation property),
or the trademark symbol (U+2122) with:
[p{Pi}p{Pf}u2122]
Regex options: None |
Regex flavors: .NET, Java, XRegExp, Ruby 1.9 |
[p{Pi}p{Pf}x{2122}]
Regex options: None |
Regex flavors: Java 7, PCRE, Perl |
If your regular expression flavor does not support Unicode categories, blocks, or scripts, you can list the characters that are in the category, block, or script in a character class. For blocks this is very easy: each block is simply a range between two code points. The Greek Extended block comprises the characters U+1F00 to U+1FFF:
[u1F00-u1FFF]
Regex options: None |
Regex flavors: .NET, Java, JavaScript, Python, Ruby 1.9 |
[x{1F00}-x{1FFF}]
Regex options: None |
Regex flavors: Java 7, PCRE, Perl |
For most categories and many scripts, the equivalent character class is a long list of individual code points and short ranges. The characters that comprise each category and many of the scripts are scattered throughout the Unicode table. This is the Greek script:
[u0370-u0373u0375-u0377u037A-u037Du0384u0386u0388-u038A↵ u038Cu038E-u03A1u03A3-u03E1u03F0-u03FFu1D26-u1D2Au1D5D-u1D61↵ u1D66-u1D6Au1DBFu1F00-u1F15u1F18-u1F1Du1F20-u1F45u1F48-u1F4D↵ u1F50-u1F57u1F59u1F5Bu1F5Du1F5F-u1F7Du1F80-u1FB4u1FB6-u1FC4↵ u1FC6-u1FD3u1FD6-u1FDBu1FDD-u1FEFu1FF2-u1FF4u1FF6-u1FFEu2126↵ U00010140-U0001018AU0001D200-U0001D245]
We generated this regular expression using the UnicodeSet web
application at http://unicode.org/cldr/utility/list-unicodeset.jsp. We
entered p{Greek}
as the input,
ticked the “Abbreviate” and “Escape” checkboxes, and clicked the “Show
Set” button.
Only Python supports this syntax for Unicode code points as we explained earlier in this recipe in the section Unicode code point. To make this regular expression work with other regex flavors, we need to make some changes.
The regex will work with many more flavors if we remove the code points beyond U+FFFF from the character class:
Regex options: None |
Regex flavors: Python |
[u0370-u0373u0375-u0377u037A-u037Du0384u0386u0388-u038A↵ u038Cu038E-u03A1u03A3-u03E1u03F0-u03FFu1D26-u1D2Au1D5D-u1D61↵ u1D66-u1D6Au1DBFu1F00-u1F15u1F18-u1F1Du1F20-u1F45u1F48-u1F4D↵ u1F50-u1F57u1F59u1F5Bu1F5Du1F5F-u1F7Du1F80-u1FB4u1FB6-u1FC4↵ u1FC6-u1FD3u1FD6-u1FDBu1FDD-u1FEFu1FF2-u1FF4u1FF6-u1FFEu2126]
Regex options: None |
Regex flavors: .NET, Java, JavaScript, Python, Ruby 1.9 |
Perl and PCRE use a different syntax for Unicode code points. In
the original regex, we need to replace ‹uFFFF
› with ‹x{FFFF}
› and ‹U0010FFFF
› with ‹x{10FFFF}
›. This regex also works with Java
7.
[x{0370}-x{0373}x{0375}-x{0377}x{037A}-x{037D}x{0384}x{0386}↵ x{0388}-x{038A}x{038C}x{038E}-x{03A1}x{03A3}-x{03E1}↵ x{03F0}-x{03FF}x{1D26}-x{1D2A}x{1D5D}-x{1D61}x{1D66}-x{1D6A}↵ x{1DBF}x{1F00}-x{1F15}x{1F18}-x{1F1D}x{1F20}-x{1F45}↵ x{1F48}-x{1F4D}x{1F50}-x{1F57}x{1F59}x{1F5B}x{1F5D}x{1F5F}-↵ x{1F7D}x{1F80}-x{1FB4}x{1FB6}-x{1FC4}x{1FC6}-x{1FD3}x{1FD6}-↵ x{1FDB}x{1FDD}-x{1FEF}x{1FF2}-x{1FF4}x{1FF6}-x{1FFE}x{2126}↵ x{10140}-x{10178}x{10179}-x{10189}x{1018A}x{1D200}-x{1D245}]
Regex options: None |
Regex flavors: Java 7, PCRE, Perl |
http://www.unicode.org is the official website of the Unicode Consortium, where you can download all the official Unicode documents, character tables, etc.
Unicode is a vast topic, on which entire books have been written. One such book is Unicode Explained by Jukka K. Korpela (O’Reilly).
We can’t explain everything you should know about Unicode code points, categories, blocks, and scripts in just one section. We haven’t even tried to explain why you should care—you should. The comfortable simplicity of the extended ASCII table is a lonely place in today’s globalized world.
Limit input to alphanumeric characters in any language in Recipe 4.8 and Limit the number of words in Recipe 4.9 solve some real-world problems using Unicode categories.
[2] You can find all the details in Unicode Standard Annex #29 at http://www.unicode.org/reports/tr29/. The “Graphemes and Normalization” section in Chapter 6 in the fourth edition of Programming Perl has more practical details on how to deal with Unicode graphemes in your software.