Create one regular expression to match all common
misspellings of calendar, so you can find this word in a
document without having to trust the author’s spelling ability. Allow an
a or
e to be used
in each of the vowel positions. Create another regular expression to
match a single hexadecimal character. Create a third regex to match a
single character that is not a hexadecimal character.
The problems in this recipe are used to explain an important and commonly used regex construct called a character class.
The notation using square brackets is called a
character class. A character class matches a
single character out of a list of possible characters. The three classes
in the first regex match either an a
or an e. They do so independently.
When you test calendar against this regex, the first
character class matches a, the second e, and the third a.
Inside a character class, only four characters have a special
function: , ^, -, and ]. If you’re using Java or .NET, the opening
bracket [ is also a
metacharacter inside character classes.
A backslash always escapes the character that follows it,
just as it does outside character classes. The escaped character can be
a single character, or the start or end of a range. The other four
metacharacters get their special meanings only when they’re placed in a
certain position. It is possible to include them as literal characters
in a character class without escaping them, by positioning them in a way
that they don’t get their special meaning. ‹[][^-]› pulls off this trick. This works with all
flavors in this book, except JavaScript. JavaScript treats ‹[]› as an
empty character class that always fails to match. But we recommend that
you always escape these metacharacters, so the previous regex should be
‹[][^-]›. Escaping the metacharacters makes
your regular expression easier to understand.
All other characters are literals and simply add themselves to the
character class. The regular expression ‹[$()*+.?{|]› matches any one of the nine
characters between the square brackets. These nine characters only have
special meanings outside character classes. Inside character classes
they are just literal text. Escaping them would only make your regular
expression harder to read.
Alphanumeric characters cannot be escaped with a backslash. Doing
so may be an error or may create a regular expression token (something
with a special meaning in a regular expression). In our discussions of
certain other regex tokens, such as in Recipe 2.2, we mention that they can be used inside
character classes. All these tokens consist of a backslash and a letter,
sometimes followed by a bunch of other characters. Thus, ‹[
]› matches a carriage return
(
) or line feed (
).
A caret (^) negates the
character class if you place it immediately after the opening bracket.
It makes the character class match any character that is
not in the list.
Caution
In all the regex flavors discussed in this book, a negated character class matches line break characters, unless you add them to the negated character class. Make sure that you don’t accidentally allow your regex to span across lines.
A hyphen (-) creates a
range when it is placed between two characters.
The range includes the character before the hyphen, the character after
the hyphen, and all characters that lie between them in numerical order.
To know which characters those are, you have to look at the ASCII or
Unicode character table. ‹[A-z]› includes all
characters in the ASCII table between the uppercase A and the lowercase z. The range includes some punctuation, so
‹[A-Z[\]^_`a-z]›
matches the same characters more explicitly. We recommend that you
create ranges only between two digits or between two letters that are
both upper- or lowercase.
Tip
Reversed ranges, such as ‹[z-a]›, are not permitted.
Six regex tokens that consist of a backslash and a
letter form shorthand character classes:
‹d›,
‹D›,
‹w›,
‹W›,
‹s› and
‹S›. You
can use these both inside and outside character classes. Each
lowercase shorthand character has an associated uppercase shorthand
character with the opposite meaning.
‹d› and ‹[d]› both match a single digit.
‹D›
matches any character that is not a digit, and is
equivalent to ‹[^d]›.
Here is how we can use the ‹d› shorthand to rewrite the “hexadecimal
character” regex from earlier in this recipe:
[a-fA-Fd]
| Regex options: None |
| Regex flavors: .NET, Java, PCRE, Perl, Python, Ruby |
‹w›
matches a single word character. A word
character is a character that can occur as part of a word. That
includes letters, digits, and the underscore. The particular choice of
characters here may seem odd, but it was chosen because these are the
characters that are typically allowed in identifiers in programming
languages. ‹W› matches
any character that is not part of such a propellerhead word.
In Java 4 to 6, JavaScript, PCRE, and Ruby, ‹w› is always identical to
‹[a-zA-Z0-9_]›. In .NET,
it includes letters and digits from all other scripts (Cyrillic, Thai,
etc.). In Java 7, the other scripts are included only if you set the
UNICODE_CHARACTER_CLASS flag. In
Python 2.x, the other scripts are included only if you pass the
UNICODE or U flag when creating the regex. In Python
3.x the other scripts are included by default, but you can make
‹w› ASCII-only with the
ASCII or A flag. In Perl 5.14, the /a (ASCII) flag makes
‹w› identical to
‹[a-zA-Z0-9_]›, while
/u (Unicode) adds all
Unicode scripts, and /l (locale) makes
‹w› depend on the
locale. Prior to Perl 5.14, or when using /d (default) or none
of the /adlu flags in Perl 5.14,
‹w› automatically
includes Unicode scripts if the subject string or the regex are
encoded as UTF-8, or the regex includes a code point above 255 such as
‹x{100}› or a Unicode
property such as ‹p{L}›.
If not, the default for ‹w› is pure ASCII.
‹d› follows the
same rules as ‹w› in
all these flavors. In .NET, digits from other scripts are always
included. In Python it depends on the UNICODE and ASCII flags, and whether you’re using Python
2.x or 3.x. In Perl 5.14, it depends on the /adlu flags. In earlier versions of Perl, it
depends on the encoding of the subject and regex, and whether the
regex has any Uncicode tokens.
‹s›
matches any whitespace character. This includes
spaces, tabs, and line breaks. ‹S› matches any character not matched by
‹s› In .NET and
JavaScript, ‹s› also
matches any character defined as whitespace by the Unicode standard.
In Java, Perl, and Python, ‹s› follows the same rules as ‹w› and ‹d›.
Notice that JavaScript uses Unicode for ‹s› but ASCII for ‹d› and ‹w›. Further inconsistency arises when we add
‹› to the mix.
‹› is not a shorthand
character class, but a word boundary. Though
you’d expect ‹› to
support Unicode when ‹w› does and to be ASCII-only when ‹w› is ASCII-only, this isn’t
always the case. The subsection Word Characters in Recipe 2.6 has the details.
(?i)[A-F0-9]
| Regex options: None |
| Regex flavors: .NET, Java, XRegExp, PCRE, Perl, Python, Ruby |
(?i)[^A-F0-9]
| Regex options: None |
| Regex flavors: .NET, Java, XRegExp, PCRE, Perl, Python, Ruby |
Case insensitivity, whether set with an external flag (see Recipe 3.4) or a mode modifier inside the regex (see Case-insensitive matching in Recipe 2.1), also affects character classes. The two regexes just shown are equivalent to the ones in the original solution.
JavaScript follows the same rule, but it doesn’t support
‹(?i)›. To
make a regular expression case-insensitive in JavaScript, set the
/i flag when creating
it. Or use the XRegExp library for JavaScript, which adds support for
mode modifiers at the start of the regex.
[a-zA-Z0-9-[g-zG-Z]]
| Regex options: None |
| Regex flavors: .NET 2.0 or later |
This regular expression matches a single hexadecimal
character, but in a roundabout way. The base character class matches
any alphanumeric character, and a nested class then subtracts the
letters g through z. This nested class must appear at the end
of the base class, preceded by a hyphen, as shown here: ‹[›.class-[subtract]]
Character class subtraction is
particularly useful when working with Unicode categories, blocks, and
scripts. As an example, ‹p{IsThai}› matches any character in the Thai
block. ‹P{N}›
matches any character that is not in the Number category. Combining
them with subtraction, ‹[p{IsThai}-[P{N}]]› matches any of the 10 Thai
digits using character class
subtraction. Recipe 2.7 has all the details
on working with Unicode properties.
Java allows one character class to be nested inside
another. If the nested class is included directly, the resulting class
is the union of the two. You can nest as many
classes as you like. The regexes ‹[a-f[A-F][0-9]]› and ‹[a-f[A-F[0-9]]]› use character class union. They
match a hexadecimal digit just like the original regex without the
extra square brackets.
The regex ‹[w&&[a-fA-F0-9s]]› uses character
class intersection to match a hexadecimal
digit. It could win a prize in a regex obfuscation contest. The base
character class ‹[w]›
matches any word character. The nested class ‹[a-fA-F0-9s]› matches any hexadecimal digit and
any whitespace character. The resulting class is the intersection of
the two, matching hexadecimal digits and nothing else. Because the
base class does not match whitespace and the nested class does not
match ‹[g-zG-Z_]›, those
are dropped from the final character class, leaving only the
hexadecimal digits.
‹[a-zA-Z0-9&&[^g-zG-Z]]› uses character
class subtraction to match a single hexadecimal
character in a roundabout way. The base character class ‹[a-zA-Z0-9]› matches any
alphanumeric character. The nested class ‹[^g-zG-Z]› then subtracts the letters g through z. This nested class must be a negated
character class, preceded by two ampersands, as shown here: ‹[›.class&&[^subtract]]
Character class intersection and subtraction are particularly
useful when working with Unicode properties, blocks, and scripts.
Thus, ‹p{InThai}›
matches any character in the Thai block, whereas ‹p{N}›
matches any character that is in the Number category. In consequence,
‹[p{InThai}&&[p{N}]]› matches any of
the 10 Thai digits using character class intersection.
If you’re wondering about the subtle differences in the
‹p› regex
tokens, you’ll find those all explained in Recipe 2.7. Recipe 2.7 has
all the details on working with Unicode properties.
Recipe 2.2 explains how to match nonprinting characters. Recipe 2.7 explains how to match Unicode characters. You can use the syntax for nonprinting and Unicode characters inside character classes.
Bat, cat, or rat in Recipe 5.3 describes some common character class mistakes made by people who are new to regular expressions.