The four kinds of lookaround groups supported by modern regex flavors have the special property of giving up the text matched by the part of the regex inside the lookaround. Essentially, lookaround checks whether certain text can be matched without actually matching it.

Lookaround that looks backward is called lookbehind. This is the only regular expression construct that will traverse the text from right to left instead of from left to right. The syntax for positive lookbehind is ‹(?<=⋯)›. The four characters ‹(?<=› form the opening bracket. What you can put inside the lookbehind, here represented by ‹⋯›, varies among regular expression flavors. But simple literal text, such as ‹(?<=<b>)›, always works.

Lookbehind checks to see whether the text inside the lookbehind occurs immediately to the left of the position that the regular expression engine has reached. If you match ‹(?<=<b>)› against My <b>cat</b> is furry, the lookbehind will fail to match until the regular expression starts the match attempt at the letter c in the subject. The regex engine then enters the lookbehind group, telling it to look to the left. ‹<b>› matches to the left of c. The engine exits the lookbehind at this point, and discards any text matched by the lookbehind from the match attempt. In other words, the match-in-progress is back at where it was when the engine entered the lookbehind. In this case, the match-in-progress is the zero-length match before the c in the subject string. The lookbehind only tests or asserts that ‹<b>› can be matched; it does not actually match it. Lookaround constructs are therefore called zero-length assertions.

After the lookbehind has matched, the shorthand character class ‹w+› attempts to match one or more word characters. It matches cat. The ‹w+› is not inside any kind of lookaround or group, and so it matches the text cat normally. We say that ‹w+› matches and consumes cat, whereas lookaround can match something but can never consume anything.

Lookaround that looks forward, in the same direction that the regular expression normally traverses the text, is called lookahead. Lookahead is equally supported by all regex flavors in this book. The syntax for positive lookahead is ‹(?=⋯)›. The three characters ‹(?=› form the opening bracket of the group. Everything you can use in a regular expression can be used inside lookahead, here represented by ‹⋯›.

When the ‹w+› in ‹(?<=<b>)w+(?=</b>)› has matched cat in My <b>cat</b> is furry, the regex engine enters the lookahead. The only special behavior for the lookahead at this point is that the regex engine remembers which part of the text it has matched so far, associating it with the lookahead. ‹</b>› is then matched normally. Now the regex engine exits the lookahead. The regex inside the lookahead matches, so the lookahead itself matches. The regex engine discards the text matched by the lookahead, by restoring the match-in-progress it remembered when entering the lookahead. Our overall match-in-progress is back at cat. Since this is also the end of our regular expression, cat becomes the final match result.

‹(?!⋯)›, with an exclamation point instead of an equals sign, is negative lookahead. Negative lookahead works just like positive lookahead, except that whereas positive lookahead matches when the regex inside the lookahead matches, negative lookahead matches when the regex inside the lookahead fails to match.

The matching process is exactly the same. The engine saves the match-in-progress when entering the negative lookahead, and attempts to match the regex inside the lookahead normally. If the sub-regex matches, the lookahead fails, and the regex engine backtracks. If the sub-regex fails to match, the engine restores the match-in-process and proceeds with the remainder of the regex.

Similarly, (?<!⋯) is negative lookbehind. Negative lookbehind matches when none of the alternatives inside the lookbehind can be found looking backward from the position the regex has reached in the subject text.

Lookahead is easy. All regex flavors discussed in this book allow you to put a complete regular expression inside the lookahead. Everything you can use in a regular expression can be used inside lookahead. You can even nest other lookahead and lookbehind groups inside lookahead. Your brain might get into a twist, but the regex engine will handle everything nicely.

Lookbehind is a different story. Regular expression software has always been designed to search the text from left to right only. Searching backward is often implemented as a bit of a hack: the regex engine determines how many characters you put inside the lookbehind, jumps back that many characters, and then compares the text in the lookbehind with the text in the subject from left to right.

For this reason, the earliest implementations allowed only fixed-length literal text inside lookbehind. Perl and Python still require lookbehind to have a fixed length, but they do allow fixed-length regex tokens such as character classes, and allow alternation as long as all alternatives match the same number of characters.

PCRE and Ruby 1.9 take this one step further. They allow alternatives of different lengths inside lookbehind, as long as the length of each alternative is constant. They can handle something like ‹(?<=one|two|three|forty-two|gr[ae]y)›, but nothing more complex than that.

Internally, PCRE and Ruby 1.9 expand this into six lookbehind tests. First, they jump back three characters to test ‹one|two›, then four characters to test ‹gray|grey›, then five to test ‹three›, and finally nine to test ‹forty-two›.

Java takes lookbehind one step further. Java allows any finite-length regular expression inside lookbehind. This means you can use anything except the infinite quantifiers ‹*›, ‹+›, and ‹{42,}› inside lookbehind. Internally, Java’s regex engine calculates the minimum and maximum length of the text that could possibly be matched by the part of the regex in the lookbehind. It then jumps back the minimum number of characters, and applies the regex in the lookbehind from left to right. If this fails, the engine jumps back one more character and tries again, until either the lookbehind matches or the maximum number of characters has been tried.

If all this sounds rather inefficient, it is. Lookbehind is very convenient, but it won’t break any speed records. Later, we present a solution for JavaScript and Ruby 1.8, which don’t support lookbehind at all. This solution is actually far more efficient than using lookbehind.

The regular expression engine in the .NET Framework is the only one in the world^[5] that can actually apply a full regular expression from right to left. .NET allows you to use anything inside lookbehind, and it will actually apply the regular expression from right to left. Both the regular expression inside the lookbehind and the subject text are scanned from right to left.

If you use lookbehind at the start of the regex or lookahead at the end of the regex, the net effect is that you’re requiring something to appear before or after the regex match, without including it in the match. If you use lookaround in the middle of your regular expression, you can apply multiple tests to the same text.

In Flavor-Specific Features (a subsection of Recipe 2.3), we showed how to use character class subtraction to match a Thai digit. Only .NET and Java support character class subtraction.

A character is a Thai digit if it is both a Thai character (any sort) and a digit (any script). With lookahead, you can test both requirements on the same character:

(?=p{Thai})p{N}

This regex works only with the three flavors that support Unicode scripts, as we explain in Recipe 2.7. But the principle of using lookahead to match the same character more than once works with all flavors discussed in this book.

When the regular expression engine searches for ‹(?=p{Thai})p{N}›, it starts by entering the lookahead at each position in the string where it begins a match attempt. If the character at that position is not in the Thai script (i.e., ‹p{Thai}› fails to match), the lookahead fails. This causes the whole match attempt to fail, forcing the regex engine to start over at the next character.

When the regex reaches a Thai character, ‹p{Thai}› matches. Thus, the ‹(?=p{Thai})› lookaround matches, too. As the engine exits the lookaround, it restores the match-in-progress. In this case, that’s the zero-length match before the character just found to be Thai. Next up is ‹p{N}›. Because the lookahead discarded its match, ‹p{N}› is compared with the same character that ‹p{Thai}› already matched. If this character has the Unicode property Number, ‹p{N}› matches. Since ‹p{N}› is not inside a lookaround, it consumes the character, and we have found our Thai digit.

When the regular expression engine exits a lookaround group, it discards the text matched by the lookaround. Because the text is discarded, any backtracking positions remembered by alternation or quantifiers inside the lookaround are also discarded. This effectively makes lookahead and lookbehind atomic. Recipe 2.14 explains atomic groups in detail.

In most situations, the atomic nature of lookaround is irrelevant. A lookaround is merely an assertion to check whether the regex inside the lookaround matches or fails. How many different ways it can match is irrelevant, as it does not consume any part of the subject text.

The atomic nature comes into play only when you use capturing groups inside lookahead (and lookbehind, if your regex flavor allows you to). While the lookahead does not consume any text, the regex engine will remember which part of the text was matched by any capturing groups inside the lookahead. If the lookahead is at the end of the regex, you will indeed end up with capturing groups that match text not matched by the regular expression itself. If the lookahead is in the middle of the regex, you can end up with capturing groups that match overlapping parts of the subject text.

The only situation in which the atomic nature of lookaround can alter the overall regex match is when you use a backreference outside the lookaround to a capturing group created inside the lookaround. Consider this regular expression:

(?=(d+))w+1

At first glance, you may think that this regex would match 123x12. ‹d+› would capture 12 into the first capturing group, then ‹w+› would match 3x, and finally ‹1› would match 12 again.

But that never happens. The regular expression enters the lookaround and the capturing group. The greedy ‹d+› matches 123. This match is stored into the first capturing group. The engine then exits the lookahead, resetting the match-in-progress to the start of the string, discarding the backtracking positions remembered by the greedy plus but keeping the 123 stored in the first capturing group.

Now, the greedy ‹w+› is attempted at the start of the string. It eats up 123x12. ‹1›, which references 123, fails at the end of the string. ‹w+› backtracks one character. ‹1› fails again. ‹w+› keeps backtracking until it has given up everything except the first 1 in the subject. ‹1› also fails to match after the first 1.

The final 12 would match ‹1› if the regex engine could return to the lookahead and give up 123 in favor of 12, but the regex engine doesn’t do that.

The regex engine has no further backtracking positions to go to. ‹w+› backtracked all the way, and the lookaround forced ‹d+› to give up its backtracking positions. The match attempt fails.