You want to find all the matches of a particular regular expression, but only within certain sections of the subject string. Another regular expression matches each of the sections in the string.

Suppose you have an HTML file in which various passages are marked as bold with <b> tags. You want to find all numbers marked as bold. If some bold text contains multiple numbers, you want to match all of them separately. For example, when processing the string 1 <b>2</b> 3 4 <b>5 6 7</b>, you want to find four matches: 2, 5, 6, and 7.

Regular expressions are well suited for tokenizing input, but they are not well suited for parsing input. Tokenizing means to identify different parts of a string, such as numbers, words, symbols, tags, comments, etc. It involves scanning the text from left to right, trying different alternatives and quantities of characters to be matched. Regular expressions handle this very well.

Parsing means to process the relationship between those tokens. For example, in a programming language, combinations of such tokens form statements, functions, classes, namespaces, etc. Keeping track of the meaning of the tokens within the larger context of the input is best left to procedural code. In particular, regular expressions cannot keep track of nonlinear context, such as nested constructs.^[6]

Trying to find one kind of token within another kind of token is a task that people commonly try to tackle with regular expressions. A pair of HTML bold tags is easily matched with the regular expression ‹<b>(.*?)</b>›.^[7] A number is even more easily matched with the regex ‹d+›. But if you try to combine these into a single regex, you’ll end up with something rather different:

d+(?=(?:(?!<b>).)*</b>)

Though the regular expression just shown is a solution to the problem posed by this recipe, it is hardly intuitive. Even a regular expression expert will have to carefully scrutinize the regex to determine what it does, or perhaps resort to a tool to highlight the matches. And this is the combination of just two simple regexes.

A better solution is to keep the two regular expressions as they are and use procedural code to combine them. The resulting code, while a bit longer, is much easier to understand and maintain, and creating simple code is the reason for using regular expressions in the first place. A regex such as ‹<b>(.*?)</b>› is easy to understand by anyone with a modicum of regex experience, and quickly does what would otherwise take many more lines of code that are harder to maintain.

Though the solutions for this recipe are some of the most complex ones in this chapter, they’re very straightforward. Two regular expressions are used. The “outer” regular expression matches the HTML bold tags and the text between them, and the text in between is captured by the first capturing group. This regular expression is implemented with the same code shown in Recipe 3.11. The only difference is that the placeholder comment saying where to use the match has been replaced with code that lets the “inner” regular expression do its job.

The second regular expression matches a digit. This regex is implemented with the same code as shown in Recipe 3.10. The only difference is that instead of processing the subject string entirely, the second regex is applied only to the part of the subject string matched by the first capturing group of the outer regular expression.

There are two ways to restrict the inner regular expressions to the text matched by (a capturing group of) the outer regular expressions. Some languages provide a function that allows the regular expression to be applied to part of a string. That can save an extra string copy if the match function doesn’t automatically fill a structure with the text matched by the capturing groups. We can always simply retrieve the substring matched by the capturing group and apply the inner regex to that.

Either way, using two regular expressions together in a loop will be faster than using the one regular expression with its nested lookahead groups. The latter requires the regex engine to do a whole lot of backtracking. On large files, using just one regex will be much slower, as it needs to determine the section boundaries (HTML bold tags) for each number in the subject string, including numbers that are not between <b> tags. The solution that uses two regular expressions doesn’t even begin to look for numbers until it has found the section boundaries, which it does in linear time.

The XRegExp library for JavaScript has a special matchChain() method that is specifically designed to get the matches of one regex within the matches of another regex. This method takes an array of regexes as its second parameter. You can add as many regexes to the array as you want. You can find the matches of a regex within the matches of another regex, within the matches of other regexes, as many levels deep as you want. This recipe only uses two regexes, so our array only needs two elements. If you want the next regex to search within the text matched by a particular capturing group of a regex, add that regex as an object to the array. The object should have a regex property with the regular expression, and a backref property with the name or number of the capturing group. If you specify the last regex in the array as an object with a regex and a backref property, then the returned array will contain the matches of that capturing group in the final regex.

This recipe uses techniques introduced by three earlier recipes. Recipe 3.8 shows code to determine the position and length of the match. Recipe 3.10 shows code to get a list of all the matches a regex can find in a string. Recipe 3.11 shows code to iterate over all the matches a regex can find in a string.