Exploring Panflute

I have broken the header classes over multiple lines; Pandoc doesn’t mind.

In the print_structure function , I ignore all document elements that are not Header—simply because I only do anything in case the element is a header element. What I do is that I print the identifier, the classes, and the attributes of a header. I print them to standard error—if I printed them to standard out, I would mess up the JSON output that makes a filter work.

The identifier is first in the curly brackets, and it starts with #. The classes begin with a dot and otherwise standalone, and the attributes are key-value mappings. The different document elements have different properties but panflute is well documented, and you can find all the attributes each document element has.

The print_structure function doesn’t explicitly return any values which means that it implicitly returns the None object. The panflute module will interpret that as saying that the function does not want to make any transformations but leave elem as it is. If we wanted to change anything, we must return an element to replace the input elem.

That classes and attributes are not used much in the output here does mean that they are useless. You just need to find a use for them yourself. We can abuse them for our own nefarious purposes in our own filters.

Conditional Inclusion of Exercise Solutions

Consider this example from the previous chapter: we have text with exercises and solutions, and we want to compile it into documents where the solutions have been removed and documents where they have not. The preprocessing solution is excellent, but now we can see how we can achieve the same thing using a filter.

The ::: SOLUTION ::: syntax is not one we have seen before because it is relatively rare. It creates a div tag in HTML—you can use it with a CSS file for formatting—and it adds a hyper reference target in LaTeX. We want neither of that, but we can use a Div structure in our filter.

The first solution here has the same syntax as before. It will set the class of the Div object to SOLUTION and leave the other two properties empty. The second solution sets an identifier and adds another class to the list. The third solution is back to a single class but adds one key to value mapping to the attributes.

We can use this information to process the example. We want to include or exclude solutions based on metadata. We could not easily do that in the preprocessor, but there we could use preprocessor variables which are harder to do here. Each method has its pros and cons.

The preceding information tells us that 'SOLUTION' will be in the class of the Div if we have an ::: SOLUTION ::: block (or with the alternative syntax an ::: { .SOLUTION }" block). We want to check the metadata to see if we should include the solutions. We can get the meta information from the doc parameter that run_filter will give our filter function. If we call doc.get_metadata() , we will get a table from which we can get metavariables. We want our filter to remove Div objects that have SOLUTION as their class unless there is a metavariable called solutions and it is true.

First, we check if 'SOLUTION' is in the classes. If not, then we don’t have a solution block and we leave the element alone by returning None. We could also have returned elem; it would make no difference. Otherwise, we get hold of the metadata. We check if "solutions" is in the metadata. If it is not in the metadata, then it definitely cannot be true, so we return Null as a replacement for elem in the output. Notice that it is Null and not None! The former is an element in Pandoc, while the latter is an element in Python; the former replaces elem with an empty block, effectively removing the solutions block, while the latter keeps elem as it is, that is, leaves the solutions block in the output. Finally, if "solutions" is in the metadata but not true, then we also remove the solutions block. The solutions metavariable can have more than true as a value, and those would be considered as true as well in the expression meta["solutions"], so I explicitly check for True.

If we do not get past the checks, we remove the solutions block. If the “solutions” is not set in the metadata, or if it is set to anything but True, we remove the block. Otherwise, we keep it by returning None.

If the block is not a solutions block, if "solutions" is not set or set to anything except true, then we have a solutions block, and we are supposed to keep it.

Pandoc uses this syntax in its output, but you can use either of the two ways to define a Div block.

Conditional Inclusions Based on Format

Sometimes we want a different text in the output conditional on the output format, for example, a different text for HTML and LaTeX.

In either case, the quoted text is only inserted if the output format matches. We saw this syntax back in Chapter 6.

The text we include here will not be processed by Pandoc, so you cannot use Pandoc’s features. That means that you cannot, for example, use the [text](link) syntax but must the hyperlinks in HTML. We will write a filter that allows us to do this.

We will use two or more classes. The first is used to tag that it is text that should only be output for some formats and another to indicate which output we want to output the text for.

For blocks of text, we have to capture Div objects, and for an inline text, we need to catch Span objects. In either case, we need to check if we have the out class. Otherwise, we leave the object alone—it could be used for something else in another filter. If we have the out class, we check if the output format is also a class. If not, we remove the object; if the format is a class, then we include it.

When we return the elements, we want to keep we do not just return elem. We don’t necessarily want to have the Span and Div show up in the output. Instead, we get the object’s contents. The script wants this as a list, so we use elem.content.list .

There is no formatting here because there is no formatting in the input.

If you use another output format, for example, Markdown, then you will not get any output with this input; all the text is only included for HTML and LaTeX.

Evaluating Code

Now take another example we considered in the previous chapter : running Python code while formatting a document and inserting the results.

The filter is straightforward. Ignore for now the run_python we call—I list it in the following text, but it is not important how it works. Focus on eval_python . I realize that I have not been that inventive with the names, but run_python_process executes a Python process that evaluates a code block, while eval_python is the filter.

The execute_code function is slightly more complicated than the way we used exec in the preprocessor. It is simpler to use exec and let it print its output in the preprocessor compared to evaluating the code in a filter where we do not want any unwanted output.

If exec writes something to standard out, it will break the JSON format and this will break the rest of the pipeline.

You do not need to understand this part of the filter to understand how the filter itself works.

The output is not in a “backtick”-block but indented. This is just another way to write the same in Markdown.

Numbering Exercises

As a final example , let us return to the exercise examples. This time, we are not concerned with including or excluding the solutions, but we want to put exercises in a LaTeX environment when the output is LaTeX and otherwise number them and add a header.

We have two Div blocks with class Exercise and some text within them. Those are the ones we want to modify. For HTML, say, we want to give them a header, and for LaTeX, we want to put them inside a LaTeX environment.

We can get the output format from the doc object . The input and output of filters are, as mentioned earlier, JSON, and if we just used shell pipes, we couldn’t know what the final output is. When we run a script as a filter, however, Pandoc knows what the final output will be, and we can get that information.

We use a global variable, no_exercise (line 3), for increasing the header number for each exercise. Inside the filter, we first check if we have a Div block with an Exercise class. If so, we get hold of the meta object from the doc element (line 10) and use it to check if its output format is LaTeX or HTML. If it is LaTeX (line 12), then we get the metavariable exercise_env (with exercises as default), and we create a new block as a replacement for the Div block.

The RawBlock is just verbatim text but only inserted if the output format is “latex.” Of course, we know that the output is LaTeX here and we could leave out the argument, but in other cases, a RawBlock can be useful when you output roughly the same text for all output formats and do not want to check for the output format.

For all other formats (line 24 and below), we use a default level of 1 and otherwise use the metavariable exercise_header_level (lines 26 and 27). We create the header text (lines 29–31), increment the no_exercise variable (line 32), and then create the Header element. Its first argument is the list of text object that should comprise the header, then the header level, and keep the classes from the Div block . We put the elem text after the header.

As you can see, we have added a header to the exercises.

Here we do not create a header but put the exercises into an exercises environment . Since we do not add headers, the header level is ignored.

in your template before egin{document}.

It gives the header the elements identifier and sets the elements identifier to the empty string, which means that it will not be inserted in the output.

We get the identifier for the Div argument —we have seen identifiers earlier—and then we insert it after the egin command . If there is no identifier, we insert the empty string.

Now we have labels we can use with ef{} commands in LaTeX (and we can insert those conditional on the output format) and we can insert links for other formats (dependent on those). LaTeX will automatically number the environments (if you have the LaTeX magic listed earlier to define the environment type), and it will automatically insert their reference number. For other formats, you need to insert the numbers yourself in the link.

and we need to run our filter before pandoc-citeproc for the same reason that pandoc-crossref must. We don’t want to interfere with references handled by these two filters, so we will give them a

Before we can handle references, however, we want to collect a map from identifiers to exercise numbers. LaTeX will handle this for environments and ef{} commands but for other formats we must. Since we are not guaranteed that we see an exercise before we reference it, we must traverse the entire document and make the map before we traverse it again and modify the document.

The easiest way to traverse the document is with the run_filter function , but we cannot run it more than once. There is another function, run_filters—notice the plural—that handles that. One filter will provide the input to the next; we don’t want to modify anything with the filter that collects the map, so we just let it return None (implicitly by not returning another value).

When we number the exercises, in the filter that modifies the document, we need to reset no_exercise . This is not easy when the two scripts are called one after another, but a straightforward solution is to use a second counter. If you do this, then the preceding number_exercises filter will work as before. It adds the numbers, but it didn’t need the map earlier, and it doesn’t need it now.

Instead, we will write a third filter that does this; let us call it handle_citations . I describe this function in the following text.

We must run collect_numbers before handle_citations, but number_exercises can go anywhere in the list.

If you get the native format for a file that contains these [@ref] references, you get a complex text, but you will see that we have a Cite object that contains a Citation element (there can be more than one, but we will only handle one here). We want to translate these elements. We really want to work with Citation, but if we filter on that, we will create an object that goes into the Cite that encloses it, so we will handle Cite objects and extract the Citation object from it.

In the first line in the filter, we check if we have a Cite element. There is nothing new there. Then we extract the element we are actually interested in, which is a Citation element.

We get the identifier, which is the citation label (line 4), and check if it is an example label, that is, starts with "ex:" (line 5). If it is not, we return the element; we do not want to modify other citation objects since these could be used by other filters.

Now we extract the prefix of the Citation object ; again we get the actual content using .list. We add the text "exercise" to the prefix (line 9), so the references will contain this text as well. We need to add a space as well to prevent the prefix from being concatenated with the "exercise" text. We also need to add a space after "exercise", but I want a non-breaking space in the LaTeX output, so there I want a tilde rather than a space (see the following text).

I am assuming that there is already a space before the reference, that is, that the reference looks like this [see @ex:ref] rather than [see@ex:ref]; if not we need to add a space before "exercise". We will need to add it after the reference. Otherwise, the prefix and the reference number will be concatenated.

We now handle LaTeX output separately (lines 11 to 15). We append the ef{} command to the prefix and return the result. We put the LaTeX code in a RawInline object. This is similar to a RawBlock object but for inline text. We do not add a space here but a tilde.

For other output formats, lines 17 to 25, we look up the exercise number from our map, assuming that there is an identifier, and then we add a space and the number to the prefix. We add a space before the number, so it isn’t concatenated to "exercise". Finally, we create a link from the reference.

Exercises