Day 2: Controlling Mutations

In Day 1, you learned about the hardened skeleton—the language features that make such an excellent general purpose language—and that the syntax, filled with sugar to simplify recurring idioms, is opinionated and rich. Even if Elixir were just a general-purpose language with no bells and whistles, it would be attractive.

Today, you’re going to learn to grow your own mutations, without radiation exposure. We’ll roll our own mini-language, building a state machine with Lisp-style macros. Before we do that, we’re going to have to lay a little more foundation. We’ll work with Mix to manage our application build process, and we’ll learn to use structs. Then, we’ll dive head first into macros.

Mix

The first examples in this chapter were all in the console. As our ideas got too big to express on a line or two, we used scripts. For the next few examples, we’re going to want to compile our code, which may be based on dependencies. We’ll use Mix for that task. If you’ve got Elixir, you’ve got Mix. If you’re ever lost, you can see what’s available by running mix help.

Mix is Elixir’s build tool, like make for C, rake for Ruby, or ant for Java. You’ll use mix to create a project with a uniform structure and to maintain your dependencies. Let’s create a new project. Navigate to a directory where you want your new project to be and type mix new states --sup (we pass --sup to generate a supervisor tree that we will need in Day 3):

Mix created a project called states in its own directory and a number of files underneath. Your tests will go in test, your source files will go in lib, and the file describing your application and dependencies will go in mix.exs. To make sure things are working, change into the states directory, compile the default application, and run tests, like this:

Mix compiled the file because it tracks dependencies between tasks. It also provides good support for your custom tasks.

Our tests are clean and green. Each . represents a test. Let’s put this new structure to use. We’re going to build a state machine.

From Concrete to Meta

Metaprogramming uses programs to write more sophisticated programs. In this section, we’re going to build a concrete state machine that will work fine but that might be difficult to reuse. Then, we’ll take that concrete implementation and use metaprogramming to morph it into something more abstract and flexible.

Sometimes, the best way to do metaprogramming is to build a simple tool that implements the code you want your metaprogramming platform to build. Said another way, if we want to build a generic state machine builder, we start by building a single state machine.

First, let’s review. Abstractly, a state machine is a graph where the nodes are states and the connections are events. Triggering an event moves the state machine from one state to the next. Concretely, think of a state machine as a set of rules that move from one state to another. We have to work with four pieces, then:

• Our state machine data. This will be some kind of data structure describing our state machine.

• Our state machine behavior. This will be a module with functions attached.

• Our application data. This will be some data structure with a state.

• Our application behavior.

For example, let’s write a state machine for an old-school brick-and-mortar video store. A state machine works well for us because:

• Videos in a store have concrete states to represent.

• The rules for transitioning between states are well defined.

• We might want to execute complex application logic when the video transitions from one state to another.

A state machine will help us organize code and manage change. Here’s the basic state machine for our simplistic old-school video store. It has three states: available, rented, and lost.

When a new video arrives on the shelves, it will go into the available state. When a customer rents a video, it goes to the rented state. When the customer returns a video, it will go back to available. Lose a video, and it goes to lost. At that point, let’s assume that our customer must buy the video, so there’s no return to any other state.

Let’s start with the data structure for a video first. We could use maps, but since videos will each have a fixed structure, we’ll use a data structure with a fixed number of named fields called structs.

Naming Fields with Structs

A struct is like a map with a fixed set of fields with the ability to attach behavior in the form of functions. We’re going to use a struct to represent our video. To keep things simple, assume there’s one video per title. We’ll represent each video with a state and a title. Place the following in states/lib/video.ex:

And take it for a spin:

I started the console via iex -S mix. Mix compiled the file and then loaded the application modules. Now, I can use the console in the context of my project.

I created a new struct with the constructor %Video{}. Struct syntax works like maps with the name of the struct between the % and {. Notice that I specified a title but picked up the default value for state.

Structs are just maps—create, update, and pattern match using the map syntax. Define them in modules, and include the functions that work on them.

Like everything else in Elixir, structs are immutable. You can create a new copy of a struct with one or more fields changed like this:

Structs are sometimes safer than maps because they will allow only keys you specify in the formal definition:

Now that we have a strategy for representing our application data, let’s move on to application behavior.

Creating Concrete Behavior

As we flesh out our video store, we’ll need three basic modules:

• VideoStore will have the implementation of business logic.

• VideoStore.Concrete will have the video store’s state machine, and state-machine behavior specific to that video store.

• StateMachine.Behavior will have generic state-machine behavior that we’ll reuse.

Our concrete video store will have application-specific behavior, lists that represent our state machine, and generic state-machine behaviors. First, a real video store will likely have some business logic that occurs whenever the video enters a new state. We’ll capture each of these in a function.

We’ll build something concrete, extract common ideas, and then generalize a state machine API. We’ll reuse the functions in VideoStore and StateMachine.Behavior. Eventually, we’ll look for patterns in VideoStore.Concrete that we can exploit with our state machine macros.

First, let’s build that business logic. Initially, we’ll build the code that will execute whenever a customer decides to rent, return, or lose one of our videos. Put the following in states/lib/video_store.ex:

For now, we’ll record interactions on a video much like a librarian would on a book’s library card. We’ll count the number of times a video is rented. You see just four functions: one to do the logging, and one for each of the events on our state machine.

Modeling the State Machine

So far, we have only created a video struct to preserve our state. Now, we’ll need to work out how the state machine works. Here’s how we’ll build it:

• The state machine will be a keyword list of the form [state_name: state].

• Each state will have a keyword list of events, keyed by event name, with each event having a name, the transition, and a possible list of callbacks.

The state machine is a straight-up keyword list of states, and the events are keyword lists as well. We don’t need to write any code, but we’re armed with what we’d like our API to look like.

We’ll be able to define a state machine like this:

We have three states: available, rented, and lost. Each has an associated list of events. Let’s write the functions that will do the bulk of the work.

Adding State-Machine Behavior

Our state machine will allow an application to fire an event on some struct with a state. Firing that event will transition the state machine to a new state, and perhaps fire some callbacks as well.

Put the following in states/lib/state_machine_behavior.ex. We’ll start with a function called fire that will fire an event, and another called activate that will invoke all of the user-defined functions associated with an event. Our goal is to add code that we’ll be able to use directly when we create our state machine macros. The functions are short and simple:

Since functional languages are immutable, a recurring pattern is to pass around some data structure using transforming functions that return new copies that evolve over the life of the program. We’ll call this evolving state the context. All of our state machine APIs will take a context struct, transform it, and pass the transformed version to the next function in the chain. One of the fields in our context will be state. This is what the functions do:

1. Our first job is to fire an event. It is easier to define that function in terms of an event and the context. The job of fire is straightforward: update the context with the new state, and then apply all of the functions provided in event.calls.

2. We provide an alternative API for convenience, one that does not require an event lookup. We just look up the event and call the other fire API.

3. The activate function chains together each function in activate as if it were a pipe. Each function takes the previous context, transforms it, and passes it to the next function. Enum.reduce provides this service. For the anonymous function &(&1.(&2)), &1 is a function from event.calls, and &2 is the result returned from the last function call (or simply context for the first call).

We could write a test of the functions so far, but the video store we’ll define in the next step will provide a far better testing opportunity. Let’s push on.

Looking for Patterns

The next step is to provide the pieces of the video store that implement one specific state machine. We’ve come to the point where we need to think a little bit about what we’ll use to represent our state machine itself. As I build a language, I prefer to rely on basic data structures, especially keyword lists, wherever possible. We’ll tweak that language a little later when we are ready to code individual macros. With that in mind, let’s look at the state machine features.

​①​

You can trigger an event on a state by simply calling a function. The context has the state name, and the function bodies have the state machine and the event names. This function is simply a convenience so that users of the class can access basic business functionality.

​②​

This function actually specifies the state machine as keyword lists. The outermost keyword list has pairs that represent {state_name, event_keyword_list}. The next level is a keyword list of events that looks like {event_name, event_metadata}. The innermost list has the event metadata that expresses the new state (to: new_state) and a list of functions that allow customized behavior as the states change (calls: [callback_functions] ).

Let’s pause to make a few observations. First, the specification of states is a little awkward because the data structure must do too much of the work. It would help us to have a macro expressing each individual state.

Second, you can see the duplication in the callback functions. We should try to create those from our macros. Keep those thoughts in the back of your mind as we build out the concrete version of the state machine. Right now, it’s time for some tests.

Writing Tests

I get nervous if I’m building something substantial and I get too far before writing a test. There’s not too much to test yet, but that will change quickly. Put the following code in states/test/concrete_test.exs:

This file looks a little different than traditional modules. First, you see the use macro that includes our macros. Also, the test looks different. You’d expect to see def of some kind at that level. test is actually a macro. A macro takes valid Elixir code and transforms it. In this case, the test macro is actually declaring a function with an API that we would find repetitive and awkward.

Macros actually get defined at an explicit point in compilation called macro expansion time. Their domain is the AST. To see what the syntax tree looks like, in the console use quote:

The quote command shows you the internal representation of Elixir code. Here, we’re using quote to look at the syntax tree. In fact, that’s exactly what the assert macro does. It looks at the quoted expression you pass in, so it can tell what comparison you’re using. For a failed comparison, assert will give you a rich message, like this:

In fact, every row in the AST is a three-tuple. It has an operator, some metadata, and an argument list. The second element has contextual metadata, and we’re not going to worry about it here. Focus on the first and last elements. This simple format is exactly what makes Elixir such a strong metaprogramming language.

Implementing should with Macros

Now, we’ll use quote to actually inject our own code. Since we’re just dealing with lists and tuples, it’s easy. I’m a fan of using “should” to describe test expectations. That means that every test will start with test "should..." but that syntax is repetitive. Let’s fix that. Let’s tell the Elixir compiler to look for the word should and replace it with our own macro.

The existing test macro takes a name and a do block. As a do block is just a keyword list, we’ll express it with options, and make our own macro that calls the test macro. Add the following to the top of states/test/test_helper.exs:

This code tells the compiler the following:

“As you are building the AST for this program, whenever you see the word should, replace it with everything inside the quote do block.”

Think of quote as diving one level deeper into the program.

Our program is an onion that has layers: programs are writing programs. When we’re inside this quote, we’re actually writing the code that will replace should.... This replacement happens in a pre-compile step called macro expansion time. We’re one level deeper into the onion. The problem is that by the time the code executes, the keyword name and the keyword options will both be undefined. Our test macro doesn’t understand name or options at all. It’s one level up. We have to go up and get it.

Think of unquote as climbing one level out of the onion that is your program.

When we use unquote, we climb up one level outside of the quote. We can now see everything that the should macro defines, including name and options. When we unquote those, we say to the compiler, “Add the value of name and the value of options as you find them, one level up.”

Now, we can change our tests to use the new structure, like this:

And run your tests. You’ll find them clean and green. A few lines of code, and we’ve streamlined the testing API. That’s the power of macros.

Writing More Tests

Make concrete_test.exs look like this:

Now, you can finally see why we’ve been working on a state machine. The should macro makes our tests easy to understand, and the state machine is a great abstraction for handling transitions of state in a functional system. Elixir’s pipe operator shows any users of this code exactly what’s happening.

We have successfully built out an application with a state machine. That’s not the point of this day, though. We’d like to allow anyone to plug in his or her own state machine, without duplicating effort.

We can’t put it off any longer. It’s time to tame the macro beast.

Writing a Complex Macro

If we were to attack a state machine without macros, we’d need to write many similar functions that looked almost alike. Macros will let us build function templates that declare those similar functions for us. There’s a cost, though. You need to be able to handle another level of complexity.

We’ll manage our macro just as we’d manage any other complex task. We’ll start with the entire problem and break it down into smaller function calls. First, let’s decide exactly how our macro should look. Here’s our target API:

You can already see how the macros will help us. Our users can break down the definition of a machine into clearly defined parts. Instead of using functions at runtime to do the work, we’ll use macros at compile time. Using this strategy, we can build templates that generate similar functions across many different applications. Using a little metadata, we’ll be able to generate all of the code that will let us manage a state machine for our application.

We have one main macro to build: the state macro. It will take a state name and a keyword list of events. We’ll need a few new tools as we build, but that will be no problem.

Understanding Compile-Time Flow

If thinking this way is a little troubling, understand this first. Elixir evaluates functions at execution time, whereas macros execute at compile time:

Simple enough. We’ve defined a one-line macro that will print “Executing…”. Now, let’s use it.

That “Executing…” message confirms macros run at compile time. Specifically, one of the compile steps is macro-expansion. Then, the compiler continues, with any expanded macro code injected into the compilation. When you see quote and unquote, don’t get confused. Those functions make it easy to reason about the code Elixir is injecting at compile time, no more and no less.

Building a Skeleton

Like Wolverine, we’ll start with a strong skeleton based on the simplest tasks.

You might have noticed that we had to fully qualify the macro print with TestMacro.print. That API would get tedious. Sure, we could solve this problem with an import directive, but we don’t want the consumers of our API to have to manually require dependencies for our macros. We can handle the imports. The magic directive for this purpose is use.

import makes a module available for consumption. require lets you use functions in a module as if they were scoped locally. Both are compile-time behaviors.

The use directive is different. The use macro will let us specify behavior that we want to happen when a user includes our module, before compile time. use will call the macro __using__. We’ll also stub out the other functions we’ll need. Here’s our initial skeleton, in states/lib/state_machine.ex:

So far, we’re just working with the mechanical devices of building a macro. We’re not actually doing much work yet. Still, let’s take a brief look at what’s going on. We’ll add detail and depth as we go.

​①​

A call to use StateMachine will trigger this __using__ macro. So far, we’re just importing the StateMachine API, so our consumers won’t have to type out StateMachine.state to invoke our macro.

​②​

This function will eventually specify the state machine as keyword lists. The outermost keyword list has pairs that represent {state_name, event_keyword_list}. The next level is a keyword list of events that looks like {event_name, event_metadata}. The innermost list has the event metadata that expresses the new state (to: new_state) and a list of functions that allow customized behavior as the states change (calls: [callback_functions] ). For now, we simply stub it out.

It’s too early to write any tests, but we can at least put our skeleton through its paces. Open the console with iex -S mix or compile the file from the console, and try it out

You can see that we’re on the right track because we get a diagnostic message for each state as Elixir compiles the module. Let’s work on the temporary variables that will hold the state. To do this, we’ll use module attributes.

Understanding Compile-time Flow, Part 2

As we put a little more flesh on the skeleton, we’ll also beef up your understanding of compile-time flow. In this section, we’re going to need module attributes, or compile-time variables expressed as @variable. At macro expansion time, Elixir does all of the following:

• An application’s module includes the macro’s module with use.

• The compiler executes the __using__ function, injecting some setup code.

• The compiler then processes the file, making any macro substitutions that it encounters.

• Individual macros may interact with module attributes to work together.

• The compiler then looks for the function identified by the @before_compile module attribute and executes it, potentially injecting more code.

Now, it’s time to see our full macro at work. Hide the annotations as you read through the code. See if you can identify each of these steps. This is the full macro:

​①​

Our __using__ function is finally complete. We’re initializing module attributes for a list of states so that the compose operator will work correctly. We also instruct the compiler to call __before_compile__ after macro substitution but before compilation.

​②​

We use simple list construction to add each state to the head of our list. Notice there’s no = sign. Module attribute assignment is not a match expression, as the runtime = would be.

​③​

Our __before_compile__ function is now doing the lion’s share of the work. We read the value of our module attributes into states. Next, we start with all states. We pipe to Keyword.values to get all of the events, pipe those to List.flatten so we have a uniform list of events, and pipe that flattened list to Keyword.keys to get a list of all names.

​④​

Finally, we inject some code. Since states was defined up one level, we must unquote it. This function is simple since we’re just returning the state machine structure.

​⑤​

Building the callbacks is a slightly bigger job, so we call a function to do that work. Since that function is defined one level up (as is events), we unquote them.

​⑥​

event_callback defines a single callback. Each callback defines a function that invokes fire. Using these, a consumer can easily fire state machine events through simple function calls.

Using Our State Machine

We finally get to see this skeleton dance. We can now tweak VideoStore to use our new version of the API. Let’s make a copy of it so you can study the old version and the new. Copy video_store.ex to vid_store.ex. The new file will be our dynamic state machine. We can just add this code to the top of VidStore, and the macro will do all of the work for us, creating everything that was in ConcreteVideoStore dynamically. Copy video_store.ex to vid_store.ex, and then change the top of vid_store.ex to look like this:

Ah. That’s much better. The state machine reads as cleanly as a book. Now, any application can make use of our state machine’s beautiful syntax. Take your tests for a spin to make sure they still pass.

While you’re at it, create a test called vid_store_test.exs that uses the VidStore API. Make sure all tests are green! Remember, you won’t need anything in the Concrete module because our state machine now creates all of that code dynamically.

We’re not done with this video store yet. We’ll build onto vid_store in Day 3 to make the file both distributed and concurrent using Elixir’s macros for Erlang’s OTP library.

What We Learned in Day 2

We packed a lot into Day 2. First, we learned to create projects with Mix. Then, we learned to create maps with named fields, called structs. We then built two versions of a state machine, a concrete one and a dynamic one. The second version used a generic state machine language that we created with Lisp-style macros.

We saw that Elixir expressions all reduce to the same structure: a three-tuple with a function name, metadata, and the function arguments. Our state machine macros used a wide range of tools:

• Our users consume our macros with the use command.

• Within our macro module, we imported our macro file and set up some variables within the __using__ macro.

• We introduced a state macro so our users could create a state.

• We used module attributes to compute a running list of states.

• We added __before_compile__ behavior to our macro to round out our macro.

When we were done, we had a unified strategy for conveniently and concisely representing a state machine. The Lisp-like syntax tree, quote and unquote, made it all possible, but the end result was syntax that was anything but Lisp-like.

Your Turn

In this set of exercises, you’re going to look at some existing Elixir modules and how they work. You’ll also try your hand at extending our macros.

Find…

• The elixir-pipes GitHub project. Look at how the macros improve the usage of pipes. Look at how pipe_with is implemented.

• The supported Elixir module attributes.

• A tutorial on Elixir-style metaprogramming.

• Elixir protocols. What do they do?

• function_exported? What does that function do? (You will need it for one of the next problems.)

Do (Easy):

• Add a find state to the state machine that transitions from lost to found. Add this code in both the concrete and abstract versions of your state machine. Which is easier, and why?

Do (Medium):

• Write tests for VidStore. What was different, and what was the same?

Do (Hard):

• Add before_(event_name) and after_(event_name) hooks. If those functions exist, make sure fire executes them.

• Add a protocol to our state machine that forces a state machine struct to implement the state field.