Day 2: Painting the Fence

Today, we will move our exploration of Factor from the Listener to source files. We’ll learn how to define words, organize them into modules called vocabularies, run them as standalone programs, and test them with unit tests.

Defining Words

We’ve been using Factor words that are defined in the library. Let’s see how we can define our own words. A word definition starts with a colon, a space, and the name of the word. Then comes the stack effect, the code for the word, and finally a space and a semicolon. For example:

That defines a word which adds 42 to the number that’s on the stack. The ( x -- y ) part is the stack effect: the number of values that the word takes from the stack and pushes back onto the stack, on the left and right side of the --, respectively. You can see examples of stack effects by typing a word in the Listener and looking at the declaration appear at the bottom of the window. Try typing +, you should see the following show up:

The stack effect of + shows that it takes two values from the stack and pushes one value back. Words are allowed to take no value from the stack or push no value back. For example, try looking at the stack effect for print and read1.

When you look at the stack effect for a word, notice that the symbols such as x and y are not variable names; they are not used in the code for the word. Instead, they are just symbols, indicating the number of values. Although the names are arbitrary, there are some naming conventions for the stack effect, such as str, obj, seq, elt, and so on, to make the definition more descriptive.

Here is a word that takes a sequence of numbers and returns their sum:

I’ll say it again: don’t forget the spaces! You need spaces after the colon, around the parentheses, around the double-dash, and before the semicolon. Forgetting those spaces is an easy mistake to make when coming to Factor from another programming language.

Returning Multiple Values

We’ve seen words that use multiple values from the stack, but what about returning more than one value? In most languages, this cannot be done without artificially returning multiple values by stuffing them into a single collection. The caller must then unpack the collection to get the multiple values back.

Not so in Factor:

Simply by declaring multiple names on the right-hand side of -- in the word’s stack effect, we can write words that return multiple, individual values.

Getting Help

Before moving on to vocabularies, I’ll mention another useful bit of information about Factor words. Within the Listener, you can quickly get to the documentation for a word. Type a backslash, a space, followed by the word and finally help:

This opens the help browser for the at word, as shown in Figure 2, ​The Factor help browser​.

---

Figure 2. The Factor help browser

Other commands you can use in the Listener to get help are about and apropos. For example:

• "sequences" about : documentation for the sequences vocabulary

• "json" apropos : show vocabularies, words, and articles that contain json

The online help is a tremendous resource for discovering the words within the Factor libraries. Now that we can also define our own words, let’s see how we organize them into vocabularies.

Working with Vocabularies

Words are organized into vocabularies. Think packages, modules, or namespaces. When you typed a word such as print in the Listener, the following appeared at the bottom of the window:

The vocabulary in which the word is defined appears after IN:. So print belongs to the io vocabulary.

For convenience, the Listener automatically loads a number of vocabularies. You can get a list of the 50 or so vocabularies that are initially loaded in the Listener by typing interactive-vocabs get [ print ] each.

If you type a word that belongs to a vocabulary that the Listener does not load by default, you’ll get an error message. For example:

You can load a vocabulary with USE: as follows:

After loading the present vocabulary, we were able to use the present word. To load multiple vocabularies, either repeat USE: for each vocabulary, or write USING: followed by the list of vocabularies to load and ending with a semicolon:

Those two ways of loading vocabularies are equivalent.

Within a vocabulary, you can define a symbol with SYMBOL: and use it to stash a value with set. You can later retrieve the value with get. For example:

You can also use on, off, and toggle for a symbol that holds a Boolean value:

For an integer value, use inc and dec:

Symbols are a convenient way to store values and communicate them between vocabularies.

Let’s march on to creating vocabularies in source files and using them in standalone programs.

Running Standalone Programs

We’ll now create a simple vocabulary and use it in a standalone program. Start from an empty factor directory and create an examples subdirectory. Within examples, create two more subdirectories: greeter and hello. Finally, create a greeter.factor and a hello.factor file within those subdirectories so that you end up with the following structure:

We’ll start by creating the greeter vocabulary. Add the following to greeter.factor.

IN: declares this as the examples.greeter vocabulary. Note the file path under factor, which is examples/greeter. That needs to match the vocabulary name.

Next, we’ve defined a greeting word that takes a name and returns a greeting. Let’s use this in another vocabulary, examples.hello, which we will then run as a standalone program. Create the hello.factor file as follows:

We’ve declared that we want to use our examples.greeter vocabulary and that we are defining the examples.hello vocabulary. Then, we created a word that uses the greeting word to print out a greeting. Finally, we used MAIN: to indicate what word to call when running this file as a standalone program. Note that the stack effect of the word called by MAIN: must be ( -- ).

Now try running the program from the command line by typing factor factor/examples/hello/hello.factor:

Factor did not find our examples.greeter vocabulary. We need to indicate the root paths from which Factor will search for vocabularies. You can do this by creating a .factor-roots file in your home directory and indicating the full paths to the root directories where you have your Factor source files, one path per line. For example, here is my .factor-roots file:

You also have another option. You can use an environment variable named FACTOR_ROOTS and set to the list of paths, separated by : if you’re on Linux or Mac, or separated by ; if you’re on Windows.

After setting up your Factor roots configuration, try running factor factor/examples/hello/hello.factor again:

What? Using swap worked fine in the Listener, but here it crashes and burns. That’s because standalone code does not automatically include vocabularies the way that the Listener does. We need to load the kernel vocabulary to use swap, and we also need sequences to use append. Here is the full greeter.factor file:

Similarly, we need to load the io vocabulary to use print in hello.factor:

Paint the fence. Up, down. In standalone programs, we need to be explicit in loading all the vocabularies that we wish to use. Now, running factor hello.factor gives us the expected result of printing “Hello, world”.

Writing Unit Tests

Exploring code in the Listener is nice. However, when you close the Listener, your code is gone. So we saw how to write and run Factor code in source files. Verifying that the code gives the correct results, though, is a manual process. Let’s learn how to automate the process with unit tests.

Unit tests not only confirm that your code is working, but they are also a great way to experiment with a language. You run your tests to verify that the results match what you expect while you are learning the language. You also get to keep all of your code in source files, and run all your tests again at any time to make sure everything still works. Now that we know how to run code standalone, let’s write some unit tests.

The Factor library includes a tools.test vocabulary with a unit-test word. To run a test, you call unit-test with two values on the stack: a sequence of values that represents the stack that you expect, and a quotation that contains the code that you want to test. If running the code produces values on the stack that match up with the expected sequence, the test passes. For example, here is a simple unit test for our greeting word:

Try running the code from the command line just like a standalone program:

When running unit tests, errors are shown when the actual result does not match the expected output. When running this test:

we get:

The output shows the expected and actual outputs for the unit test that failed.

Remember that each test consists of a sequence containing the values that we expect to be on the stack, followed by a quotation of the code we are trying out, and ending with the unit-test word.

Next, let’s see how we can write unit tests that match up with the vocabularies that we create, and how to run a whole suite of tests with a single command.

Running a Test Suite

Remember how we defined an examples.greeter vocabulary in the examples/greeter/greeter.factor file. To write corresponding tests for our vocabulary, we used the examples/greeter/greeter-tests.factor file. Following this convention makes it easy to run all unit tests defined in examples.* vocabularies with a test suite, as follows:

Let’s break that down.

​①​

After importing the vocabularies that we need with USING:, we import the vocabulary that we are testing, examples.greeter with USE:. We could have included examples.greeter on the USING: line, but importing it separately makes our intent clearer and makes it cleaner to add more examples.* vocabularies to test, one per line.

​②​

Simply calling "examples" test runs all tests for all loaded vocabularies that start with examples. However, the output shows the code for all the unit tests, whether passing or failing, making it somewhat difficult to see the tests results at a glance. By using with-null-writer, we are suppressing that output.

​③​

After running the tests, the test-failures symbol contains a list of failures. We retrieve the list with get, and verify whether it contains any values with empty?.

​④​

Depending on whether the list of failures is empty, we either print the "All tests passed" message to the output, or call the :test-failures word, which prints out the test failures.

Now, we can run our unit tests:

When we create another vocabulary, we can simply add a USE: line to test-suite.factor to include its unit tests in the test suite.

Excellent. This is a good stopping point for today. Now, let’s hear from Slava Pestov, creator of Factor. He has some interesting thoughts on his journey.

An Interview with Slava Pestov, Creator of Factor

Us:

Why did you write Factor?

Slava:

Factor was originally a scripting language for a 2D game I built with a friend in college, back in 2003. We never got very far with the game itself, but it was enough to flesh out the basics of the language and have something to apply them to.

The scripts I wanted to write here would consist of lists of strings, and not code. I didn’t realize it at the time, but I wanted something with homoiconic syntax. There were a few other scripting languages for Java at the time, such as Groovy, and even a Common Lisp implementation (ABCL), which had fancier syntax for these things. I decided to write my own language, though, because I wanted something really simple, and also just because it would be fun.

I picked postfix syntax on a whim. At the time, I had heard of Forth but didn’t know much about it. Then I came across Joy, a language by Manfred von Thun which incorporated elements of Forth and Lisp by replacing Forth’s control structures with higher order functions that took code blocks as parameters. A code block in Joy is just a list literal. It was very elegant, and I managed to implement something similar in a few hundred lines of Java code.

Factor was more like Joy than Forth in the early days. In particular, the lexical syntax was fixed, with no “parsing words.” It wasn’t until I started working on the native implementation of Factor that I took the time to properly learn Forth. At this point I had another epiphany and realized that most of the actual parser can be scrapped, and syntax elements such as “[” and “]” can become ordinary words (functions) in the language.

Eventually I cleaned the code up and split it off from the game engine and released Factor to the concatenative mailing list.

Us:

Why did you decide to move Factor away from the JVM?

Slava:

I wanted to experiment with an image-based runtime and a self-hosting parser written in Factor, which was not really possible on the JVM. Also, once again with the learning experience aspect, I wanted to learn enough to implement the minimal possible VM instead of relying on something else.

While working on the Java implementation I had already figured out that over time, I can replace primitives with Factor definitions making use of lower-level primitives, and simplify the design that way. Before starting work on the C implementation, I tried to really limit the scope of what would be written in C. I wrote a simple C interpreter that read an image file containing a heap dump. At the same time, I wrote a Factor parser in Factor which would read source code and output it in the form of this image file. I ran this parser in the Java implementation, and used it to generate image files for testing the C implementation. It was a while before the C implementation could itself run the parser. I was learning C at the same time, and even a simple copying garbage collector is quite tricky the first time you do it!

Once the C implementation was good enough to bootstrap itself, I stopped working on the Java port. The initial version of the C VM was only something like 7,000 lines of C, with the rest in Factor. There was something very satisfying about being able to do so much with so little “real” code. Compilation to native x86 code came very soon after. Writing a JIT in C is actually really simple. You allocate some memory with the right protection bits, write some machine code there, and cast it to a function pointer. Optimizations are the hard part...

Us:

What do you like the most about Factor?

Slava:

I enjoyed working in an interactive development environment. From the start the goal of Factor was to enable quick experimentation with fast turnaround for testing changes, so I took care to structure the system so that almost any type of change to the source code could be reflected in the running program without having to restart. Smalltalk and Lisp environments were a big inspiration for me in this regard.

There is something very hypnotizing about concatenative syntax. Once you learn to think in it without mentally translating back and forth, certain refactorings that can be hard to envision in an applicative language become simple sequences of copy/paste operations. I beseech every programmer to read the Joy papers, even if you have no interest in using a concatenative language, just to learn about the combinators that abstract over many common recursion and iteration patterns in a novel way.

With Factor, often when you first write some code it looks very haphazard and messy. Then you think about it really hard, whip it into shape, come up with some new abstractions, and maybe even also apply them to a previously written library or two. It can feel like a lot of work but in the end what you’re producing feels more like a general set of tools, rather than a single piece of code. To a large extent, concatenative languages were uncharted territory. By encouraging contributors to submit code to the main repository, Factor really pushed the bar in terms of what could be done with concatenative languages, by facilitating the development of new idioms and abstractions.

Finally, the best parts of working on Factor were the simple joy of learning about language design, compilers, and operating systems, and collaborating with all of the extremely talented contributors.

What We Learned in Day 2

Today was about taking our Factor code to the next level, from the experimental ground of the Listener to a more durable form, source files. We saw how to run the code as a standalone program. We spent the rest of the day learning how to write unit tests and running a complete test suite from the command line.

Your Turn

You can always use the Listener for feeling your way around with Factor code, but ultimately the goal in today’s exercises is to write code in source files.

Find…

• A third way of adding directories to the list of vocabulary roots. Remember that the two ways we discussed were the .factor-roots file and the FACTOR_ROOTS environment variable.

• The tool that Factor provides to deploy a program as a truly standalone application, meaning that the executable can be run without Factor being installed on the target machine.

Do (Easy):

• Create an examples.strings vocabulary and write a word named palindrome? that takes a string from the stack and returns t or f according to whether or not the word is a palindrome (a word that is spelled the same frontward and backward, such as racecar).

• In the appropriate vocabulary for associating tests with the examples.strings vocabulary, write two unit tests for palindrome?, one that expects t and one that expects f.

• Add the examples.strings to the test suite so that its tests are included when running test-suite.factor.

Do (Medium):

• Create an examples.sequences vocabulary and write a find-first word that takes a sequence of elements and a predicate, and returns the first element for which the predicate returns true. Write a corresponding unit test that confirms its behavior. What happens if none of the elements satisfy the predicate?

• In an examples.numberguess vocabulary, write a standalone program that picks a random number from 1 to 100 and asks the user to guess, printing out “Higher”, “Lower”, or “Winner” accordingly.

Do (Hard):

• Enhance the test-suite.factor program so that it prints out how many tests have run, and in case of failures, how many tests failed.

• Make test-suite.factor interactive by turning it into a command-line program that asks the user which vocabularies to test via the console, then runs the tests and outputs the results.