A Relational Example

You're probably already familiar with the data model of relational databases. They represent data in tables, also called “relations” in formal terminology, with a row for each “thing” in the table and a column for each piece of information about the thing. For example, a table of contacts might look like what you see in Table 6.1.

As Rich Hickey described in his talk “The Value of Values,” this is a model of information that's all about “place.” If you want to know Jane Doe's email, you go to the location of Jane Doe's row and retrieve it from the location of the email column. If Jane Doe changes her email, you go back to its location, delete what's there, and replace it with the new one, but the old email lost forever. This model quickly starts to look a lot less like what we as Clojure developers think of as data and more like a pile of mutable references.

Inflexibility is another drawback of the relational model. You must declare the structure of all your data in very strict terms ahead of time, and you must shoehorn the dynamic heterogeneity of the real world into fixed and structured tables. Relationships between things must be carefully modeled in advance, typically requiring multiple extra tables whose sole purpose is to describe relationships between things in other tables, and every query that uses them must be written with explicit joins, while paying careful attention to the specifics of the schema.

Datomic Datoms

With that in mind, let's compare Datomic's data model. Datomic stores a collection of 5-tuples called Datoms, which are its fundamental unit of data. The Datomic documentation describes it as a database of immutable facts, and each one of these facts is expressed as a Datom, which look like this:

[e a v t added?]

• e means “entity”. It is roughly analogous to the “id” field in the previous relational table. It contains a Datomic entity id, a Long. Also similar to the way primary keys are typically handled in relational databases is that you don't set entity ids directly when adding data; they are internal to Datomic and are generated automatically by the transactor, Datomic's writer.
• a means “attribute.” It's a bit like a column in a relational table, or in more familiar Clojure terms it's like the key in a hash map. The attribute part of a Datom contains the Datomic entity id of an attribute entity. You'll see what this means later, in the section on schema and modeling data.
• v means “value.” It's similar to a cell or field in a relational table. In Datomic, the value part of a Datom contains either an entity id, if the Datom is describing a relationship between the e entity and the v entity, in which case it's called a “reference,” or some data value like a string, number, date, and so forth.

Together, entity-attribute-value gives you the basic facts. Before talking about what t and added? mean, let's take a look at what the data from the previous relational example might look like as Datoms with just the [e a v] part. You'll notice that keywords are being used in the a position rather than entity ids (Longs). In the schema section you'll see why this is the case, and how Datomic resolves them to the entities they refer to.

[[101 :first-name "Jane"]
 [101 :last-name "Doe]
 [101 :email "[email protected]"]
 [101 :emergency-contact 115]]

It's very important to keep this [e a v] pattern in mind when thinking about data in Datomic because Datalog—Datomic's query language—is all about matching these patterns with variables and values. This is covered later in the section on querying.

The t and added? parts of the Datom together are what give Datomic its immutable characteristics, point-in-time query features, and database-as-a-value abstraction. This is also where Datomic departs almost completely from the paradigm of relational databases, because there is no real relational analog for this part of the data model.

t means “transaction.” Each write to Datomic contains one or more Datoms bound in a transaction. As part of the write process, the transaction itself is created as an entity, which has an attribute with a timestamp that corresponds to the time of the transaction. Every Datom in the transaction has the transaction's entity id in the t position.

Storing transaction entities in this way as part of the data model means that every fact in the database can be traced back to the exact point in time that it entered the system.

added? is a Boolean flag that will be set to true if the Datom is an assertion, and false if it's a retraction. Assertion means the fact that a Datom is conveying is true as of the transaction time, retraction means it's no longer true. Let's look at an example using Jane Doe and a new attribute:favorite-color. As of the time of transaction t1, Jane Doe's favorite color is red. The state of the database looks like this.

[[101 :favorite-color "red" t1 true]]

We have a Datom for Jane Doe (entity id 101), which was added as part of transaction t1, and it's an assertion because the added? flag is true. Now let's assume Jane Doe is going through a rebellious teenage phase, and as of the time of transaction t2 she decides that her favorite color is no longer red, but it's now black. Here's the new state of the database after t2.

[[101 :favorite-color "red" t1 true]
 [101 :favorite-color "red" t2 false]
 [101 :favorite-color "black" t2 true]]

You see what happened! Rather than an update-in-place, where you would have completely deleted the idea that Jane Doe's favorite color was ever red, you have added a retraction Datom for that fact since it is no longer true, and added an assertion Datom that tells you Jane's new favorite color.

In the most regular usage of Datomic, you don't directly deal with the complete Datom. Most of the APIs work in terms of [e a v]—entity, attribute, and value. The transaction and added flag are typically underneath an abstraction where, given a point in time of a particular transaction value, you only see the entity, attribute, and value parts of Datoms that are true at that time. The transaction and added flag of the Datoms are used to build a projection of the current state of the data as of that time. For example, as of transaction t1 the state looked like this:

[[101 :favorite-color "red"]]

Whereas, as of transaction t2 the state looked like this:

[[101 :favorite-color "black"]]

This is a very important abstraction to keep in mind, because most of the time you're working in Datomic the t and added? parts of Datoms are abstracted away from you. There are also numerous ways to powerfully leverage the full Datoms, and understanding the complete Datomic data model is very important in designing systems to use it properly, so you need to be able to think about your data in both ways.

It's an abstraction that fits in powerfully with Clojure's broader view of data and state. As a Clojure programmer, you usually think of state as something to be carefully managed and minimized in your systems, because you realize that much of the most tedious, error-prone, and dangerous aspects of programming revolve around state management. The opinionated design of Clojure works very hard to keep state to a minimum, and places more restrictive semantics around mutable state so that developers know to handle it with care. Whenever possible, data is preferable to state.

Datomic's data model works along very similar lines. It looks at the database as a data structure to which you can only add additional facts, and treats state as a projection of that data structure.

Entity Maps

When using Datomic, one of the common ways that you'll interact with data is as entity maps. There is an API for this, which is covered in more detail in the Clojure API section, but in the meantime let's examine the basic concept. You've already seen how to describe all of the facts about an entity in terms of Datoms. Another way to look at that same data is in an entity map format.

Entity maps are conceptually quite simple. You start the projection of the state of an entity as of a given transaction time—in other words, only the [e a v] parts of Datoms representing facts that are true as of that time. After that, it's a fairly straightforward matter to represent the Datoms about a particular entity as a map. The one catch is where the entity id goes. In Datomic entity maps, the entity id is in a special key called :db/id. Here's what Jane Doe would look like as an entity map.

{:db/id 101
 :first-name "Jane"
 :last-name "Doe"
 :email "[email protected]"
 :emergency-contact 115}

Syntax

There are two ways to write Datalog queries: the list form and the map form. They are semantically equivalent, but are typically used in different scenarios. The list form is idiomatically most often what's used for queries written by humans, because many people find it somewhat easier to read and it requires fewer inner lists. Here's a simple query in the list form, which will return the first name and last name of the entity whose email address is [email protected].

'[:find ?first-name ?last-name
  :where [?e :email "[email protected]"]
         [?e :first-name ?first-name]
         [?e :last-name ?last-name]]

The map form is more often used when building queries programmatically, because it's much easier to destructure and manipulate the various parts of the query when they're in map keys. This is the same query in the map form.

'{:find [?first-name ?last-name]
  :where [[?e :email "[email protected]"]
          [?e :first-name ?first-name]
          [?e :last-name ?last-name]]}

The two forms are quite similar, but the map form requires that all of the query components be wrapped in lists. You'll notice that both forms have been quoted—the preceding '. This is required when used in Clojure code, because queries are written using symbols, like ?first-name and ?last-name, which throw exceptions when evaluated.

Since it's more idiomatic for queries written by humans, the list form will be used for the rest of this chapter, except in places discussing how to build and manipulate queries programmatically.

Here is what the equivalent SQL might look like. Notice that, in this simple example, the SQL version has at least a vague resemblance to the Datalog.

"SELECT first_name, last_name FROM contacts
 WHERE email='[email protected]'"

Let's take a closer look at that simple query example.

Find… What Am I Looking For?

The “find” part of the query—the first line starting with :find—is roughly analogous to the SELECT part of a SQL SELECT query. It says what you want to return, and in what format you want it. In this example, here is the find specification:

:find ?first-name ?last-name

In Datalog, variables are symbols prefixed with ?, so this find specification is returning the values for the two variables ?first-name and ?last-name, as defined later in the where section of the query. This type of find specification—where the variables are simply positionally listed after the :find keyword—returns a “relation.” This is the most common type of find specification in most queries you'll see and write because it's the most general form, and its behavior is the most similar to that of SQL SELECT.

The result, when run against the earlier sample data, is a set of tuples:

#{["Jane" "Doe"]}

:find [?first-name ...]

["Jane"]

:find [?first-name ?last-name]

["Jane" "Doe"]

"SELECT first_name, last_name FROM contacts
 WHERE email='[email protected]'
 LIMIT 1"

:find ?first-name .

:where Is Where the Magic Happens

The body of the query, where almost all of the query logic and important structure live, are in the where clauses. This part of the query does almost everything that you'd do in a SQL query–conditions, joins, inner queries, filters. As such, there are probably more features that deal with what you can do in the where part of a query than for any other aspect of Datalog.

There's a great breadth of topics to cover to get a full idea of what is possible in where clauses. Truly, the level of power and flexibility that Datalog gives you in expressing query logic here is amazing, and throughout the rest of the chapter you'll see a more comprehensive set of this functionality.

You can think of the where clauses as patterns that describe what the data in the database needs to look like to satisfy the pattern. These patterns match against the entity, attribute, and value parts of Datoms as of a given transaction-time. Let's first get a basic grasp of what's going on in our simple query. The where clauses look like this:

:where [?e :email "[email protected]"] ;; 1
       [?e :first-name ?first-name]          ;; 2
       [?e :last-name ?last-name]            ;; 3

In order to return a solution, these patterns require that three Datoms exist in the database:

1. A Datom with any entity id (which will be bound to the variable ?e), the attribute :email, and the value [email protected].
2. A Datom with the same entity id as the previous Datom—in other words, a Datom that's about the entity that has the :email of [email protected]—has the attribute :first-name, and has any value. The value gets bound to the variable ?first-name.
3. A Datom with the same entity id as the previous two, the attribute :last-name, and any value. The value is, as before, bound to a variable: ?last-name.

There are a few things to note here. When values are “bound” to Datalog variables, that same value is used for the rest of the solution. In this case that means several things.

• If there was no Datom with attribute :email and value [email protected], then these where clauses would return no solutions. The result would be an empty set.
• The entity id bound to ?e in the first pattern must also have Datoms that match the second and third patterns. In other words, that entity-id must also have Datoms with :first-name and :last-name; otherwise the where clauses would again return no solutions.
• If there were multiple entities that matched all of the patterns, then multiple solutions are returned.

It's important to understand how this variable binding and solution finding process works. It's possible for patterns to match multiple Datoms and have multiple possible value bindings for Datalog variables, and each one of these bindings is tested against the rest of the patterns in turn. If Datoms exist in the database that satisfy all of the patterns, then that solution is considered “unified” and is returned as part of the results. This process can be recursive, since variable bindings from earlier patterns can be linked to additional variable bindings in later patterns, which must in turn each be tested against subsequent patterns.

Let's go through a somewhat more involved example to see how this process works.

Variables and Joins

The initial example was fairly simple. Let's look at one that involves data with more relationships and a query with more variables. First, we'll extend the data to include more than one contact, and the concept that contacts can have friends. For simplicity, let's keep using only the [e a v] part of the Datoms.

[[101 :first-name "Jane"]
 [101 :last-name "Doe"]
 [101 :email "[email protected]"]
 [101 :friend 102]
 [101 :friend 103]
 [102 :first-name "Ada"]
 [102 :last-name "Lovelace"]
 [102 :email "[email protected]"]
 [102 :friend 104]
 [103 :first-name "Robert"]
 [103 :last-name "Heinlein"]
 [103 :email "[email protected]"]
 [104 :first-name "Jane"]
 [104 :last-name "Smith]]

You still have your old friend Jane Doe, but she has two new friends. Using this data, let's look at a slightly more complex query.

'[:find ?first-name ?last-name
  :where [?e :email "[email protected]"]
         [?e :first-name ?first-name]
         [?e :last-name ?last-name]
         [?e :friend ?f]
         [?f :first-name "Robert"]
         [?f :last-name "Heinlein"]]

In this query, you are still looking for the first name and last name of an entity whose email is [email protected], but you have added some more clauses. You now have a join! The entity must also have a friend—another entity?f—whose first name is “Robert” and last name is “Heinlein.” As it happens, “Jane Doe” is friends with “Robert Heinlein,” so the result is the same as before: #{["Jane" "Doe"]}

Let's walk through the execution of this query to help clarify how Datalog works with multiple variables and relationships.

As before, the first three patterns will match Datoms where there is an entity with the specified value for :email, which also has :first-name and :last-name, and as before you bind those to the variables ?first-name and ?last-name. The fourth pattern binds any value from a Datom with that same entity-id and the attribute :friend to the variable ?f. This means that ?f will contain two possible values: 102 and 103.

To satisfy the next pattern, the query will try to find matches where ?f is 102 or 103. Trying 102 will not return a solution, because there is no Datom where 102 is the entity, and the :first-name is “Robert.” But the second value, 103, will return a solution because it matches both of the last two patterns.

This query can still be written in SQL, although it begins to look more unwieldy.

"SELECT e.first_name, e.last_name FROM contacts AS e
 INNER JOIN contacts AS f
 ON e.friend_id = f.id
 AND e.email = '[email protected]'
 AND f.first_name = 'Robert'
 AND f.last_name = 'Heinlein'"

Going Deeper

To help you understand this more fully, let's look at an even more complex query that shows some more of the power of logic programming, and the use of variables across joins.

'[:find ?first-name ?last-name
  :where [?e :first-name ?first-name]  ;; 1
         [?e :last-name ?last-name]    ;; 1
         [?e :friend ?f]               ;; 2
         [?f :friend ?g]               ;; 3
         [?g :first-name ?first-name]] ;; 4

This query returns the same solutions as the previous queries—#{["Jane" "Doe"]}, but why? In words, what this query is looking for is:

1. The first name and last name of anyone
2. Who is friends with someone
3. Who is friends with another person
4. Who has the same first name as the first person

In the data, “Jane Doe” is friends with “Ada Lovelace” who is friends with “Jane Smith,” so “Jane Doe” is returned as the solution. Here you start to see the magic of logic programming. You did not have to delve into any of the mechanics of how to traverse the data, or the structure and storage of intermediate results, or explicitly call out any joins between different entities. In this query you declaratively describe the data pattern at a high level, so the constraints and relationships you are interested in for the solution, and the Datalog engine, finds all of the answers that match.

The SQL equivalent for this query is left as an exercise to the reader.

However, with great power comes great responsibility. This particular query does almost nothing to narrow down the possible set of matches, so as the number of contacts and size of the friends graph in our data grows, this query will have to churn through more and more possible solutions to find the one that matches. In production systems this can become unfeasibly expensive, so exercise caution and try to write queries that are as selective as possible, as you saw in earlier queries that required a matching email.

Kaboom! A Combinatorial Explosion

Just as in SQL, it's possible to write queries that will result in combinatorial explosions. Here's one such query.

'[:find ?first-name ?last-name
  :where [?a :first-name ?first-name]
         [?b :last-name ?last-name]]

Notice how the two patterns in the where clause are disjointed. This query returns every combination of first and last name in the database.

Low-level List Syntax

The list form follows this format:

[command e a v]

• command is either :db/add or :db/retract—the first means that this fact should be added to the database, the second means it should be retracted. This will map directly to the added? part of the Datom.
• e is an entity reference, which is an extended version of the e part of the Datom. This is covered in greater detail in the section on using Datomic in Clojure. For now, let's simplify this and say that it is either the entity id of an existing entity, or it's a temporary id structure that Datomic translates into a permanent entity id as part of the transaction.
• a and v are the same as in the previous examples.

Perhaps you're wondering, “I see e a v and added?, but where's t?” The answer is that it's generated for you. Datomic automatically creates a transaction entity and attaches it to all of the Datoms in the transaction as part of the transaction process.

For example, a transaction in list form to add the data about “Jane Doe” to the system for the first time might look like:

[[:db/add jane-temp-id :first-name "Jane"]
 [:db/add jane-temp-id :last-name "Doe"]
 [:db/add jane-temp-id :email "[email protected]"]]

This adds an entity for Jane with her first and last name, and email, assuming you have generated a temporary id for her with the Datomic API and bound it to jane-temp-id. Now suppose you want to add some friends for Jane. First, you need to find Jane's actual entity id with this query.

'[:find ?jane-id .
  :where [?e :email "[email protected]"]]

Then you can add her friends with a transaction like this:

[[:db/add heinlein-temp-id :first-name "Robert"]
 [:db/add heinlein-temp-id :last-name "Heinlein"]
 [:db/add heinlein-temp-id :email "[email protected]"]
 [:db/add jane-id :friend heinlein-temp-id]
 [:db/add lovelace-temp-id :first-name "Ada"]
 [:db/add lovelace-temp-id :last-name "Lovelace"]
 [:db/add lovelace-temp-id :email "[email protected]"]
 [:db/add jane-id :friend lovelace-temp-id]]

Again, let's assume you have generated temporary ids for the two new entities being added, and you've bound Jane's permanent id from the previous query to jane-id.

All of these transactions are assertions, so let's take a quick look at what a retraction looks like. Let's assume that Jane Doe, having read some of Heinlein's later works, is shocked by his views on morality, and decides that she no longer wants to be his friend. First, you need to look up Heinlein's permanent entity id using a query—left as an exercise to the reader—quite similar to the above query to find Jane's permanent id. The transaction to retract the friendship looks like this.

[[:db/retract jane-id :friend heinlein-id]]

You can see that the list form is somewhat verbose and unwieldy. It does offer the advantage of having direct, low-level control over the transaction and, unlike the map form, allows you to retract individual Datoms. However, you will almost always end up using the higher-level map form for all your transactions, with higher-level APIs to handle things like retraction. So let's take a look at the map form.

Map Syntax

The map syntax is, as mentioned before, higher-level than the list syntax and it's also generally considered more idiomatic. It represents transaction data as maps, where attributes are the keys, and entity ids are represented by a special :db/id key, just like the format of entity maps. Let's take a look at the Jane transaction from before expressed in this syntax.

[{:db/id jane-temp-id
  :first-name "Jane"
  :last-name "Doe"
  :email "[email protected]"}]

In this form, as with entity maps, the entity id is represented by the :db/id key. Under the hood, Datomic translates this into the list form, exactly the way it was in the previous section, before transacting. But you can see that this is a much more concise, convenient, easy-to-read way of expressing that transaction data.

You can also do something really nifty that would take quite a bit more work to do using the other syntax: add Jane and her friends at the same time. Watch this:

[{:db/id jane-temp-id
  :first-name "Jane"
  :last-name "Doe"
  :email "[email protected]"
  :friend #{{:first-name "Robert"
             :last-name "Heinlein"
             :email "[email protected]"}
            {:first-name "Ada"
             :last-name "Lovelace"
             :email "[email protected]"}}}]

That's right, the map syntax supports nesting for related entities! One thing you'll notice is that Mr. Heinlein and Ms. Lovelace don't have temporary ids. That's because Datomic will generate them for you when building a transaction this way.

There are a few caveats to this. Don't worry, you won't understand them until you read about Datomic schema, but they're included here in case you're reviewing this section later and want this information all in one place. When you use the nested map syntax this way, if you don't provide temporary ids, then the nested entities must either be related to the containing entity via an isComponent attribute, or they must have an identity attribute. Also, their generated ids will be in the same partition as the containing entity's id.

Set Semantics

One thing to keep in mind: Datoms have set semantics. In other words, asserting the same [e a v] more than once is a no-op; the Datom will only exist in the database once. As a quick example, if you transact this data:

[{:db/id jane-id
  :first-name "Jane"
  :last-name "Doe"
  :email "[email protected]"}]

And then transact this data:

[{:db/id jane-id
  :first-name "Jane"}]

The second transaction will succeed and it will add a transaction entity, but it will not add another Datom about Jane. This is covered in greater detail in the section about the Clojure API.

eavt Index

The eavt index represents data in a way that is conceptually similar to the way relational databases store rows. Datoms in this index are first sorted by entity id, so Datoms about each entity are grouped together and then by attribute, then value, then transaction id. Datoms in this index would be stored quite similarly to the way we've been representing them so far, using a dummy transaction id:

[[101 :email "[email protected]" 'some-t]
 [101 :first-name "Jane" 'some-t]
 [101 :friend 102 'some-t]
 [101 :friend 103 'some-t]
 [101 :last-name "Doe" 'some-t]
 [102 :email "[email protected]" 'some-t]
 [102 :first-name "Ada" 'some-t]
 [102 :last-name "Lovelace" 'some-t]
 [103 :email "[email protected]" 'some-t]
 [103 :first-name "Robert" 'some-t]
 [103 :last-name "Heinlein" 'some-t]]

Since facts about each entity are grouped together, it's easy to retrieve all of the data about a given entity quickly. For obvious reasons, the entity API that's used to retrieve entity maps uses this index. This API is covered in more detail in a later section.

All Datoms are stored in this index.

aevt Index

The aevt index groups all Datoms about an attribute together, allowing for the quick determination of which entities have values for that attribute and, secondarily, what those values are. As the Datomic docs point out, this index is somewhat similar to a columnar store, and is one of the ways that Datomic enables flexible querying that supports mixed workloads. For this example data, the index looks something like this:

[[:email 101 "[email protected]" 'some-t]
 [:email 102 "[email protected]" 'some-t]
 [:email 103 "[email protected]" 'some-t]
 [:first-name 101 "Jane" 'some-t]
 [:first-name 102 "Ada" 'some-t]
 [:first-name 103 "Robert" 'some-t]
 [:friend 101 102 'some-t]
 [:friend 101 103 'some-t]
 [:last-name 101 "Doe" 'some-t]
 [:last-name 102 "Lovelace" 'some-t]
 [:last-name 103 "Heinlein" 'some-t]]

One thing to note here is that we are making a slight oversimplification. Attributes are not stored and sorted based on their keyword name, but rather on their underlying attribute entity id. This doesn't make a huge amount of difference in practice, but it's something to keep in mind.

All Datoms are stored in this index.

avet Index

The avet index is subtly different from the aevt index, in that it's designed to very quickly retrieve attributes with specific values. It's also, as the Datomic docs point out, the most expensive index to build and maintain. For this reason it's optional on a per-attribute basis, whether or not Datoms are stored in this index. Assuming indexing is enabled on all of the example attributes, here is what the data looks like in this index:

[[:email "[email protected]" 103 'some-t]
 [:email "[email protected]" 101 'some-t]
 [:email "[email protected]" 102 'some-t]
 [:first-name "Ada" 102 'some-t]
 [:first-name "Jane" 101 'some-t]
 [:first-name "Robert" 103 'some-t]
 [:friend 102 101 'some-t]
 [:friend 103 101 'some-t]
 [:last-name "Doe" 101 'some-t]
 [:last-name "Heinlein" 103 'some-t]
 [:last-name "Lovelace" 102 'some-t]]

The decision about whether to enable the avet index for a given attribute involves a similar tradeoff to the decision about what columns to index in a relational database. It adds cost to writes, but depending on the kinds of queries you're doing, it may be essential. Any query that relies on the specific value of an attribute, without knowing anything about which entity it belongs to, will be enormously sped up by this index. Best practices here are similar to those in the relational world: if you think you'll query against the attribute's value, then enable indexing.

Fortunately, if you miss out on adding an attribute to this index that you end up needing later, you can always enable indexing on that attribute with a run-time schema update.

vaet Index

The vaet index is for representing reverse references. That is, when you have a reference-type attribute, which means it represents a relationship between two entities, this index stores that relationship in the opposite direction. Naturally, this index is only enabled for reference type attributes. For this example data, the vaet index might look like the following:

[[102 :friend 101 'some-t]
 [103 :friend 101 'some-t]]

You can grasp the general idea. The other indexes can be used in various ways to easily find outgoing entity references, but the vaet index is required to efficiently determine incoming references.

The log

The log is structured somewhat differently from the indexes. The Datomic docs put it quite simply: the log is an ordered collection of transactions, each of which contains an unordered set of Datoms. Queries do not directly use the log; rather it's accessed via several specialized APIs that can be used as part of queries or to examine the history directly in your application.

Index structure

The above means that every Datom is stored in at least 3 and as many as 5 copies. In the past, this kind of data replication may have been cause for concern or perhaps it was even infeasible. Indeed, the idea that storage is scarce motivated many of the original design decisions that are now deeply embedded in relational database designs. Today this is a rather antiquated assumption: storage is cheap and fast. Datomic's design makes time vs. space trade-offs in the direction of optimizing for speed, because modern hardware and system architectures make data replication and storage size a much lesser concern.

The indexes themselves are stored as shallow (3-level) trees, with an extremely high branching factor. The structure looks like what is shown in Figure 6.1.

It's structured as a root node, directory nodes, and segments. Datomic optimizes read performance by keeping the root node and as many of the directory nodes in the memory of the peers and the transactor possible to minimize the number of trips to storage.

Each of the segments contains up to approximately 50Kb of Datoms, which have been serialized to binary using the Fressian format and then gzipped. Depending on the size of the values, each segment can contain many thousands of Datoms. This is the most important thing to grasp about how Datomic stores data: to get at a single Datom in a given segment, the entire segment must be transferred, unzipped, and deserialized into memory. Thus, when thinking about performance, the biggest factor is going to be the number of segment accesses required to complete any operation.

The Life and Times of a Transaction

To get a clearer understanding of the process of how data gets into Datomic and how the different parts of the system are involved, let's walk through the life and times of an example transaction.

1. A peer submits a transaction to the transactor.
2. The transactor processes transactions one at a time order of receipt, and the transaction waits in line until its turn.
3. The transaction is processed by the transactor, transaction functions run, temporary ids resolved, etc.
4. The transactor persists the transaction to the transaction log in the storage layer.
5. The transactor adds the new Datoms to its memory index, and sends the new Datoms to every connected peer.
6. Some time later, the transactor begins a re-indexing job, incorporating all new Datoms into the indexes.

Schema Basics

Let's take a look at what a basic schema transaction contains. First, we'll create the schema attributes for the tasks. The first thing a task needs is a description.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/description
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "description of the task"
  :db.install/_attribute :db.part/db}]

The first key assigns the attribute a :db/id, which is the entity id that's required for every entity in the system, including attributes. The value is a reader macro implemented as part of the Datomic API, which creates a temporary entity id in the :db.part/db partition. All attribute entities are required to be in the :db.part/db partition.

A :db/ident keyword is required for all attribute entities. It provides a universal identifier that can be used to refer to the attribute in queries and in transactions. In general an ident can be used anywhere an entity id is accepted. You can also give :db/ident values to data entities, in addition to attribute entities. Be careful of abusing this feature, though, because all the :db/ident values in the database are loaded on startup into every Datomic peer.

This attribute has the ident of :task/description. The best practices laid out in the Datomic documentation recommend that you use namespaced keywords for all your attribute idents, to organize them into categories, and to prevent name collisions. If you haven't read through the best practices section of the Datomic documentation, you should probably take a few minutes and do so now. It's an immensely useful document that's been distilled from the experience of nearly four years that Datomic has been in production.

Attribute cardinality is specified using the required :db/cardinality attribute. There are two possible values for this attribute: :db.cardinality/one or :db.cardinality/many. If it's :db.cardinality/one, every entity can only have one value true at any given time for this attribute—if you try to assert a different value, then the previous value will be retracted and replaced with the new value. Otherwise for :db.cardinality/many there is no restriction. Since tasks should only have one title, :db.cardinality/one is the right choice.

Every attribute needs to have a :db/valueType, which specifies the type that its v value must have. There are two broad categories of valueTypes: reference, which means that the value must be an entity ID, or some data type, such as string, float, integer, URI, and so forth. You want your titles to be strings, and :db.type/string is Datomic's string type.

One optional attribute that you can use with both attribute and data entities is :db/doc. This is broadly equivalent to a Clojure docstring or simple Javadoc, but there are no hard-coded ways in which it is used, so you can adapt it to whatever purposes suit the needs of your application. Here, we'll treat this attribute as a docstring-like annotation, which is only meant to be used by developers and other internal users.

Finally, there's the somewhat confusing :db.install/_attribute key. Whenever you see a Datomic attribute where the name part of the keyword begins with an underscore, you know you're seeing a reverse reference. Reverse references are where the entity and value parts have been swapped.

{attribute-temp-id :db.install/_attribute :db.part/db}

What this really means is this:

{:db.part/db :db.install/attribute attribute-temp-id}

The :db.install/attribute attribute is a special Datomic attribute used to add attributes to Datomic's schema, and attributes always need to be installed in :db.part/db.

Partitions

Datomic best practices call for creating some partitions specific to your app, rather than using the built-in :db.part/user that's intended more for development and experimentation. Creating partitions is simple: just create an entity with a :db/ident and the partition install command. Let's create two partitions: one for the tasks, and one for accounts that will contain data about users, their accounts, and their payments and charges.

[{:db/id #db/id[:db.part/db]
  :db/ident :db.part/task
  :db.install/_partition :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :db.part/account
  :db.install/_partition :db.part/db}]

Fulltext Indexing

Datomic supports fulltext indexing on specific attributes using a Lucene index, and supports fulltext search on those attributes as part of queries. This can be very useful if you have one-off needs for fulltext searching that are limited enough to not call for a dedicated search database. Let's add a task title attribute that is fulltext-indexed.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/title
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "title of the task"
  :db/fulltext true
  :db.install/_attribute :db.part/db}]

Use the :db/fulltext attribute to enable fulltext indexing for the title attribute. By default, it's disabled, but there are a few things to bear in mind about fulltext indexing.

• The underlying Lucene indexes are not ACID, but they are eventually consistent and will be updated during transactor re-indexing. If your app requirements include fulltext search that's immediately available after data is added, then you need to use an external system for search and your app needs to pipe searchable data into it, perhaps using Datomic's transaction report queue.
• The decision about whether an attribute is fulltext indexed is final. It is not currently possible to enable or disable fulltext indexing on an attribute after it's been installed.

Enums in Datomic

Many relational databases support the concept of an enum datatype to create columns that require one of a specific set of enumerated values. Datomic emulates this function using reference types and enumeration entities. Let's take a look at an example schema, and add a task status attribute that takes an enum.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/status
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref
  :db/doc "task status - an enum"
  :db.install/_attribute :db.part/db}]

There is nothing particularly special required to create an enum attribute. Here the valueType is set as :db.type/ref, because the enumerated values are entities, and a note was added in the attributes doc to make it clear that it is specifically enum entities that you want to use as the refs for this attribute.

For task status values, it would be nice if they had an attribute that provided some kind of display label that could be used when showing the status. Let's add an attribute that can be used to add a label to any entity.

[{:db/id #db/id[:db.part/db]
  :db/ident :label
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "display label of an entity"
  :db.install/_attribute :db.part/db}]

Now, let's create the enumeration values for the task status. It's quite simple to do: just create an entity with a :db/ident. You need to add task statuses for “To Do,” “In Progress,” and “Done.”

[{:db/id #db/id[:db.part/task]
  :db/ident :task.status/todo
  :label "To Do"}
 {:db/id #db/id[:db.part/task]
  :db/ident :task.status/in-progress
  :label "In Progress"}
 {:db/id #db/id[:db.part/task]
  :db/ident :task.status/done
  :label "Done"}]

You'll notice a few things here. First, since these are not schema attributes, you can't put them in the :db.part/db partition, so put them instead in the :db.part/task partition that was created for the tasks earlier. Next, you'll notice that the namespace of their ident keywords is “task.status,” corresponding to the name of the attribute that they're to be used with. This corresponds with the best practice recommended for naming enumerated values in Datomic. Finally, you can give them some display-friendly labels that the app can use, avoiding any hard-coding in the app about what specific status values should be called, or about how to translate the ident keywords into something for display.

Identity Attributes

Next, let's look at identity attributes. These are attributes whose values are meant to be external keys, and can be used to refer to the entity they belong to. In this way, they're somewhat analogous to primary keys in the relational world, although an entity can have more than one of them. It's very important to include identity attributes in the data model for your key domain entities.

It's considered an anti-pattern to use Datomic's entity id to refer to an entity, except in cases where you just have queried for it, and it's a particularly bad practice to ever store entity ids outside of Datomic. Entity ids are meant to be internal references inside the database, and they are not guaranteed to remain the same if you do a backup and restore.

Let's give the tasks in the app an identity attribute that represents their “issue id,” which is sort of like the ticket numbers you often see in other task and bug tracking systems.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/issue-id
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "task's issue ID, for external reference"
  :db/unique :db.unique/identity
  :db.install/_attribute :db.part/db}]

This is marked as an identity attribute by setting the :db/unique attribute to :db.unique/identity. Setting something as an identity attribute has several effects.

• The attribute must be cardinality-one.
• Values of an identity attribute have enforced uniqueness—two entities can't have the same value for an identity attribute.
• The value of this attribute can be used to transparently look up the entity to which it belongs, and using an identity attribute as part of a transaction makes that an upsert transaction.

Identity attributes are covered in more detail later in the Datomic's Clojure API section.

Many-to-Many and Hierarchical Relationships

One of the biggest benefits of Datomic's data model is the flexibility and simplicity of expressing relationships between entities. Datomic's model of many-to-many relationships, in particular, is a welcome relief to those used to the relational way of modeling them. Here are four little words that are music to the ears of every developer who has struggled with many-to-many in SQL: “no more join tables.”

To show how many-to-many is done in Datomic, let's create some attributes for adding tags to tasks. This will allow users to tag tasks with things like “Home,” “Work,” “Shopping,” “Family Vacation,” and so forth. Let's add the task to tag relationship, and then add a tag attribute that allows us to give tags names.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/tag
  :db/cardinality :db.cardinality/many
  :db/valueType :db.type/ref
  :db/doc "task tags"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :tag/name
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "tag's name, used as identity"
  :db/unique :db.unique/identity
  :db.install/_attribute :db.part/db}]

You see how doing many-to-many in Datomic is as simple as adding a reference-type attribute with cardinality-many? Since you're not constrained by the fixed number of columns in relational tables, it's simple to model the kind of data that typically requires specialized extra tables in SQL databases.

Also, notice that you've made the name of tags an identity attribute. This makes them upsertable, and will make it easier to reuse tags in multiple tasks.

Let's next look at how to create an entity relationship that represents a hierarchy. In the task app you want to create subtasks, which are children of the parent tasks. And perhaps those subtasks can in turn have subtasks of their own, and so on… Some people have very complicated lives, after all, with activities that have deep dependencies on other things being completed. Since subtasks are often attached to parent tasks, moved around, and reorganized, let's model the relationship on the child side.

[{:db/id #db/id[:db.part/db]
  :db/ident :task/parent
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref
  :db/doc "parent of the task, establishing arbitrary hierarchy"
  :db.install/_attribute :db.part/db}]

This is a fairly straightforward attribute. It says that a task can have at most one parent, and the parent is a reference. Thus, tasks can have parents tasks, which can in turn have parents, allowing for hierarchies of arbitrary depth.

Datomic allows you to model relationships in either direction, so this can be modeled as :task/subtask and be an attribute of the parent, rather than the subtask. The decision largely depends on how you prefer to think about the relationship, and how your application will use it. In this case, the deciding factor was that one of the key operations you want to support is assigning an existing task to be a subtask of some other task, possibly removing it as a subtask from another task. Modeling it this way allows this operation to be done very easily: asserting a new parent for the subtask will automatically remove any existing parent relationship, since this is a cardinality-one attribute.

Let's take a moment to consider something interesting about Datomic that particularly distinguishes it from SQL. In this case we have named the parent relationship attribute such that it appears to be task-specific, but there's no reason that it has to be—Datomic won't enforce that constraint. With a single attribute, you could say that tasks can have “idea” entities as parents, and subtasks or comments or any number of differently structured entities as children. The flexibility of Datomic makes these kind of heterogeneous relationships much simpler to model.

Identity versus Unique

Let's create a simple model for users. It's quite similar to what you might see in a relational schema, but it has a few important differences.

[{:db/id #db/id[:db.part/db]
  :db/ident :user/login
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "user's login name and display name"
  :db/unique :db.unique/identity
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :user/password
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "crypted pasword"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :user/email
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/string
  :db/doc "User's email address"
  :db/unique :db.unique/value
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :user/account
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref
  :db/isComponent true
  :db/doc "The account linked to the user"
  :db.install/_attribute :db.part/db}]

This model adds four attributes: the user's login, an encrypted password, an email address, and a link to an account that will be modeled in a moment. For this system, separate logins are used instead of email addresses to allow people to choose their own clever nicknames like “weavejester” and “hyPiRion,” or more pedestrian ones like “rhickey.” We do, however, want to store people's email addresses and only allow one user per email address.

Notice that login is an identity attribute, and email address is modeled as a :db.unique/value attribute. They have slightly different semantics. Identity attributes, as discussed before, enforce uniqueness, but also allow for upserts: if you assert new data that contains an identity value that already exists in the database, then that data updates the entity to which that identity belongs. That means you can refer to your users by their login everywhere, and Datomic will resolve that to the correct entity.

The other type of uniqueness is set with :db.unique/value: if you assert new data that duplicates an existing :db.unique/value Datomic will throw an exception and the entire transaction will fail.

It's generally bad practice to have more than one identity attribute for a single type of entity, unless they correspond to external keys for different systems. If you need additional unique attributes for a type of entity, you should usually use :db.unique/value attributes.

Component Relationships

An attribute can also define a “component” relationship. This describes a relationship where the component entity is owned by the parent entity, and it isn't meant to exist independently. Datomic retracts component entities when you retract the parent entity, and will also return data about component entities when you fetch data about the parent in several key parts of the API.

Let's set up an account system for users that will handle paid accounts and keep track of charges and payments. First let's add an account type attribute, modeled as an enum.

[{:db/id #db/id[:db.part/db]
  :db/ident :account/type
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref
  :db/doc "account type (eg paid, free), an enum"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/account]
  :db/ident :account.type/free
  :label "Free account"}
 {db/id #db/id[:db.part/account]
  :db/ident :account.type/paid
  :label "Paid account"}]

This is just like the enum covered earlier in this chapter: a cardinality-one reference type attribute, with entities with appropriately named idents as the enumeration values. Now let's add in the charges and payments, which are called "transactions" here against the account.

[{:db/id #db/id[:db.part/db]
  :db/ident :account/transaction
  :db/cardinality :db.cardinality/many
  :db/valueType :db.type/ref
  :db/isComponent true
  :db/doc "transactions (payments and charges) against the account"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :transaction/type
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/ref
  :db/doc "transaction type (eg charge, payment, adjustment) - an enum"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/db]
  :db/ident :transaction/amount
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/bigdec
  :db/doc "amount of the transaction"
  :db.install/_attribute :db.part/db}
 {:db/id #db/id[:db.part/account]
  :db/ident :transaction.type/charge
  :label "Charge"}
 {:db/id #db/id[:db.part/account]
  :db/ident :transaction.type/payment
  :label "Payment"}
 {:db/id #db/id[:db.part/account]
  :db/ident :transaction.type/adjustment
  :label "Adjustment"}]

You can see the addition of three more attributes. The last two should be familiar. The first is an enum for the transaction type—either charge, payment, or adjustment for those times when you need to correct mistakes. The second is an attribute for the transaction amount. Datomic includes support for a decimal type corresponding with java.math.BigDecimal objects, which you've used earlier to model the transaction amounts so as not to worry about any stray precision errors from floats.

The first attribute, :account/transaction, models transactions as components of accounts. You do this by setting :db/isComponent to true. It has implemented here because these transactions completely depend on the account for them to have any meaning, and if the account entity is retracted it makes no sense for them to remain.

Disabling History

While Datomic is designed around the idea that data is immutable and the complete history of your data should be available, there are some cases where you don't want that to happen. For example, if you have attributes with a high churn rate or represent calculations that are materialized for the sake of convenience, sometimes it's just not worth it to keep a history of every single value the attribute has ever had.

Fortunately, Datomic allows you to disable history on a per-attribute basis. Let's add an attribute that will represent the current balance of the account. This is a value that will change every time there's any transaction, and furthermore it's something that you cancalculate at any point by summing up all of the transactions. It can be convenient to have this value materialized, because then it's easier to do queries about, for example, accounts that have more than a certain balance. Here's how that is modeled in a Datomic schema:

[:db/id #db/id[:db.part/db]
  :db/ident :account/current-balance
  :db/cardinality :db.cardinality/one
  :db/valueType :db.type/bigdec
  :db/doc "current balance of the account"
  :db/index true
  :db/noHistory true
  :db.install/_attribute :db.part/db}]

By setting :db/noHistory to true, you have disabled history. What this means practically is that during every indexing job, the transactor will drop both retraction Datoms and assertions that have been retracted for this attribute, keeping only the most recently asserted value. You can see then that it's possible for some history data about this attribute to be present between indexing jobs, so you shouldn't write your application logic to depend on there never being any history present for the attribute.

Schema and Example Data

First, let's put the schema that was created in the previous chapter in a file in the resources folder of the project, called schema.edn. This way you can easily work with it in the project.

Next, add some example data in resources/example-data.edn. You can use this data to help you while exploring the Datomic API, developing the data access code, and later it can even form the basis of test cases for testing.

;; A test user, with account and charges
[{:db/id #db/id[:db.part/account]
  :user/login "janed"
  ;; bcrypted "totalanon"
  :user/password "$2a$11$W2juQqRpaxVqXt4u..4qz.asyhbfR53K1a3stjQ3wpUYOCcagH8VK"
  :user/email "[email protected]"
  :user/account
  {:account/type :account.type/paid
   :account/transaction
   #{{:transaction/type :transaction.type/charge
      :transaction/amount 7.99M}
     {:transaction/type :transaction.type/payment
      :transaction/amount -7.99M}
     {:transaction/type :transaction.type/charge
      :transaction/amount 2.55M}}
   :account/current-balance 2.55M}}]
;; A couple of tasks
[{:db/id #db/id[:db.part/task]
  :task/title "Write to Robert about Number of the Beast"
  :task/description "He should know the first part was meandering and slow, and the
                     ending was just self indulgent. "Meet Lazarus, live happily
                     ever after." Please!"
  :task/status :task.status/todo
  :task/issue-id "HOME-11"
  :task/tag #{{:tag/name "Home"}
              {:tag/name "Writing"}}
  :task/user {:user/login "janed"}}
 {:db/id #db/id[:db.part/task]
  :task/title "Disappear into anonymity"
  :task/description "Jane Doe can't be a public figure."
  :task/status :task.status/in-progress
  :task/issue-id "WORK-1"
  :task/tag #{{:tag/name "Work"}
              {:tag/name "Important"}}
  :task/user {:user/login "janed"}}
 ;; Add a subtask
 {:db/id #db/id[:db.part/task]
  :task/title "Pack clothes"
  :task/description "Focus on neutral colors, hoodies."
  :task/status :task.status/todo
  :task/issue-id "WORK-2"
  :task/user {:user/login "janed"}
  :task/parent {:task/issue-id "WORK-1"}}]

A test user, “janed,” has been created, so you can continue the saga of Jane Doe. Her password is “totalanon,” but it's been encrypted with bcrypt. She has an account with some transactions, and a few tasks including one that's a subtask.

Setting Up for Development

While developing you often want to quickly iterate and experiment with an in-memory database. Let's create a dev namespace that lets you do just that. Create a dev folder in the project root (remember dev was added to the source-paths) and add a file called dev.clj to that folder, which looks like this:

(ns dev
  (:require [clojure.java.io :as io]
            [clojure.pprint :refer (pprint)]
            [datomic.api :as d])
  (:import datomic.Util))
(def dev-db-uri
  "datomic:mem://dev-db")
(def schema
  (io/resource "schema.edn"))
(def example-data
  (io/resource "example-data.edn"))
(defn read-txs
  [tx-resource]
  (with-open [tf (io/reader tx-resource)]
    (Util/readAll tf)))
(defn transact-all
  ([conn txs]
   (transact-all conn txs nil))
  ([conn txs res]
   (if (seq txs)
     (transact-all conn (rest txs) @(d/transact conn (first txs)))
     res)))
(defn initialize-db
  "Creates db, connects, transacts schema and example data, returns conn."
  []
  (d/create-database dev-db-uri)
  (let [conn (d/connect dev-db-uri)]
    (transact-all conn (read-txs schema))
    (transact-all conn (read-txs example-data))
    conn))
(defonce conn nil)
(defn go
  []
  (alter-var-root #'conn (constantly (initialize-db))))
(defn stop
  []
  (alter-var-root #'conn
                  (fn [c] (when c (d/release c)))))

You can see the usage of the Datomic API, conveniently called datomic.api as a dependency. The entire Clojure API for Datomic is in this namespace, so you will be seeing this dependency quite a bit. The other import is datomic.Util, which includes a handy static method for reading schema and other Datomic data from files.

Next, configure the URI for the in-memory Datomic db, datomic:mem://dev-db. The format of URIs for a Datomic DB varies based on the type of storage medium in use, but the general format is datomic:<type>://<connection params specific to the storage>/<db name>.

We have two utility functions, read-txs and transact-all, that deal with reading transaction data from a file and writing a collection of transactions. We also have a function, initialize-db, that sets up the database for development, adding both the schema and the example data. Let's return to these in a moment.

The two main functions used in our example are go and stop. The go function initializes the in-memory database and connects to it, binding the Datomic connection object to the conn var. The stop function drops the in-memory database and releases the connection.

This provides a nice development workflow: set up a database, work with it in the REPL, and reset it to the starting state.

Connections and dbs

The Datomic connection object represents Datomic's connection to both the transactor and the storage layer. Since you're using an in-memory database, both of these are in-process. You can get a db value from a connection with the d/db function of the Datomic API.

dev> (def db (d/db conn))

So, what's the difference between a connection and a db? The connection is a handle, and it's used when you are sending writes to the transactor. A db, however, has value semantics. It represents the state of the database at a particular time. When you call d/db you get the latest state of the database that the Datomic peer in your application knows about. You can use the db to do all kinds of different reads.

Highlight Tour of the Read API

You can examine attributes with attribute:

dev> (d/attribute db :task/title)
#AttrInfo{:id 67 :ident :task/title :value-type :db.type/string
          :cardinality :db.cardinality/one :indexed false
          :has-avet false :unique nil :is-component false
          :no-history false :fulltext true}

You can see the transaction number, or t value as the Datomic docs describe it, of the most recent transaction in a db with basis-t:

dev> (d/basis-t db)
1021

You can retrieve the transaction entity id for a given t value with t->tx.

dev> (d/t->tx 1021)
13194139534333

You can get low-level access to the Datoms in a specific index with datoms.

dev> (first (d/datoms db :aevt))
#datom[0 10 :db.part/db 13194139533312 true]
dev> ;; or seek to specific parts of the index
dev> (first (d/datoms db :aevt :db/doc :task/title))
#datom[67 62 "title of the task" 13194139534313 true]

You can then get the entity id for a given ident with entid.

dev> (d/entid db :task/title)
67

And then back again with ident:

dev> (d/ident db 67)
:task/title

The Entity API

One of the most commonly used features in Datomic is the entity API. It gives you access to entity data in a convenient, idiomatic way for Clojure. Let's take a look at Jane Doe. But how do you access her record? She wasn't given an ident, so you can't just use that keyword. You have to access her entity id somehow.

Here you can take advantage of a great feature of identity attributes: they can be used in place of entity ids in every place that accepts them. When used this way, they're called “lookup refs” and the format is:

[attribute-name value]

We know Jane's login from the example data is “janed,” and since login is an identity attribute, you can use that as a lookup ref for her entity. The entity API is accessed with the entity function.

dev> (d/entity db [:user/login "janed"])
{:db/id 285873023222776}

You might be thinking that this looks strange—there's definitely supposed to be more data about Jane than just an id. Now is a good time to step back and learn a few things about the entity maps returned by Datomic's entity API.

They are lazily loaded. Only the :db/id is present at first.

• They work with most of Clojure's map functions.
• You navigate to related entities through normal map access.

Here are some examples of how you can treat entities like maps:

dev> (def jane (d/entity db [:user/login "janed"]))
#'dev/jane
dev> (keys jane)
(:user/login :user/password :user/email :user/account)
dev> (:user/login jane)
"janed"
dev> (vals jane)
("janed" "$2a$11$W2juQqRpaxVqXt4u..4qz.asyhbfR53K1a3stjQ3wpUYOCcagH8VK"
 "[email protected]" {:db/id 285873023222777})
dev> (into {} jane)
{:user/login "janed", :user/password "$2a$11$W2j ...<snip>",
 :user/email "[email protected]", :user/account {:db/id 285873023222777}}
dev> (get-in jane [:user/account :account/type])
:account.type/paid

The Datomic API also includes a function that loads in all of the entity's attributes, as well as every component entity, touch. Here it is used on Jane's account entity.

dev> (d/touch (:user/account jane))
{:db/id 285873023222777, :account/type :account.type/paid,
 :account/current-balance 2.55M,
 :account/transaction #{
  {:db/id 285873023222779, :transaction/type :transaction.type/payment,
   :transaction/amount -7.99M}
  {:db/id 285873023222778, :transaction/type :transaction.type/charge,
   :transaction/amount 7.99M}
  {:db/id 285873023222780, :transaction/type :transaction.type/charge,
   :transaction/amount 2.55M}}}

You see how the results contain all of the attributes of Jane's account, plus all of the related transactions since they're components.

The Query API

There are two ways to access the query API: the more traditional q function, and a query function that takes arguments in a slightly different format and supports setting a query timeout. Let's use the q function here, as it's considered more idiomatic for most types queries.

Here's a simple example, adapted from the earlier section on queries.

(d/q '[:find ?login :in $ ?email
       :where [?user :user/email ?email]
              [?user :user/login ?login]]
     db "[email protected]")
#{["janed"]}

This query looks up the login of the user with a given email. You see that q takes the query as the first parameter. After that, there are some new things here. First, you see this :in form that wasn't in the previous queries. This is used to specify the inputs to the query, in the same positional order as they are passed to q.

Here the db is the first argument, and it's called $ in the inputs. This is how you refer to the db or dbs—it's possible to query against more than one db!—that the query is targeting. Much like the reader-macro syntax for Clojure anonymous functions, if there's only one input db you can just call it $, and it's used as the implicit database for all of your where patterns. If there are multiples, you need to delineate them with $db1 $db2, for example, and each pattern needs to begin with the datasource it's targeting.

The second argument is an input binding that can be used anywhere in the query. In this query you're passing in the email address that you're looking for. You can have an arbitrary number of these inputs. You can also pass in collections, in which case you need to slightly change the format of the input binding. Here is the same query, except this time you're looking for a number of email address logins.

(d/q '[:find ?login :in $ [?email ...]
       :where [?user :user/email ?email]
              [?user :user/login ?login]]
      db
      ["[email protected]" "[email protected]" "[email protected]"])

Notice how this uses ?email as the binding form. This is the required format for bindings where you're binding a collection.

The Pull API

Often you want to find some specific entities with a query, and then retrieve more information about those entities. You can do this with the entity API, but that can get somewhat verbose and isn't very declarative and elegant, particularly when compared to the queries. To solve this problem, Datomic introduced the pull API. This let's you describe the “shape” of data you want to return about an entity or entities, and you can even integrate it directly into your queries.

When you use the pull API, normal Clojure data structures are returned: vectors for collections of results, maps for data about entities, and sets for cardinality-many relationships. Let's look at an example, building on the previous query, but returning the user's login and some information about their account.

dev> (d/q '[:find (pull ?user [:user/login
                               {:user/account [{:account/type [:db/ident]}]}])
            :in $ ?email
            :where [?user :user/email ?email]]
           db "[email protected]")
[[{:user/login "janed", :user/account
   {:account/type {:db/ident :account.type/paid}}}]]

In pull specs, vectors specify lists of attributes to return, and the maps specify relationships. Here you are specifying that you want the :user/login, and then in the account via the :user/account relationship, you want the :db/ident of the :account/type.

Using pull specs as part of queries is often the best way to retrieve data that's meant to be returned to an external system, since the results can be directly serialized as edn and sent over the wire without any further transformation.

Transactions

You can perform transactions with the transact function. It accepts a connection and the transaction data, and returns a future containing a transaction result map. Let's take a look at an example, and what the various parts of the result map mean.

dev> (pprint @(d/transact conn [{:db/id (d/tempid :db.part/user)
                                 :task/title "Hello world"}]))
{:db-before datomic.db.Db@c7f1e174,
 :db-after datomic.db.Db@d62413b7,
 :tx-data
 [#datom[13194139534349 50 #inst "2016-02-06T20:22:19.369-00:00"
         13194139534349 true]
  #datom[17592186045454 67 "Hello world" 13194139534349 true]],
 :tempids {-9223350046623220340 17592186045454}}

The result map contains a :db-before key that contains the db value from immediately before the transaction, and a :db-after key that contains the db value immediately after it. You can pull either of these dbs directly out of the map and work with them as with any other db value. Using the :db-after in particular is quite common.

The :tx-data key contains a collection of all of the Datoms written during this transaction. Here you see there are two: one for the :task/title Datom that was asserted, and another for the transaction entity.

Finally, the :tempids key contains a mapping from the tempids created for the transaction to the actual entity ids written in the database. In this case the entity created has an id of 17592186045454. Let's see what happens when you try to assert a duplicate Datom for this entity.

dev> (pprint @(d/transact conn
                          [[:db/add 17592186045454
                            :task/title "Hello world"]]))
{:db-before datomic.db.Db@d62413b7,
 :db-after datomic.db.Db@10110222,
 :tx-data
 [#datom[13194139534351 50 #inst "2016-02-08T20:27:16.653-00:00"
         13194139534351 true]],
 :tempids {}}

You tried to add the Datom [17592186045454 :task/title "Hello world"], but that Datom already existed. Notice that the transaction succeeds, but the :tx-data key only has the Datom about the transaction entity. You can see that the Datom assertion is idempotent. If you did the same experiment in the other direction—that is, we retracted the same Datom twice—you'd see a similar result: only the first transaction would have had that retraction Datom in its :tx-data, and the second would only include the transaction Datom.

Time Travel Will Never Be Impossible Forever

Once confined to fantasy and science fiction, time travel is now simply an engineering problem.

—Michio Kaku

Datomic's treatment of time as first class, and the database as a value, gives you as a developer the power of time travel. Let's look at the more obvious type of time travel in Datomic: journeying to the past.

Let's start by simulating time advancing forward by adding some data. Create a new task with an issue-id, and then use the issue-id to upsert several new values of the task's description. Each time you'll capture the resulting db value.

dev> (def db1 (:db-after @(d/transact conn [{:db/id (d/tempid :db.part/user)
                                        :task/issue-id "Hello"}])))
dev> (def db2 (:db-after @(d/transact conn [{:db/id (d/tempid :db.part/user)
                                        :task/issue-id "Hello"
                                        :task/description "First description"}])))
dev> (def db3 (:db-after @(d/transact conn [{:db/id (d/tempid :db.part/user)
                                        :task/issue-id "Hello"
                                        :task/description "Second description"}])))

It's time to introduce another Datomic API function: as-of, which takes a db value and a t value—the result of calling basis-t, a transaction id, or a date—and returns a new db value that is of that point in time. Now let's see how you can travel back in time.

dev> (def now-db (d/db conn)) ;; get the most current db
dev> (:task/description (d/entity now-db [:task/issue-id "Hello"]))
"Second description"
dev> (:task/description (d/entity (d/as-of now-db (d/basis-t db2))
                                  [:task/issue-id "Hello"]))
"First description"

You can use the db from the past anywhere in the Datomic API that expects a db: even queries.

dev> (def description-query '[:find ?i . :in $ ?desc
                              :where [?i :task/description ?desc]])
dev> (d/q description-query now-db "Second description")
17592186045446
dev> (d/q description-query (d/as-of now-db (d/basis-t db2)) "First description")
17592186045446

What's perhaps more unexpected is that you can travel into the future—at least, into a speculative possible future like the kind that Ebenezer Scrooge was brought to in A Christmas Carol. This is done with the somewhat innocuously named with function in the Datomic API. It takes a db and some transaction data, and returns the result of what would occur if the data were transacted, including a :db-after key with a new db value. This lets you chain multiple simulated transactions together, like so:

dev> (def future-db (-> now-db
                        (d/with [{:db/id (d/tempid :db.part/user)
                                  :task/issue-id "Hello"
                                  :task/description "Third description"}])
                        :db-after
                        (d/with [{:db/id (d/tempid :db.part/user)
                                  :task/issue-id "Hello"
                                  :task/title "Hello world"}])
                        :db-after))
dev> (d/touch (d/entity future-db [:task/issue-id "Hello"]))
{:db/id 17592186045446, :task/description "Third description",
 :task/title "Hello world", :task/issue-id "Hello"}

Ensuring Data Integrity with Transaction Functions

Next, let's add the function that creates users. Since this function needs to perform a transaction, it will take a Datomic connection rather than db. However, there's one nagging problem here. Since user logins are upsertable, you need a way to ensure that the user you're creating doesn't have a login that's already in use.

You could try using the availability check function to see if it already exists. This would cover most of the cases, but if you have a lot of experience with databases you might already see the problem with this approach. Whenever you do a read and then a write in this way, you open yourself up to a race condition where, in this case, a user with that login gets created by someone else between the read and the write.

Relational databases solve this with transactions spanning both reads and writes and complicated locking semantics. Datomic doesn't need that because it solves the problem with transaction functions.

Transaction functions are run by the transactor as part of transaction processing. Datomic comes with two built-in funtions: a compare-and-swap operation, and an entity retraction operation. But neither of these functions quite meets our needs. What is needed is a transaction function that will look to see if the ident value exists at the time of the transaction.

In Datomic, functions are represented as entities with a :db/fn attribute that has the value of a Datomic function object. You can create function objects by calling the Datomic API's function method with a map of information about that function, or with the #db/fn reader literal with a map in that same format. The map describes what language the function is in (either Java or Clojure), what parameters it accepts, a string containing the function body, and optional import and require lists.

Another nifty feature of Datomic function objects is that they are callable as regular Clojure functions. Here's a simple example that adds two numbers:

(def add
  (d/function {:lang "clojure"
               :params '[x y]
               :code "(+ x y)"}))
(add 1 2) ;; => 3

Transaction functions are a special type of Datomic function. They have some additional requirements:

• The first argument is a Datomic db.
• They should return transaction data, or throw an exception to fail the transaction.

Let's create a transaction function that will ensure that an identity value for a given attribute doesn't exist, and add it to the schema file.

{:db/id #db/id[:db.part/task]
 :db/ident :add-identity
 :db/fn #db/fn{:lang "clojure"
               :params [db e ident-attr value]
               :code "(if (d/entity db [ident-attr value])
                        (throw (ex-info (str value " already exists for "
                                             ident-attr)
                                        {:e e
                                         :attribute ident-attr
                                         :value value}))
                        [[:db/add e ident-attr value]])"}}

This function takes a db value, an entity id to add the identity for, the identity attribute, and the value. The function looks to see if that identity value already exists, and if so it throws an exception. Otherwise, it returns transaction data that adds the identity to the entity id. Give it an ident, :add-identity, so you can refer to it in transactions.

You can make use of this in the function to create users.

(defn create
  "Attempts to create a new user entity with given login, password,
   and email. If paid? is true, creates a paid account and link to the
   user, otherwise creates a free account. Returns the transaction
   data if succesful. Will throw an exception if the login or email
   already belong to another user."
  ([conn login password email]
   (create conn login password email false))
  ([conn login password email paid?]
   (let [tempid (d/tempid :db.part/account)
         user-tx [{:db/id tempid
                   :user/email email
                   :user/password (password/encrypt password)
                   :user/account {:account/type (if paid?
                                                  :account.type/paid
                                                  :account.type/free)}}
                  ;; db function ensures login doesn't exist already
                  [:add-identity tempid :user/login login]]]
     @(d/transact conn user-tx))))

You can see how the calling syntax for database functions is fairly close to the way the list-syntax for transaction data works. The first list element is the transaction function name, the following elements are the arguments. You didn't need to use this transaction function for emails since the email was made with a unique value attribute, and Datomic throws an exception that aborts the transaction if there's a duplicate.

Wrapping Up

Notice that it took very little actual code to power a large amount of functionality, as shown in this chapter. Datomic's powerful API, and the investments made in our application data model, really pay off when it comes to building the application and getting a lot of leverage out of concise code.

Testing is something outside of the scope of this chapter, but you can take advantage of the value semantics of Datomic dbs, and the ease with which you can stand up in-memory Datomic databases to write a thorough automated test suite for your application code with a minimum of machinery.

Hard (and Soft) Limits

The most obvious limitation of Datomic is that there is only one writer, the transactor. This means that write scalability is limited to vertical scalability only, and applications built on Datomic can only handle the write volume that the single transactor node can handle. In practice, this is not generally a critical limitation, since it's not difficult to achieve hundreds of sustained writes per second through the transactor, with bursts into the thousand-per-second range.

If your application is working at this kind of data scale, then generally you are already outside the range where you would be able to get good performance out of a relational database without tuning and designing your schema to optimize your read patterns. In other words, you've already given up many of the benefits of a relational database, and one of the other NoSQL databases designed around this kind of data volume might be a better fit than Datomic.

Another limit is the number of “schema” elements. This includes both schema attributes and partitions. There is a hard maximum of 2∧20 of these—a little over 1 million. It is unlikely you'll ever reach this limit—in fact it's likely that something has gone terribly wrong if you have—but it's something to be aware of.

The max number of entity ids in any partition is 2∧42, which is around 4.4 trillion. This is purely a theoretical limitation, included here just so that you don't worry about it, because it is absolutely impossible to get this many entities into Datomic, period. Principally this is due to the final, most imposing limitation.

There is a soft limit on the total number of Datoms in a database of around 10 billion. This includes assertions, retractions, and history. This is a soft limit, because nothing will stop you from adding more, and in fact Cognitect is aware of a production Datomic database with more than 14 billion Datoms. This is a limit to be concerned with just the same for three reasons.

• Cognitect's automated testing of Datomic includes database sizes of up to 10 billion Datoms, so there's reason to be confident that Datomic will work at that volume. Above that, however, you are officially in uncharted waters.
• The index data structures start to become quite bloated at this size, such that query performance is likely to suffer and there will be increased resource demands on every peer node.
• Transactor performance will take a real hit as the number of directory nodes and segments increase, the magnitude of which depends on how varied your write patterns are and how well you have designed your partitions. If the transactor's re-indexing job needs to rewrite an excessively large number of data segments because the indexes have become very large and your writes are spread across many segments, it is possible that it will not be able to complete and your database will simply fall over to the point of requiring a restore from a previous safe backup.