Technology

Over the last several years, many new systems for storing, querying, and retrieving data have become available. We’ve looked at the various types of data storage systems that a project will have to deal with: document stores, graph databases, key/value stores, and other new types of databases that each bring unique capabilities to bear on the issues at hand. However, it is when these abilities are combined in a multi-model database that a development team can truly achieve data agility.

Multi-model databases allow flexible representation of a wide variety of data, with the details depending on which models the database supports. Once such a database is adopted, the team can develop expertise in that specific tool. By comparison, the polyglot persistence approach, in which several tools are combined to provide different capabilities, leads to developers becoming experts in a subset of those tools, making it harder to see how a change in one system will affect others.

Another benefit to combining storage in one multi-model system is the simplification of the application logic that interacts with it. If multiple stores hold the application’s data, then the application tier needs to synthesize any behaviors that cross those storage barriers. If one system holds the data in a format that isn’t a natural representation for that data, then the application tier might need to reassemble the data back into a useful form.

The designers of an application also need to take into account that modern databases vary in more than just storage formats—capabilities that were once considered critical in a database are now left out of some. Choose a database that provides the functionality you need. If you need support for ACID transactions, don’t try to implement that in your application layer; choose a database that provides that support. If you need zero-latency search after updates, make sure your database provides the search and query capability that you’ll need. Let your database be the database; save your work in the application layer for your business logic.

Process

New databases won’t solve our problems by themselves. If we take the same approach with them as we have taken with traditional databases, we will only get some of the benefit. How can we maximize our return?

When we choose a database that provides flexibility, we have to instill discipline in how we use that flexibility. If we decide that “anything goes,” we’ll break more than we fix.

The short answer is to take what we’ve learned from software engineering and DevOps and apply it to how we manage our data.

Skills

This report has covered a lot of ground. It has been suggested that we as software developers might need to accelerate our learning to better accommodate the needs of our stakeholders and to use the capabilities of our tools. Considering our perspective, we might well feel our rate of learning might have already maxed out.¹ The old adage to work smarter, not harder, applies here. By looking at a bigger picture, we may well be able to identify a far smaller set of new processes and tools on which to focus our efforts than we had originally imagined, or feared.

With that said, to move forward in the accelerating software development world, better skills with whatever data management tools you select will be useful. Learning to utilize all of the facilities of a multi-model database could have extraordinary impact because whatever procedures you pass to the database will be done to whatever scale you might require. The simple API call you make might ultimately be run in parallel in 100 or more servers. As the “typical” server grows from 8 to 16 cores, for instance, your application could well double its performance if you have chosen correctly. A skillset along these lines will pay off in the long term because, as your system scales, the same tool will simply be called on to do more. You will not only be learning new tools, but will be getting better at applying the tools you have already learned.

Conclusion

Information processing is a huge field with plenty of work for everyone. We as developers can choose to hold defensively to an old model that works for us or vigorously pursue the ideas just described. Either way, we might experience some level of personal and professional success. Plenty of us—and plenty of organizations—will survive using every choice described here…good or bad.

If we fearfully cling to our old ways of doing things, we will certainly not see our jobs getting any easier as time passes. Or they may disappear entirely. Pain is suffered by different professionals in different industries at different times. It’s true that: “If nothing changes, nothing changes. If you keep doing what you’re doing, you’re going to keep getting what you’re getting. You want change, make some.”²

The value we bring to our organizations comes from solving business problems, not from wiring components together. The more our tools and processes allow us to focus on producing that value, the better our careers will be. Agility isn’t just for software; it’s for us too.

We hope that the material in this report has helped inform your ongoing conversations with your fellow developers, as well as with DBAs, system administrators, and others in your organization.

Let’s all listen to what the data is saying!