Preface

At TheLadders, we’ve been using Storm since it was introduced to the world (version 0.5.x). In those early days, we implemented solutions with Storm that supported noncritical business processes. Our Storm cluster ran uninterrupted for a long time and “just worked.” Little attention was paid to this cluster, as it never really had any problems. It wasn’t until we started identifying more business cases where Storm was a good fit that we started to experience problems. Contention for resources in production, not having a great understanding of how things were working under the covers, sub-optimal performance, and a lack of visibility into the overall health of the system were all issues we struggled with.

This prompted us to focus a lot of time and effort on learning much of what we present in this book. We started with gaining a solid understanding of the fundamentals of Storm, which included reading (and rereading many times) the existing Storm documentation, while also digging into the source code. We then identified some “best practices” for how we liked to design solutions using Storm. We added better monitoring, which enabled us to troubleshoot and tune our solutions in a much more efficient manner.

While the documentation for the fundamentals of Storm was readily available online, we felt there was a lack of documentation for best practices in terms of dealing with Storm in a production environment. We wrote a couple of blog posts based on our experiences with Storm, and when Manning asked us to write a book about Storm, we jumped at the opportunity. We knew we had a lot of knowledge we wanted to share with the world. We hoped to help others avoid the frustrations and pitfalls we had gone through.

While we knew that we wanted to share our hard-won experiences with running a production Storm cluster—tuning, debugging, and troubleshooting—what we really wanted was to impart a solid grasp of the fundamentals of Storm. We also wanted to illustrate how flexible Storm is, and how it can be used across a wide range of use cases. We knew ours were just a small sampling of the many use cases among the many companies leveraging Storm.

The result of this is Storm Applied. We’ve tried to identify as many different types of use cases as possible to illustrate how Storm can be used in many scenarios. We cover the core concepts of Storm in hopes of laying a solid foundation before diving into tuning, debugging, and troubleshooting Storm in production. We hope this format works for everyone, from the beginner just getting started with Storm, to the experienced developer who has run into some of the same troubles we have.

This book has been the definition of teamwork, from everyone who helped us at Manning to our colleagues at TheLadders, who very patiently and politely allowed us to test our ideas early on.

We hope you are able to find this book useful, no matter your experience level with Storm. We have enjoyed writing it and continue to learn more about Storm every day.