Contents

About the Author

About the Technical Reviewer

Acknowledgments

Introduction

 Part I: Introducing Hadoop and Its Security

 Chapter 1: Understanding Security Concepts

Introducing Security Engineering

Security Engineering Framework

Psychological Aspects of Security Engineering

Introduction to Security Protocols

Securing a Program

Non-Malicious Flaws

Malicious Flaws

Securing a Distributed System

Authentication

Authorization

Encryption

Summary

 Chapter 2: Introducing Hadoop

Hadoop Architecture

HDFS

Inherent Security Issues with HDFS Architecture

Hadoop’s Job Framework using MapReduce

Inherent Security Issues with Hadoop’s Job Framework

Hadoop’s Operational Security Woes

The Hadoop Stack

Main Hadoop Components

Summary

 Chapter 3: Introducing Hadoop Security

Starting with Hadoop Security

Introducing Authentication and Authorization for HDFS

Authorization

Real-World Example for Designing Hadoop Authorization

Fine-Grained Authorization for Hadoop

Securely Administering HDFS

Using Hadoop Logging for Security

Monitoring for Security

Tools of the Trade

Encryption: Relevance and Implementation for Hadoop

Encryption for Data in Transit

Encryption for Data at Rest

Summary

 Part II: Authenticating and Authorizing Within Your Hadoop Cluster

 Chapter 4: Open Source Authentication in Hadoop

Pieces of the Security Puzzle

Establishing Secure Client Access

Countering Spoofing with PuTTY’s Host Keys

Key-Based Authentication Using PuTTY

Using Passphrases

Building Secure User Authentication

Kerberos Overview

Installing and Configuring Kerberos

Preparing for Kerberos Implementation

Implementing Kerberos for Hadoop

Securing Client-Server Communications

Safe Inter-process Communication

Encrypting HTTP Communication

Securing Data Communication

Summary

 Chapter 5: Implementing Granular Authorization

Designing User Authorization

Call the Cops: A Real-World Security Example

Determine Access Groups and their Access Levels

Implement the Security Model

Access Control Lists for HDFS

Role-Based Authorization with Apache Sentry

Hive Architecture and Authorization Issues

Sentry Architecture

Implementing Roles

Summary

 Part III: Audit Logging and Security Monitoring

 Chapter 6: Hadoop Logs: Relating and Interpretation

Using Log4j API

Loggers

Appenders

Layout

Filters

Reviewing Hadoop Audit Logs and Daemon Logs

Audit Logs

Hadoop Daemon Logs

Correlating and Interpreting Log Files

What to Correlate?

How to Correlate Using Job Name?

Important Considerations for Logging

Time Synchronization

Hadoop Analytics

Splunk

Summary

 Chapter 7: Monitoring in Hadoop

Overview of a Monitoring System

Simple Monitoring System

Monitoring System for Hadoop

Hadoop Metrics

The jvm Context

The dfs Context

The rpc Context

The mapred Context

Metrics and Security

Metrics Filtering

Capturing Metrics Output to File

Security Monitoring with Ganglia and Nagios

Ganglia

Monitoring HBase Using Ganglia

Nagios

Nagios Integration with Ganglia

The Nagios Community

Summary

 Part IV: Encryption for Hadoop

 Chapter 8: Encryption in Hadoop

Introduction to Data Encryption

Popular Encryption Algorithms

Applications of Encryption

Hadoop Encryption Options Overview

Encryption Using Intel’s Hadoop Distro

Step-by-Step Implementation

Special Classes Used by Intel Distro

Using Amazon Web Services to Encrypt Your Data

Deciding on a Model for Data Encryption and Storage

Encrypting a Data File Using Selected Model

Summary

 Part V: Appendices

 Appendix A: Pageant Use and Implementation

Using Pageant

Security Considerations

 Appendix B: PuTTY and SSH Implementation for Linux-Based Clients

Using SSH for Remote Access

 Appendix C: Setting Up a KeyStore and TrustStore for HTTP Encryption

Create HTTPS Certificates and KeyStore/TrustStore Files

Adjust Permissions for KeyStore/TrustStore Files

 Appendix D: Hadoop Metrics and Their Relevance to Security

Index