Monitoring with Cloud Monitoring

Monitoring is the practice of collecting measurements of key aspects of infrastructure and application performance. Examples include average CPU utilization over the past minute, the number of bytes written to a network interface, and the maximum memory utilization over the past hour. These measurements, which are known as metrics, are made repeatedly over time and constitute a time series of measurements.

Dashboards

Dashboards are visual displays of time series. For example, Figure 8.1 shows a simple dashboard with several time-series metrics.

Dashboards are customized by users to show data that helps monitor and meet service-level objectives or diagnose problems with a particular service. Dashboards are especially useful for determining correlated failures. For example, one dashboard may indicate that you are not meeting your service-level objective of response time for purchase transactions. You might then switch to a dashboard that shows performance metrics for your application server, database, and cache. From there, you could determine which of the main three components of the systems are having performance issues. If, for example, the cache hit rate is unusually low, that can lead to greater IO load on the database, which can increase the latency of purchase transactions. Crafting informative dashboards is often an iterative process. You may know key performance indicators that you should monitor when you first create a service, but over time you may monitor additional metrics. For example, if you had an incident because of a lag in a messaging service, you may want to monitor several metrics of the messaging service to catch potential problems before they occur.

Of course, watching a dashboard for potential problems is not the most efficient use of DevOps engineers' time. Reliable systems also depend on alerting and automation to notify engineers when a problem arises and needs their attention.

Alerting with Cloud Monitoring

Alerting is the process of monitoring metrics and sending notifications when some custom-defined conditions are met. The goal of alerting is to notify someone when there is an incident or condition that cannot be automatically remediated and that puts service-level objectives at risk. If you are concerned about having enough CPU capacity for intermittent spikes in workload, you may want to run your application servers at an average of 70 percent utilization or less. If utilization is greater than 80 percent, then you may want to be notified.

Logging with Cloud Logging

Cloud Logging is a centralized log management service. Logs are collections of messages that describe events in a system. Unlike metrics that are collected at regular intervals, log messages are written only when a particular type of event occurs. Here are some examples:

• If a new user account is added to an operating system and that account is granted root privileges, a log message with details about that account and who created it may be written to an audit log.
• When a database connection error message is received by an application, the application may write a log message with data about the time and the type of operation that would have been performed.
• Applications running on the Java virtual machine may need to reclaim memory that is no longer in use. This process is called garbage collection. Log messages may be written at the start and end of garbage collection.

Cloud Logging provides the ability to store, search, analyze, and monitor log messages from a variety of applications and cloud resources. An important feature of Cloud Logging is that it can store logs from virtually any application or resource, including GCP resources, other cloud resources, or applications running on premises. The Cloud Logging API accepts log messages from any source.

Some of the most important features of Cloud Logging are its search and analysis features. In addition to text searching, logs can be easily exported to BigQuery for more structured SQL-based analysis. Logging is also integrated with Cloud Monitoring, so log messages can trigger notifications from the alerting system. Metrics can also be created from log messages, so the functionality of Cloud Monitoring is available for logs as well.

Applications and resources can create a large volume of log data. Cloud Logging will retain log messages for 30 days. If you would like to keep logs for longer, then you can export them to Cloud Storage or BigQuery for longer retention periods.

Log Analytics is a new feature in Pre-GA at the time of writing that allows you to directly query log data using BigQuery and SQL. When using Log Analytics, your log data is stored in a data set that is managed by BigQuery.

Logs can also be streamed to Cloud Pub/Sub if you would like to use third-party tools to perform near real-time operations on the log data.

To summarize, Cloud Operations Suite provides three essential tools for observing and understanding the state of services: monitoring, alerting, and log management. Monitoring collects basic data about a wide array of resource characteristics and their performance over time. Alerting evaluates metric data to determine when some predefined conditions are met, such as high CPU utilization for an extended period of time. Logging collects messages about specific events related to application and resource operations. The combination of all three provides a foundation for understanding how complicated systems function and can help improve reliability.

Open Source Observability Tools

In addition to Google Cloud–specific services like Cloud Monitoring and Cloud Logging, you may also encounter widely used open source tools for monitoring and visualization. Prometheus and Grafana are two such open source tools.

Continuous Delivery

Continuous delivery (CD) is the practice of releasing code soon after it is completed and after it passes all tests. CD is an automated process—there is usually no human in the loop. This allows for rapid deployment of code, but since humans other than the developer do not need to review the code prior to release, there is a higher risk of introducing bugs than there would be if a human were in the loop.

In practice, software developers should write comprehensive tests to detect bugs as early as possible. Teams that regularly write comprehensive tests and deploy in small increments find the trade-off favors rapid release. Other teams may find that they optimize their release management practice by having a human quality assurance (QA) engineer review the code and write additional tests.

When a human such as a QA engineer is involved, code may be ready for deployment but not deployed until it is reviewed.

Continuous Integration

Continuous integration (CI) is the practice of incorporating new code into an established code base as soon as it is complete. For example, a software engineer may have a new algorithm to reduce the time and CPU required to perform a calculation. The code change is isolated to a single function. The software engineers tested the change in their local development and environment and are ready to perform the final tests and release the code to the staging environment.

If the software engineers had to build and compile the code manually, execute tests, and then deploy the code to the production environment for each change, it would make sense to wait until there were multiple changes and then release the code. After all, each of those steps can be time-consuming, and a software engineer's time should be maximized for developing features and services. Bundling multiple changes to code and releasing them as a batch was a common practice before the advent of DevOps. Today, we understand that continually integrating changes into baseline code and then testing and releasing that code is a faster and more efficient way to build software.

Another consequence of manual build and test is that it is much more likely than an automated process to introduce random and unexpected inconsistencies in the process.

Software engineers are able to integrate new code continuously into the production code base because of automated tools.

Tools include version control and test and build tools such as GitHub and Cloud Source Repository. GitHub is a widely used code repository, and Cloud Source Repository is Google Cloud's version control system. Developers can more easily collaborate when they all use a single repository that can keep track of changes to every component in a system. GitHub and Cloud Source Repository support other common practices, such as automatically executing tests or other checks such as verifying stylistic coding conventions and using the results as quality gates. Lint, for example, is the first static code analyzer developed for C. Today there are static code checkers for the many commonly used programming languages.

Jenkins is a widely used CI tool that builds and tests code. Jenkins supports plugins to add functionality, such as integration with a particular version control system or support for different programming languages. Google Cloud Build is a GCP service that provides software building services, and it is integrated with other GCP services, such as Cloud Source Repository.

CI and continuous delivery (CD) are widely used practices that contribute to systems reliability by enabling small, incremental changes to production systems while providing mechanisms to quickly roll back problematic changes that disrupt service functionality.

Overload

One thing that service designers cannot control is the load that users may want to place on a system at any time. It is a good practice to assume that, at some point, your services will have more workload coming in than they can process. This can happen for a number of reasons, including the following:

• An external event, such as a marketing campaign that prompts new customers to start using a service
• An external event, such as an organizational change in a customer's company that triggers an increase in the workload that they send to your service
• An internal event, such as a bad deployment that causes an upstream service to generate more connection requests than ever generated previously
• An internal event, such as another development team releasing a new service that works as expected but puts an unexpected load on your service

While software engineers and site reliability engineers (SREs) cannot prevent overloads, they can decide how to respond to them.

One factor to consider when responding to overload is the criticality of an operation. Some operations may be considered more important than others and should always take precedence. Writing a message to an audit log, for example, may be more important from a business perspective than responding to a user query within a few seconds. For each of the following overload responses, be sure to take criticality into account when applying these methods.

Cascading Failures

Cascading failures occur when a failure in one part of a distributed system causes a failure in another part of the system, which in turn causes another failure in some other service, and so on. For example, if a server in a cluster fails, additional load will be routed to other servers in the cluster. This could cause one of those servers to fail, which in turn will further increase the load on the remaining healthy servers.

Consider the example of a database failure. Imagine that a relational database attempts to write data to persistent storage but the device is full. The low-level data storage service that tries to find a block for the data fails to find one. This failure cascades up the database software stack to the code that issued the corresponding INSERT operation.

The INSERT operation in turn returns an error to the application function that tried to write the data. That application code detects the error and decides to retry the operation every 10 seconds until it succeeds or until it tries six times and fails each time. Now imagine that multiple calls to the database are failing and responding with the same strategy of retrying.

Now the application is held up for up to an additional 60 seconds for each transaction. This slows the processing of that service's workload. This causes a backlog of requests that in turn causes other services to fail. This kind of ripple effect of failures is a significant risk in distributed systems.

Cascading failures are often caused by resource exhaustion. Too much load is placed on system resources like memory, CPU, or database connections until those resources run out. At that point, services start to fail. A common response to this is to limit the load on failing services.

One way to deal with cascading failures is to use the Circuit Breaker design pattern described earlier. This can reduce the load of services and give them a chance to catch up on their workloads. It also gives calling services a chance to degrade gracefully by implementing a degraded quality-of-service strategy.

Another way to deal with cascading failures is to degrade the quality of service. This conserves resources, such as CPU, by reducing the amount of resources allocated to each service call.

When there is a risk of a cascading failure, the best one can hope for may be to contain the number and range of errors returned while services under stress recover.

To reduce the risk of cascading failures, load test services and autoscale resources. Be sure to set autoscaling parameters so that resources are added before failures begin. When setting autoscale parameters, it is important to be aware of the initialization time required before a new instance of a resource is fully available. For example, if the startup time for a service instance is more than 2 seconds, you shouldn't expect to see an improvement in the load until 5 to 15 seconds after the service instance starts. Also, consider the time that is required to bring a resource online. Plan for some period of time between when the need to scale is detected and the time that resource becomes available. Also consider when to release resources when the load drops.

If you are quick to drop resources, you may find that you have to add resources again when the load increases slightly. Adding and releasing resources in quick succession, known as thrashing, should be avoided. It is better to have spare capacity in resources rather than try to run resources at capacity.

Testing for Reliability

Testing is an important part of ensuring reliability. There are several kinds of tests, and all should be used to improve reliability. Testing for reliability includes practices used in CI/CD but adds others as well. These tests may be applied outside of the CI/CD process. They include the following:

• Unit tests
• Integration tests
• System tests
• Reliability stress tests

Incident Management and Post-Mortem Analysis

Incidents are events that have a significant adverse impact on a service's ability to function. An incident may occur when customers cannot perform a task, retrieve information, or otherwise use a system that should be available.

Incidents may have a narrow impact, such as just affecting internal teams that depend on a service, or they can have broad impact, such as affecting all customers. Incidents are not minor problems that adversely affect only a small group, for example, a single team of developers. Incidents are service-level disruptions that impact multiple internal teams or external customers.

Incident management is a set of practices that is used to identify the cause of a disruption, determine a response, implement the corrective action, and record details about the incident and decisions made in real time.

When an incident occurs, it is good practice to do the following:

• Identify an incident commander to coordinate a response.
• Have a well-defined operations team that analyzes the problems at hand and makes changes to the system to correct or remediate the problem.
• Maintain a log of actions taken, which is helpful during post-mortem analysis.

The goal of incident management is to correct problems and restore services to customers or other system users as soon as possible. There should be less focus on why the problem occurred or identifying who is responsible than on solving the immediate problem.

After an incident is resolved, it is a good practice to conduct a post-mortem analysis. During a post-mortem meeting, engineers share information about the chain of events that led to the incident. This could be something as simple as a bad deployment or resource exhaustion, or it could be more complicated, such as a failure in a critical service, for instance, DNS name resolution or an unexpected combination of unlikely events that alone would not have been problematic but together created a difficult-to-predict set of circumstances that led to the incident.

The goal of a post-mortem analysis is to identify the causes of an incident, understand why it happened, and determine what can be done to prevent it from happening again. Post-mortems do not assign blame; this is important for fostering an atmosphere of trust and honesty needed to ensure that all relevant details are understood. Post-mortems are opportunities to learn from failures. It is important to document conclusions and decisions for future reference.

1. As an SRE, you are assigned to support several applications. In the past, these applications have had significant reliability problems. You would like to understand the performance characteristics of the applications, so you create a set of dashboards. What kind of data would you display on those dashboards?
   1. Metrics and time-series data measuring key performance attributes, such as CPU utilization
   2. Detailed log data from syslog
   3. Error messages output from each application
   4. Results from the latest acceptance tests
2. After determining the optimal combination of CPU and memory resources for nodes in a Kubernetes cluster, you want to be notified whenever CPU utilization exceeds 85 percent for 5 minutes or when memory utilization exceeds 90 percent for 1 minute. What would you have to specify to receive such notifications?
   1. An alerting condition
   2. An alerting policy
   3. A logging message specification
   4. An acceptance test
3. A compliance review team is seeking information about how your team handles high-risk administration operations, such as granting operating system users root privileges. Where could you find data that shows your team tracks changes to user privileges?
   1. In metric time-series data
   2. In alerting conditions
   3. In audit logs
   4. In ad hoc notes kept by system administrators
4. Release management practices contribute to improving reliability by which one of the following?
   1. Advocating for object-oriented programming practices
   2. Enforcing waterfall methodologies
   3. Improving the speed and reducing the cost of deploying code
   4. Reducing the use of stateful services
5. A team of software engineers is using release management practices. They want developers to check code into the central team code repository several times during the day. The team also wants to make sure that the code that is checked is functioning as expected before building the entire application. What kind of tests should the team run before attempting to build the application?
   1. Unit tests
   2. Stress tests
   3. Acceptance tests
   4. Compliance tests
6. Developers have just deployed a code change to production. They are not routing any traffic to the new deployment yet, but they are about to send a small amount of traffic to servers running the new version of code. What kind of deployment are they using?
   1. Blue/Green deployment
   2. Before/After deployment
   3. Canary deployment
   4. Stress deployment
7. You have been hired to consult with an enterprise software development that is starting to adopt Agile and DevOps practices. The developers would like advice on tools that they can use to help them collaborate on software development in the Google Cloud. What version control software might you recommend?
   1. Jenkins and Cloud Source Repositories
   2. Syslog and Cloud Build
   3. GitHub and Cloud Build
   4. GitHub and Cloud Source Repositories
8. A startup offers a software-as-a-service solution for enterprise customers. Many of the components of the service are stateful, and the system has not been designed to allow incremental rollout of new code. The entire environment has to be running the same version of the deployed code. What deployment strategy should they use?
   1. Rolling deployment
   2. Canary deployment
   3. Stress deployment
   4. Blue/Green deployment
9. A service is experiencing unexpectedly high volumes of traffic. Some components of the system are able to keep up with the workload, but others are unable to process the volume of requests. These services are returning a large number of internal server errors. Developers need to release a patch as soon as possible that provides some relief for an overloaded relational database service. Both memory and CPU utilization are near 100 percent. Horizontally scaling the relational database is not an option, and vertically scaling the database would require too much downtime. What strategy would be the fastest to implement?
   1. Shed load
   2. Increase connection pool size in the database
   3. Partition the workload
   4. Store data in a Pub/Sub topic
10. A service has detected that a downstream process is returning a large number of errors. The service automatically slows down the number of messages it sends to the downstream process. This is an example of what kind of strategy?
    1. Load shedding
    2. Upstream throttling
    3. Rebalancing
    4. Partitioning