The tools described in the previous section are not application-specific. For a long-running process, it often necessary to provide information about the internal state to either a monitoring a graphing solution such as Ganglia or Graphite, or just display it in a servlet. Most of these solutions are read-only, but in some cases, the commands give the control to the users to modify the state, such as log levels, or to trigger garbage collection.
Monitoring, in general is supposed to do the following:
- Provide high-level information about program execution and application-specific metrics
- Potentially, perform health-checks for critical components
- Might incorporate alerting and thresholding on some critical metrics
I have also seen monitoring to include update operations to either update the logging parameters or test components, such as trigger model scoring with predefined parameters. The latter can be considered as a part of parameterized health check.
Let's see how it works on the example of a simple Hello World web application that accepts REST-like requests and assigns a unique ID for different users written in the Scalatra framework (http://scalatra.org), a lightweight web-application development framework in Scala. The application is supposed to respond to CRUD HTTP requests to create a unique numeric ID for a user. To implement the service in Scalatra, we need just to provide a Scalate template. The full documentation can be found at http://scalatra.org/2.4/guides/views/scalate.html, the source code is provided with the book and can be found in chapter10 subdirectory:
class SimpleServlet extends Servlet {
val logger = LoggerFactory.getLogger(getClass)
var hwCounter: Long = 0L
val hwLookup: scala.collection.mutable.Map[String,Long] = scala.collection.mutable.Map()
val defaultName = "Stranger"
def response(name: String, id: Long) = { "Hello %s! Your id should be %d.".format(if (name.length > 0) name else defaultName, id) }
get("/hw/:name") {
val name = params("name")
val startTime = System.nanoTime
val retVal = response(name, synchronized { hwLookup.get(name) match { case Some(id) => id; case _ => hwLookup += name -> { hwCounter += 1; hwCounter } ; hwCounter } } )
logger.info("It took [" + name + "] " + (System.nanoTime - startTime) + " " + TimeUnit.NANOSECONDS)
retVal
}
}First, the code gets the name parameter from the request (REST-like parameter parsing is also supported). Then, it checks the internal HashMap for existing entries, and if the entry does not exist, it creates a new index using a synchronized call to increment hwCounter (in a real-world application, this information should be persistent in a database such as HBase, but I'll skip this layer in this section for the purpose of simplicity). To run the application, one needs to download the code, start sbt, and type ~;jetty:stop;jetty:start to enable continuous run/compilation as in Chapter 7, Working with Graph Algorithms. The modifications to the file will be immediately picked up by the build tool and the jetty server will restart:
[akozlov@Alexanders-MacBook-Pro chapter10]$ sbt [info] Loading project definition from /Users/akozlov/Src/Book/ml-in-scala/chapter10/project [info] Compiling 1 Scala source to /Users/akozlov/Src/Book/ml-in-scala/chapter10/project/target/scala-2.10/sbt-0.13/classes... [info] Set current project to Advanced Model Monitoring (in build file:/Users/akozlov/Src/Book/ml-in-scala/chapter10/) > ~;jetty:stop;jetty:start [success] Total time: 0 s, completed May 15, 2016 12:08:31 PM [info] Compiling Templates in Template Directory: /Users/akozlov/Src/Book/ml-in-scala/chapter10/src/main/webapp/WEB-INF/templates SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [info] starting server ... [success] Total time: 1 s, completed May 15, 2016 12:08:32 PM 1. Waiting for source changes... (press enter to interrupt) 2016-05-15 12:08:32.578:INFO::main: Logging initialized @119ms 2016-05-15 12:08:32.586:INFO:oejr.Runner:main: Runner 2016-05-15 12:08:32.666:INFO:oejs.Server:main: jetty-9.2.1.v20140609 2016-05-15 12:08:34.650:WARN:oeja.AnnotationConfiguration:main: ServletContainerInitializers: detected. Class hierarchy: empty 2016-15-05 12:08:34.921: [main] INFO o.scalatra.servlet.ScalatraListener - The cycle class name from the config: ScalatraBootstrap 2016-15-05 12:08:34.973: [main] INFO o.scalatra.servlet.ScalatraListener - Initializing life cycle class: ScalatraBootstrap 2016-15-05 12:08:35.213: [main] INFO o.f.s.servlet.ServletTemplateEngine - Scalate template engine using working directory: /var/folders/p1/y7ygx_4507q34vhd60q115p80000gn/T/scalate-6339535024071976693-workdir 2016-05-15 12:08:35.216:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@1ef7fe8e{/,file:/Users/akozlov/Src/Book/ml-in-scala/chapter10/target/webapp/,AVAILABLE}{file:/Users/akozlov/Src/Book/ml-in-scala/chapter10/target/webapp/} 2016-05-15 12:08:35.216:WARN:oejsh.RequestLogHandler:main: !RequestLog 2016-05-15 12:08:35.237:INFO:oejs.ServerConnector:main: Started ServerConnector@68df9280{HTTP/1.1}{0.0.0.0:8080} 2016-05-15 12:08:35.237:INFO:oejs.Server:main: Started @2795ms2016-15-05 12:03:52.385: [main] INFO o.f.s.servlet.ServletTemplateEngine - Scalate template engine using working directory: /var/folders/p1/y7ygx_4507q34vhd60q115p80000gn/T/scalate-3504767079718792844-workdir 2016-05-15 12:03:52.387:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@1ef7fe8e{/,file:/Users/akozlov/Src/Book/ml-in-scala/chapter10/target/webapp/,AVAILABLE}{file:/Users/akozlov/Src/Book/ml-in-scala/chapter10/target/webapp/} 2016-05-15 12:03:52.388:WARN:oejsh.RequestLogHandler:main: !RequestLog 2016-05-15 12:03:52.408:INFO:oejs.ServerConnector:main: Started ServerConnector@68df9280{HTTP/1.1}{0.0.0.0:8080} 2016-05-15 12:03:52.408:INFO:oejs.Server:main: Started @2796mss
When the servlet is started on port 8080, issue a browser request:
Tip
I pre-created the project for this book, but if you want to create a Scalatra project from scratch, there is a gitter command in chapter10/bin/create_project.sh. Gitter will create a project/build.scala file with a Scala object, extending build that will set project parameters and enable the Jetty plugin for the SBT.
http://localhost:8080/hw/Joe.
The output should look similar to the following screenshot:
Figure 10-1: The servlet web page.
If you call the servlet with a different name, it will assign a distinct ID, which will be persistent across the lifetime of the application.
As we also enabled console logging, you will also see something similar to the following command on the console:
2016-15-05 13:10:06.240: [qtp1747585824-26] INFO o.a.examples.ServletWithMetrics - It took [Joe] 133225 NANOSECONDS
While retrieving and analyzing logs, which can be redirected to a file, is an option and there are multiple systems to collect, search, and analyze logs from a set of distributed servers, it is often also important to have a simple way to introspect the running code. One way to accomplish this is to create a separate template with metrics, however, Scalatra provides metrics and health support to enable basic implementations for counts, histograms, rates, and so on.
I will use the Scalatra metrics support. The ScalatraBootstrap class has to implement the MetricsBootstrap trait. The org.scalatra.metrics.MetricsSupport and org.scalatra.metrics.HealthChecksSupport traits provide templating similar to the Scalate templates, as shown in the following code.
The following is the content of the ScalatraTemplate.scala file:
import org.akozlov.examples._
import javax.servlet.ServletContext
import org.scalatra.LifeCycle
import org.scalatra.metrics.MetricsSupportExtensions._
import org.scalatra.metrics._
class ScalatraBootstrap extends LifeCycle with MetricsBootstrap {
override def init(context: ServletContext) = {
context.mount(new ServletWithMetrics, "/")
context.mountMetricsAdminServlet("/admin")
context.mountHealthCheckServlet("/health")
context.installInstrumentedFilter("/*")
}
}The following is the content of the ServletWithMetrics.scala file:
package org.akozlov.examples
import org.scalatra._
import scalate.ScalateSupport
import org.scalatra.ScalatraServlet
import org.scalatra.metrics.{MetricsSupport, HealthChecksSupport}
import java.util.concurrent.atomic.AtomicLong
import java.util.concurrent.TimeUnit
import org.slf4j.{Logger, LoggerFactory}
class ServletWithMetrics extends Servlet with MetricsSupport with HealthChecksSupport {
val logger = LoggerFactory.getLogger(getClass)
val defaultName = "Stranger"
var hwCounter: Long = 0L
val hwLookup: scala.collection.mutable.Map[String,Long] = scala.collection.mutable.Map() val hist = histogram("histogram")
val cnt = counter("counter")
val m = meter("meter")
healthCheck("response", unhealthyMessage = "Ouch!") { response("Alex", 2) contains "Alex" }
def response(name: String, id: Long) = { "Hello %s! Your id should be %d.".format(if (name.length > 0) name else defaultName, id) }
get("/hw/:name") {
cnt += 1
val name = params("name")
hist += name.length
val startTime = System.nanoTime
val retVal = response(name, synchronized { hwLookup.get(name) match { case Some(id) => id; case _ => hwLookup += name -> { hwCounter += 1; hwCounter } ; hwCounter } } )s
val elapsedTime = System.nanoTime - startTime
logger.info("It took [" + name + "] " + elapsedTime + " " + TimeUnit.NANOSECONDS)
m.mark(1)
retVal
}If you run the server again, the http://localhost:8080/admin page will show a set of links for operational information, as shown in the following screenshot:
Figure 10-2: The admin servlet web page
The Metrics link will lead to the metrics servlet depicted in Figure 10-3. The org.akozlov.exampes.ServletWithMetrics.counter will have a global count of requests, and org.akozlov.exampes.ServletWithMetrics.histogram will show the distribution of accumulated values, in this case, the name lengths. More importantly, it will compute 50, 75, 95, 98, 99, and 99.9 percentiles. The meter counter will show rates for the last 1, 5, and 15 minutes:
Figure 10-3: The metrics servlet web page
Finally, one can write health checks. In this case, I will just check whether the result of the response function contains the string that it has been passed as a parameter. Refer to the following Figure 10.4:
Figure 10-4: The health check servlet web page.
The metrics can be configured to report to Ganglia or Graphite data collection servers or periodically dump information into a log file.
Endpoints do not have to be read-only. One of the pre-configured components is the timer, which measures the time to complete a task—which can be used for measuring scoring performance. Let's put the code in the ServletWithMetrics class:
get("/time") {
val sleepTime = scala.util.Random.nextInt(1000)
val startTime = System.nanoTime
timer("timer") {
Thread.sleep(sleepTime)
Thread.sleep(sleepTime)
Thread.sleep(sleepTime)
}
logger.info("It took [" + sleepTime + "] " + (System.nanoTime - startTime) + " " + TimeUnit.NANOSECONDS)
m.mark(1)
}Accessing http://localhost:8080/time will trigger code execution, which will be timed with a timer in metrics.
Analogously, the put operation, which can be created with the put() template, can be used to either adjust the run-time parameters or execute the code in-situ—which, depending on the code, might need to be secured in production environments.
Note
JSR 110
JSR 110 is another Java Specification Request (JSR), commonly known as Java Management Extensions (JMX). JSR 110 specifies a number of APIs and protocols in order to be able to monitor the JVM executions remotely. A common way to access JMX Services is via the jconsole command that will connect to one of the local processes by default. To connect to a remote host, you need to provide the -Dcom.sun.management.jmxremote.port=portNum property on the Java command line. It is also advisable to enable security (SSL or password-based authentication). In practice, other monitoring tools use JMX for monitoring, as well as managing the JVM, as JMX allows callbacks to manage the system state.
You can provide your own metrics that are exposed via JMX. While Scala runs in JVM, the implementation of JMX (via MBeans) is very Java-specific, and it is not clear how well the mechanism will play with Scala. JMX Beans can certainly be exposed as a servlet in Scala though.
The JMX MBeans can usually be examined in JConsole, but we can also expose it as /jmx servlet, the code provided in the book repository (https://github.com/alexvk/ml-in-scala).