The data we are concerned with relates to the internal states of our running application. Collecting this data gives us more information about what's going on inside our service. The data may be to do with the goal the application is designed to achieve, or the runtime data intrinsic to the application. Obtaining this data often requires us to manually instruct our program to expose the internal data to the monitoring pipeline because only we, the service owner, know what data is meaningful, and also because it's often hard to get information such as the size of records in the memory for a cache service externally.

The ways in which applications interact with the monitoring system differ significantly. For example, if we need data about the statistics of a MySQL database, we could set an agent that periodically queries the information and performance schema for the raw data, such as numbers of SQL queries accumulated at the time, and transform them to the format for our monitoring system. In a Golang application, as another example, we might expose the runtime information via the expvar package and its interface and then find another way to ship the information to our monitoring backend. To alleviate the potential difficulty of these steps, the OpenMetrics (https://openmetrics.io/) project endeavours to provide a standardized format for exchanging telemetry between different applications and monitoring systems.