Business Activity Monitoring (BAM) provides real-time business intelligence by capturing data as it flows through a business system. By using BAM, you can monitor a business process in real time and generate alerts when the business process needs human intervention. Although many vendors, including IBM and Oracle, sell BAM solutions, this book focuses on the Microsoft BAM tools. Microsoft’s BAM is a feature of Microsoft BizTalk Server that was first introduced in BizTalk Server 2004. The current version of BAM is a flexible infrastructure that captures data from Microsoft’s full line of service-oriented architecture (SOA) offerings: Windows Communication Foundation (WCF), Windows Workflow Foundation (WF), .NET applications, and BizTalk Server.

In this book, you’ll gain an in-depth understanding of the BAM tools in BizTalk Server 2009 and learn how to apply the tools to your project, whether you’re building on BizTalk Server, WCF, WF, or some other set of tools. If you’re a BizTalk developer, you’ll learn how to provide your end users with information about how your BizTalk application behaves. However, BAM provides value to any application built on the Microsoft platform, whether or not you’re using BizTalk Server. If you’re using WCF or WF, you can use BAM to instrument your application, often without modifying your code. If you’re writing .NET code, you can call the BAM API to make use of the low-latency data capture provided by BAM. BAM allows information workers to monitor a business process in real time, regardless of which tools are being used to implement the process.

This chapter provides an introduction to BAM. The chapter first outlines some of the technology trends that provide the context for the development of BAM. We’ll briefly review service-oriented architecture, business process management (BPM), and business intelligence (BI). Next, we describe the benefits of BAM, followed by an overview of the architecture and functions of BizTalk Server 2009. Finally, the chapter ends on an overview of BAM, describing the data sources that can be used with BAM, the observation models that define the data to capture, and the ways to present BAM data.

Technology Trends

In this section, we’ll review some recent trends in enterprise architecture. BAM builds on service-oriented architecture, business process management, and business intelligence, and a short review of these trends will be helpful in your understanding of BAM.

Service-Oriented Architecture and BAM

BAM is often used to capture data from applications built using service-oriented architecture principles. SOA is a style of architecture based upon message passing between systems, where the goal is to expose data and services in a loosely coupled fashion that makes it easy to create composite services. SOA projects typically follow four principles:

• Boundaries are explicit: Calling a service may require crossing a network, crossing a security boundary, or marshalling data between different representations. Service-oriented designs pay attention to the costs of boundary crossing; messages are passed explicitly instead of being implicitly created by method calls.
• Services are autonomous: Services are versioned and managed independently of their clients. Changes in implementation details are irrelevant to the clients.
• Services share schema and contract, not class: In SOA, there’s a bright line between the external interfaces to the service and the internal implementation of the service. The interfaces to the service are typically defined using Web Services Description Language (WSDL). Implementation details, such as classes, are hidden from the client. All interaction with the service is through endpoints defined by contracts.
• Service compatibility is based upon policy: Information such as service-level agreements and security policy isn’t captured by a service’s WSDL. Instead, tools such as WS-Policy are used to capture information on service policies.

Typical goals of SOA projects include opening up data in legacy systems to new uses, reusing functionality, composing services into new capabilities, promoting business agility, and lowering the cost of developing new systems. Often, systems within the enterprise are exposed as web services. These services can be composed into new capabilities, which are consumed by new applications. BAM is commonly used to capture messages flowing between services, and Microsoft’s implementation of BAM does an excellent job of monitoring a system built on Microsoft’s SOA tools: WCF, WF, and BizTalk Server. Several examples in the book will teach you how to use BAM in an SOA.

Business Process Management and BAM

Many BAM projects are used to monitor the behavior and performance of business processes in a business process management system. As the name implies, BPM systems manage business processes. There are two major flavors of BPM: Integration-Centric Business Process Management Systems (IC-BPMS) and Human-Centric Business Process Management Systems (HC-BPMS).

IC-BPMS are concerned with moving data between systems. The IC-BPMS marketplace evolved out of the Enterprise Application Integration (EAI) space, and IC-BPMS are focused on moving messages between software systems. As an example, consider an insurance company. When a new policy is written, the underwriting system must notify the accounting system, so it can start collecting premiums. When a claim comes in, the claims system should verify coverage using the underwriting system, and then notify the accounting system that the claim should be paid. Each of these business processes could be managed by IC-BPMS. Microsoft’s IC-BPMS offering is BizTalk Server.

HC-BPMS are concerned with workflows that connect people. As an example, consider how a distributor of medical devices manages its product catalog. A salesperson visits a hospital and receives a request for a device that is not in the catalog. The salesperson may request that the device be added to the catalog, but several departments must review the request before the item can be sold. The catalog desk will review the request and determine whether an equivalent device from another manufacturer is already carried. Perhaps the distributor carries a similar product in its own brand that it would prefer to sell. If the product is approved by the catalog desk, vendor management will need to approve the vendor, and a purchasing agent will need to agree to a price with the manufacturer. Each of these steps will require human intervention. HC-BPMS have capabilities for delegating tasks when a person is on vacation and escalating tasks if they don’t get done within the expected time. Microsoft’s HC-BPMS offering is Microsoft Office SharePoint Server with workflows created using Share-Point Designer.

In many cases, HC-BPMS and IC-BPMS are used together to implement a business process. Let’s return to the insurance claim example we described previously. In most cases, the claim will be paid immediately, and IC-BPMS (in this case, BizTalk Server) will verify coverage with the underwriting system and then notify the accounting system to pay the claim. However, the claims system runs a fraud detection algorithm that identifies suspicious claims. For those claims, IC-BPMS will initiate a human-centric workflow in HC-BPMS (in this case, Microsoft Office SharePoint Server) that will ensure the claim is reviewed before payment. This would be implemented by using BizTalk’s SharePoint adapter to write a message from the claims system into a SharePoint document library, and then initiating a SharePoint workflow based on the document arriving in the library.

One of the most common uses for BAM is to track the progress of a business process. BAM can be used to report on the status of an individual process, to summarize the state of a collection of business processes by providing aggregate data, or to send alerts to a person or system when issues arise with a process. In our example, BAM provides end-to-end visibility within both systems.

Business Intelligence and BAM

BAM adds real-time capabilities to business intelligence systems. Forrester defines BI as “A set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making.” Typical BI systems transfer data from operational systems to a data warehouse using some kind of extract-transform-load (ETL) tool. Data in the warehouse is aggregated to allow for ad hoc analysis using reporting tools. Information workers work with data in the data warehouse to understand the state of the enterprise, analyze trends, and perform “what-if” analyses. One way to think about a BI system is to view it as a way to make data in the underlying operational systems visible to the business.

BI initiatives are a powerful tool for aligning execution with strategy. Corporate Performance Management (CPM) tools can provide direct feedback on how well the tactics adopted by the organization are supporting the strategic goals established at the executive level. Well-chosen key performance indicators (KPIs) can measure the performance of the organization and allow analysis of causes of success or failure.

Traditional BI systems have limitations, however. Many BI systems don’t work in real time. Usually, the data load happens on a periodic basis: hourly, daily, or weekly. Because of this, data in a typical BI system is primarily historical and cannot be used to generate an alert when business conditions change or action is needed by management. Many BI systems expect to load their data from a traditional database and don’t support grabbing data from an in-flight process in the BPM. Adding BAM capabilities increases the value of a BI system by providing real-time data.

Don’t think of BAM as a competitor to BI, and don’t assume that your organization needs to make a choice between deploying traditional BI systems and deploying BAM. In many cases, the BAM infrastructure will be used to feed data into a traditional data warehouse. A comprehensive performance management strategy will use both BI and BAM to view both the current status of processes and the trends that drive performance.

The Benefits of BAM

BAM captures data from your business systems, providing real-time monitoring of a business process. In the same way that a BI system makes data in your transactional database visible to information workers, BAM captures data from your BPM system or SOA solution and makes it available to nontechnical users. The main benefits of BAM are

• BAM automates data capture and aggregation.
• BAM provides real-time visibility into business processes.
• BAM can alert users.
• BAM can capture data asynchronously.

Let’s review each of the advantages of BAM before drilling into the implementation details.

BAM Automates Data Capture and Aggregation

There’s an important difference between the typical data sources used by BAM and BI. BI systems typically either load data from an operational database via ETL or use hybrid online analytical processing (HOLAP) to build data aggregations over relational data. BAM loads data from events as they are processed by an application or BPM system. Events can be Biz-Talk messages, WCF service calls, WF activities, or API calls. BAM makes life very easy for the developer who wants to monitor processes built on BizTalk applications, WCF services, or WF workflows by automatically capturing data without requiring any application changes.

Once data has been captured, it can be automatically aggregated into summary views, either in real time or during a scheduled cube build.

BAM Provides Real-Time Visibility into Business Processes

The difference in data sources leads to a different emphasis between the way BAM and BI data is used. BI data is often used for historical and “what-if” analysis, or in an attempt to understand performance trends. BAM data is typically used for a short-term picture of what’s happening in an organization right now or to analyze the behavior of a business process.

BAM Can Alert Users

Because BAM provides a real-time, event-driven infrastructure, it can alert your users when something requires their attention. Alerts can be based either on a specific instance of a process, such as an order that hasn’t shipped 48 hours after being received, or on the aggregate behavior of a process, such as more than 500 orders being received in a week, a business situation that calls for extra staffing to be brought in. BAM alerts can be sent to one or more people, for example by e-mail, or can be used to initiate an automated process such as a BizTalk orchestration.

BAM Can Capture Data Asynchronously

Obviously, it would be easy for a developer to write .NET code that logged to a database each time a BizTalk message was transmitted. This would have performance implications, however, because each service call would be delayed by a network round-trip to the database server. Because BAM provides a technology known as a buffered event stream within the BAM API, with background services that capture events and write them to the BAM Primary Import (BAM PI) database, BAM can capture event data with minimal impact on the performance of the system that’s being monitored.

BizTalk Server Overview

Because BAM ships as a feature of BizTalk Server, and many BAM projects use BizTalk Server as one of the sources of data, this section will provide a brief overview of BizTalk Server. Readers with BizTalk experience will probably want to skip to the next section.

Microsoft BizTalk Server 2000 was marketed as a “.NET Enterprise server that unites, in a single product, enterprise application integration (EAI) and business-to-business (B2B) integration.” It included such features as messaging between applications, document tracking, process orchestration, and XML tools. While BizTalk’s goals have remained more or less consistent since that time, it’s grown to encompass numerous industry, business, and architectural trends. The current version integrates several technologies that make transformational change possible. BizTalk Server 2004 was a major change in the BizTalk architecture, as BizTalk moved from the component object model (COM) to .NET. Since that time, BizTalk Server 2006, BizTalk Server 2006 R2, and BizTalk Server 2009 have built on the architecture, adding support for radio frequency identification (RFID), electronic data interchange (EDI), and rich integration with the Visual Studio Team System project model.

Many Microsoft server products provide a rich set of tools focused on a common theme. As an example, SQL Server provides several tools focused on data: the core database engine, SQL Server Analysis Services, SQL Server Reporting Services, and SQL Server Integration Services. Another example is Microsoft Office SharePoint Server, which provides several tools focused on collaboration: document libraries, enterprise search, content management, forms-driven business process, team sites, and the business data catalog. The BizTalk Server tools are focused on systems integration. The tools include a messaging engine, adapters for external systems, orchestration and long-running business processes, a business rules engine, EDI tools, an RFID infrastructure, endpoint and trading partner management, and BAM. Very few BizTalk projects use every tool in the toolbox, but together these tools make up a powerful kit for building integration solutions.

Today, BizTalk is a hugely successful integration product, with over 8,000 customers worldwide. In late 2008, BizTalk’s customer base included

• 90% of the Fortune Global 100
• 12 of the 15 largest retailers in the world
• 5 of the 10 largest hotel chains in the world
• 6 of the 8 largest U.S. pharmaceutical companies
• 4 of the 5 largest U.S. electronics parts manufacturers
• 9 of the10 largest U.S. telecommunications companies
• 5 of the 8 largest U.S. chemical companies
• 4 of the 5 largest railroads in the U.S.
• 9 of the 10 largest insurance companies in the world
• 23 of 27 EU member governments

BizTalk Messaging

The core of BizTalk Server is the messaging engine, illustrated in Figure 1-1. BizTalk Messaging provides a publish-subscribe engine built on a SQL Server database called the message box. The BizTalk administrator configures a receive port that tells an adapter to listen at a specific location. As an example, a receive port might listen for XML files dropped into a specific folder. Once the adapter picks up an inbound interchange, the adapter hands it to a receive pipeline. The receive pipeline is a collection of .NET components that can validate message structure, disassemble messages from text or binary formats into XML, break batches of messages into individual messages, or do any other custom processing that the developer chooses. The receive pipeline may call a map, which transforms the inbound message into another format, and then the message is published into the message box, a SQL Server database.

Figure 1-1. Publish-subscribe messaging in BizTalk Server

The message box is the heart of BizTalk; it contains information on in-flight messages and subscriptions. Two types of service can subscribe to an inbound message: send ports and orchestrations. A send port is in some ways the mirror image of a receive port. When a send port picks up an outbound message, the message can be mapped to another format, and is then passed to a send pipeline. The send pipeline can assemble the message into a non-XML format, validate the message, encrypt the message, or perform other custom processing. Next the message is passed to an adapter that transmits the message. As an example, a send port might use the FTP adapter to transmit the file to a server on the Internet via File Transfer Protocol (FTP).

BizTalk Adapters

Adapters are the secret sauce that allows BizTalk to talk to other systems. An adapter is a piece of software that can connect to an external data source, technology, or application. Adapters have been part of BizTalk since the first version. In BizTalk Server 2004, a new adapter framework based on .NET was included in the new BizTalk architecture. In BizTalk Server 2006 R2, an additional adapter framework built on WCF was added. With most adapters, connecting to an external system is as simple as a few mouse clicks.

Most adapters can be configured as either send or receive ports, although there are a few exceptions. The SMTP adapter is an example of a send-only adapter, and the POP3 adapter is a receive-only adapter. Adapters are loosely categorized into adapters that work with specific applications, such as the SAP adapter, adapters that work with data, such as the IBM DB2 adapter, and adapters that work with technologies or protocols, such as the HTTP and MSMQ adapters. Table 1-1 lists the adapters available in BizTalk Server 2009.

BizTalk Orchestration

The orchestration engine builds on messaging to provide support for business logic and long-running business processes. The screenshot in Figure 1-2 is an example of a BizTalk orchestration that implements a design pattern called parallel convoy. This orchestration is part of a purchase order process. In this system, a purchase order can only be processed when two messages have arrived: an order detail message from the web site and a credit card approval message from a third-party credit card processor. The orchestration waits for both messages to arrive, and then uses a map to create a message that contains both order details and credit card payment information. This message is then sent to the ERP system for processing.

Orchestration is part of the publish-subscribe messaging model. Orchestrations receive incoming messages by creating subscriptions in the message box. When an orchestration sends a message, it is published to the message box, where it can be picked up by a send port or another orchestration.

Two facilities provided by BizTalk Orchestration for dealing with long-running business processes are correlation and compensation. As an example of correlation, imagine an orchestration that sends a purchase order to a supplier and needs to receive the corresponding purchase order acknowledgment. Using correlation, the orchestration would initialize a correlation set that contains the order number when the outgoing message is sent. BizTalk will use the correlation set to create a special subscription in the message box that receives acknowledgments with the correct purchase order and routes them to the correct instance of the orchestration.

Figure 1-2. A BizTalk orchestration

Compensation is a technique to deal with long-running transactions and can be thought of as the “undo” to the execution of an orchestration. Traditional database transactions are atomic, which means that they succeed or fail as a unit. Integration processes often need transactions that last too long to be implemented as atomic transactions. To continue the example, sending the purchase order to the supplier, and receiving a response, should be implemented as a transaction. However, it may take several hours to receive the response from the supplier, so this can’t be implemented as an atomic database transaction. Also, sending the message to the supplier has an effect outside BizTalk, so it can’t be simply rolled back. Compensation is the set of actions that need to be taken if a transaction needs to be cancelled after it has started. Unlike rolling back an atomic transaction, compensation may take additional steps to recover from the failed transaction, such as updating other systems or e-mailing users to let them know about the failed transaction.

Business Rules Engine

The business rules engine allows business logic to be separated from the application. As an example, consider a car dealer who has branches in several states. The taxes on a car depend on the state where the car is sold, the state where the new owner lives, the type of car, and the status of the owner. In some states, Native Americans or veterans receive special treatment for car taxes. As well, tax rules can change with each jurisdiction’s financial year. The company wants to open additional branches in new states, so it will need to add new tax logic.

It would certainly be possible to implement the tax rules as .NET code, but that would require a developer to get involved every time the rules were modified. Instead, a business analyst could create and modify tax policies in the business rules engine, and the application would call the tax policy during the sales process. By using the business rules engine, it’s possible to achieve a clean separation between the application and business rules that are subject to change.

The business rules engine, or BRE, may be called from within an executing BizTalk orchestration or directly using C# code from an interfacing application. Rules are compiled and deployed to the engine, and may be dynamically altered as conditions change. This level of flexibility provides the robust conditional logic processing functionality required by BPM and real-time systems.

EDI

In addition to traditional EAI within an organization, BizTalk is widely used for B2B connectivity. The traditional EDI formats, X12 and EDIFACT, are the most widely used formats in B2B; X12 is the dominant format in the United States, and EDIFACT is the dominant format in the rest of the world. In the United States, transaction formats mandated by the Health Insurance Portability and Accountability Act (HIPAA) are widely used in the healthcare industry; the HIPAA formats are based on the X12 standards.

BizTalk provides pipeline components that can assemble and disassemble messages in X12 and EDIFACT. The pipeline components can deal with batching and debatching of EDI messages. They can also generate acknowledgments, such as X12’s 997 Functional Acknowledgement or AS2’s Message Disposition Notification (MDN). Special pipelines are provided that implement the AS2 protocol, which transmits EDI files securely over the Internet using HTTP and SMIME. BizTalk ships with schemas for all the popular versions of X12 and EDI-FACT. The BizTalk Administration Console provides features for administering EDI trading partners and setting the EDI options for each partner.

RFID

Radio frequency identification is used to track the movement of physical goods. One example of RFID that’s familiar to commuters in the Northeast region of the United States is the E-ZPass system. An E-ZPass is a tag that’s mounted on a car windshield. As the car passes through a highway tollbooth, the tag is automatically detected by an antenna in the tollbooth, and the toll is charged to the driver’s credit card. RFID tags are widely used in manufacturing, distribution, and retailing to track the movement of inventory. BizTalk provides a software infrastructure for RFID that uses a driver model to interface with a wide variety of RFID hardware.

BizTalk Server 2009 adds support for RFID devices that run the Windows Mobile and Windows CE operating systems.

Management and Operations

BizTalk provides a variety of tools for developing BizTalk solutions and managing BizTalk servers. A set of add-ins for Visual Studio 2008 allows message schemas, maps, orchestrations, and pipelines to be developed. The BizTalk Administration Console allows administrators to create receive and send ports, manage services, set up EDI trading partners, view subscriptions, and handle suspended messages.

System Monitor (PerfMon) can be used to check the performance of BizTalk applications, and System Center Operations Manager (SCOM) can be used to monitor the health of your BizTalk group. Send ports, receive ports, and orchestrations can be configured to track messages. Tracked messages are copied to a special database after processing so that it’s possible to examine all the messages that have passed through the system. In earlier versions of BizTalk Server, a separate tool called Health and Activity Tracking (HAT) was used to examine tracked data, but in BizTalk Server 2009, tracked data was added to the Administration Console, shown in Figure 1-3.

Figure 1-3. The BizTalk Administration Console

What’s New in BizTalk Server 2009

There are a number of new features in BizTalk Server 2009:

• Platform alignment: BizTalk Server now supports Windows Server 2008, SQL Server 2008, Visual Studio 2008, and virtualization using Windows Server 2008’s hypervisor technology (Hyper-V). BizTalk artifacts are now fully integrated into Visual Studio Team System’s application life-cycle management, supporting TFS source code control, automated builds via MSBuild, and unit testing via Visual Studio Test. Visual Studio now supports debugging BizTalk maps, pipeline components, and orchestrations.
• Administration: The Administration Console now supports access to Health and Activity Tracking (HAT) data.
• SOA registry: BizTalk Server now provides a new registry based on UDDI 3.0.
• New adapters: New adapters based on the WCF Adapter Framework have been provided for Oracle e-Business Suite and Microsoft SQL Server.
• New legacy integration features: BizTalk Server now includes a WCF channel for communicating with WebSphere MQ. The WCF Service for Host Applications exposes the Transaction Integrator to .NET developers.
• Mobile RFID: RFID devices running Windows Mobile and Windows CE are now supported.
• Messaging improvements: BizTalk Messaging has improved support for recoverable interchange processing. The WCF adapter has improved support for transactions.
• EDI improvements: The EDI pipelines now support multiple message attachments, automessage resend, file name preservation, and improved reporting. The system has been recertified by Drummond.
• ESB Guidance 2.0: The Enterprise Service Bus (ESB) Guidance is available as a download from Microsoft’s Patterns & Practices group. It provides code and guidance that is useful for implementing an ESB on BizTalk Server. ESB Guidance 2.0 is built on BizTalk Server 2009. It improves itinerary processing and itinerary modeling, provides a new resolver-adapter pack, and adds new features to the ESB Portal.
• And best of all, new BAM Features: BAM can now use SQL Server 2008 Unified Dimensional Model (UDM) cubes, and BAM works with PerformancePoint Server 2007.

BAM Overview

Business Activity Monitoring is a feature of BizTalk Server that provides a flexible infrastructure for capturing data as it flows though a business process. Data is captured from one or more data sources, and then flows through the BAM infrastructure to a set of databases, where it is available for display to an end user or can be consumed by some other application. It’s important to realize that BAM is useful not only for capturing data flowing through BizTalk applications, but also for capturing data in WCF, WF, or any .NET application via the BAM API, as shown in Figure 1-4.

Figure 1-4. Data flow in BAM

When you implement a BAM project, you have to answer three key questions:

• What data should be captured?
• Where does the data come from?
• How will the data be presented to the user?

The following sections will look at each of these three areas in turn.

Observation Model: What Data Should Be Captured?

All BAM projects start by creating an observation model. The observation model defines the data that is captured from the business system and the way the data is displayed. On most projects, the observation model is defined by a business analyst working in Excel, using the BAM Add-In for Excel. Observation models contain two types of artifacts: activities and views. It’s important to understand that the observation model doesn’t define how the data is captured; it only defines the data of interest and the way that data will be presented to data consumers.

A BAM activity is a collection of data fields and milestones. The data may be text data, or it may be numeric data captured as either floats or integers. Milestones are points in time when something occurred, for example, the date and time that a message was received by BizTalk or a WCF service was called. As an example, if we were monitoring a sales order process, we might capture the data listed in Table 1-2.

If you examine the data, you’ll see that not all the data in the purchase order is captured by BAM. The order also includes the customer street address, but this isn’t captured because none of the users who use BAM data are interested in this level of detail.

A BAM view provides information on how the data in the activity will be presented to the user. Often, there will be several views onto the same activity, supporting different roles within the organization. In addition to simple data and milestones, a view can include aggregations and dimensions.

In our purchase order example, we might have three audiences for BAM data: the sales manager, the shipping manager, and the IT help desk.

The sales manager is primarily interested in tracking the performance of the sales team by industry and state. The sales manager’s view might be a four-dimensional Excel pivot table, with the dimensions being order date, salesperson, customer industry, and geography (see Table 1-3). At each cell in the cube, the measures would be total orders, average order value, and number of orders.

The shipping manager is concerned with getting orders out in a timely manner, but is too busy to look at a report on a regular basis. Instead, BAM should notify the shipping manager when an order takes more than 48 hours to ship. Occasionally, the shipping manager needs to report on average time to ship and would like a report that shows the average number of hours to ship an order by week.

The shipping manager will use the BAM Portal to display his view on the data and will also have an alert configured to e-mail the order number when an order has not shipped within 48 hours (see Table 1-4).

The IT Help Desk is called when the status of an order needs investigation. The help desk wants to be able to determine the date and time an order was received, sent to the ERP system, and invoiced. However, the order values are considered confidential information that should not be visible to the help desk. The help desk will also use the BAM Portal to view data, and the help desk view is just a subset of the original activity without any aggregation, as shown in Table 1-5.

Data Sources: Where Does the Data Come From?

The four data sources most commonly used with BAM are BizTalk applications, WCF services, WF workflows, and .NET applications that work with the BAM API.

• BizTalk applications: BAM can capture data from fields in BizTalk messages or BizTalk context properties. BAM can also capture milestones from the date and time a BizTalk message is received or sent. Data capture from BizTalk is configured using the Tracking Profile Editor (TPE), shown in Figure 1-5. The TPE provides a simple click-and-drag user interface that makes it easy for the developer to specify the data that is captured.

• WCF services: BAM can capture data from the SOAP envelope in a WCF request, response, or fault message. The captured data is specified using an interceptor configuration (IC) file, an XML file that is created by a developer. The service’s app.config is modified to load the BAM components at run time.

• WF workflows: BAM can capture milestones from WF activities. Like WCF services, an IC file is used to specify the data that must be captured, and BAM components must also be loaded at run time.

• BAM API: .NET code can write data directly into BAM using an API. Code that isn’t executing in BizTalk will use either the BufferedEventStream or DirectEventStream classes. These two classes have different performance and reliability characteristics, but the bottom line is that it’s easy to write .NET code that interfaces with the BAM API. If you need to instrument code on another platform, such as J2EE, all the standard Java/.NET interop techniques can be used, including creating a web service in .NET that calls the BAM API.

Figure 1-5. The Tracking Profile Editor

Databases and Consuming Data: How Will the Data Be Presented to the User?

After events pass through the BAM infrastructure, they are initially stored in a database called BAM Primary Import. This database contains a set of tables for each BAM activity.

There are two types of views: real-time aggregation and scheduled. Real-time aggregation views will be created as SQL Server views in BAM PI. Scheduled views are implemented by building OLAP cubes on a periodic basis. Older data may be migrated out of BAM Primary Import to the BAM Archive database.

The tables and views that bm.exe builds are quite straightforward; standard reporting tools, such as SQL Server Reporting Services or Microsoft Office PerformancePoint Server, can easily consume the data. Developers familiar with ADO.NET or LINQ will have no difficulty programming against the databases. You can assume that the database structures will be maintained in future product versions.

In many cases, you won’t need to build your own client. BizTalk provides two prebuilt clients for BAM data: the BAM Portal and BAM Excel spreadsheets. The BAM Portal, shown in Figure 1-6, is an ASP.NET application that can be installed with BizTalk Server, if your system is running IIS. It will automatically display activities and views, if the user has rights to see them. The BAM Portal is able to examine the BAM PI database, and automatically display a user interface that allows activity data to be searched and views to be displayed both graphically and as a pivot table. The BAM Portal can also be used to create new alerts and subscribe to them. For many users, the BAM Portal will be the only tool they need to work with BAM data.

Figure 1-6. The BAM Portal

However, some users will prefer to work with the Excel client. Each time you build a new view with the BAM Add-In for Excel, a live pivot table is built in the spreadsheet that is linked to the BAM data. If you have users who love Excel, they will be delighted to learn that they can use their favorite tool to analyze BAM data. You can enable them to connect to the BAM databases by just e-mailing them the spreadsheet that is built by the add-in.

BAM alerts are built on SQL Server Notification Services. When SQL Server Notification Services are available, alerts can be created that will either send an e-mail or create a file when a particular condition occurs. BAM Alerts are often used to notify a person, typically by e-mailing a human-readable message. However, a BAM alert can also write an XML file, and this can be read by a receive port that initiates a BizTalk orchestration, allowing BizTalk to act on the alerted condition.

Summary

This chapter introduced BAM in the context of several important technology trends. The first section introduced SOA, BPM, and BI. The next three sections described the benefits of BAM, and then provided an overview of BizTalk Server and the new features of BizTalk Server 2009. The final section introduced BAM as a tool for capturing data that flows through business systems, especially systems built on BizTalk Server, WCF, WF, and .NET. That section also mentioned three important questions: What data does BAM need to capture? Where does the data come from? How is the data presented to the user?

In the remainder of Part 1, you’ll learn more of the basics of BAM, starting with configuration options and how to install BAM in Chapter 2.