images

In this chapter, you’ll learn about managing BAM. BAM is all about data flows, and the chapter starts by reviewing the databases and data flows in BAM. This section also contains exercises that teach two important administrative functions: setting up a tracking host and scheduling data archiving. The health of a BizTalk server is dependent on the underlying databases. If BizTalk’s databases are healthy, BizTalk itself is usually healthy. To keep the databases healthy, certain database jobs should be scheduled, and you’ll learn about these here.

The next section provides a reference for the commands available in the BAM Management utility, bm.exe. The final section provides tips and tricks for managing BAM, including step-by- step instructions for some advanced administration techniques.

Data Flow in BAM

The heart of the BAM architecture is several SQL Server databases, shown in Table 13-1.

images

images

Figure 13-1 shows the way data flows between the BAM databases.

images

Figure 13-1. Data flows in BAM

The data flows fall into three categories:

  • TDDS: The Tracking Data Decode Service (TDDS) runs in a BizTalk host instance and despools data from the Message Box database to the BAM Primary Import database and the BizTalk Tracking database.
  • DTS packages: Each scheduled view has an associated job that builds the view’s cube in the BAM Analysis OLAP database. In addition, each activity has a job that moves old data out of BAM Primary Import into the BAM Archive database.
  • SQL Server Notification Services: The BAM Notification Services Instance (Alerts Main) and Application (Alerts Application) databases are used to support BAM alerts, and are populated by SQL Server Notification Services.

BizTalk Hosts, Host Instances, and Groups

TDDS runs in a BizTalk host called the tracking host. In order to understand the concept of a tracking host, you’ll need to know a little about BizTalk hosts, host instances, and groups. Readers who are BizTalk experts may want to skip ahead to the next section. In BizTalk, receive ports, send ports, and orchestrations are run by host instances. Each host instance is a Windows service running on a particular machine.

Figure 13-2 shows the Windows services on a server with four running BizTalk host instances. BizTalkServerApplication is the default name of a host that is created when BizTalk is installed. This host instance is currently stopped. OneSampleHost, AnotherSampleHost, and ThirdSampleHosts were created later. All three are currently running.

images

Figure 13-2. Services Administration Console showing four BizTalk host instances

So how does BizTalk decide what to run in a specific host instance? It uses the concept of a host. Each BizTalk port or orchestration is associated with a host. Each host may have one or more instances, and each instance runs the ports and orchestrations associated with that host, as shown in Figure 13-3.

images

Figure 13-3. The properties for the sample orchestration used in Chapter 6. This orchestration runs in a host called BizTalkServerApplication.

A BizTalk group is a collection of BizTalk servers that share a single management database. BizTalk groups provide BizTalk with fault tolerance, because each server in a group may run an instance of each host.

As an example, imagine a BizTalk group that contains two servers, and each server has an instance of a host called SampleHost. Further imagine there is an orchestration that runs in SampleHost. The multiple host instances provide scalability to the orchestration; BizTalk will run instances of the orchestration on both machines to improve performance. When a BizTalk orchestration is waiting for an inbound message, it can be dehydrated. This means that its state is saved to the Message Box database, and the orchestration is removed from memory. Each time the orchestration is rehydrated, it will be loaded on the first host instance that has available capacity. Orchestrations can move from server to server to maximize scalability of the BizTalk group.

The multiple host instances also provide fault tolerance. If one server fails while running the orchestration, the other server will be able to continue running the orchestration from the last point where the orchestration was persisted to the Message Box database. Finally, multiple host instances provide separation between applications. Imagine an orchestration that calls a rogue component that crashes the system occasionally. By putting the orchestration in a separate host, it’s possible to ensure that the orchestration can’t take down anything else.

Now that the review is over, you’re ready to learn about tracking hosts.

The Tracking Data Decode Service

The Tracking Data Decode Service, which is also known as the BAM Event Bus Service, is responsible for the following:

  • Moving BAM data from the Message Box database to the BAMPrimaryImport database.
  • Moving Health and Activity Tracking (HAT) data from the Message Box database to the BizTalkDTAdb database. This book doesn’t cover HAT, as we don’t recommend using it in high-performance environments.

TDDS runs in a BizTalk host instance. We recommend that you create a separate host that does only tracking, and turn off tracking in any host that contains ports or orchestrations, to ensure that tracking is insulated from ports and orchestrations. This provides better scalability and fault tolerance.

EXERCISE 13-1. SETTING UP A TRACKING HOST

Archiving BAM Data

BAM is intended to provide information about the status of active business processes; it’s not recommended that you keep historical data in the BAM Primary Import database. Instead, you should move old data out of BAM Primary Import and into the BAM Archive database.

Each time you use bm.exe to load a new BAM activity, a SQL Server Integration Services (SSIS) package called BAM_DM_<Activity> is created. As an example, when you created the Orders activity in Chapter 5, you created a package called BAM_DM_Orders, shown in Figure 13-7, that is stored in MSDB. This package is used to move old BAM data into the BAM Archive database.

images

Figure 13-7. The BAM_DM_Orders package in Visual Studio 2005

If you want to inspect the package, you can use the dtutil utility to export it to a file. The following code line exports the package called BAM_DM_Orders stored in MSDB to a file called C:ssisam_dm_orders.dtsx:

dtutil /SQL BAM_DM_Orders /COPY "FILE;c:pro bam sampleschapter
  fourteenam_dm_orders.dtsx"

After you have exported the package to a file, you can open SQL Server Business Intelligence Development Studio from the SQL Server folder in the Start menu and open the file.

The package has two functions:

  • It partitions the data in BAM Primary Import.
  • It removes old data from BAM Primary Import. By default, it moves it into the BAM Archive database.

Data is partitioned by creating new tables in BAM Primary Import to contain activity data. An example is shown in Figure 13-8. The archive package has just run and partitioned the bam_Orders_Completed table by creating an additional partition table called bam_Orders_4BD3B6B1_763A_4698_BF74_9986E9155576. The bam_Orders_CompletedRelationships table has been partitioned by the creation of the bam_Orders_4BD3B6B1_763A_4698_BF74_9986E9155576_Relationships table.

Partitioning data into separate tables is a performance improvement, especially for real-time aggregations. It also simplifies the archival process. When the package runs, it looks for partitions that are available to archive, and processes the entire partition.

images

Figure 13-8. Detail from SQL Server Management Studio after running BAM_DM_Orders. Additional partition tables have been created on the order view.

imagesCaution If you write code against the BAM Primary Import database, it’s essential to use the views, not the underlying tables. If you write code directly against the underlying tables, BAM data may disappear when the job runs and partitions the table.

A partition is available to archive if all scheduled views that depend on the partition have been processed, and the partition is older than the online window for the activity. By default, the online window for an activity is six months. You can change the online window using BM. As an example, the following command changes the online window for the orders activity to 14 days:

bm set-activitywindow -activity:orders -timelength:14 -timeunit:day

imagesNote bm.exe is in the folder C:Program FilesMicrosoft BizTalk Server 2009Tracking, so set the path appropriately before running the command.

It’s important to bear in mind that the package only archives partitions older than the online window. As an example, imagine an activity where the archive package has never been run. There is a year’s worth of data in the BAM Primary Import database, but there are no partitions. The online window is the default: six months. Now consider what happens if you schedule the archive package to run each day. The first day the package runs and creates a partition table in BAM Primary Import. This partition table contains a year’s worth of data. However, there are no partitions older than six months, so no data is removed from BAM Primary Import. The next day, the package runs again, and a second partition table is created. This partition will contain one day’s worth of data. The partition table created the previous day is only 24 hours old, and so it will be left in BAM Primary Import. If the online window remains six months, it will be a full six months after the job is scheduled before any data is removed from BAM Primary Import, but then a full year of data will be removed in one day.

You can configure whether or not data is moved to the BizTalk Archive database using BM. This command causes old data to be purged from the Orders activity, instead of moving it to the BizTalk Archive:

bm set-archive -activity:orders -shouldarchive:false

This command causes old data to be archived to the BizTalk Archive database:

bm set-archive -activity:orders -shouldarchive:true

The purge flag and the online window can be confirmed by examining the BAM_Metadata_Activities table in the BAM Primary Import database.

You have two design decisions to make when setting up archiving: the length of the online window and how often the archive job runs. The length of the archive window is driven by business needs for historical data. Longer online windows will require more disk space, and will also have an impact on the time taken to back up the BizTalk databases. In very large installations of BizTalk, the total time taken to back up the BizTalk databases can become a significant issue.

The archive job should be run on a regular basis. For many organizations, running this job daily or weekly will be a good choice. Running the job more often will create more partitioning in the database, which may provide a slight improvement in write performance at the cost of complicating the database structure by creating many more partitions.

EXERCISE 13-2. SCHEDULING DATA ARCHIVING

Scheduling a Cube Build

Scheduled aggregated views require the aggregations to be built before they are available to users. In Chapter 7, you learned how to run the BAM_AN_Marketing DTS package from SQL Server Management Studio to build the Marketing view on the Orders activity. In production, it won’t work to build views by hand; they should be built using scheduled SQL Server jobs.

Whenever you create a new view that requires a cube to be built, follow the steps in Exercise 13-2, but use the package name BAM_AN_<View> to schedule the cube build.

imagesNote Chapter 7 contains instructions on building the cube for an aggregated view using SQL Server Management Studio.

Backing Up the BizTalk Databases

Your organization may have a standard approach to backing up SQL Server databases, and the DBA team may tell you they have backups under control. If they haven’t dealt with BizTalk before, don’t believe them.

Several BizTalk databases participate in cross- database transactions. These include Biz-Talk Message Box, BizTalk Management, BizTalk Tracking (DTA), and BizTalk Primary Import. If you back up each of these databases separately, the backups will be useless to you. That’s because additional transactions that complete after the first backup but before the second backup will make the databases transactionally inconsistent.

The only way to back up BizTalk is to use the Backup BizTalk Server job that is installed in SQL Server by the BizTalk installation. This job will put a mark in the transaction log for each of the BizTalk databases at a specific point in time. When you restore the databases, you will restore to the transaction log mark, ensuring that all databases are transactionally consistent.

To check the list of databases that will be backed up for you by the Backup BizTalk Server job, take a look at the admv_BackupDatabases view in the BizTalk Management database. (It’s fine to look at this view, just don’t edit it.)

Cleaning Up the Alerts Chronicle Table

If you are using BAM alerts, there’s a table called BAM_Metadata_AlertChronicles in the BAM Alerts Application database that audits all alerts fired by the notification services infrastructure. Each time an alert is fired, a new row is added to this table. For each view you create, you will also see a job called bam_<View Name>_<Activity View>_DelAlertHistJob. This job cleans up the auditing data.

You should run each of these jobs occasionally, perhaps monthly or quarterly, to clean up old audit information.

Enjoy Your BM

Many aspects of a BAM solution are managed using bm.exe. You’ve already seen some examples of bm.exe. It was used in Chapter 5 to deploy activities and views, and in Chapter 3 to deploy interceptor configuration files. It was used earlier in this chapter to adjust archiving behavior.

This section documents the complete set of commands implemented by bm.exe in BizTalk Server 2009. Some of these commands are not available in earlier versions of BizTalk Server.

Help Commands

The BM utility provides help for its many commands. help gets a description for each of the BM commands.

Database and Infrastructure Commands

The BM utility provides commands to manage databases and infrastructure, as listed in Table 13-2.

images

images

Figure 13-12. The BM ­get-­changes command

Observation Model Commands

The commands listed in Table 13-3 focus on administration of objects contained within an observation model.

images

images

Archive and Aggregation Management

When it comes to management of BAM databases, the commands listed in Table 13-4 are useful for archive and aggregation management.

images

Alert Commands

Administration of BAM alerts visible in the BAM Portal is greatly enhanced by the commands listed in Table 13-5.

images

Interceptor Configuration Commands

With the BAM interceptors available, bm.exe also includes commands to manage artifacts related to the interceptors, as shown in Table 13-6.

images

Tracing BAM Management

You can trace the behavior of the BAM Management utility to investigate its behavior. This will provide a log of the internal operations of the utility.

EXERCISE 13-3. TRACING BAM MANAGEMENT

Advanced Topics

This section includes information on some advanced topics in BAM management.

Removing Orphaned Tracking Profiles

Imagine the following scenario: you create and deploy a BAM activity, and then you use the TPE to create a tracking profile that captures data from your BizTalk application. Later, you undeploy the activity. What’s going to happen? The tracking profile is still active and will attempt to capture data and store it in the activity. All the tables that support the activity, however, will be missing from the database.

In this scenario, you will start seeing errors showing up in the Application event log. The text will be similar to this:

Event Type: Error
Event Source: BAM EventBus Service
Event Category: None
Event ID: 6
Date: 03/22/2009
Time: 11:33:17
User: N/A
Computer: SQL2K5BAM
Description:
Execute event error. Error(s) occurred while executing events, see
TDDS_FailedTrackingData table for more details. SQLServer: SQL2K5BAM,
Database: BAMPrimaryImport.

The TDDS_FailedTrackingData table will contain entries similar to this:

Could not find stored procedure 'dbo.bam_Orders_PrimaryImport'.
TDDS failed to execute event. Could not find stored procedure
'dbo.bam_Orders_AddRelationship'.
TDDS failed to execute event. Could not find stored procedure
'dbo.bam_Orders_EnableContinuation'.
TDDS failed to execute event. Could not find stored procedure
'dbo.bam_Orders_PrimaryImport'.
TDDS failed to execute event. Parameter is not valid.

This is known as the orphaned tracking profile problem. In order to fix it, you need to use TPE to remove the orphaned tracking profile, so you’ll need to run through the following steps:

  1. Find the file that defines the activity.
  2. Deploy the activity using BM.
  3. Start TPE, select the activity you just loaded, and then retrieve the tracking profile that corresponds to that activity.
  4. Undeploy the tracking profile.
  5. Remove the activity using BM.

This sounds easy enough, but sometimes during development you will be deploying and removing activities for testing, editing the definitions as you go. A few hours later, you notice the signs of an orphaned profile, realize you deleted the activity definition file, and have no choice but to say “Oh, Fiddlesticks!”

At that point, the urge to start editing BAM Primary Import by hand will become overwhelming. If you’re on a production box, don’t even think about it. If you mess with the database, your server is immediately unsupported by Microsoft support, and your best choice is to go back to a recent backup before the problem started. However, if you’re willing to attempt an unsupported fix, here’s the process.

imagesCaution This process is unsupported. Back up your databases before using this process. Do not use this process on production systems.

  1. Compare BAM_Metadata_TrackingProfiles with the list of activities returned by bm get-activities. If you have orphaned profiles, there will be activity names in BAM_ Metadata_TrackingProfiles that aren’t currently deployed.
  2. Take a backup of all BizTalk databases.
  3. Run TRUNCATE TABLE against the following tables in BizTalkMgmtDB:
    • BAM_TrackingProfiles
    • BAM_TrackPoints
    • StaticTrackingInfo
  4. Run TRUNCATE TABLE against the following tables in BAMPrimaryImport:
    • BAM_Metadata_TrackingProfiles
    • BAM_Metadata_Annotations

Tracing TDDS

You can trace the behavior of TDDS. If you have issues with the performance of tracking, it might be worth tracing TDDS to examine the details of its behavior.

EXERCISE 13-4. TRACING TDDS

Regenerating a Live Data Workbook

If you have lost the files associated with an activity, you can still create an Excel workbook with live data by using bm.exe.

EXERCISE 13-5. REGENERATING A LIVE DATA WORKBOOK

Changing BAM Runtime Settings

Several configuration settings can be changed using bm.exe. The get-config command creates a file that contains the BAM configuration. To change a setting, such as changing the name of a database, you can edit the file and then reload the configuration using the update-config command.

Listing 13-1 shows an example configuration file.

Listing 13-1. Sample Generated by Using BM Get-Config -filename:config.xml

<?xml version="1.0" encoding="UTF- 8" ?>
<BAMConfiguration xmlns="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM">
   <GlobalProperty Name="BAMVRoot">http://sql2k5bam:80/BAM</GlobalProperty>
   <GlobalProperty Name="MaxAlertableActivityViews">63</GlobalProperty>
   <DeploymentUnit Name="PrimaryImportDatabase">
     <Property Name="ServerName">SQL2K5BAM</Property>
     <Property Name="DatabaseName">BAMPrimaryImport</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="StarSchemaDatabase">
     <Property Name="ServerName">SQL2K5BAM</Property>
     <Property Name="DatabaseName">BAMStarSchema</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="AnalysisDatabase">
     <Property Name="ServerName">SQL2K5BAM</Property>
     <Property Name="DatabaseName">BAMAnalysis</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="ArchivingDatabase">
     <Property Name="ServerName">SQL2K5BAM</Property>
     <Property Name="DatabaseName">BAMArchive</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="CubeUpdateDTS">
     <Property Name="ConnectionTimeOut">120</Property>

     <Property Name="UseEncryption">0</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="DataMaintenanceDTS">
     <Property Name="ConnectionTimeOut">120</Property>
     <Property Name="UseEncryption">0</Property>
   </DeploymentUnit>
   <DeploymentUnit Name="Alert">
     <Property Name="DBServer">SQL2K5BAM</Property>
     <Property Name="ApplicationDatabaseName">Application</Property>
     <Property Name="InstanceDatabaseName">BAMAlerts</Property>
     <Property Name="GeneratorServerName">SQL2K5BAM</Property>
     <Property Name="ProviderServerName">SQL2K5BAM</Property>
     <Property Name="DistributorServerName">SQL2K5BAM</Property>
     <Property Name="SmtpServerName">SQL2K5BAM</Property>
     <Property Name="AlertMailFrom">[email protected]</Property>
     <Property Name="FileDropUNC">\SQL2K5BAMalerts</Property>
   </DeploymentUnit>
</BAMConfiguration>

EXERCISE 13-6. ENCRYPTING TRAFFIC TO ANALYSIS SERVICES

Summary

In this chapter, you learned about data flows in BAM, and learned to schedule data archiving and cube builds. You also saw how to perform several advanced maintenance procedures. The primary tool for administering BAM is bm.exe, and you’ve now seen examples of almost all the functions provided by this tool. While you may encounter other scenarios while managing BAM, these tend to be the most common you’ll run across. As always, the Internet provides discussion forums on some of the more esoteric scenarios, as well as prescriptive guidance on how to remedy them.

You probably noticed while reading this chapter that BAM has relatively few utilities for its management and administration. In fact, out of the box, BizTalk provides applications and utilities that address the more common scenarios of using BAM and managing BizTalk. The BizTalk Server 2009 Administration Console is a very useful tool for managing BizTalk, but a custom BAM solution provides additional capabilities the Administration Console does not.

The next chapter will introduce how to monitor BizTalk using BAM as a hands-on exercise.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset