Chapter 1
Preparing for Your Upgrade
Every BizTalk migration and upgrade is different. This chapter outlines the key considerations that need to be taken into account when planning your migration, and includes details about performing the installation and configuration. You may have a solution that can be migrated virtually “as-is,” or you may have something that will require some recoding and recompiling in order to get it to deploy properly. In either case, you will have an opportunity to examine your solution(s) in detail and make improvements before pushing into a brand-new BizTalk 2016 environment.
Upgrading to the new version of any enterprise application, such as BizTalk, is an opportunity not only to look at specifics of doing an in-place migration, but also looking at how to improve your solutions.
Migration Considerations
There is no way to know in advance exactly what will be required for any migration to BizTalk 2016 from an older version of BizTalk Server. Some solutions are very simple; some are very complex. Some will need to be recompiled with code updates, while others may be able to be migrated using the same compile code you already have. You will want to take all of the following into consideration as you plan your upgrade.
1.You must set up a brand-new BizTalk 2016 environment – new server(s), new database(s). There are no “in place” migrations of BizTalk. An overview of installing and configuring BizTalk 2016 is given later in this chapter.
2.In parallel with the first step, make sure you have the right licensing in place. We’ll look at Standard Edition vs. Enterprise Edition licensing later in this chapter, in the section on licensing. This will also give an overview of the different server architectures that can be put into place using these different licenses.
3.It is possible that you can migrate your solution exactly as-is, without recompiling any DLLs. This is true even when migrating from BizTalk 2006. The compiled .NET code, in some cases, will work without any modification. We’ll look at this in detail in Chapter 2. Your deployment may be as simple as exporting an MSI or a binding file, and deploying your .NET assemblies to the Global Assembly Cache (GAC).
In some cases, especially with a simple BizTalk solution, you can migrate compiled code to BizTalk 2016 exactly as-is without making any modifications. However, in most cases, you will need to make minor changes to maps, orchestrations, etc. and recompile these in Visual Studio 2015.
4.You’ll want to decide how you move objects between environments. You can export an MSI and the associated Binding files, but these won’t always be the best method to migrate. You may need to deploy some of your objects manually, while others use the imports of the exported MSIs. In cases where you are using Batching and complex party setups in an EDI solution, you may have to manually modify the binding files before importing. There are several scenarios outlined in Chapter 7 for how to deploy using different models.
5.You will want to take some time to consider if there are ways to improve your solution(s) prior to migrating. If, for example, you must recompile your code, you may want to take the opportunity to restructure your Visual Studio solution(s) and project(s). Often, developers will place all the code files for a BizTalk solution into a single project, making it very difficult to make changes or do deployments postproduction.
Take a look at Figure 1.1 as an example of a single project with many maps. To make an update to a single map, you must recompile the entire project and deploy a large DLL. In a production setting, this would require impacting multiple running processes, and would be very cumbersome and stressful to work with. If your code is not split out into small, componentized structures (like those shown in the good example in Figure 1.2), then you should definitely spend some time rethinking this structure before deploying into a fresh new BizTalk 2016 environment. We’ll cover this topic in more detail in Chapter 3.
6.Look at simplifying your maps. Complex maps are very common in BizTalk, and first time BizTalk developers will often create exceptionally intricate maps that are extremely difficult to modify, test, and maintain. Using XSLT, C#, and other external technologies to the BizTalk UI Mapper is not only good practice, it can substantially reduce the amount of time to develop your maps. As you can see in Figure 1.3, using traditional functoids can create a very complex mapping environment. Additionally, there are a number of mapping scenarios that simply cannot be solved through using the standard functoids.
You will want to make sure that you can develop in XSLT, which will ensure that you can solve every mapping scenario and also keep your maps easy to read and update (see Figure 1.4). More information on mapping can be found in the mapping discussions in Chapter 3, and the .NET and SQL utilization discussions in Chapter 4.
Your migration is the best time to examine how you can simplify your existing maps, and how you can create a new paradigm for future mapping work that you may do. You must learn to incorporate XSLT into your mapping options.
7.Try to eliminate complex orchestrations and delivery patterns. In many cases, what might have been developed 10 years ago on BizTalk 2006 may work more or less “as is” when upgrading to BizTalk 2016, but the architecture is grossly outdated. There are patterns available that can take what required multiple maps and orchestrations the first time around and replace it with only a very finite number of well-constructed and simple-to-build BizTalk components.
If you open your Visual Studio solution and are unable to immediately piece together how components interact, or you look at your BizTalk Admin console and don’t fully understand how things have been deployed and how your integrations are working, then you must take the time to piece all of it together. You should take advantage of that work and use that time to come up with a simpler model for your upgraded environment. Chapter 3 has more information about simplifying orchestrations.
8.You will need to plan appropriately for resourcing and costs associated with your migration. In some cases, you’ll be able to do this with a minimum of effort – but in most cases, you will need to set aside a substantial amount of time to work through this and complete it with success. Chapter 6 will give more detail about how to approach resourcing around your migration.
Take the time to do a proper assessment of the time it will take to do your BizTalk migration. There may be more (or less!) to it than you think. In a matter of a few hours, a BizTalk expert should be able to come up with a fairly accurate estimate of time and costs.
9.Look at what options may be available in utilizing Azure. Many corporations are looking at ways to push things to the cloud. BizTalk was the original platform that Azure was born from, and there is a lot of talk about BizTalk in the cloud. Understand what is available, and how the various models might make sense for you. Chapter 5 outlines options and considerations around BizTalk and Azure.
While there are a number of other things that may come to mind as you plan your migration, these are the most critical considerations. Given everything you must consider, the first step is to have a BizTalk 2016 environment where you can experiment with your current codebase and see how various aspects of it will migrate. Therefore, the very first thing you will want to do is get your new BizTalk 2016 environment up and running.
BizTalk 2016 Licensing and Installation
You must install BizTalk in a new environment. There is no option for an “in place” migration, as there is for many other platforms. You must stand BizTalk 2016 up in a new environment and move all your code to that new environment. Therefore, the most logical first step in preparing for your migration is getting a new server infrastructure set up, configured, and installed. BizTalk consists of two primary components – the application server, which is where BizTalk and any related code components are installed (such as DLLs, WCF services, etc.), and the SQL Server database server, where all the BizTalk databases are created.
Very little has changed with the installation of BizTalk Server 2016 from previous editions. We’ll walk through a basic installation and configuration in a moment, but before you do an installation, take some time to think about the options available to you. It may be time to give your solution a boost in performance and availability, or, conversely, it may be time to scale back the resources you’ve dedicated to it in the past.
The cost of BizTalk licenses has changed with BizTalk 2016, and is based on a per-core model. Figure 1.5 shows the current pricing as the time of this writing. You can find out more at https://www.microsoft.com/en-us/cloud-platform/biztalk-pricing.
The Simple Standard Edition Solution
Standard Edition is a cheap-license and an easy-deployment model. For smallscale integrations that do not require real-time or highly available implementations, the Standard Edition can be a cost-effective way to get BizTalk into your environment. When migrating, you may find that the original license you purchased was an Enterprise Edition license, but you never really used the benefits of this license. For example, many organizations will have a single BizTalk application server with the Enterprise Edition installed on it. The core reason to have the Enterprise Edition is to scale out your architecture and have multiple BizTalk Group boxes sharing load.
If you have only a single server, or a dual server (BizTalk and SQL) architecture, you likely only need BizTalk Standard Edition. You may have to reorganize your BizTalk applications so that you have only five, but this is a small effort, and using Standard over Enterprise can lead to sizable cost savings.
There are two models that you can have with BizTalk Standard Edition. The first, shown in Figure 1.6, is a single box with both BizTalk and SQL installed on it. This architecture is recommended only in a development or test scenario. In this scenario, you usually have Visual Studio as well as other development tools in place. Only in cases where your company is very small and you have limited funds or hardware availability should this model be used in a production setting.
A production environment with BizTalk Standard Edition should be based on the second model, shown in Figure 1.7. In this model, the BizTalk application server is separated from the SQL database server. This ensures that the load on the CPU is balanced more effectively, and allows you to license and structure the RAM and CPUs on both boxes appropriately. For example, many BizTalk implementations use far more system resources on the SQL box than they do on the BizTalk box. Therefore, you may be able to create a solution where you have a BizTalk Server with two CPUs (and therefore are paying only for that cost) and a SQL Server with four CPUs (and are paying that for the SQL licenses). There is a lot of flexibility in how you configure your boxes and what type of licensing is proper for you.
The Standard Edition is often overlooked when organizations look to purchase BizTalk. Generally, this is because of the idea that it will handle only “five applications.” An application in BizTalk is not the same as a separate unit of work, and does not mean that you will be limited to only five solutions – really, it is more like a file directory and an organizational tool than anything else. Figure 1.8 shows where these applications are located. You can deploy anything you want within a single application, even integrations that have nothing in common.
Prior to migrating your older BizTalk solution – which may be on the Enterprise Edition – take time to look at Standard and see if you can fit your solution into this cheaper pricing model.
The Simple Multi-Server Enterprise Edition Solution
The Enterprise Edition allows for multiple BizTalk boxes to be added to a BizTalk Group. A group is one or more servers that share load across each-other – similar to a SQL cluster. If you have two servers in a group, the second one will process data when the first one is consumed. There is no exact science to when one BizTalk box within the group will take over from the other, but the basic concept is that if one server is under load, the second server will pick up newly inbound processes. And – if one of the servers in the group fails, there is no impact to your running processes, as the other server will continue to process. Figure 1.9 shows a simple multi-server environment where two BizTalk servers are part of a single Group and share a single SQL Server database.
A new server can be added to a BizTalk Group at any time. This means that if you have low load the first year, and your requirements change in the second year, you can add the second server to the group at that time.
The Highly Available Enterprise Edition Solution with SQL Clustering
The simple BizTalk multi-server environment shown above in Figure 1.9 is an extremely common production solution, and is often thought of as being highly available. This is not accurate. It is only highly available in the sense that if one of the BizTalk Servers goes down, the other one will continue to run. However, the database is still a single point of failure. So, it is only halfway to the “highly available” target. To get true failover availability, clustering SQL Server is a requirement. This model is shown in Figure 1.10.
Notes on Web Services
One other item to take into consideration is external facing apps that may be part of your BizTalk solution, but are built on different technologies. The best example of this is the use of web services. BizTalk orchestrations can be published as web services, which will create proxy web service files in the local BizTalk Server’s Internet Information Server (IIS). In a highly available BizTalk environment, shown above in Figure 1.11, the IIS portion of the solution is highly available only if there is some sort of web load balancer sitting in front of the two BizTalk boxes brokering requests. Each box has its own IP and name, so each would have its own web URL for external apps to call. If you have no load balancing, then applications would have to reference either the first box or the second box directly, as shown in Figure 1.11.
For this web portion of the solution to be highly available, a load balancer or similar would be required. This load balancer would translate a single URL request and rotate it to each of the servers on the backend, something like the architecture shown in Figure 1.12. The external apps hit a single URL; that URL is then split to the individual backend BizTalk boxes through the load balancer’s configuration. New BizTalk boxes added to the group can then share load in the future by simply adding them into the Web Balancer’s list of boxes to send requests to.
Exercise: BizTalk 2016 Installation and Configuration
This exercise will walk you through the basic installation and configuration for BizTalk Server 2016. There are several patterns available to you in an installation, the most common of which are the single-server install used in a development/testing environment and a multi-server install used in a production environment. Both are very similar in the steps taken during the install and configuration. The section that follows the walkthrough demonstrated below will note those things you must consider when deploying in a multi-server environment.
Installation
The installation of the BizTalk software is very easy and usually takes about 30 minutes to complete. Once you click on the Setup executable, you have several options – allowing the setup utility to automatically download and install the prerequisites is the ideal option. If for some reason, you have heavy security on your box, and Internet Explorer is set to not allow downloads, you will have to download these files on a separate computer that has access to the internet and then transfer them manually to the server where you are installing BizTalk. The installation of the prerequisite files is shown in Figure 1.13.
Once the required components have been installed, the installation wizard will move on to install the core BizTalk engine and files. There isn’t much to do for the installation other than watch it progress. The real work comes in the configuration of BizTalk. When the installation is complete, the wizard will prompt you to start the configuration, as shown in Figure 1.14. If you miss this or want to do the configuration at a later time, you can access the BizTalk Server Configuration application via your Start menu.
Configuration Options
The initial configuration screen (shown in Figure 1.15) allows for several things. The first is determining whether you will be doing a basic or custom configuration. The basic configuration installs all of the BizTalk components, which is generally unnecessary for most organizations. For example, not every organization will use BAM or EDI or the BizTalk Rules Engine. The basic configuration should be used only for a development environment and only in a single-server setup (where the database and BizTalk application sit on the same server). The custom configuration is for multi-server setups (where you have a minimum of two servers, the database server and the BizTalk application server separated from one another) and for configurations where you need to have control over groups, users, and permissions (virtually every production deployment).
The next item to determine is the database server name where the BizTalk databases will be installed. BizTalk installs a number of databases, depending on which options in the configuration you select. In a production environment, you will want to have a dedicated BizTalk database server. In a development or test environment, you can share a database server that has other databases installed on it. Only one version of BizTalk can be installed on any single database instance; for example, you cannot use the same database instance for your BizTalk 2016 environment as you would for your BizTalk 2010 instance from which you may be upgrading.
The final item you will need to determine is the service account you will use. In a production environment, or any multi-server install, you will need to set up a number of Windows domain groups and users, one of which will be a BizTalk service account (usually called something like BTService). This is the account that all the services will run under (in Windows Services), and which will be used to log these services into the underlying databases. The full list of Windows users and groups that will need to be set up can be found in a number of places, including the following link.
https://msdn.microsoft.com/en-us/library/aa577661.aspx
Configuring Enterprise SSO
When accessing the numerous systems across a network that are common in an integration environment, there are many security credentials that most likely will be necessary. Even in a multi-server environment where BizTalk must access its own databases from multiple BizTalk Group servers, management of security is an issue. Enterprise Single Sign on (SSO) is BizTalk’s tool for managing these credentials and security domains. In 99% of BizTalk configurations, configuring this is the only time you as an administrator or developer will ever see SSO. Enterprise SSO can be used to store and manage credentials that allow BizTalk to access systems across the network. However, a much easier and arguably just as secure method is to store these credentials in the BizTalk configuration file. Configuring SSO is shown in Figure 1.16.
A new server can be added to a BizTalk Group at any time. This means that if you have low load the first year, and your requirements change in the second year, you can add the second server to the group at that time.
Configuring the BizTalk Group
When using the Enterprise Edition of BizTalk Server, you have the option to add one or more BizTalk application servers to the group. The first group server configured must use the “Create a new BizTalk Group” option, as shown in Figure 1.17. The second group server must run the BizTalk configuration tool separately and select the “Join an existing BizTalk Group” option.
When adding a second BizTalk server to an existing BizTalk group, you will need to run the BizTalk configuration tool from that second (or third, etc.) server. You will not need to configure anything except the group settings (where you will “Join” an existing group), as the rest of the environment has already been configured.
Note that with the Standard Edition, you will only be able to have a single BizTalk Group box in your environment – only Enterprise Edition allows for multiple servers within a group.
Configuring the BizTalk Runtime
This portion of the configuration can be a little confusing, but it is basically setting up the host instance information that will be used to run the BizTalk adapters, orchestrations, and any other component deployed within BizTalk. The most common configuration is shown in Figure 1.18. These settings can be altered once the configuration is complete by accessing the hosts and host instances within the BizTalk Administration tool. You may also create additional hosts that can be used to separate adapters, applications, etc.
In a highly available environment, there are some adapters that will instantiate on each BizTalk server if using the default BizTalk host. For example, if you are using the SQL adapter and have a receive port deployed using the default Host instance, it will trigger simultaneously on both servers, resulting in your solution executing twice. There are several solutions to this, the easiest of which is to create a host application that runs on only one of the servers and tying this adapter to that host application.
Configuring the Business Rules Engine
One of my least favorite tools within BizTalk is the Business Rules Engine (BRE). The concept is great, but the implementation is poor, difficult to work with, and often results in very complex solutions that are difficult to maintain and extend. My experience has taught me that virtually any scenario where a configurable rule or property is required can most easily be done through a C# helper library or intelligent logic within the database tier. If you have a solution that uses the BRE and you are looking at migrating it to BizTalk 2016, you may want to migrate things exactly as they are, and therefore use the BRE. But, if you are considering new development or have the opportunity to rethink your architecture during your migration, eliminating the BRE would be my recommendation.
Creating your own approach to managing business rules is generally a better option than using the BizTalk Business Rules Engine. You can do some amazing work using database components that are easy to develop and can be maintained by far more developers than the BRE.
Configuring Business Activity Monitoring (BAM)
Another overly complex aspect to BizTalk that often leads to unintended and unnecessary complexity is BAM. Getting at metrics and analyzing your custom solutions can be done in so many ways that BAM is generally unnecessary. You can create your own logic to track where a process is and push it into more common reporting tools like SQL Server Reporting Services (SSRS) or existing reporting frameworks that your organization already has. I generally enable BAM only when working with EDI and AS2 solutions. There is some good reporting you can get out of the box for applications using these engines. But for solutions I build for clients, I always work to tie in reporting to something they are already familiar with and can easily configure and modify.
Architecting your integrations to tie in with existing reporting and analysis technologies that your organization already uses is your best option.
Configuring EDI and AS2
Many BizTalk environments have no use for the EDI engine. If you are using EDI within your environment, you’ll need to enable the Runtime, as shown in Figure 1.19. AS2 is a rare need, but if you will be doing direct AS2 transactions with trading partners, you will want to enable this piece of the solution. Some extended reporting is available if you have BAM installed (which requires SQL Server Analysis Services). If you want this, you’ll select the third option shown in the figure.
Notes on Configurations within a Multi-Server Environment
When installing BizTalk on multiple servers, several items differ from single-server configurations. The key differences are as follows:
1.You must use Windows domain groups and users (on a single-server instance you can use local Windows accounts). This has been covered in an earlier section within this chapter.
2.You will need to enable Microsoft Distributed Transaction Coordinator on all servers. MSDTC is used to ensure communications to and from the database to the BizTalk Group application servers can be completed or rolled back, depending on the interruptions in the network or other connectivity. To enable MSDTC, you must open Component Services and click on the Properties of DTC, as shown in Figure 1.20. Once the MSDTC property window has opened, you will need to set the properties as shown in Figure 1.21.
The MSDTC settings must be set identically on each of the boxes within the BizTalk environment. This includes the SQL database server(s) and the BizTalk Group server(s).
Notes on SQL Server Maintenance
SQL Server maintenance is often an afterthought in a BizTalk deployment. Make sure you are conscious of it, and are treating it as any other SQL database on your network. Truncating tables and log files, monitoring disk usage and overall health – anything you do for other databases needs to be done for the BizTalk databases. There is nothing special about these databases, and in many cases they can be left untouched for years without impact. However, there are several things to be aware of and to address, as shown in the following list.
You should have SQL database administrators assigned to your BizTalk installation so that they are familiar with it from the start. Many database administrators will ignore BizTalk SQL databases, because they are unfamiliar with BizTalk. The general rule of thumb is to maintain and monitor your BizTalk databases just as you would any other SQL Server environment within your network.
1.Some applications, such as large-scale EDI implementations, will use a tremendous amount of disk space. The actual amount will vary based on tracking settings enabled on application, ports, orchestrations, etc. and the type of server architecture you have (multi-server environments will have different tracking and database usage than single servers). Make sure you are monitoring for database and log file growth, and that you have some plan for truncating data and log files when they get very large.
2.There are several SQL jobs that should be enabled. These are installed by default with BizTalk, but are not enabled. You’ll want to turn all the BizTalk jobs on. If you experience any performance issues, you can turn these off – they are not mission critical, and can be left off for long periods of time. They handle database cleanup, especially the elimination of unneeded messages from the MessageBox database. Figure 1.22 shows the BizTalk jobs interspersed with other jobs unrelated to BizTalk. Your environment will vary, but you can easily pick out the BizTalk-specific jobs. Make sure and monitor the history of these for any exceptions that may be thrown.
Summary
You should at this point have a firm idea of what you want to accomplish in your migration. You will be installing and configuring BizTalk in a new server environment, and must decide if you want a simple architecture or one that is highly available. When you move your solution, you’ll have to decide if you want to keep it exactly as it has been programmed to date or if you’ll be re-architecting it to better use available technology and better align with your business. There are many things to consider and experimenting with the code in a BizTalk 2016 environment will be your best litmus test for how much work will be required in your migration. In Chapter 2, we will begin looking at this process of experimentation.