CHAPTER
2
Understanding the Components of Performance Tuning
Ask an experienced JD Edwards system administrator what worries him or her, and the answer you will most likely hear is, “the small things.” When things go very wrong, it seems that the problem can often be traced back to one small item. Keep this in mind when you approach tuning. The system needs to behave like an orchestra: If one instrument is out of tune, it can ruin the entire symphony.
Of course, as with music, people have different listening abilities, and the same is true with performance tuning. You may be able to get the system to run even faster, but to do so, you might have to dedicate more time and resources than is reasonable for the return on your investment. As you review the components of performance tuning, ask yourself, “What does my organization really require?” This will help you avoid spending too much time and money without seeing an equitable return.
In this chapter, we cover many important concepts that can help you realize an equitable return on your investment of time, money, sweat, and tears:
You’ll learn how to create a performance tuning strategy, a plan to help you implement your performance tuning approach. You’ll set the system boundaries, where you’ll decide where, exactly, you declare success. You’ll learn about the different operating systems that you may encounter. This information will help you to lock tasks into your strategy. You’ll read about disk layout, which can directly affect system performance of the system. EnterpriseOne allows you to use multiple databases, so these must also tie into your performance strategy. General tuning levers are covered next. These “big” levers can influence performance across your system. You’ll learn about setting up your network to help achieve the best performance with your EnterpriseOne system. You’ll learn about direct application tuning, including changes that are common during performance tuning engagements.Creating a Performance Tuning Strategy
An old friend of mine used to say, “Plan to plan or you plan to fail.” This adage holds true for ensuring system performance as well as life in general. The first step is to understand that you need a performance tuning strategy. The next step is the actual planning of that strategy. To do this, you must ensure that you dedicate enough time to accomplish the required tasks and that you involve the appropriate people. You need to look at the pillars of your system to determine who needs to be in the room for any strategy session.
Here are the tiers of the implementation:
System boundaries Operating system Operating system levels Patches for the operating system Disk considerations Database components General tuning levers Network Direct JD Edwards application tuningA strategy session should occur during your implementation, but it can occur anytime, so don’t panic if you don’t have a formal strategy yet. Based on the tiers of the implementation just listed, you will need to include people with the appropriate skill sets in this session to help you produce a formal performance tuning strategy document.
Here are the required roles:
JD Edwards system administrator/Configurable Network Computing (CNC) Database administrator Operating system engineer (infrastructure) Virtual machine administrator (such as VMware or Oracle VM) Network administrator Application server administrator (IBM WebSphere or Oracle WebLogic) Key business unit representatives Executive sponsorUnless you define what constitutes acceptable performance, you can’t really create a strategy for achieving it. You need metrics to measure what performance means to your business. These metrics will be different for each business, and all areas of the business are not equal.
We are sure some of you are wondering about including business unit representatives in a technical discussion. If you do not first identify what your organization’s requirements are, you are wasting your time. These business unit representatives are usually the only ones who can give you the true detail of what is happening at the end user level for performance. We cannot tell you how many times we have been called when problems occur and a CIO or an IT director says the system is “slow.” Our response is, “The sky is blue.”
Performance metrics are real and can be measured by taking samples; this helps you to truly know how your system is performing, day-to-day and month-to-month. This type of information is extremely useful. It allows you to chart the impact of specific business cycles. For example, you can measure what performance looks like during open enrollment, at a quarter close, or at a year-end close.
If you are the IT director, performance metrics are critical when you need to justify the purchase of additional hardware to meet the business requirements. You can also establish a Service Level Agreement (SLA) with the business, so you can spend time tuning the system instead of trying to agree about what “slow” really means. If you cannot reach agreement on what this metric is up front, you are in for a difficult fight. Without defining what constitutes acceptable performance, you will never be able to say whether the system is fast or slow—everything will be subjective to each area.
How do you determine what to include in these metrics? A better question might be, What are the top five to eight reports and applications required for your business? Start with these and work from there. Find your top reports and applications, then determine which of these are not performing up to what the business expects. Reports are fairly simple: they normally need to be completed with a certain amount of data within a certain period of time. You’ll need to take a closer look at the applications.
Consider a sales order entry application, for example. How do you determine the time it takes to complete a sales order? You start by breaking the process apart. First, determine what an average order is. Then break down the process so you can see where a slowdown in the process might occur. For example, measure how long it takes to open the application, then how long it takes for the add screen to come up, and then how long it takes to calculate everything on the line of an order, such as advanced pricing. This provides a baseline of details you’ll need to determine whether a performance decrease or increase has occurred. It offers real data points, and quite a bit of data must be looked at. Later in this chapter, in the section, “Common Performance Tools,” we’ll discuss some of the tools you can use to collect this type of information.
System Boundaries
As systems have become more complex, the integrations have become real time, and bolt-on applications have become critical to the performance of the system. The business does not care that a specific screen affects another application that is maintained by a different team. But you should care, and it is critical that you determine the true boundaries of your system and create the metrics to measure it. This is also where having an executive sponsor for your tuning strategy comes in very handy.
To determine your system boundaries, examine your system architecture diagram and interface diagram (such as Figure 2-1). Identify where transactions “leave” the EnterpriseOne system. Say, for example, you are using EnterpriseOne Business Services to make a web services call to a third-party system. This call occurs at the end of a sales order process: a customer is on the phone or waiting at your company’s web page for a customer self-service application to process an order.
FIGURE 2-1. Example systems boundary diagram
When determining the system boundaries in this scenario, you need to work with the other team that manages the third-party system as well, to track the metrics of how long it takes to send data to the system and receive the return. You’ll also need to calculate how long it takes to handle erroneous data. Multiple performance issues can be tracked to bad error handling. Even though you do not own or manage the third-party system, the team that does manage it is smack-dab in the middle of a critical process for your business. So you need to create a metric for both systems—JD Edwards EnterpriseOne and the third-party system. This will let you know whether a performance problem is due to your EnterpriseOne system or whether you need to involve the other team.
Operating System
When it comes to performance tuning, our first stop is the operating system. One of the great strengths of JD Edwards EnterpriseOne is that it is platform-independent. Several different operating systems can be included in your architecture and the software can handle it without issue. This also means that from a performance tuning perspective, you need to be familiar with multiple operating systems.
All implementations will have two operating systems in their architecture: the development clients require one level of a Microsoft operating system and the servers require a different level, even if you are solely a Microsoft shop. If you are an i-Series (AS/400) or Linux shop, you have a few more operating systems to contend with.
Let’s start this discussion by considering the different types of operating systems that you may have in your architecture:
OS/400 Linux Microsoft Windows Solaris IBM AIX HP-UX (Hewlett-Packard UniX) | NOTE
HP-UX support is not in place for version 9.1 currently. You should reference the Oracle Platform Statement of Direction, as Oracle has said it plans to support HP-UX Itanium. We mention this as there are customers running HP-UX. |
OS/400
The OS/400 operating system ships with the I-Series. There are many different levels of this operating system. V5R4 has been around for a long time but is being deprecated with EnterpriseOne Tools 9.1, so customers will need to be using V6R1 or higher. The OS/400 has its own unique requirements to tune with regard to JD Edwards EnterpriseOne and requires an experienced system administrator.
Linux
Oracle has announced its Unbreakable Enterprise Kernel for Linux. To run Oracle virtual machines in your architecture, you’ll need to be running this operating system. If you want to drop your support costs, Linux may be an operating system worth considering, because Oracle does not charge a large amount for support.
Microsoft Windows
Oracle supports several versions of Microsoft Windows. With Tools 9.1, you should be running 64-bit operating systems, which will help to ensure performance on the system. Your servers should be running Windows 2008 and your development workstations running Windows 7. You will need to be aware of the patches that come from Microsoft and ensure that you test against these patches.
Solaris is an operating system developed by Sun Microsystems. This operating system is very scalable and reliable. Oracle has continued support of this operating system after purchasing Sun.
AIX
AIX is the operating system for IBM’s RS/6000. This operating system is a form of Unix. To ensure success, make sure that a member of your team is deeply familiar with this operating system.
HP-UX
Because of changes in its strategic direction, Oracle is discontinuing support of Hewlett-Packard UniX. As of Tools 9.1, Oracle was no longer testing, validating, or supporting HP-UX on HP PA RISC. If your system is currently running HP-UX, you should attempt to find a partner to help migrate your enterprise to one of the supported operating systems. Oracle has indicated its plans to support HP-UX on Itanium, but currently this is not validated for EnterpriseOne 9.1.
Operating System Levels
You can choose from among several different levels of operating systems and still be supported. As OS vendors move forward with new releases, Oracle will certify these releases against different tools releases and software releases. Oracle’s product strategy team communicates which operating system levels they support and which will be decremented in the JD Edwards EnterpriseOne Platform “Statement of Direction.” This document can be found on Oracle’s metalink site, at http://metalink3.oracle.com. It is important for your team to know which direction Oracle is going with regard to your OS.
| NOTE
The JD Edwards EnterpriseOne Tools releases contain the code that handles the operating systems and databases. A developer in EnterpriseOne does not have to worry about what type of OS or database their report will be used against; the Tools release handles all of this. Each Tools release is aligned or certified with specific releases of the JD Edwards software. These applications and business logic deliver functionality to the end users. For example, you can run EnterpriseOne release 9.0 with Tools 9.1, or you can run EnterpriseOne release 9.1 with Tools 9.1, but you cannot run EnterpriseOne release 8.12 or below with Tools release 9.1. |
You may be wondering how to choose which OS level you should run. This is a very valid and important consideration. The answer depends upon how you answer questions about several components: Where are you at in your implementation? Are you just planning now, in testing, or are you about to go live? It also depends upon how easily your organization can absorb testing and change.
If you are just starting your implementation, you should always try to aim for the latest version of the operating system, which will help to keep your system up-to-date and functioning correctly. Ensure that your third-party and bolt-on software can handle the new operating system as well; sometimes these vendors may lag behind on the certification of a new operating system.
Suppose, for example, that your system has been live for a while, and you are running V6R1 on an iSeries. How do you decide when it is time to move to V7? Although it’s not as large as an implementation, the upgrade of an OS level in your JD Edwards EnterpriseOne architecture needs to be taken seriously. Before you undertake such a change, you need to plan for the change and dedicate the proper resources to the endeavor.
As for when you should do this, our advice is that you should always strive to keep your operating system and your EnterpriseOne software up-to-date. One way to accomplish this is to set aside some time and money biannually or annually to perform updates. This will ensure that you are using current software and will already have the resources set aside to help with the testing. When you are scoping out your project, don’t forget to inform the CIO and CFO that you have set aside time and resources to keep your system up-to-date.
| TIP
An automated testing tool such as the Oracle Automated Testing Suite (OATS) will help you test and reduce the level of effort required by your business to test new systems. |
Patching the Operating System
When reviewing performance tuning, the operating system is a key component of your approach. If you do not maintain your operating system, your chances of experiencing performance issues in EnterpriseOne will increase significantly. No matter which supported operating system you are using with EnterpriseOne, it needs to be maintained.
Many operating systems are supported, including the following:
iSeries Linux Solaris Windows 2008We recommend that you patch your operating systems in a controlled manner and that you schedule time to install these patches. You can coordinate patch installations with the application of EnterpriseOne Tools releases, as support for new operating system levels come from these releases. This works well if the patch is significant.
You should develop a test script to be executed when applying operating system patches. The patches should be applied in a separate area that does not affect production—that is, your architecture should allow you to have production separated from development and prototyping to allow for isolated testing (see Figure 2-2). As a best practice, your architecture should support testing of operating system patches without affecting production.
FIGURE 2-2. Architecture showing separation of development and prototyping from production
Often, organizations with this type of separation can take a month to apply system patches to production after the patches have been installed in development and prototyping systems. This practice ensures that they run the system after patch installation for a full month. This helps to isolate issues and ensure production uptime.
If the customer has defined metrics to track performance, you can also gather data points on the effect, if any, of the operating system patch. This type of information can help to show where performance improves or drops off. To ensure accuracy, do not change too many variables at the same time.
| TIP
Although, as an administrator, you may sometimes feel alone, you are not. You can take advantage of many message boards to gather information about how patches have affected other sites. You’ll find such message boards on Oracle’s web site www.oracle.com, the Quest user group web site www.questdirect.org, and the famous JDE List www.jdelist.com. Use these tools to help you; if others are having consistent issues on either operating system levels or patches, you can get support. |
Disk Considerations
You don’t expect a monster truck to go fast; you expect it to be powerful and have the ability to crush other cars. But when you look at a smaller racecar or sports car, you do expect it to be quick. In some ways, you can think of your disk in the same manner. For example, a slower disk is less expensive, but it can directly affect system performance.
When thinking about disk performance, you should consider the following:
Disk speeds Disk layout Storage area network (SAN) versus network area storage (NAS) optionsDisk Speeds
Most of your users will always feel the need for speed. To help your system perform, you must first analyze your disk speed requirements. You need to look at not only the speed of the disk, but the seek time on the arm. If you have fast disks with fast seek times, you should have very little disk latency in your system. This will allow for a faster overall system.
As with everything in life, there are compromises. The faster the disk, the more expensive it is. However, there is nothing to say you have to use the same disk across your entire system architecture. You can use a slower disk for the development and prototyping servers, for example.
Disk Layout
The layout of your disk also can greatly affect your performance. Layout means how many disk arms you have included in the architecture, as well as what kind of Redundant Array of Independent Disks (RAID) system you use in your system (see Figure 2-3). There are several different types of RAID:
FIGURE 2-3. Common disk configurations with EnterpriseOne
RAID 0 RAID 1 RAID 2 RAID 3 RAID 4 RAID 5The following information was gathered from professional experience as well as referencing Wikipedia (http://en.wikipedia.org/wiki/RAID).
RAID 0
RAID 0 is commonly striped, which means that the data is written across both disks, so there is no parity and thus no redundancy. So you may be asking yourself, Why on Earth would I ever use this type of configuration? The answer is speed. When speed is more important than data integrity, RAID 0 is an option. An example is a drive that you use for temporary files only.
RAID 1
RAID 1 is a very common and simple configuration. In this case, the data is written to two disks identically. So the exact same data that is written to one disk will be written to the second disk, unlike RAID 0, where the data is striped and all the data is split across the disks. RAID 1 gives you some redundancy, meaning you can lose a disk and not lose all your data.
Because the same data is written to both disks at the same time, it takes longer in this configuration to write data to the disks. However, reading data is quick with this configuration. If you have operations that are read-specific, without lot writing, then RAID 1 may be the configuration for you. For example, RAID 1 may be appropriate if you have a database that is used only for reporting and is read-only.
RAID 2
RAID 2 is not really relevant anymore with the advance of disk technology. This configuration consisted of striping data at the bit rather than block level. It used Hamming code for error correction, which allowed for single-bit corruption recovery. However, as disk technology advanced, the Hamming code was implemented directly onto the hard disks.
RAID 3
This technology has striping with parity on a dedicated disk. We have not really experienced RAID 3 in use in the real world. This is probably because all activity on the disks need synchronized disk arms. This obviously has an effect on performance of the disk and thus the system.
RAID 4
RAID 4 is a little different from the others because it also uses striping across the disks with a separate parity disk. However, unlike RAID 3, which uses bit striping, RAID 4 stripes in blocks. This makes it act more like RAID 5. However, like RAID 3, RAID 4 has only one parity disk. This technology, like RAID 3, has become obsolete. It was replaced by RAID 5.
RAID 5
This is the RAID configuration most of you are probably familiar with. Almost everyone is familiar with RAID 5 on some level, because it is one of the most common configurations (see Figure 2-4). We see this type of configuration a lot in the industry with servers that use native disks.
FIGURE 2-4. Example of RAID 5 with EnterpriseOne
Storage Area Network (SAN)
A disk is a disk is a disk, right? Well, not quite. You’ll find many different vendors of SAN systems, such as EMC, IBM, and HP. Each offers its own solutions to the disk storage issues. Which one is best will depend upon your needs, budget, and preference.
EnterpriseOne will likely be only one of the systems utilizing the disk storage solution. Although those of us who work on software sometimes think everything should revolve around our system, this is not always the case.
Does EnterpriseOne care about which solution you choose? To answer this question, you need to think about what is actually required. From a technical perspective, EnterpriseOne does not care which product you choose for storage. As long as you can get a drive letter or volume recognized by the operating system, EnterpriseOne does not know or care whether you’ve opted for a native disk or a SAN solution. Does that mean all of the solutions will perform the same? No, of course not; almost any technology has pros and cons.
Although you are probably getting tired of hearing this—yep, you guessed it—to ensure success, you must plan, plan, and plan some more. With SAN solutions today, you must choose among a variety of options, such as how much cache capability is built into the solution. This capability will keep data that is accessed frequently in memory so that the access speed is much faster.
The SAN solutions also allow for your database administrator not to have to worry so much about what data is stored on what disk. With a native disk, this is a necessary exercise to ensure that there is no disk arm contention. A common mistake for SQL Server databases, for example, is for people to put their TEMPDB on the same disk as their database. Because TEMPDB is accessed all the time, this slows down access to the database due to contention. With current SAN technology, this is not an issue. The SAN takes care of what disks are written to and where the data is stored. To the database and to EnterpriseOne, the data is simply on a volume or drive letter.
So what do you need to plan for? What could possibly go wrong with all this useful technology? A lot can still go wrong, even with all this useful technology. Consider a real-world example: We had a system up and running that experienced great performance. All of the machines used a SAN solution. Development used a machine with an older disk and production used one with a faster disk. Both of these SAN solutions were shared with other systems in the company’s IT portfolio.
One day, everything came to an absolute crawl. We looked everywhere. The network team proved that the network was performing correctly and data was being sent back and forth. The database administrator saw a slowdown on some SQL statements, but he couldn’t identify any kind of pattern in the performance. We were finally able to track this down to a problem with the switch that EnterpriseOne was assigned to access the disks. Even with a cache and high-speed disks, we still needed a path to the data. Someone had put EnterpriseOne in the same data path as another system that was using a lot of bandwidth to the disk. So EnterpriseOne got strangled, and thus slowed down and then sped back up randomly as the other system used the SAN.
Here’s the moral of this story: Don’t assume everything will be fine because you have a SAN system in place and it offers lots of great technology. Bring your SAN engineer into the discussion. Have him or her sit down with you and discuss the path to the data. Ask the following questions:
How many other systems share this path? Once you start testing the EnterpriseOne system, what sort of reporting can you expect? If you have a cache, how often will you get your data out of cache versus getting it from the disk? Are too many systems or requests coming down the same path?Disk performance is a key metric to keeping your system running effectively. Insist on regular reports you can compare with your other data to show your system’s performance.
| TIP
SAN systems usually use fairly inexpensive disks, but the cabinets that contain the disks are not quite so inexpensive. You should always know how much disk space is available before a new cabinet is required to house additional disk drives. Also ensure that you communicate your projected disk growth to your infrastructure team so they can help plan accordingly. This will save you a lot of budgetary and time headaches down the road. |
Figure 2-5 shows an example of a SAN system with EnterpriseOne. EnterpriseOne sees this disk as simply a logical disk volume—drive F: or G: or mount point /u01. The system is not aware that this disk is actually a SAN system. As long as the system can access the disk, the system will function. As discussed previously, you cannot forget about disk speed as you undergo performance tuning. This one component can kill your entire system performance. So be sure to have a person available to your EnterpriseOne support team who knows this system and can monitor its performance.
FIGURE 2-5. Example of EnterpriseOne using a SAN system
Network Area Storage (NAS)
Another system that is not really used that often with EnterpriseOne is network area storage (NAS). A NAS system is slightly different from a SAN system. (Some of you are probably asking, Does this industry have enough acronyms?)
Let’s take a minute to talk about the differences between a SAN and a NAS. The main thing to keep in mind is that on a SAN system, your machine’s operating system handles the file system. This means that the disk will show up as a mount point or a drive letter. However, with a NAS system, the NAS handles the file system. It’s similar to mapping a drive to a file server. This is why you normally do not see a NAS used in an EnterpriseOne architecture.
A NAS is usually appropriate for media objects. The EnterpriseOne software, upon install of clients, platform pack, and deployment server, will look for a true drive letter, not a mapped drive to another machine. However, as you probably know, EnterpriseOne uses the Universal Naming Convention (UNC) to find data, such as check-in locations and media object files, so these types of files could be placed on a NAS.
Database Components
The database knows all and sees all. Let’s pull back the curtain a little bit and see how this component can affect your system performance. But before we do this, we need to cover a few items.
First, we’ll discuss several types of databases you can use with EnterpriseOne software. Because EnterpriseOne is platform-independent, you have a variety of choices when it comes to the type of database you want to use. Your database layout is another important consideration. We will briefly cover this here, and you’ll learn a lot more about this topic in later chapters. Finally, we will touch on general tuning levers, including some high-level items that you need to be aware of for your database.
Types of Databases
One of the true powers of the EnterpriseOne system is that it supports a variety of database types. This gives customers who use the software a lot of flexibility when architecting their system. Oracle’s Minimum Technical Requirements documents for database servers 9.1 can be found on Oracle’s support web site at http://myoraclesupport.com. Currently, the EnterpriseOne 9.1 release supports the following databases:
Oracle SQL Server DB/400 DB2These database types run on different supported platforms. Currently Oracle supports the following platforms for EnterpriseOne 9.1:
Linux Solaris AIX Windows | NOTE
Some of you are probably wondering why HP 9000 is not in this list. It used to be a supported platform, but Oracle no longer offers support for HP 9000. How do you know which direction Oracle is going? Well, we are glad you asked. Oracle provides a “JD Edwards EnterpriseOne Platform Statement of Direction” for EnterpriseOne clients that will let you know what platforms and operating systems are supported and when support of some operating systems will be deprecated. This document can be found on http://myoraclesupport.com. We recommend that you review this document at least once a quarter to ensure that you are working with supported platforms, operating systems, and databases. Obviously, if you are not using a supported architecture, the chances of good performance are significantly reduced. |
Oracle Database
When selecting a database, you will at least want to consider Oracle Database. Currently, the EnterpriseOne Tools release 9.1.0.x is compatible with EnterpriseOne releases 9.0 and 9.1. At this Tools release level, you need to run an Oracle 11g database.
SQL Server
SQL Server is a very popular database type. Currently Oracle supports SQL Server 2008 and 2008 R2. This database is a very good choice for shops that already have the skill set in-house for other applications. The database also scales and performs quite nicely when configured properly.
DB2 for iSeries
If you have been working in the industry a long time, you probably still call the platform database runs on an AS/400 instead of IBM iSeries. DB2 i-series is a popular midrange system with an integrated database. Currently Oracle supports versions 6.1 and 7.1. The advantage here is that you don’t need a separate DBA; you simply need a system administrator. This is an advantage, but also it requires that the administrator have a specialized skill set. A DBA will know the details of the database only, and not the operating system necessarily. In SQL Server and Oracle, there are very specialized skill sets to handle this. An iSeries administrator does need to know the database details, but simply the operating system. Most of the details are masked from the administrator.
DB2
With Tools 9.1.0.x, Oracle supports DB2 releases 9.7.0.1 and 9.7.0.5. We do not see this database in use as often as others with regard to EnterpriseOne. Like the others, however, it is a solid and viable database option.
Database Layout
When you are working with different databases, you’ll need to consider the layout of each. There are many different examples of layouts, but in this chapter, we are going to keep it at a fairly high level because we go into much greater detail in Chapter 8.
Oracle Database
When you architect the database, it’s important that you consider the database layout and involve your DBA. And remember that just because your CNC knows about Oracle Database does not mean he or she is a DBA. Be sure that you involve the DBA early in this process. The DBA should help plan the layout of the disk for your database as well as the System Global Area (SGA) and Program Global Area (PGA).
SQL Server
With SQL Server, you also need to ensure that a DBA is involved up front. Some of the issues you need to look at, other than disks, are where you are going to store different databases, because EnterpriseOne installs multiple databases. In addition, if you are a large company, you may need to divide the data from specific tables, such as the general ledger F0911, across different disks. You also need to ensure the TEMPDB, which is active constantly, is on its own disk with the correct RAID configuration. In addition, when you work with SQL Server, you must know how many CPUs and how much memory you are going to dedicate to the database. This will help to ensure a good foundation for building your tuning approach.
DB2 iSeries
The DB2 iSeries database is unique, because it is integrated into the platform itself. In this case, you need to involve your iSeries administrator. Several parameters need to be set, including disk and memory pools. Also, if you have a large iSeries server system, you may be using a logical machine or logical partition (LPAR). How you configure and set up this system will affect the performance of your EnterpriseOne system.
DB2
With DB2, you also need to ensure that disks are laid out correctly and that the memory is allocated correctly. Work with an experienced DBA.
General Tuning Levers
It is important at this point to discuss a few general tuning levers. These levers are items that can change performance across the entire system, instead of just one small part, such as sales order entry. To help you understand the framework for starting with a system or starting to look at tuning your database, we’ll cover the following:
SGA PGA TEMPDBSystem Global Area (SGA)
Related to Oracle Database, the SGA is a group of shared memory areas that are dedicated to the database. SGA is one of the important large tuning levers, so you must be sure to allocate enough memory to your database to accommodate SGA. You might, for example, start with only 150 concurrent users and expand later to 500. As your user base grows, you need to ensure that you allocate enough memory to accommodate this. You may also need more SGA, depending on how you are using the database—that is, are you doing mostly reads or a lot more writing to the database?
Program Global Area (PGA)
The PGA stores the control data for your database. How you set up PGA depends on the size of your database, the number of users, and what the database is being used for. The PGA setup can affect the speed of your database. The PGA holds the sort area, among other things: The sort area in memory handles different types of sorting for your database, such as hash joins. If insufficient memory is allocated to the PGA and you are doing lots of sorting, you will experience slower performance. Worse, the performance will vary for the users. Most EnterpriseOne users do not understand or care whether their application or report is producing a large sort that requires more memory. They simply want their report or application to perform in a quick and efficient manner.
TEMPDB
With SQL Server, the TEMPDB is the temporary workspace for storing temporary tables, storing intermediate results, and processing sorting and queries. This workspace is constantly active. If TEMPDB shares a disk arm with another database, it can slow down your database. Some people use solid memory for their TEMPDB; a solid state drive is simply circuitry and does not have the moving parts like a “normal” drive. This means that it can be very fast for finding data.
If you research “tuning SQL Server” on Microsoft’s web site you will find many articles related to the use of the TEMPDB. This type of tuning occurs at a base database level, so it can apply to systems other than JD Edwards EnterpriseOne. Therefore, any other items you are running from your SQL Server database can also benefit.
| NOTE
Be sure that TEMPDB has its own disk arm! |
Network
If you have implemented JD Edwards EnterpriseOne within a large company, you’ve probably dealt with sites over a wide area network (WAN). Sometimes these sites are located across town, and sometimes they are across the globe. Following are some real-world examples of network issues across a WAN.
At a client’s corporate headquarters, everything was working great on the system, but operations in Finland were not going so well. The users were constantly complaining that the system took too long to navigate and enter data onto screens. The batch performance seemed fine once the job was submitted, but the interactive parts were performing unacceptably.
You can guess the first place that was suspected as the problem: It must be the network! Break out the tar and feathers. When you look at these types of performance issues, take the network into account, but realize that it might not be the whole problem. Let’s examine what we did in this case.
We tested timings from multiple sites. We started by sending simple pings from the client machine to the web server—we started at a DOS command line and typed ping WEBSERVERNAME. This gave us information on the trace route over the network that the client machine was taking to get to our web server. (Sometimes, you’ll find interesting things when doing this exercise.)
We found a serious latency issue. The pings from Finland where taking 120 milliseconds, and when we pinged from the corporate office we experienced times of less than 8 milliseconds. Even when we pinged across the WAN in the United States, it took 40 to 80 milliseconds.
So why is this information important? Isn’t some latency just a fact of life with a WAN? Yes, it is. Latency will always be an issue in any system—to a point. The trick is to minimize the latency whenever possible. Now, that does not mean we stormed into the network manager’s office and demanded less latency. Because, as with all things in life, nothing is free!
| NOTE
It is rare that a network administrator gets to start from scratch and design the perfect network. Most administrators inherit history from someone else and must work with systems that are not always the best, but that were what the company could manage when the network was created. So sometimes you find networks that take “the long way” around. Such problems, however, can be fairly simple to fix with correct resources and plan. |
Our WAN example was further complicated by the fact that the company did not technically own the network it was using. They had paid another company for office space, and thus the actual wiring and switches did not belong to them. They asked for the hardware to be upgraded and got a polite “no” for an answer.
Users were suffering due to this network issue, but the company had no control over the network. But, even so, all was not lost. We identified the problem—the latency was caused by all the turns. Perhaps performance would improve if all those requests were placed closer to the web server. So this is exactly what we did.
A multitude of software packages will compress some of your web traffic requests and help reduce latency. However, we took an easier approach. We decided that, instead of sending all of these requests across the WAN, the network would simply send screen scrapes, that is, pictures of the browser results rather than all information on the browser. This meant a lot fewer turns and less information would be flowing back and forth. Even though EnterpriseOne uses only about 5 to 6 KBps of bandwidth, it sends a lot of requests. We used a Citrix solution to publish our browser. The company had already been using Citrix to publish other applications, so this solution did not require a lot of investment or training.
Citrix was perhaps not our only choice, but it was a simple choice for us. Using Citrix made a huge difference in network performance. It also really helped the IT team by reducing the number of tickets. The browser was set at a standard level, with no extra software or toolbars, such as Yahoo! toolbar or other items. This greatly helped to reduce the number of complaints about an application’s performance on one machine versus another.
| NOTE
Oracle does not test against Citrix. Oracle looks to Citrix to validate Citrix software against different solutions. You can refer to the minimum technical requirements documentation at www.myoraclesupport.com. |
You can always reduce latency on your network, but the question is, is it worth the cost? Let’s return to the base problem. Why is latency such an issue for the EnterpriseOne software? JD Edwards EnterpriseOne software is very chatty across the network. In order for the software to be user friendly and to validate data as it is entered, the software takes a lot of turns. This means it communicates with the server and then an answer is returned to the client.
An easy example of this is when you tab out of a user-defined field (UDC). The EnterpriseOne software will instantly validate the entry. This is a turn, because it sends a request to the web server, and then information is sent back to the client. The EnterpriseOne software did not always require this process, however. In the infancy of the HTML solution for EnterpriseOne, the software waited until all the data was entered and then the user submitted it. This might sound great on paper, but when you are a sales order entry clerk and you just typed in more than 100 lines of data, only to find out you have an error on every line, it is not very user friendly. Thus, the software was changed to be more interactive with the user.
| TIP
As with other components for performance tuning, you need to plan your approach and ensure that you have a repeatable process. This is not really a one-size-fits-all solution. Different sized companies have different network configurations. Involve your network administrator up front. He or she can often also provide information from third-party monitoring. It should be rare that you actually have to get to the packet-sniffing level, but it can happen. If you have already mapped out our requirements and monitoring with your network team, you will be able to identify quickly whether the network is really part of the issue. |
Network Cards
Network cards are simple, right? You have gigabyte cards in the data center and a lot of bandwidth in the data center. So it’s all good—right? Well, not so much. Although EnterpriseOne has gotten a lot better, sometimes it will experience an issue with network interface card (NIC) teaming. You should be aware of how NICs are configured on the EnterpriseOne Servers. If addition, you should not have auto-detect turned on for these cards, if possible, because this is a waste of time. As long as you have configured your network so that you know the connection speed no matter what, auto-detect is not required.
Packet Priority
Another issue when dealing with WAN issues is packet priority. By working with your network administrator, you can give specific information a higher priority on the network (although this is not always the best thing to do). This is sometimes a good solution for remote offices, and not just with regard to EnterpriseOne performance. Sometimes, certain traffic to your web servers gets higher priority than general e-mail and other network traffic. Adjusting priority can help to ensure that requests to your web server get a higher priority. Again, nothing is free, so work with the network engineer to determine whether this is necessary and/or beneficial.
Generate Server Layout
Suppose you are starting from scratch and installing a new EnterpriseOne system or servers for the next release. You know that these servers talk to each other a lot. They constantly send traffic saying they are on the network, through JDENet. They also send traffic through the server manager agents, which monitor and help administer each of these machines.
Keeping all this in mind, do you really want the servers located on different subnets? Do you want them to have to go through a switch every time they need to communicate with each other? The answer is, as usual, it depends. For some companies, putting everything together is the best option. Others have strict controls and need to isolate all their productions servers from the rest of the network. (For example, a life sciences customer is required to meet specific guidelines for how it handles IT processes.) This is an area where you need to do a little up-front planning.
You also need to consider whether you are going to use a network appliance to help load balance for your EnterpriseOne configuration. You can use a content switch to help balance the load across your web servers. You can also use this solution to help load balance your business logic and Universal Batch Engines (UBEs). This would mean EnterpriseOne would see an alias instead of the actual application server names.
| NOTE
You can read one of several white papers on how to do this at www.myoraclesupport.com. One that might be a useful start is “Configure JD Edwards EnterpriseOne Servers on IBM WebSphere Application Server with F5 Big IP.” This is a more complex configuration, so if you are going to take this on, be sure you allocate enough resources and time to configure and test the solution. |
Direct Application Tuning
In this final section, we will discuss some things that directly affect your tuning of the EnterpriseOne application. We will cover the following areas:
User interface Application tier Batch tier Interfaces Common performance tools ReportingUser Interface
You might be wondering how the user interface (UI) can affect performance. EnterpriseOne gives you a lot of flexibility, including the user interfaces you choose. However, as you know, nothing is free, and the UI you choose can impact performance. Often this impact is very small, but sometimes it can make a huge difference.
When you log into EnterpriseOne 9.1, you will see a screen similar to the one shown in Figure 2-6. This screen includes a lot of information, but it’s not normally where you can see hits to your performance. You need to look a little deeper for that.
FIGURE 2-6. Example of EnterpriseOne 9.1 screen after login
To see performance hits, you need to look at an application screen; you’ll often see some performance impacts on application screens with grids. For example, consider a sales order entry application; this screen is commonly used and also has a large grid. The number of columns in the grid will need to be rendered on the workstation, and this can sometimes cause performance slowdowns.
Oracle has taken steps to help resolve this. For example, the newer tools release will render only what is shown on the screen, so unless you scroll all the way over you don’t have as much of an impact. Still, it may be worth limiting the number of columns in your applications that appear on screen, not only to help with performance, but to get them out of the user’s way if they are not useful. You can use user overrides to change the appearance of the application so that specific groups, or everyone, can view only the information that’s necessary for the work to be done. These overrides can also be promoted to higher path codes using Object Management Workbench (OMW).
Application Tier
The application tier handles all of the business logic in the EnterpriseOne architecture. This should be separated from report handling for performance reasons, because UBEs and reports use a lot of CPU and memory, which “steals” memory from the business logic or call object kernels that perform work for the interactive piece of the software. Thus, performance can fluctuate as UBEs grab CPU and memory.
To avoid this issue, you can change your architecture to dedicate machines to handle only business logic or only reports/UBEs. You can also combine your database server with your logic or UBE server, but again this is not recommended because of performance issues. Figure 2-7 shows how you can effectively divide up your architecture to handle the load on your system.
FIGURE 2-7. Example architecture to maximize impact of the application tier
Batch Tier
The batch tier handles your reports or UBEs. These reports can be somewhat CPU and memory intensive. Sometimes they are long-running. So when looking at performance, you need to consider whether your system has enough computing power to handle all of the reports that you require.
To determine whether your system can handle the reports, you need to estimate your reporting requirements. If you’ve not used EnterpriseOne in the past, this may be difficult, but it’s still very manageable. Work with an experienced applications consultant by describing your processes to determine the standard reporting load you will need. The module you are running also affects the load. If your system is performing manufacturing resource planning (MRP), for example, this will have a much larger impact on the system than accounts receivable. If you know how much your estimated reporting load is going to be, this will help you correctly size the machines for the batch tier.
Next, consider your job queues. You can have the biggest machine on the block, but it won’t help you much if you are sending your jobs through a tiny pipeline, one at a time to the machine to process. An experienced applications consultant can help you with your job stream. The job queues should be set up in a logical way. Many jobs in EnterpriseOne are single-threaded jobs, which means they can run only one at a time through a job queue. That being said, if you design your system in the appropriate way, you can set up multiple single-threaded job queues. This requires that the data selection for these jobs be set correctly so that double-posting does not occur.
When setting up job queues, you can specify a default job queue within the UBE version itself, so that a specific version of the report will always go to a specific job queue. However, be aware that this information is stored in a binary large object (BLOB) field, so unless you are using special software or you write your own program to update this field, once you set all your versions to be overridden to specific job queues it can be difficult to change them. So, for example, if your business analysts are going to copy a version, make sure you verify the job queue prior to promoting it. Also, be sure to publish the job queues and the approach to the business. This may sound simple and obvious, but we can tell you that we’ve experienced people who are in a rush to get started fail to pay attention to this detail; it is more difficult, then, to get the reports aligned to it later.
The JDE.INI settings must also be addressed on your batch tier. JDE.INI settings are covered in detail in Chapter 7. For now, you should know that these INI settings control your system’s kernels. When you tune your batch server INI file, make sure you have enough JDENet kernels as well as UBE and call object kernels. The JDENet kernels are often forgotten. These kernels are like the system’s postal workers. Their job is to open messages and route them to the correct kernels for work. If your system has too many packages and too few workers, your system will slow down, even though your CPU and memory may look fine on your batch server. Be sure to use Server Manager to monitor the system. This will show you are having a slowdown due to too few JDENet kernels.
Interfaces
It’s important that you map out the system interfaces and the processes that use them. While planning for interface use, you need to account for growth as well as actual performance requirements. Then you need to ask, How am I going to monitor the performance on these interfaces? Often, a different team, and sometimes a different company, handles the “other side of the interface.” You must ensure that you can measure your system performance up to the point of porting the data over to the other system.
You also need to ensure that you have an agreement with the team that maintains the system into which your system interfaces. If performance issues are noticed, how are you going to approach the problem? Can the other team gather metrics on their side and report them to you, and vice versa, so that issues can be identified early and before users complain?
Metrics is an area that is sometimes tricky and neglected. The end users don’t care that the data goes “over the wall” of the EnterpriseOne system. All they know is that they need to get through a business process, and if it is not performing to expectations, they can’t get their work done. Metrics help you establish what is acceptable performance and help ensure that everyone’s expectations of the system are the same.
| NOTE
Many third-party tools on the market can help you establish system metrics. One example is Oracle’s plug-in to their Enterprise Manager. They have a plug-in for JD Edwards as well as other systems to help monitor performance. |
Common Performance Tools
One of the first questions you should ask when looking for performance tools is, What am I trying to measure specifically? Sometimes you want to get lots of detail about your database. Other times you want to monitor the server’s operating system or storage system. Of course, you also want to see what is going on inside the EnterpriseOne system.
To do all this, you need to use different tools. A variety of network, database, and operating system tools are on the market—way too many for us to cover in this book. As you’re considering which tools you need, consider which tools best handle what you want to monitor. Consult your network team, your DBA, and your infrastructure team to select the appropriate performance tools, and ensure that the reporting can be pulled in and combined with information gathered from other tools to give you a complete view of your system.
Several tools can assist you in monitoring, tracking, and troubleshooting performance issues in EnterpriseOne, particularly the following:
Server Manager Performance Workbench JD Edwards EnterpriseOne Application Pack for Oracle Enterprise Manager Cloud ControlServer Manager
Server Manager will be covered in multiple chapters later in the book. Server Manager allows you to monitor and administer your EnterpriseOne system. The software is generally installed on the deployment server, and agents are installed on each of the different types of EnterpriseOne servers. These agents communicate with the Server Manager software, telling it about what is occurring on each of the servers. The agents also work with the installed software to allow the administrator to administer the machines through an HTML user interface. This means you can start and stop your web servers, check log files, or check to see what activities are occurring on the system from pretty much anywhere on your network. Figure 2-8 shows how Server Manager works.
FIGURE 2-8. Server Manager overview
Performance Workbench
Performance Workbench is often an unsung hero when it comes to troubleshooting performance issues with EnterpriseOne. It has been around for a while and is kind of like the crescent wrench of EnterpriseOne performance tuning. As we have discussed, when you’re approaching performance tuning, you need to know where to look for the performance issues. This tool can actually help you with that. It quickly runs through your logs and tells you where the system is spending its time during a process and if it spots any errors.
If you have been working with EnterpriseOne for a while, you know that if you turn on the debug logs they can get large very quickly. Running through these by hand is not a fun process. This is where Performance Workbench, as shown in Figure 2-9, really helps out.
FIGURE 2-9. Performance Workbench
As you can see in Figure 2-9, this tool offers many options. You can use it to parse a JDEDEBUG log from a client workstation or an enterprise server. It allows you to look for different levels of detail, including business function, open table and close table counts, or you can look at everything—that is, counts of business functions, selects, inserts, deletes, updates, and all information related to business functions. It can also generate a call stack file, which is useful when you’re debugging an application error. This parse functionality can also be used with a JAS debug log.
This means that you can trace a process, such as entering a sales order, all the way through to see what is happening. You can gather logs for issues on both the HTML server and Enterprise server to see if an error occurred on either server. Thus you can see what is happening at a specific point in time during a process.
This tool is useful not only for finding errors, but it includes a setting to generate a timing gap file that shows how much time various processes on your system are taking, down to the millisecond in releases 8.9 and later. So, for example, you can see if SQL selects are using up the majority of the time when you perform your trace or if a business function is taking more time during a transaction.
JD Edwards EnterpriseOne Application Pack for Oracle Enterprise Manager Cloud Control
We haven’t seen this tool in use as often as we’d like. It is a very neat tool. It’s actually a plug-in for Enterprise Manager, the console that you use to manage Oracle databases. The Cloud Control plug-in allows you to do a lot of performance monitoring with your EnterpriseOne servers. You can use it to gather metrics and even monitor the system for an SLA. It requires that you obtain an additional license fee from Oracle, but depending upon your enterprise size and requirements, this tool might be worth the cost.
Figure 2-10 shows an example of using the Enterprise Manager to see JD Edwards EnterpriseOne servers. You can see the types of servers as well as their status. You can also use this software to set up specific monitoring of your EnterpriseOne system, including web servers and enterprise servers.
FIGURE 2-10. Enterprise Manager E1 Application Pack
Let’s start with the web server, because this is where a lot of really good information can be gathered using the Cloud Control plug-in. You can gather the following metrics using this tool:
Average execution time Cache group Call object statistics Number of applications that are open Database data source information General system uptime JDENet connection manager JDENet connection pool socket Java heap memory used Response Number of system errors Time-out errors Total number of users logged onto the system User sessionsThis type of information can tell you what is happening at any moment on your HTML server. This software also allows you to collect information for reports. This means you can see whether you are within an SLA. So, for example, if a user complains that the user interface—the HTML server—is slow, you can see the actual performance of the particular server itself.
You can also use this software to monitor and gather information on your enterprise servers. This is very useful because it can tell you how each machine is performing. Following are some of the important metrics that you can gather:
Average CPU Monitor kernel processes Response from server Number of database connections Number of incoming network connections Number of outstanding requests Number of users for serverReporting Using the Cloud Control Plug-in One of the main advantages of the plug-in is the reporting you can get on the actual performance of your system. Having this information at your fingertips allows you to see quickly where the problems exist in the system. With performance tuning, you sometimes have to sort through a lot of data to determine where problems reside. This tool allows you to see all of the data in an easily digestible format—in other words, you get quality information instead of just quantity.
You can configure and set up dashboards to show what is happening on your system. All areas of your business are not equal, so, for example, if part of your EnterpriseOne system is customer-facing, you’ll give this a much higher priority than back-office functions. (After all, it takes only a few seconds to lose a customer.) Figure 2-11 shows an example of how you can view the data gathered. This data shows how your HTML server is performing at a particular point in time.
FIGURE 2-11. Browse performance data
You can also use the tool’s dashboard functionality to enable a view in which individual business units or divisions may be interested. This functionality makes it very easy for executives to see and evaluate performance in their areas of the business. You can also restrict the dashboard view to show only areas that are of particular interest, at a system level, to you. Figure 2-12 shows an issues overview as well as the availability of the different servers.
FIGURE 2-12. Issues overview and availability
This plug-in offers even more, but here we’ve given you a high-level overview so that you can consider using this type of solution in your JD Edwards EnterpriseOne architecture. Performance issues become more manageable when you can actually show baseline measurements and thus get the business to agree on a specific service level agreement on performance. This will allow you to see any performance changes, for example, after you apply new patches to your system.
| TIP
The JD Edwards EnterpriseOne Application Management Suite also allows you to monitor an iSeries. So do not think that just because you are using a blue stack architecture that you cannot take advantage of this tool. |
Handy Third-Party Tools
Over the course of working on the JD Edwards software package, the authors have found several handy third-party tools that you can leverage to help you in performance tuning. These tools are fairly specific to circumstances. They can make the difference between an issue that takes a few hours to resolve and one that takes a few weeks to resolve.
IBM Pattern Modeling and Analysis Tool (PMAT) for Java Garbage Collector can be downloaded from the Internet. It allows you to analyze verbose garbage collection. This will help you quickly to identify any issues in garbage collection. The tool also provides a graph view that can help you quickly identify issues.
Another tool, SunSpider JavaScript Benchmark, can also be downloaded from the Web. This tool lets you benchmark a browser’s performance, which can be handy if you are trying to determine whether performance issues are directly related to the browser or are within EnterpriseOne. If a problem occurs with a browser, support staff or your CNC consultant may ask you to try performing the same task using a different browser to see if the problem persists. If, for example, Firefox seems to be fast and Internet Explorer is slow, you can determine whether the slowness is really a speed issue or the browser.
Wireshark is helpful if you suspect that the trouble may be at the network level. This tool is basically a package analyzer that lets you see what is happening with the traffic from the web server to the database server or from client to web server. This is a fairly technical tool, and you should always have permission from the network team to employ this type of tool before using it. Wireshark can help to pinpoint an issue or prove that the issue is not at the network level.
Summary
In this chapter, we covered a lot of topics. The idea was not to go into too much detail here, but to help provide the basics of the components of performance tuning. We covered each of the different tiers, dove into the supported platforms and their operating systems, and talked about considerations while patching these operating systems.
We also spent some time on disk considerations for system performance. We talked about the different disk types as well as disk layouts. We covered how you could use a SAN system with your software as well. We then moved on to the database component of the system and covered several database types that are supported by EnterpriseOne. You learned about different layouts and the general tuning levers for databases.
We discussed some real-world examples of how a WAN can affect your EnterpriseOne system. This chapter wrapped up with a discussion of direct application tuning, including a discussion on the user interface, the application tier, batch tier, database tier, interfaces, and performance-tuning tools.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.