Extending Functionality via Plugins

One of the design goals of Oracle Enterprise Manager 12c Cloud Control (the current release at the time of this writing) is to allow Oracle’s developers to be more flexible. The previously used framework was quite rigid and made it difficult to change code. It was equally difficult for Oracle to deploy new functionality. The new approach was to make the whole architecture a lot more flexible. In addition to easier development, the new architecture also allows the administrator to deploy a smaller subset of functionality. Instead of deploying all functionality during the installation, the new release starts with a specific number of plugins. Any further functionality or upgrades to an existing plugin can be downloaded and implemented later.

Enterprise Manager relies a lot more on communication with My Oracle Support, and both are tightly integrated. The Management Service will periodically poll Oracle’s websites to see if there is new or updated functionality available. This function is called Self Update and will be discussed later in this chapter in greater detail. When new functionality is available the administrator can download it to the so-called software library and apply it during a change window. Plugins are not limited to the Management Service, but also exist for targets and agents. New functionality can equally be added to the agents, allowing for a smaller initial storage footprint.

The Role of the Agent

The Oracle agent is the part of the infrastructure responsible for gathering information about the monitored targets. The agent can be installed via the Enterprise Manager console, or alternatively via the command line. Oracle supplies a response-file based silent installation as well as an RPM for the Linux platform. The latter however proves more trouble than it is worth due to the way it is implemented. When verifying the contents of the RPM you get lots of missing files, which is not a defect but rather how that particular RPM is designed.

The agent has its own package and library prerequisites, which are usually met already when installing it on an Oracle database server. The agent image will require 1 GB free hard disk space. Unlike Oracle Enterprise Manager 11.1 and earlier, you can no longer download the agent from Oracle’s website. The agent has become an integral part of Enterprise Manager and as such is internally managed via a new OEM 12c feature named “Self-Update”. You always get the agent for your OMS platform as part of the installation. Management agents for other platforms need to be downloaded from My Oracle Support.

The agent has been specifically created with a less-than-optimal network connectivity to the Management Service in mind. It is able to tolerate network disconnects and poorly performing infrastructure without severely impacting the Enterprise Management deployment as a whole. Although an unreliable communication between an agent and the OMS will be felt for the targets monitored by the agent, it should not cause an effect that can be felt system-wide.

High latencies and disconnects between the repository and Management Service however are poison to the deployment of Enterprise Manager. The impact of poor network quality between the Management Host and the repository database is severe and likely to be felt by every user connecting to the console.

The Oracle Management Service

The Oracle Management Service or OMS is at the heart of the Enterprise Manager infrastructure. Agents upload their metric information to the OMS, and users connect to the Enterprise Manager console. Information, current and historic, is available from the repository database at the application’s backend. At its heart, the OMS is a Java application using WebLogic as the execution environment. As such there are considerations about the amount of memory and the number of CPU cycles available for users. Oracle documentation contains a lot of information on the sizing of the OMS, and there is also a section about sizing considerations in the chapter you are reading. Although it tries to list the main points, it cannot replace the official documentation so have a look at that as well.

You are not limited to having only one OMS, Enterprise Manager is built to scale out horizontally by adding more OMSs to the configuration. For larger deployments it might be beneficial to add an optional load balancer into the infrastructure (while at the same time not forgetting to take high availability precautions for the database). Instead of communicating directly with an individual OMS the agents will communicate with the load balancer’s IP address. Since the load balancer is a perfect single point of failure, you should ensure with the network department that there is resilience built into the design there too!

A lot of important functionality is available from the Cloud Control console, such as notifications and incident management which are going to be covered later in the chapter.

The Enterprise Manager Repository

The repository database is the place where all the information gathered by agents throughout the estate is stored. Not every database release is certified to act as a management repository. You should check the certification matrix on My Oracle Support first. As a rule of thumb, if your release is still in premium support and Patch Set Updates are produced, then there is a high probability that your database is indeed certified. You may have to wait a little bit though for support of Oracle 12c as a repository database. Unlike some previous versions, the Oracle Universal Installer does not create a database for you, and there is no option to install everything on the same host as in Enterprise Manager 10g. Instead, you have to point the installer to an existing database.

Depending on the number of targets and the amount of history you want to keep you have to set aside quite a bit of storage for the repository. The official Oracle documentation has dedicated a chapter to the maintenance of the repository. You should ensure that the network communication between the repository and the OMS is of low-latency and high-bandwidth quality.

Who Should Look After Enterprise Manager?

Very often the Oracle support team is given the honor of looking after Oracle Enterprise Manager. After all, OEM is an Oracle product so it should naturally fall into the realm of the database administrator, right? This attitude is found in many environments, but it can be counter-productive to a successful deployment. Anyone looking after Enterprise Manager needs knowledge about the architecture, the processes such as the data upload from the agents, maintenance of the repository database and many more. Furthermore debugging problems with the user-visible console require knowledge about WebLogic and ultimately how Java Enterprise Applications are deployed.

Oracle has successfully hidden a lot of the configuration details of the WebLogic applications from the user, but every now and then you still need to know where a specific log-file resides to troubleshoot a problem. For this reason you should invest in the people that look after Enterprise Manager. They could have a DBA background, but a Middleware administrator is likely even better suited. As with any complex multi-tier application procedures need to be developed to define what to do when a problem arises, how to escalate a problem to third line support or—in the worst case—how to fail over to the disaster recovery site.

Sizing Considerations

An installation of Oracle Enterprise Manager has to be well thought through, since it is potentially such an important part of the infrastructure! The sizing exercise for the OEM infrastructure is a crucial part of the deployment process. The following should be taken into account:

• Number of managed targets
• Latency and bandwidth of the network infrastructure leading to the load balancer/Management Service
• Expected number of concurrent connections to the console
• Expected or required availability of the service

The following sections in this chapter assume that repository database and management hosts are physically separate machines.

Choosing the Operating System for the Management Host

The Management host and repository database do not need to have the same operating system, although you cannot really make a mistake using x86-64 Linux. At the time of writing Linux was the development platform for Enterprise Manager, and problem fixing is expected to be quickest and most efficient on the Linux platform. Additionally, as laid out in the chapter about hardware, the performance of the current x86 processors is more than adequate to display the Enterprise Manager console. The Enterprise Manager OMS is certified for Oracle Linux 5 update 2 and later. The OMS is also certified on Oracle Linux 6 after the application of a mandatory patch.

As with any planned deployment of Oracle products on a specific combination of hardware and software you should check the certification matrix first. Enter the following after logging into My Oracle Support’s “Certifications” tab:

• “Enterprise Manager Base Platform - OMS” in the Product field.
• “12.1.0.x.0” as the Release.
• Your target platform in the Platform field.

Ensure you read the additional certification information for your platform by clicking on the link named after your platform to find out more about mandatory patches for the OMS, or the operating system or both. In the remaining chapter Linux x86-64 will be used for the management host and repository database server. For this chapter it is assumed that a suitable and certified database is available for the installation of the repository. Refer back to the section “Sizing considerations” for more information about the repository database.

Whichever operating system you choose, please ensure that you download the software for your platform, and unzip it in a suitable location. This chapter assumes that /mnt has been used to mount the installation media via NFS.

Preparing the Linux Operating System

As with the database, the operating system must be prepared for the installation of the Management Service. Before you begin to be more serious about the installation process you should consider which user account will own the OMS installation. In most situations this will be the Oracle account, which is created similar to a database installation. First, create the inventory group-oinstall, then the oracle account if they have not already been created by the build process. Please consult your standards document to check if there are fixed numeric IDs for these groups, especially if you are using directory services. For any Oracle deployment it is very desirable to use the same user and group IDs throughout the enterprise. Alternatively you could simply use the RDBMS preinstall RPM which will always create the oracle user and groups in the same way.

[root@oem12oms1 ∼]# groupadd oinstall
[root@oem12oms1 ∼]# useradd –g oinstall –G oinstall oracle
[root@oem12oms1 ∼]# passwd oracle

Note that groups dba and oper are not needed for the installation of a Management Service. In the next step you should review the use of SELinux and iptables for host-based firewalls. Configuration of these is outside the scope of this section, for the sake of simplicity it is assumed that the firewalls are deactivated, and SELinux is in permissive mode. A production system should obviously be better protected, and you have to check with the security team about the correct settings!

You can either go through the list of prerequisites manually or use the prerequisite checker which is part of the installation source. In order to run the prerequisite check, change to the location where you unzipped the installation media. The prerequisite checker resides in the ./install directory, and has to be invoked as in this example:

[oracle@oem12oms1 stage]$ ./install/runInstaller -prereqchecker 
> PREREQ_CONFIG_LOCATION=../stage/prereq 
> -entryPoint oracle.sysman.top.em_noseed_Core -prereqLogLoc /tmp  -silent
 
Starting Oracle Prerequisite Checker...
[...]
Oracle Prerequisite Checker version Version 11.1.0.9.0 Production
Copyright (C) 1999, 2012, Oracle. All rights reserved.
 
Starting execution of Prerequisites...
Total No of checks: 11
 
Performing check for CertifiedVersions
S_CHECK_CERTIFIED_VERSIONS
Expected result: One of enterprise-5.6,enterprise-6.2,enterprise-6.0,redhat-6.2,
redhat-6.0,redhat-5.6,enterprise-5.5,enterprise-5.4,enterprise-5.3,enterprise-5.2,
enterprise-5.1,enterprise-5,asianux-3,redhat-5.5,redhat-5.4,redhat-5.3,redhat-5.3,
redhat-5.2,redhat-5.1,redhat-5,SuSE-11,SuSE-10
Actual Result: enterprise-5.6
Check complete. The overall result of this check is: Passed
 
Check complete: Passed
========================================================
Performing check for Packages
S_CHECK_PACKAGES
Checking for make-3.81; found make-1:3.81-3.el5-x86_64. Passed
Checking for binutils-2.17.50.0.6; found binutils-2.17.50.0.6-20.el5-x86_64.   Passed
Checking for gcc-4.1.1; found gcc-4.1.2-52.el5-x86_64.  Passed
Checking for libaio-0.3.106; found libaio-0.3.106-5-x86_64.     Passed
Checking for glibc-common-2.3.4; found glibc-common-2.5-81-x86_64.      Passed
Checking for libstdc++-4.1.1; found libstdc++-4.1.2-52.el5-x86_64.      Passed
Checking for setarch-1.6; found setarch-2.0-1.1-x86_64. Passed
Checking for sysstat-5.0.5; found sysstat-7.0.2-11.el5-x86_64.  Passed
Checking for rng-utils-2.0; found rng-utils-1:2.0-5.el5-x86_64. Passed
Checking for glibc-devel-2.5-49-i386; found glibc-devel-2.5-81-i386.    Passed
Checking for glibc-devel-2.5-49-x86_64; found glibc-devel-2.5-81-x86_64.       Passed
Check complete. The overall result of this check is: Passed
 
[...]

Check complete: Passed
========================================================
PrereqChecks complete

If the prerequisite checker returns no failures no further action has to be taken. If the checker does return failures, review which check has failed and correct the problem. If you are planning a graphical installation for the Management Host you should add the necessary packages and their dependencies in order to install via the Virtual Networking Protocol (VNC). You are of course free to use whichever other means to display Oracle Universal Installer, the author has good experience with VNC. Please refer back to Chapters 5 and 6 in this book for more information about VNC and preparations for a graphical installation of the software as well as package management via yum. The reference for packages required for every Linux distribution and release can be found in Chapter 3,  “Enterprise Manager Cloud Control Basic Installation Guide”. Unlike the database, which requires many modifications of the kernel parameters, only one parameter needs to be changed for the OMS installation: shmmax. Edit /etc/sysctl.conf with your favorite text editor and ensure that the shmmax is set to 4294967295. This unusual value is 1 bit less than 4 GB.

The only change necessary to the per user limits is to increase the maximum number of open files from 1024 to 4096 for the OMS owner in /etc/security/limits.conf:

oracle          soft    nofile          4096
oracle          hard    nofile          4096

These changes only take effect the next time you log in. If you had a VNC Server session active, you need to kill it and create a new one.

You should have at least 10GB of disk space available for the installation of the OMS on the local host, preferably more. This is purely for the installation of the software required to run the OMS, not the software library. If you can, you should allocate more space for the installation, ideally on a logical volume and an online-resizable file system to be able to deal with space shortages.

Installing Enterprise Manager

Before starting the installation you should ensure that communication between the repository database and the management host is possible. Review any potential firewall rules and make any necessary changes to the configuration in a change window before the OEM installation. If you decided to use VNC for the installation, start a vncserver process as oracle on the Management Host. Connect to your virtual display from your desktop machine to begin the installation.

In this section an advanced OMS installation will be performed. At the end of the installation you will end up with an OMS and an agent on the same host plus a repository schema in a database. Begin the installation by changing to the location where you unzipped the binaries and execute the runInstaller command as usual. After a little while the splash screen appears followed by the first configuration screen shown in Figure 8-3.

As with the Oracle database, you have the option to be informed via email of issues with the product. Enter whatever you feel appropriate and click on the “Next” button. You’ll be taken to Step 2 in Figure 8-4.

Figure 8-4 shows another screen which looks similar to one used in the RDBMS installation wizard. If you are installing the software to a host that is connected to the Internet then you could search for updates online by selecting the radio button “My Oracle Support”. In this case, enter your username and password in the relevant input fields.

The whole step has been skipped in the example. Skipping the step takes you to Figure 8-5.

Figure 8-5 lets you define the inventory location. In this example, the directory /u01/app/oraInventory has been entered to comply with the Oracle Flexible Architecture. If you configured the oracle account in the same way shown earlier in this chapter then your only option is to select oinstall as the inventory group in the drop-down in the center of the screen. Clicking on the “Next” button moves you to step 4 shown in Figure 8-6.

Figure 8-6 shows the same output from the prerequisite checks that you performed earlier on the command line. Before launching the Oracle Universal installer you should have addressed the problems reported, so you do not expect issues at this stage. If however there are problems reported and a status does not return “Succeeded” please address anything outstanding before proceeding. This is a good time to double-check your firewall and SELinux settings! Once you are confident these settings adhere to the security guidelines and policies while at the same time allowing the installation to proceed, click on the “Next” button to advance to step 5, as shown in Figure 8-7.

Figure 8-7 shows one of the main screens in the Enterprise Manager installer. It allows you to choose an installation method. You can chose from the following options:

• Create a new Enterprise Manager System: This option lets you install a new Oracle Enterprise Manager system. Furthermore you can perform a simple installation or an advanced one. As the names suggest the simple install, which really should have been called simplified installation, does not prompt you for options related to WebLogic and assumes safe default values. The advanced installation gives you a lot more control over the process.
• Upgrade and Existing system: Oracle Enterprise Manager greatly enhanced the upgrade method. Whereas in previous versions many users opted to perform a tech refresh and install the new version independently of the existing OMS, an upgrade is a real possibility with Cloud Control. Upgrading OEM is out of the scope of this chapter. If you are interested and want to know more about the difference between a one-system and a two-system upgrade, please refer to the Oracle® Enterprise Manager Cloud Control Upgrade Guide.
• Install Software only: This option allows you to install the software only and perform the configuration later. It is clearly aimed at more experienced users and not covered here.

If you would like to follow the example in this chapter, then please select the option to install a new system using the advanced configuration. In the next screen which is not shown here you will need to define a location for the Middleware Home, the Agent Base Directory and a Host Name. The following values have been chosen for this example’s installation:

• Middleware Home Location: /u01/app/oracle/oem12c
• Agent base directory: /u01/app/oracle/agent
• Host name: oem12oms1.example.com

Once you’ve entered the relevant information click on the “Next” button, which leads you to step 7 as depicted in Figure 8-8.

The plugin selection screen shown in Figure 8-8 allows you to select optional plugins at installation time. Bear in mind that these can always be installed later! Some plugins have dependencies that are automatically resolved when you select them. When you are happy with your selection proceed to the next step where you configure the WebLogic Server, shown in Figure 8-9.

Oracle Enterprise Manager is deployed as a Java application in WebLogic, and this becomes very apparent in the screen in Figure 8-9. Here you get down to the details of the WebLogic deployment with regard to users and passwords in the underlying application server instance. If you are unsure what to fill in here, leave the defaults and only fill in the passwords. The defaults are based on previous input. Remember to only choose safe passwords before clicking “Next”. Step 9, which allows you to define the connection to the repository database, is shown in Figure 8-10.

Enterprise Manager would not be complete without a repository database. In Figure 8-10 this is what you specify. Enter the name of the database server host, the port, Oracle SID or service name, and a sys password. The installer will need this information to create the repository information. One of the few differences between the installation in 12c Release 2 to earlier releases is the drop-down field “deployment size”. Here you defined the targeted size of your OEM deployment. You read earlier in this chapter how Oracle defines a small, medium, and large deployment.

You might get a pop-up with warnings if the installer finds that the database does not meet the requirements. OUI needs more information to complete the repository schema creation, and you need to fill the missing values in the next screen, shown in Figure 8-11.

You are required to supply the following information:

• SYSMAN password: The SYSMAN account is the Enterprise Manager super-user. Whoever has control over it has complete control over the whole Enterprise Manager configuration. The SYSMAN password therefore is very sensitive and should not be given out to regular users. Just like the UNIX root account, only qualified and properly trained staff should have the SYSMAN password. Ideally the use of the SYSMAN account is restricted to the initial configuration. For the reasons just outlined the password must be sufficiently complex.
• Registration password: In order for agents to initially establish communication with the Oracle Management Service agents need a password. This password is called the registration password which you need to enter here.
• Tablespace locations: The Oracle Universal Installer will create three tablespaces in the repository database. Their location can be defined in the last three input fields. Choose appropriate locations and click on “Next” to specify the port ranges in step 11, shown in Figure 8-12.

Oracle Enterprise Manager uses network ports for various purposes: the agents need to be able to upload information using the HTTP and HTTPS protocols, there must be free ports for the Enterprise Manager console and so forth. You can normally use the defaults listed in Figure 8-12, unless you have very specific requirements. You are taken to the summary screen, shown in Figure 8-13 after you click on “Next”.

The summary screen as shown in Figure 8-13 gives you the option to review the choices made. It is the last opportunity to go back and make any changes. If you are happy with the settings click on “Install” to begin the installation and configuration of Enterprise Manager Cloud Control. You can now change your focus to another task since the process is almost certainly going to take a while, even on fast hardware.

You will have to wait for the installation and configuration steps to finish before you can run the root scripts. Eventually the well-known “execute as root” pop-up screen will appear. As usual with Oracle products, open a root shell on the OMS and execute the root scripts as indicated in the pop-up window. If you do not have root access to the host, you will need to ask or ring the friendly Linux administrator on duty to perform that task for you.

Close the pop-up windows after the root scripts completed, and wait for the final screen summarizing the port information and initial access to appear. The information shown in it is important, but thankfully Oracle preserved the text in the Management Services home directory in the setupinfo.txt file. Nevertheless you should take note of the EM console URL, which you need for logging in.

Congratulations, you have successfully installed and configured the initial Enterprise Manager 12 Cloud Control environment! Before you log in for the first time there is one very important step to be performed. The repository is encrypted with a secret key you should back up immediately:

[oracle@oem12oms1 bin]$ ./emctl exportconfig oms -dir /path/to/safe/location/

Now you are ready to log in for the first time.

The Initial Configuration Steps

The first user account you have available after a fresh installation as just shown is the SYSMAN user. After the OEM welcome screen has loaded, connect as SYSMAN using the password you defined at installation time. You will be asked for a default home screen, a choice that does not require a lot of consideration at this point. The enterprise summary screen seems a good candidate for the default screen, but you can always change your home screen. Figure 8-2 earlier in this chapter showed the summary screen.

Enterprise Manager 12c has a central configuration menu to be used, named “Setup”. It can be found near the top-right corner of the screen and will be used extensively throughout the next sections. The example assumes a connection of the OMS to the Internet—either direct or via proxy—which Oracle refers to as the “online” mode. The alternative is named an offline connection and is not covered in this chapter.

Creating and Managing Enterprise Manager Accounts

Users in Cloud Control are referred to as administrators. The authentication and authorization system provides a wealth of different methods to let users access the Enterprise Manager console. Users can be authenticated outside of Enterprise Manager as well from a variety of other sources, including Microsoft’s Active Directory. Covering all of these authentication methods is not possible in this chapter, which is why the default so-called repository-based authentication will be used as one way to identify users. With repository-based authentication every user provisioned in the Enterprise Manager system will be created as a named user in the repository database. The advantage of this approach is that every user password can be maintained with password profiles as any other database account. On the downside EM administrators need to remember yet another username and password combination.

In Enterprise Manager you can create two types of users: administrators and super administrators. Be careful granting the super-administrator role to anyone because real damage can be done with it! The difference between these two classes of users is their scope: the super administrator can create and manage any other administrator and his objects in Cloud Control whereas the administrator is limited to his own objects and therefore more restricted. As is the rule with the root account on UNIX systems, you should limit the use of the super administrator accounts to global setup operations and configuration. You should definitely not use it on a routine basis. Also bear in mind that for auditing purposes it is essential that you use named users rather than generic accounts: another good reason to keep the SYSMAN password very safe.

The privilege system in Enterprise Manager 12c is very flexible, and is a real revelation after the rather strict model in previous generations of the software. Generally speaking the system differentiates between:

• Grants on specific managed targets
• Grants applicable to all targets
• Grants on the Enterprise Manager system itself, so-called resource privileges

To simplify account management, individual privileges can be rolled up into roles. Just as with the database roles can further be granted to other roles, creating nested hierarchies. There are predefined roles available out-of-the-box, but you can of course create your own if you like. The management of users via roles greatly simplifies the management of accounts since changes have to be made only once for the role, not for every individual account. Oracle supplies almost 20 roles for typical tasks, such as patching, target management, deploying and plugins. In addition to these roles you are free to define a set of privileges to the PUBLIC role. This role does not have any privileges by default. But since it is always granted to new users you can define a basic set of privileges every user should have to it, simplifying new user creation. The EM_USER is another essential role, similar to the database’s CONNECT role and it allows the user to connect to the Enterprise Manager system.

There are two types of target privileges: those applicable to all targets like “view any target” and those that apply to a named managed target only. There are more than 20 target-specific privileges available that should make it very easy to properly define access to targets. Luckily you do not need to refer to the documentation when reviewing what a privilege can be used for. The Enterprise Manager console does a better job describing what a privilege allows you to do as well as which other privileges are implicitly granted as well.

The final group of privileges refers to the various parts or modules of Enterprise Manager itself. Depending on the plugins deployed—and therefore new functionality—available you can grant privileges for patching, compliance monitoring, the job system and many more.

The creation of a new user follows the process shown in Figure 8-14:

The following sections detail the user creation.

Deploying an Agent to a New Host

The agent deployment can be performed either via the OEM console, or alternatively from the command line. The command line installation is more suitable for a mass deployment of the agent, and it will be covered here. When installing the agent you will notice a slightly different directory structure than the previous versions of the agent. During the installation you need to define an agent base directory, which contains all agent related files. There are three directories of higher importance:

• The agent home directory
• The agent instance directory
• The plugins directory

The agent instance directory contains all configuration files, and especially the all-important sysman directory containing the log files for the agent. The instance home is not to be confused with the agent home directory, which is below $AGENT_BASE/core/version. The agent home contains the binaries used to run the agent on the server, but confusingly the emctl executable recommended for use with the agent is in the instance home. The plugins directory contains the plugins currently installed for the agent.

You can make use of the Enterprise Manager command line interface for mass-deploying the agent. Log in as the OMS software owner on an arbitrary management host. You then create a zip file with the agent software for a specific platform. If you wanted to create the deploy image for the Linux x86-64 agent, you should follow these steps:

[oracle@oem12oms1 ∼]$ emcli login -username=sysman
Enter password
 
Login successful

Once logged in you can query the agents currently available on the management host:

[oracle@oem12oms1 ∼]$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /u01/app/oracle/oem12/gc_inst/em/EMGC_OMS1/sysman/emcli/
setup/.emcli/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.2.0
 Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

Armed with that information it is possible to get the agent image and deploy it to the staging directory on the OMS as shown here:

[oracle@oem12oms1 ∼]$ mkdir /u01/temp
[oracle@oem12oms1 ∼]$ emcli get_agentimage -destination=/u01/temp 
> -platform="Linux x86-64"
Platform:Linux x86-64
Destination:/u01/temp
 === Partition Detail ===
Space free : 14 GB
Space required : 1 GB
Check the logs at /u01/app/oracle/oem12/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/
.emcli/get_agentimage_2013-01-01_21-03-28-PM.log
Setting property ORACLE_HOME to:/u01/app/oracle/oem12/oms
calling pulloneoffs with arguments:/u01/app/oracle/oem12/oms/u01/app/oracle/oem12/oms/sysman/
agent/12.1.0.2.0_AgentCore_226.zip12.1.0.2.0linux_x64
Check this logs for more information: /u01/app/oracle/oem12/oms/sysman/prov/
agentpush/logs

The result of this process’s execution is a deployable agent image for Linux x86-64. This process only needs to be completed once, the resulting zip file can be reused on as many hosts as you like. You could move the resulting zip file to a shared area to make it easier to deploy the zip file later. Whichever way you chose to make the image available, the newly created zip file needs to be accessible on the destination host. Most users decide to install the agent using the same user ID as they chose for the RDBMS binaries, as does this example. Once logged in as “oracle” on the destination host, prepare for the installation. If your host already has an Oracle database installed you should be fine with regards to the agent installation prerequisites. If not, you should review Table 6-1 in the Advanced Installation and Configuration Guide. Next you unzip the zip file to a convenient location, it is /tmp/agent12c in this example. Before the agent can be installed you need to provide a response file. An example response file is shown below, stripped of comments and empty lines:

[oracle@oem12db agent12c]$ sed -e '/^#/d' -e '/^$/d' agent.rsp
RESPONSEFILE_VERSION=2.2.1.0.0
OMS_HOST=oem12oms1.example.com
EM_UPLOAD_PORT=4900
AGENT_REGISTRATION_PASSWORD=<secretPassword>
b_startAgent=true
EM_INSTALL_TYPE="AGENT"

The full description of the response file parameters is listed in Table 6-3 in the Oracle® Enterprise Manager Cloud Control Advanced Installation and Configuration Guide. The above example is complete and can be used after changing it to match your environment. If you are unsure about the ports to be used, you can interrogate the OMS directly, using the command shown here:

[oracle@oem12oms1 ∼]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        :oem12oms1.example.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7799
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4900
EM Instance Home           : /u01/app/oracle/oem12/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/oem12/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL:https://oem12oms1.example.com:7799/em
Upload URL:https://oem12oms1.example.com:4900/empbs/upload
 
WLS Domain Information
Domain Name      : GCDomain
Admin Server Host:oem12oms1.example.com
 
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host:oem12oms1.example.com
WebTier is Up
Oracle Management Server is Up
[oracle@oem12oms1 stage]$...

Supply the values shown in bold to the response file for the agent. The agent registration password is more difficult to recover. If you forgot your agent registration password you can create a new one in Setup Security Registration Passwords. Once you have completed the creation of the response file, begin with the installation of the agent. Change the agent base directory to match your directory standards.

[oracle@server1 agent12c]$ ./agentDeploy.sh AGENT_BASE_DIR=/u01/app/oracle/agent12c 
> RESPONSE_FILE=/tmp/agent12c/agent.rsp
 
Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u01/app/oracle/agent12c/core/12.1.0.2.0/jdk/bin/java -classpath 
/u01/app/oracle/agent12c/core/12.1.0.2.0/jlib/agentInstaller.jar:/u01/app/oracle/
agent12c/core/12.1.0.2.0/oui/jlib/OraInstaller.jar 
oracle.sysman.agent.installer.AgentInstaller /u01/app/oracle/agent12c/core/12.1.0.2.0
/tmp/agent12c /u01/app/oracle/agent12c -prereq
 
Validating oms host & port with url:http://192.168.1.223:4900/empbs/genwallet
Validating oms host & port with url:https://192.168.1.223:4900/empbs/genwallet
Return status:3
[...]
Agent Configuration completed successfully
 
The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
 /u01/app/oracle/agent12c/core/12.1.0.2.0/root.sh
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts
Agent Deployment Successful.
Agent deployment log location:
/u01/app/oracle/agent12c/core/12.1.0.2.0/cfgtoollogs/agentDeploy/agentDeploy_2013
-01-01_16-23-36-PM.log
Agent deployment completed successfully.

The agent deployment takes very little time, and at the end all you need to do is to execute the root.sh script from the agent home as shown here:

[root@server1 ∼]# /u01/app/oracle/agent12c/core/12.1.0.2.0/root.sh
Finished product-specific root actions.
/etc exist
 
Creating /etc/oragchomelist file...
Finished product-specific root actions.

Congratulations, you just installed the Oracle agent. If for some reason the prerequisites for the agent installation are not met, the silent installation will fail. In such a case you should check the output of the log file to see which particular problem prevented the installer from completing its task. More information about the agent software prerequisites can be found in Chapter 3 of the “Enterprise Manager Cloud Control Basic Installation Guide”, Table 3-2.

Before moving on to the next task you should quickly check if the agent can indeed communicate with the OMS. The command to use is shown here, together with the success message taken from a different machine:

[oracle@oem12oms1 ∼]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : 12.1.0.2.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/agent12c/agent_inst
Agent Binaries    : /u01/app/oracle/agent12c/core/12.1.0.2.0
Agent Process ID  : 3948
Parent Process ID : 3902
Agent URL         :https://server1.example.com:3872/emd/main/
Repository URL    :https://oem12oms1.example.com:4900/empbs/upload
Started at        : 2013-01-20 20:24:30
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2013-01-20 21:45:46
Last attempted upload                        : 2013-01-20 21:45:46
Total Megabytes of XML files uploaded so far : 0.16
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 66.09%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2013-01-20 21:45:51
Last successful heartbeat to OMS             : 2013-01-20 21:45:51
Next scheduled heartbeat to OMS              : 2013-01-20 21:46:51
 
---------------------------------------------------------------
Agent is Running and Ready

If you like, you can add the agent home to the oratab file just as you would with a database:

agent:/u01/app/oracle/agent12c/core/12.1.0.2.0:N

You can then use the oraenv utility to switch back and forth Oracle homes as you do with the database. You do not need to worry about a start script, the agent installation creates such a script named /etc/init.d/gcstartup to automatically start and stop the agent when a runlevel changes.

The installation process can easily be scripted and packaged in the form of an RPM for example. All you need to do in order to fully automate the agent deployment is to:

• Provide a common, well-known location for the file.
• Configure the sudoers file to allow Oracle to execute the root.sh script as root.
• Make the zipfile available on the destination host for your platform.
• Provide the response file.
• Execute the agentDeploy.sh script as part of the post-install scriptlet in RPM.
• Change the sudoers file again and revoke the permission to execute root.sh.

In very security-conscious environments it might be difficult to get permissions to change the sudoers file. In these environments you need to ensure that a qualified system administrator with root access to the destination host is available to execute the final step in the installation procedure.

Getting Agent Software for Different Platforms

Unlike with Enterprise Manager 11.1 and before you cannot download the agent software from Oracle’s OTN website anymore. This is quite startling at first, but the new process is very elegant and compensates you for the missing agents. You read earlier in this chapter that Enterprise Manager is built around a plugin infrastructure. As such only required functionality is deployed initially, updates and new functionality are available via Setup Extensibility Self Update. To use the Self-Update functionality you need to have configured access to My Oracle Support. Figure 8-19 shows the main interface to the Self-Update functionality.

The Self-Update is periodically refreshed via an Enterprise Manager job which executes daily. However if you are impatient you can always start a manual refresh by clicking on the “Check Updates” button. Information about the agents is shown in the “Agent Software” row. As you can see from the screen print there are 9 updates available, and the initial agent is installed (“Applied Updates” is 1). Clicking on the link “Agent Software” takes you to the detail screen. A value of “Available” in the Status column indicates that an agent is available, clicking on the “Download” button will trigger the agent download to the previously configured software library. A small dialog window prompts you for a time to download the agent. It might be a good idea to download the agent software at a quiet time instead of immediately. A job will be submitted to Enterprise Manager which will be in charge of downloading the agent software.

Once the status for the agent has changed to “Downloaded” you need to click on the “Apply” button to make the agent available for deployment. Another job will take care of the deployment, which should be completed in little to no time. A manual page refresh later the agent status has changed to “Applied”. You can view the success of the operation by querying the supported platforms as you did during agent deployment. Remember that only the Linux x86-64 agent was initially available. Now the second agent has been made available as well:

[oracle@oem12oms1 ∼]$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /home/oracle/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list  ...
-----------------------------------------------
Version = 12.1.0.2.0
 Platform = Linux x86-64
-----------------------------------------------
Version = 12.1.0.2.0
 Platform = Linux x86
-----------------------------------------------
Platforms list displayed successfully.

Repeat these steps for any additional agent you need.

The Manual Target Discovery Process

After the agent has been deployed on the host you can start configuring the discovery process. There are two ways to discover targets using the Enterprise Manager console. The well-known, user-defined way of manually discovering targets in Grid Control is now called “Guided discovery” in Cloud Control. A second, more elegant way of target discovery is available as well, which makes a lot of sense in the consolidated environment. After a new host has been rolled out with an agent installed it makes sense to start a manual discovery of the components that have so far been installed. In most cases these components are the database home, a listener, and the ASM instance if applicable. Once these are registered one could define automatic target discovery to take the dynamic nature of the Database-As-A-Service environment into account.

Let’s begin the discovery process by manually discovering targets. Navigate to Setup Add Target Add Targets Manually. In the resulting screen you need to select the radio button “Add Non-Host Targets Using Guided Process (Also Adds Related Targets)”. Select “Oracle Database, Listener and Automatic Storage Management” from the drop-down menu “Target Types”. Click on the “Add using Guided Discovery” button to start the discovery process. Figure 8-20 shows this screen.

On the next screen, titled “Add Database Instance Target: Specify Host” you need to select the host to which the target belongs using the magnifying glass or type the name in directly. If the host you want to add your target from does not exist in the list-of-values search you should investigate agent communication problems between the OMS and the host. Clicking on “Next” will initiate the target discovery process.

Once that process is completed you are presented with an overview of discovered targets as shown in Figure 8-21. The next step is to configure them by clicking on the wrench icon in the “configure” field.

The target configuration screens are usually self-explanatory. For databases all you have to add is the dbsnmp password. Note that for newer database targets the dbsnmp is often locked and needs unlocking first. An additional screen allows you to define the Target Global Properties such as a contact, lifecycle status, and more. After saving the changes in the summary screen a new target becomes active after a slight delay. There is no need to panic if the target does not appear straight away or if there is a watch depicted as the status: the agent has to gather metric information about the target before it can be fully accessed.

If you look at Figure 8-21 closely you will notice a change from previous versions of Oracle Enterprise Manager. Instead of only adding the database instance, Enterprise Manager 12c Cloud Control will automatically create what it calls a system for a database. A system in OEM terminology is a logical group of targets that need to be up in order to support a given service. By default, the database system created uses the suffix “SYS” to indicate its function. You can view systems via Targets Systems. A typical system created on ASM in an Oracle Restart configuration will have these members:

• ASM instance
• Listener
• The database instance
• The Oracle Restart (Grid Infrastructure) software home
• The RDBMS software home

This is very useful when it comes to patching these components as Enterprise Manager can also track the Oracle homes, including their patch levels. Click on the Finish button to finish the configuration of the newly discovered targets. You are presented a summary of all discovered targets to be added which you should acknowledge by clicking on the “Save” button. Enterprise Manager will then display a status page indicating it is saving the targets, and finally show the successful operation summary.

Automatic Target Discovery Process

In addition to the manual process Oracle Enterprise Manager Cloud control allows you to continuously monitor hosts which are already managed via an agent. The target discovery will be initiated automatically on a daily basis, however this interval can be configured. Navigate to Setup Add Target Configure Auto Discovery to configure options. The discovery module of choice is named “All Discovery Modules”. Click on the wrench icon next to it to configure it. You can select the target type the agent should discover by clicking on a host in the list, as shown in Figure 8-22.

Click on the “OK” button to save any changes made to the settings in Figure 8-22 to take you back to the previous screen. You can optionally wait until the discovery is automatically started on a daily basis, or if you are impatient run the discovery now by selecting the host and clicking on the “Run discovery now” button. A short prompt will appear asking if you are sure before the target discovery starts. At the end of it a pop up informs you of success or failure. If new targets have been discovered you will see a number in the discovered targets column for your host. If you click on the number you are taken to the auto discovery results screen, which is shown in Figure 8-23.

Here you can see all the targets discovered on server1.example.com. No manual target discovery has been performed: the tab “Host Targets” shows 0 managed targets, but there are 8 new targets found. You can now choose to either propagate the target to a managed target or ignore it. To promote a target, highlight it and click on the “Promote” icon. Depending on the target type you will have to add some more configuration information before promoting it. The configuration screens are the same for manual and automatic discovery, and almost all are self-explanatory. After the configuration information has been entered you have the option to promote the target to managed status. Some target types such as Oracle homes can be configured but not promoted, leaving that to a later stage. After a target has been promoted it will disappear from the list of non-host targets.

Support for Pluggable Databases

Unlike previous generations of the Enterprise Manager software OEM 12c has support for Pluggable Databases built in. You read earlier in the chapter that OEM functionality can be added and updated via plugins. The author’s system used for this chapter was OEM 12c Release 2 or 12.1.0.2 with database plugin 12.1.0.3.0. Since then Enterprise Manager 12c Release 3 has been made available roughly at the same time as the database became generally available. The database plugin allows viewing the container database including the PDBs as you can see in Figure 8-24.

The layout of the screen has been slightly modified. Normally you would see the summary in the top-left corner but for the sake of demonstration it has been moved up. Each PDB detected in the system has its own status as you can see. The performance panel allows you to view the load on the database broken down per container. You find the CDB$ROOT as well as each individual PDB. The rest of the screen is the same as for a non-CDB, you get to see the resources as well as a panel with statements for which you could invoke the SQL Monitor reports.

To drill down on a PDB simply click its name or select it from the drop-down list next to the database name. As you would expect a PDB behaves very much like a “regular” database from a management point of view. All the features and restrictions laid out for PDBs in the previous chapter still apply.

Standardized Monitoring

Enterprise Manager 12c has a very well thought-through feature named Administration Group. With Administration Groups it is possible to automatically group similar targets together based on a common attribute. OEM allows you to define attributes for targets, such as department, lifecycle status, location, and many more. Based on these attributes you can categorize targets. For example, you could put all production databases from Europe, Asia, and America into different groups. With the Administration Group created you can assign monitoring templates to every group, ensuring that members of a group are always consistently monitored. The beauty of Administration Groups lies in the fact that if you need to change the monitoring template, the change to the template is automatically propagated to all the targets it applies to, keeping the targets in sync with the monitoring requirements. With Administration Groups in place you need to worry less about newly deployed databases. As soon as they are classified by lifecycle status and location, they will be added to a group and monitored exactly as any other member of that group.

This sounds fairly abstract so let’s use an example. Consider a number of databases in your estate, some of which are production; some are staging or user acceptance test environments plus all the inevitable test environments that you need to support. As soon as your database targets are registered in Enterprise Manager you can assign the appropriate lifecycle status to them. In a next step you define monitoring templates, which really are checks against the target to ensure that everything functions normally. A monitoring template for the listener for example could trawl the listener log file and check for occurrences of TNS-xxxx errors. If such an error is found the template can report that condition back to the Enterprise Manager console, where an incident will be created. Your monitoring templates can report normal execution, or alternatively warnings and critical states back. These are based on metrics you can collect. For example, if a tablespace is 75% full you might want to issue a warning. If the tablespace gets up to 85% you may want to report a critical condition back.

Administration Groups are not limited to a flat hierarchy. You can create arbitrarily complex group mappings based on multiple target properties. If you wanted different monitoring templates for production databases in EMEA from AMERICAS then you could create a hierarchy consisting of the lifecycle status and location. The basic principle however is identical for simple and complex setups, which is why you will only find a hierarchy with a maximum depth of one in this section.

[oracle@oem12oms1 ∼]$ emcli login -username=sysman
Enter password
 
Login successful
[oracle@oem12oms1 ∼]$ emcli  get_target_properties -target_type="oracle_database"
Comment
Contact
Cost Center
Customer Service Identifier
Department
LifeCycle Status
Line of Business
Location
Operating System
Platform
Target Version
Target properties fetched successfully

[oracle@oem12oms1 ∼]$ emcli set_target_property_value 
-property_records=" emrep_sys:oracle_dbsys:LifeCycle Status:Production" 
-propagate_to_members

Incident Management

Incident management builds upon settings and configuration steps explained in previous sections. One almost imperative task is the creation of Administration Groups, and more specifically the definition of monitoring templates. After your important systems are properly configured for standardized monitoring then the next step is to define alerting. Elaborate workflows can be designed in Enterprise Manager, and if you like you can use OEM as your only incident management solution.

Oracle however knows that many organizations already have some kind of Trouble Ticket System (TTS) in place, and provides connectors for the most commonly used ones. Enterprise Manager collects metrics from targets. Lots of metrics are collected, such as CPU usage, database free space, and many more. In addition, OEM records when a target is down or a job did not execute successfully. The management agents also make extensive use of the Automatic Diagnostic Repository (ADR) to detect problems with a listener, ASM, or database instance. This section is about using Enterprise Manager 12c for the complete workflow. The workflow is based on notifications which you need to set up first.