CELLCLI and DCLI
You have already seen many references to dcli and cellcli in the previous chapters. Although the syntax appears quite intuitive, a more thorough discussion of what you can do with the tools is certainly in order. You probably ended up here following a reference in another chapter. This appendix is not meant to be a comprehensive discussion of what you can do with the tools. You have the Exadata Storage Server Software User's Guide Chapter 8 (Using the CellCLI Utility) and Chapter 9 (Using the dcli Utility) for this. The appendix is rather your guide to getting started and understanding two of the most useful configuration tools available for the Exadata system. The cellcli part is a bit longer, paying tribute to the more powerful of the two utilities.
cellcli is a command interpreter through which you can manage a storage cell. Understandably, it is not available on the compute nodes. cellcli is to a cell what SQL*Plus is to a database instance. The other utility to introduce in more detail in this appendix is dcli. It is a utility that sends a single command to all your database servers and/or storage cells in one go. Other functionality includes copying files to multiple locations and copying SSH keys. Everyone who has been a RAC administrator before supporting Exadata will probably join the author in thinking that dcli is something that every RAC system should have available. It may not sound so useful in the eighth or quarter rack case, but as soon as you manage half racks or full racks, you will start to appreciate the ability to execute a command once across all cluster nodes.
An Introduction to CellCLI
As the name implies, Exadata storage software uses the cellcli utility as its command-line interface. We complained about the lack of syntax reference about cellcli in the first edition of the book. At the time, the only help you had was the online help or this appendix. Things have thankfully changed for the better, and the Exadata Storage Administrator’s Guide has a chapter dedicated to the use of cellcli. However, since you may not be reading this book on an electronic device (or may not have one nearby), we decided to include a little reference anyway. When writing this appendix, we also found out that the documentation is lagging behind in some places, so we thought we would include a few of the things we learned while working with it.
It is interesting that Oracle chose to write an entirely new command-line tool for managing the storage cell. Oracle could have used SQL*Plus, which has become the most well-known tool for managing databases and ASM. Be that as it may, cellcli is the tool you will use for managing the storage cells. The syntax is somewhat different from SQL*Plus, but there are similarities, particularly with the LIST command. LIST is used to execute queries, and it looks very similar to the SELECT command that DBAs have become accustomed to. Like SELECT, it has WHERE and LIKE keywords that allow you to filter out unwanted information from the output.
Following is our top-ten list of things you should know about cellcli:
- cellcli does implement a handful of SQL*Plus commands (START (@), SET ECHO ON, SPOOL, DESCRIBE, REM, and HELP).
- SELECT is replaced by LIST, and it must be the first keyword on the command line.
- There is no FROM keyword (the LIST keyword must be immediately followed by the ObjectType, which is equivalent to a table name).
- Column names are specified with the ATTRIBUTES keyword followed by the columns you wish to be displayed.
- There is a DESCRIBE command, which displays the attributes (columns) that make up an ObjectType (table).
- There is a default set of columns for each ObjectType that will be returned if the ATTRIBUTES keyword is not specified.
- There is a WHERE clause that can be applied to any attribute and multiple conditions can be ANDed together; however, there is no support for OR.
- Unlike in the first edition of this book, there is an ORDER BY equivalent in 12.1.2.1.0 and later, and you can limit the output to a maximum of 200 lines.
- The DETAIL keyword can be appended to any LIST command to change the output from column-oriented to row-oriented.
- The LIKE operator works, but instead of the standard SQL wildcard, %, cellcli uses a simple form of regular expressions, so the % we know from SQL*Plus becomes the .*, matching any character zero or more times.
After a little bit of practice, you will feel at ease with cellcli. For the most part, you will use the LIST command to query different (performance) aspects of the cell.
Invoking cellcli
When you execute cellcli on the command line, you actually launch a bash script. The script is a wrapper with sanity checks, sourcing in environment variables and eventually executing Java code. By default, you will be dropped into an interactive session when entering cellcli on the command line while connected to the cell. The privileges you enjoy in your session depends on the account you used to connect to the cell. You have the option to connect as root, celladmin, or cellmonitor, where root has most privileges (careful!) and cellmonitor the least.
When invoking it, you can pass a few command line options. The most relevant ones are “-e” to execute a command in a non-interactive session and “-xml” to generate XML output. If you are running into problems, you can use “-v” to “-vvv” to generate more verbose logging. Other options exist but are of no relevance for this chapter.
Since cellcli creates output on STDOUT, you can use your favorite UNIX tools as well: just pipe the output of cellcli to sed, awk, grep, sort, uniq, nl, or any other of the myriad of UNIX tools helping you to process text. These provide a great way to perform data slicing and dicing for pre 12.1.2.1.0 storage cells. Here is an example on how to simulate the missing order by clause—first on 12.1.2.1.0, to show you how you can do this with a current Exadata software release:
# cellcli -e list metriccurrent where objectType = 'SMARTIO' order by metricValue limit 30
SIO_IO_EL_OF_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_FC_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_RQ_FC_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RD_RQ_FC_SEC SMARTIO 0.0 IO/sec
SIO_IO_OF_RE_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_RQ_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RV_OF SMARTIO 0.000 MB
SIO_IO_RV_OF_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_FC_SEC SMARTIO 0.000 MB/sec
SIO_IO_SI_SV_SEC SMARTIO 0.000 MB/sec
SIO_IO_PA_TH SMARTIO 0.000 MB
SIO_IO_WR_FC_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_WR_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_PA_TH_SEC SMARTIO 0.000 MB/sec
SIO_IO_WR_RQ_FC_SEC SMARTIO 0.0 IO/sec
SIO_IO_WR_RQ_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RD_FC_HD SMARTIO 577 MB
SIO_IO_RD_RQ_FC_HD SMARTIO 582 IO requests
SIO_IO_WR_FC SMARTIO 106,126 MB
SIO_IO_WR_RQ_FC SMARTIO 106,564 IO requests
SIO_IO_OF_RE SMARTIO 128,723 MB
SIO_IO_WR_HD SMARTIO 225,426 MB
SIO_IO_WR_RQ_HD SMARTIO 226,354 IO requests
SIO_IO_SI_SV SMARTIO 235,694 MB
SIO_IO_RD_FC SMARTIO 408,460 MB
SIO_IO_RD_RQ_FC SMARTIO 412,032 IO requests
SIO_IO_RD_HD SMARTIO 573,819 MB
SIO_IO_RD_RQ_HD SMARTIO 574,816 IO requests
SIO_IO_EL_OF SMARTIO 1,481,339 MB
In an older version of the cell software you will notice that the syntax is not supported. On the other hand, the UNIX sort command can be used instead, as you see in the second example.
# cellcli -e cellcli -e list metriccurrent where objectType = 'SMARTIO' order by metricValue
CELL-01504: Invalid command syntax.
# cellcli -e list metriccurrent where objectType = 'SMARTIO' | sort -n -k 3 | head –n30
SIO_IO_EL_OF_SEC SMARTIO 0.000 MB/sec
SIO_IO_OF_RE_SEC SMARTIO 0.000 MB/sec
SIO_IO_PA_TH_SEC SMARTIO 0.000 MB/sec
SIO_IO_PA_TH SMARTIO 0.000 MB
SIO_IO_RD_FC_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_FC_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_RD_RQ_FC_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RD_RQ_FC_SEC SMARTIO 0.0 IO/sec
SIO_IO_RD_RQ_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RV_OF_SEC SMARTIO 0.000 MB/sec
SIO_IO_RV_OF SMARTIO 0.000 MB
SIO_IO_SI_SV_SEC SMARTIO 0.000 MB/sec
SIO_IO_WR_FC_SEC SMARTIO 0.000 MB/sec
SIO_IO_WR_HD_SEC SMARTIO 0.000 MB/sec
SIO_IO_WR_RQ_FC_SEC SMARTIO 0.0 IO/sec
SIO_IO_WR_RQ_HD_SEC SMARTIO 0.0 IO/sec
SIO_IO_RD_FC_HD SMARTIO 577 MB
SIO_IO_RD_RQ_FC_HD SMARTIO 582 IO requests
SIO_IO_WR_FC SMARTIO 106,126 MB
SIO_IO_WR_RQ_FC SMARTIO 106,564 IO requests
SIO_IO_OF_RE SMARTIO 128,723 MB
SIO_IO_WR_HD SMARTIO 225,426 MB
SIO_IO_WR_RQ_HD SMARTIO 226,354 IO requests
SIO_IO_SI_SV SMARTIO 235,694 MB
SIO_IO_RD_FC SMARTIO 408,460 MB
SIO_IO_RD_RQ_FC SMARTIO 412,032 IO requests
SIO_IO_RD_HD SMARTIO 573,819 MB
SIO_IO_RD_RQ_HD SMARTIO 574,816 IO requests
SIO_IO_EL_OF SMARTIO 1,481,339 MB
Use the UNIX tools and be creative! This could be a good moment to review the sed, awk, and perl tutorials.
Getting Familiar with cellcli
As any good command-line utility, cellcli comes with an online help. This built-in help was more accurate than the official documentation, and, as a tip, you should check the official HTML documentation against the output of the help command. To show you what you can do with cellcli in Exadata 12.1.2.1.0, here is the output:
CellCLI> help
HELP [topic]
Available Topics:
ALTER
ALTER ALERTHISTORY
ALTER CELL
ALTER CELLDISK
ALTER FLASHCACHE
ALTER GRIDDISK
ALTER IBPORT
ALTER IORMPLAN
ALTER LUN
ALTER PHYSICALDISK
ALTER QUARANTINE
ALTER THRESHOLD
ASSIGN KEY
CALIBRATE
CREATE
CREATE CELL
CREATE CELLDISK
CREATE FLASHCACHE
CREATE FLASHLOG
CREATE GRIDDISK
CREATE KEY
CREATE QUARANTINE
CREATE THRESHOLD
DESCRIBE
DROP
DROP ALERTHISTORY
DROP CELL
DROP CELLDISK
DROP FLASHCACHE
DROP FLASHLOG
DROP GRIDDISK
DROP QUARANTINE
DROP THRESHOLD
EXPORT CELLDISK
IMPORT CELLDISK
LIST
LIST ACTIVEREQUEST
LIST ALERTDEFINITION
LIST ALERTHISTORY
LIST CELL
LIST CELLDISK
LIST DATABASE
LIST FLASHCACHE
LIST FLASHCACHECONTENT
LIST FLASHLOG
LIST GRIDDISK
LIST IBPORT
LIST IORMPLAN
LIST KEY
LIST LUN
LIST METRICCURRENT
LIST METRICDEFINITION
LIST METRICHISTORY
LIST PHYSICALDISK
LIST QUARANTINE
LIST THRESHOLD
SET
SPOOL
START
Each command can be further described using the HELP command, as in HELP LIST or HELP LIST CELL, for example.
CellCLI> help list
Usage: LIST <object_type> [<name> | <filters>] [<attribute_list>] [DETAIL]
[ORDER BY <order_by_attribute_list>] [LIMIT integer]
Purpose: The LIST command displays attributes for Oracle Exadata Server Software objects.
Objects displayed are identified by name or by filters.
The attributes displayed for each object are determined by the specified attribute
list.
Arguments:
<object_type>: The type of existing object to be displayed.
<name>: The name of the active request to be displayed.
<filters>: an expression which determines which active requests should
be displayed.
<attribute_list>: The attributes that are to be displayed.
ATTRIBUTES {ALL | attr1 [, attr2]... }
<order_by_attribute_list>: The attributes that are to be ordered by.
{attr1 [asc|desc] [, attr2 [asc|desc]]}
Options:
[DETAIL]: Formats the display as an attribute on each line, with
an attribute descriptor preceding each value.
[ORDER BY]: Orders the objects by attributes in ascending or descending order.
The default is ascending.
[LIMIT]: Sets the number of displayed objects.
Enter HELP LIST <object_type> for specific help syntax.
<object_type>: {ACTIVEREQUEST | ALERTHISTORY | ALERTDEFINITION | CELL
| CELLDISK | DATABASE | FLASHCACHE | FLASHLOG | FLASHCACHECONTENT
| GRIDDISK | IBPORT | IORMPLAN | KEY | LUN
| METRICCURRENT | METRICDEFINITION | METRICHISTORY
| PHYSICALDISK | QUARANTINE | THRESHOLD }
CellCLI> help list cell
Usage: LIST CELL [<attribute_list>] [DETAIL]
Purpose: Displays specified attributes for the cell.
Arguments:
<attribute_list>: The attributes that are to be displayed.
ATTRIBUTES {ALL | attr1 [, attr2]... }
Options:
[DETAIL]: Formats the display as an attribute on each line, with
an attribute descriptor preceding each value.
Examples:
LIST CELL attributes status, cellnumber
LIST CELL DETAIL
The output shown in the previous listing is taken from a cell with Exadata software version 12.1.2.1.0. As you can see, the help system allows you to see the syntax for each command. You may also have noticed a couple of SQL*Plus carryovers. SET, SPOOL, and START work pretty much as expected. Note that the @ character is equivalent to the SQL*Plus START command and that the only things you can use SET for are ECHO and DATEFORMAT. Now, here are a few examples of queries using the LIST command:
CellCLI> describe metriccurrent
name
alertState
collectionTime
metricObjectName
metricType
metricValue
objectType
CellCLI> list metriccurrent where objectType = 'FLASHCACHE' and name not like '.*SEC' -
> attributes name, metricType, metricValue
FC_BYKEEP_OVERWR Cumulative 0.000 MB
FC_BYKEEP_USED Instantaneous 0.062 MB
FC_BY_ALLOCATED Instantaneous 337,591 MB
FC_BY_DIRTY Instantaneous 166,816 MB
FC_BY_STALE_DIRTY Instantaneous 373 MB
FC_BY_USED Instantaneous 363,063 MB
[many more skipped]
FC_IO_RQ_W_SKIP_NCMIRROR Cumulative 0 IO requests
CellCLI> list metriccurrent where objectType = 'FLASHCACHE' and name not like '.*SEC' -
> and metricValue not like '0.*' attributes name, metricType, metricValue -
> order by metricValue desc limit 20
FC_IO_RQ_W Cumulative 1,325,800,847 IO requests
FC_IO_RQ_R Cumulative 1,323,518,596 IO requests
FC_IO_RQ_W_OVERWRITE Cumulative 1,288,444,158 IO requests
FC_IO_RQ_W_SKIP Cumulative 545,358,635 IO requests
FC_IO_RQ_W_FIRST Cumulative 35,716,955 IO requests
FC_IO_BY_W Cumulative 11,196,640 MB
FC_IO_BY_W_OVERWRITE Cumulative 10,702,060 MB
FC_IO_BY_R Cumulative 10,518,906 MB
FC_IO_RQ_R_SKIP Cumulative 4,422,859 IO requests
FC_IO_RQ_REPLACEMENT_ATTEMPTED Cumulative 4,051,159 IO requests
FC_IO_RQ_R_SKIP_NCMIRROR Cumulative 4,029,287 IO requests
FC_IO_RQ_R_DW Cumulative 2,614,954 IO requests
FC_IO_RQ_W_SKIP_LG Cumulative 2,558,104 IO requests
FC_IO_BY_W_SKIP Cumulative 2,467,692 MB
FC_IO_RQ_W_POPULATE Cumulative 1,639,734 IO requests
FC_IO_RQ_R_MISS Cumulative 1,581,764 IO requests
FC_IO_BY_W_SKIP_LG Cumulative 989,725 MB
FC_IO_RQ_DISK_WRITE Cumulative 800,551 IO requests
FC_IO_RQ_REPLACEMENT_FAILED Cumulative 455,425 IO requests
FC_IO_BY_R_DW Instantaneous 404,121 MB
CellCLI> list metriccurrent where objectType = 'FLASHCACHE' and name not like '.*SEC' -
> and metricValue not like '0.*' attributes name, metricType, metricValue -
> order by metricType, metricValue desc limit 20
FC_IO_RQ_W Cumulative 1,325,801,132 IO requests
FC_IO_RQ_R Cumulative 1,323,520,259 IO requests
FC_IO_RQ_W_OVERWRITE Cumulative 1,288,444,442 IO requests
FC_IO_RQ_W_SKIP Cumulative 545,359,973 IO requests
FC_IO_RQ_W_FIRST Cumulative 35,716,956 IO requests
FC_IO_BY_W Cumulative 11,196,644 MB
FC_IO_BY_W_OVERWRITE Cumulative 10,702,065 MB
FC_IO_BY_R Cumulative 10,518,931 MB
FC_IO_RQ_R_SKIP Cumulative 4,424,593 IO requests
FC_IO_RQ_REPLACEMENT_ATTEMPTED Cumulative 4,051,159 IO requests
FC_IO_RQ_R_SKIP_NCMIRROR Cumulative 4,030,357 IO requests
FC_IO_RQ_R_DW Cumulative 2,614,954 IO requests
FC_IO_RQ_W_SKIP_LG Cumulative 2,558,201 IO requests
FC_IO_BY_W_SKIP Cumulative 2,467,735 MB
FC_IO_RQ_W_POPULATE Cumulative 1,639,734 IO requests
FC_IO_RQ_R_MISS Cumulative 1,581,764 IO requests
FC_IO_BY_W_SKIP_LG Cumulative 989,754 MB
FC_IO_RQ_DISK_WRITE Cumulative 800,551 IO requests
FC_IO_RQ_REPLACEMENT_FAILED Cumulative 455,425 IO requests
FC_IO_BY_W_FIRST Cumulative 378,110 MB
The DESCRIBE verb works similarly to the way it does in SQL*Plus, but it must be fully spelled out; you cannot use the familiar DESC as an abbreviation. Notice that there are no headings for column-oriented output. Many of the LIST commands were strung across multiple lines by using the continuation operator (-). The LIST commands look a lot like SQL, except for LIST being used instead of SELECT and the regular expressions for matching when using the LIKE keyword. You can see that the ATTRIBUTES and WHERE keywords can be anywhere on the command line after the LIST ObjectType keywords. In other words, these two keywords are not positional; either one can be used first.
The first example in the previous listings can be executed on any Exadata cell version. The LIST command is used to display a certain set of attributes related to the cell’s FLASHCACHE performance metrics. More specifically, those metrics that calculate values “per second” are excluded.
The second example expands on the previous one by specifying an ORDER BY, which is new for 12.1.2.1.0 and later. Very often when an ORDER BY is specified, a LIMIT clause has to be provided as well, or else an error similar to this one is raised:
CellCLI> list metriccurrent where objectType = 'FLASHCACHE' and name not like '.*SEC' -
> and metricValue not like '0.*' attributes name, metricType, metricValue -
> order by metricValue desc
CELL-02026: The LIMIT parameter is mandatory for "LIST METRICCURRENT" command when using the ORDER BY option.
One alternative to circumvent the limit would be to pipe the output into the UNIX sort command as shown earlier. Note that you can order by in ascending (ASC, the default) or descending (DESC) order. You are not limited to order by just one column. The last example shows you can sort by more than one column.
Sending Commands from the Operating System
In addition to running cellcli interactively you just saw in these examples, you read in the introduction that you can specify the -e option to pass in cellcli commands from your operating system prompt (even via dcli as you will see in a minute). For example, the following listing shows how the -e option can be used to query the status of cellsrv directly from the OS command line:
[root@enkcel04 ~]# cellcli -e "list cell detail"
name: enkcel04
bbuStatus: normal
cellVersion: OSS_12.1.2.1.0_LINUX.X64_141206.1
cpuCount: 24
diagHistoryDays: 7
fanCount: 12/12
fanStatus: normal
flashCacheMode: WriteBack
[...]
usbStatus: normal
cellsrvStatus: running
msStatus: running
rsStatus: running
Among other things, the –e option is helpful when you want to invoke cellcli from within an operating system shell script. You need to be careful when passing commands to cellcli using the –e argument—escaping quotes can become very important, as you will see later in the chapter.
Using cellcli XML Output in the Database
Parsing the output of cellcli in tabular form can be a bit of a challenge. Thankfully, another output method—XML—is available to the performance analyst. Combining the –e and –xml flags allows you to read the cellcli output into the database and process it using SQL. Here is an example how to load cellcli information in XML format into the database.
The first step is to create a directory object in the database, using the create directory command in SQL*Plus. The directory used in the following paragraphs is ORADIR. With the directory in place, you can create a table of XMLType. There are alternative ways of storing the XML information but this one works for the author.
SQL> create table metrics_xmltab of xmltype;
Table created.
This particular table will hold the XML representation of cellcli performance data. Before XML information can be loaded, it needs to be extracted from the cell first. In order to keep the example simple, just a single cell’s performance data will be loaded. You could alternatively use dcli to capture the information simultaneously. The XML file with performance metrics related to the SMARTIO category can be read using SSH and placed into the directory ORADIR. This is just an example. You can fire off any query against the cell:
ssh cellmonitor@enkcel04 "cellcli -xml -e list metriccurrent where objectType = 'SMARTIO'"
> > oradir/metrics.xml
The XML file will have the following contents; the important bits to be displayed later are found in the <metric> tags:
[oracle@enkdb03 ~]$ head -n20 oradir/metrics.xml
<?xml version="1.0" encoding="utf-8" ?>
<cli-output>
<context cell="enkcel04" realm="" ossStartTimestamp="1430673687318" iormResetTimestamp="782230633"/>
<metric> <name>SIO_IO_EL_OF</name>
<alertState>normal</alertState>
<collectionTime>1431455744000</collectionTime>
<metricObjectName>SMARTIO</metricObjectName>
<metricType>Cumulative</metricType>
<metricValue>634659.296875</metricValue>
<objectType>SMARTIO</objectType>
</metric>
<metric> <name>SIO_IO_EL_OF_SEC</name>
<alertState>normal</alertState>
Well-formed XML—all you need. In the next step, this can be loaded into the database:
SQL> insert into metrics_xmltab values (xmltype
2 (bfilename('ORADIR','metrics.xml'), nls_charset_id('AL32UTF8')));
1 row created.
SQL> commit;
Commit complete.
SQL> select count(*) from metrics_xmltab;
COUNT(*)
----------
1
All that remains to do now is to use some XMLDB magic to transform the XML information into something tabular for easier digestion by the human eye. The XMLTABLE construct can be used to generate a report. Later on, when inserting more data, you can distinguish records based on the collection time. The XML document reports the time in UNIX epoch time.
SQL> select x.name, TO_DATE('1970-01-01', 'YYYY-MM-DD') + x.collectionTime/86400000
2 as collectedAt, x.metricType, x.metricValue, x.objectType
3 from metrics_xmltab m,
4 xmltable('//metric'
5 passing object_value
6 columns
7 name varchar2(25) path 'name',
8 collectionTime number path 'collectionTime',
9 metricType varchar2(20) path 'metricType',
10 metricValue number path 'metricValue',
11 objectType varchar2(50) path 'objectType') x;
NAME COLLECTEDAT METRICTYPE METRICVALUE OBJECTTYPE
------------------------- ------------------- -------------------- ----------- -------------
SIO_IO_EL_OF 12.05.2015 18:35:44 Cumulative 634659.297 SMARTIO
SIO_IO_EL_OF_SEC 12.05.2015 18:35:44 Rate 0 SMARTIO
SIO_IO_OF_RE 12.05.2015 18:35:44 Cumulative 14336.6719 SMARTIO
SIO_IO_OF_RE_SEC 12.05.2015 18:35:44 Rate 0 SMARTIO
SIO_IO_PA_TH 12.05.2015 18:35:44 Cumulative 0 SMARTIO
SIO_IO_PA_TH_SEC 12.05.2015 18:35:44 Rate 0 SMARTIO
SIO_IO_RD_FC 12.05.2015 18:35:44 Cumulative 60078.5391 SMARTIO
SIO_IO_RD_FC_HD 12.05.2015 18:35:44 Cumulative 3831.76563 SMARTIO
You can extend the example by creating another XML table with the metric names and their definition taken from the metricdefinition found on every cell and then joining both to get more easily readable output. Views help reduce the amount of typing to extract the same information.
Configuring and Managing the Storage Cell
cellcli is also used in a number of ways for configuring everything from disk storage to cell alerts. You can also use cellcli for management tasks such as startup and shutdown. Following are a few examples of how to use cellcli to configure and manage the storage cell.
Cell services can be shut down one at a time or all at once. The following commands are used to shut down cell services:
-- Shutdown cell services one at a time --
CellCLI> alter cell shutdown services cellsrv
CellCLI> alter cell shutdown services ms
CellCLI> alter cell shutdown services rs
-- Shutdown all cell services --
CellCLI> alter cell shutdown services all
Cell services may also be started up one-by-one or all at once. Note that the RS process must be started first or cellcli will throw an error such as the following:
CellCLI> alter cell startup services cellsrv
Starting CELLSRV services...
CELL-01509: Restart Server (RS) not responding.
The following commands are used to start up cell services:
-- Startup cell services one at a time --
CellCLI> alter cell startup services rs
CellCLI> alter cell startup services ms
CellCLI> alter cell startup services cellsrv
-- Startup all cell services --
CellCLI> alter cell startup services all
To show the current status of cellsrv, use the LIST CELL command:
CellCLI> list cell attributes name,cellsrvStatus,cellVersion,flashCacheMode,-
> msStatus,rsStatus,status detail
name: enkcel04
cellVersion: OSS_12.1.2.1.0_LINUX.X64_141206.1
flashCacheMode: WriteBack
status: online
cellsrvStatus: running
msStatus: running
rsStatus: running
Several of the settings you see in the output of list cell detail can be changed using the ALTER CELL command. These settings may be configured one at a time or together by separating them with a comma. For example:
-- Configure notification level for alerts --
CellCLI> ALTER CELL notificationPolicy='critical,warning,clear'
-- Configure the cell for email notifications --
CellCLI> ALTER CELL smtpServer='smtp.example.com', -
smtpFromAddr='[email protected]', -
smtpFrom='exa01', -
smtpToAddr='[email protected]', -
notificationPolicy='critical,warning,clear', -
notificationMethod='mail'
By the way, if you have not already stumbled across this feature, cellcli stores a command history similar to the bash shell. You can scroll up and down through your history and edit commands using the arrow keys. And thanks to the regular expression support in queries, you have a very powerful pattern-matching capability at your disposal. The cellcli syntax will be something new to system administrators and DBAs alike, but once you understand the logic, it really is not difficult to master at all.
An Introduction to dcli
dcli is a tool by which you can execute a single command across all cells or compute nodes. Having worked on various clustered systems over the years, we have come to appreciate the importance of keeping scripts (and some configuration files) identical across all nodes. It is also very handy to have a facility for executing the same command consistently across all nodes of a cluster. Oracle provides the dcli command to do just that—unfortunately, not on ordinary clustered systems. Among other things, the dcli command allows you to do the following:
- Configure SSH equivalency across all storage cells and/or database servers
- Distribute a file to the same location on all servers/cells in the cluster
- Distribute and execute a script on servers/cells in the cluster
- Execute commands and scripts on servers/cells in the cluster
dcli uses SSH equivalency to authenticate your session on the remote servers. If you do not have SSH equivalency established across servers/cells, you can still use it, but it will prompt you for a password for each remote system before executing the command. dcli executes all commands in parallel, aggregates the output from each server into a single list, and displays the output on the local machine.
Unlike cellcli just discussed, dcli is not a bash script. It is still a script but written in python. The script can take the following command line options, taken from a 12.1.2.1.0 compute node:
[oracle@enkdb03 ~]$ dcli -h
Distributed Shell for Oracle Storage
[...]
Usage: dcli [options] [command]
Options:
--version show program's version number and exit
--batchsize=MAXTHDS limit the number of target cells on which to run the
command or file copy in parallel
-c CELLS comma-separated list of cells
-d DESTFILE destination directory or file
-f FILE files to be copied
-g GROUPFILE file containing list of cells
-h, --help show help message and exit
--hidestderr hide stderr for remotely executed commands in ssh
-k push ssh key to cell's authorized_keys file
-l USERID user to login as on remote cells (default: celladmin)
--maxlines=MAXLINES limit output lines from a cell when in parallel
execution over multiple cells (default: 100000)
-n abbreviate non-error output
-r REGEXP abbreviate output lines matching a regular expression
-s SSHOPTIONS string of options passed through to ssh
--scp=SCPOPTIONS string of options passed through to scp if different
from sshoptions
--serial serialize execution over the cells
--showbanner show banner of the remote node in ssh
-t list target cells
--unkey drop keys from target cells' authorized_keys file
-v print extra messages to stdout
--vmstat=VMSTATOPS vmstat command options
-x EXECFILE file to be copied and executed
In most cases, you will find yourself executing commands against either all the cells or all compute nodes. The tool relies on a simple text file with names of machines, delimited by carriage returns, and passed using the –g parameter. Usually the files are called cell_group, dbs_group, and all_group respectively, which is what the onecommand tool leaves after installation. These files may be described as follows:
- dbs_group: This file contains the management hostnames for all database servers in your Exadata configuration. It provides a convenient way to execute dcli commands on the database servers.
- cell_group: This file contains the management hostnames for all storage cells in your Exadata configuration. It provides a convenient way to execute dcli commands limited to the storage cells.
- all_group: This file is a combination of the dbs_group and cell_group files and contains a list of the management hostnames for all database servers and storage cells in your Exadata configuration. Using this file, you can execute dcli commands on all database servers and storage cells with care.
Here is an example for a quarter rack:
[root@enkdb03 ~]# cat cell_group
enkcel04
enkcel05
enkcel06
In addition to the –g parameter, you will most often provide the user ID using the –l argument to dcli to specify which user should connect against the systems specified in the group file. Remember that you can use root, celladmin, and cellmonitor when connecting to a cell. When invoking dcli to interact with other compute nodes, root and oracle are the most obvious candidates for user IDs.
dcli is particularly useful when you want to collect information from all storage cells using the cellcli commands. The following example shows how dcli and cellcli commands can be used together to report the status of all storage cells in a quarter rack cluster:
[oracle@enkdb03 ~] $ dcli -g cell_group -l cellmonitor cellcli -e list cell
enkcel04: enkcel04 online
enkcel05: enkcel05 online
enkcel06: enkcel06 online
Any of the cellcli commands discussed in this appendix may be executed from a central location using dcli. In fact, the only restriction is that the command cannot be interactive, (such as requiring user input during execution). For example, the following listing illustrates collecting current performance metrics from the storage cells:
[oracle@enkdb03 ~]$ dcli -l cellmonitor -g cell_group
> cellcli -e "LIST METRICCURRENT ATTRIBUTES name,metricValue,
> collectionTime where objecttype='FLASHLOG' and name like '.*FIRST'"
enkcel04: FL_DISK_FIRST 525,232,661 IO requests 2015-05-12T14:33:45-05:00
enkcel04: FL_FLASH_FIRST 11,914,368 IO requests 2015-05-12T14:33:45-05:00
enkcel05: FL_DISK_FIRST 555,655,999 IO requests 2015-05-12T14:34:06-05:00
enkcel05: FL_FLASH_FIRST 12,418,630 IO requests 2015-05-12T14:34:06-05:00
enkcel06: FL_DISK_FIRST 555,696,351 IO requests 2015-05-12T14:34:08-05:00
enkcel06: FL_FLASH_FIRST 12,086,961 IO requests 2015-05-12T14:34:08-05:00
Using cellcli from dcli requires creative quoting in the where clause as you can see.
Summary
There are many more uses for dcli and cellcli than what we cover here. System administrators will find it useful for creating new user accounts on the database servers using the useradd and groupadd commands, for example. DBAs will find dcli useful for distributing scripts and other files to other servers in the cluster. And using dcli and cellcli together provides a convenient way of managing, extracting, and reporting key performance metrics from the storage cells. Scheduling these as cron jobs and loading them into the database as shown in the little example allows you to keep a nice repository of performance-related information for later analysis.