CellCLI is a command interpreter through which you can manage a storage cell. It is to a cell what SQL*Plus is to a database instance. dcli
is a utility by which you can send a single command to all your database servers and/or storage cells in one go. We describe both utilities briefly in this appendix.
Exadata storage software uses the CellCLI utility as its command-line interface. Unfortunately, although the documentation set that comes with Exadata does have many examples of CellCLI commands, and even a chapter dedicated to CellCLI, it does not include any reference material on the syntax itself (particularly the LIST
command). So we thought we would include a few of the things we learned while working with it.
It's interesting that Oracle chose to write an entirely new command-line tool for managing the storage cell. They could have used SQL*Plus, which has become the most well-known tool for managing databases and ASM. Be that as it may, CellCLI is the tool you will use for managing the storage cells. The syntax is somewhat different from SQL*Plus, but there are similarities, particularly with the LIST
command. LIST
is used to execute queries, and it looks very similar to the SELECT
command that DBAs have become accustomed to. Like SELECT
, it has WHERE
and LIKE
keywords that allow you to filter out unwanted information from the output.
Following is our top-ten list of things you should know about CellCLI:
START (@),
SET ECHO ON
, SPOOL
, DESCRIBE
, and HELP
).SELECT
is replaced by LIST
, and it must be the first keyword on the command line.FROM
keyword (the LIST
keyword must be immediately followed by the ObjectType
, which is equivalent to a table name).DESCRIBE
command, which displays the attributes (columns) that make up an ObjectType
(table),ATTRIBUTES
keyword followed by the columns you wish to be displayed.ObjectType
that will be returned if the ATTRIBUTES
keyword is not specified.WHERE
clause that can be applied to any attribute and multiple conditions can be AND
ed together; however, there is no support for OR
.ORDER BY
equivalent.DETAIL
keyword can be appended to any LIST command to change the output from column oriented to row oriented.LIKE
operator works, but instead of the standard SQL wildcard, %
, CellCLI uses regex, so %
becomes the .*
A good way to begin to get familiar with CellCLI is to explore its help interface. Following is an example of invoking online help:
CellCLI> help
HELP [topic]
Available Topics:
ALTER
ALTER ALERTHISTORY
ALTER CELL
ALTER CELLDISK
ALTER GRIDDISK
ALTER IORMPLAN
ALTER LUN
ALTER THRESHOLD
ASSIGN KEY
CALIBRATE
CREATE
CREATE CELL
CREATE CELLDISK
CREATE FLASHCACHE
CREATE GRIDDISK
CREATE KEY
CREATE THRESHOLD
DESCRIBE
DROP
DROP ALERTHISTORY
DROP CELL
DROP CELLDISK
DROP FLASHCACHE
DROP GRIDDISK
DROP THRESHOLD
EXPORT CELLDISK
IMPORT CELLDISK
LIST
LIST ACTIVEREQUEST
LIST ALERTDEFINITION
LIST ALERTHISTORY
LIST CELL
LIST CELLDISK
LIST FLASHCACHE
LIST FLASHCACHECONTENT
LIST GRIDDISK
LIST IORMPLAN
LIST KEY
LIST LUN
LIST METRICCURRENT
LIST METRICDEFINITION
LIST METRICHISTORY
LIST PHYSICALDISK
LIST THRESHOLD
SET
SPOOL
START
CellCLI> help list
Enter HELP LIST <object_type> for specific help syntax.
<object_type>: {ACTIVEREQUEST | ALERTHISTORY | ALERTDEFINITION | CELL
| CELLDISK | FLASHCACHE | FLASHCACHECONTENT | GRIDDISK
| IORMPLAN | KEY | LUN
| METRICCURRENT | METRICDEFINITION | METRICHISTORY
| PHYSICALDISK | THRESHOLD }
CellCLI> help list FLASHCACHECONTENT
Usage: LIST FLASHCACHECONTENT [<filters>] [<attribute_list>] [DETAIL]
Purpose: Displays specified attributes for flash cache entries.
Arguments:
<filters>: An expression which determines the entries to be displayed.
<attribute_list>: The attributes that are to be displayed.
ATTRIBUTES {ALL | attr1 [, attr2]… }
Options:
[DETAIL]: Formats the display as an attribute on each line, with
an attribute descriptor preceding each value.
Examples:
LIST FLASHCACHECONTENT DETAIL
As you can see, the help system allows you to see a bit of the syntax for each command. You may also have noticed a couple of SQL*Plus carry-overs. SET
, SPOOL
, and START
work pretty much as expected. Note that the @
character is equivalent to the SQL*Plus START
command and that the only things you can use SET
for are ECHO
and DATEFORMAT
. Now, here are a few examples of queries using the LIST
command:
CellCLI> describe flashcachecontent
cachedKeepSize
cachedSize
dbID
dbUniqueName
hitCount
hoursToExpiration
missCount
objectNumber
tableSpaceNumber
CellCLI> set echo on
CellCLI> @fc_content
> CellCLI> list flashcachecontent –
where dbUniqueName like 'EXDB' –
and hitcount > 100 –
attributes dbUniqueName, objectNumber, cachedKeepSize, cachedSize, -
hitCount, missCount
EXDB 2 0 4194304 600 208
EXDB 40 0 2424832 376 60
EXDB 224 0 1802240 115 80
EXDB 267 0 458752 128 9
EXDB 383 0 2547712 157 27
EXDB 423 0 1867776 180 41
EXDB 471 0 4071424 552 85
EXDB 472 0 1277952 114 22
EXDB 474 0 13246464 286 326
EXDB 475 0 5914624 519 124
EXDB 503 0 5308416 669 455
EXDB 5710 0 3735552 363 90
EXDB 6207 0 393216 112 9
EXDB 6213 0 3842048 359 147
EXDB 6216 0 1245184 184 29
EXDB 6373 0 3481600 222 61
EXDB 56085 0 4194304 822 129
EXDB 66849 0 438763520 1221 3322
EXDB 71493 0 5636096 302 127
EXDB 71497 0 1351680 320 22
EXDB 71573 0 2760704 101 37
EXDB 71775 0 1801412608 34994 46315
CellCLI> list flashcachecontent where dbUniqueName like 'EX.?.?' –
and hitcount > 100 -
attributes dbUniqueName, objectNumber, cachedKeepSize, cachedSize
EXDB 2 0 4194304
EXDB 18 0 1179648
EXDB 37 0 622592
EXDB 40 0 2424832
EXDB 63 0 524288
EXDB 104 0 688128
EXDB 224 0 3407872
EXDB 267 0 458752
EXDB 383 0 2670592
EXDB 420 0 1507328
EXDB 423 0 1867776
EXDB 424 0 720896
EXDB 471 0 4071424
EXDB 472 0 1277952
EXDB 473 0 2351104
EXDB 474 0 13574144
EXDB 475 0 5521408
EXDB 503 0 5308416
EXDB 5702 0 262144
EXDB 5709 0 2416640
EXDB 5710 0 3735552
EXDB 6207 0 393216
EXDB 6210 0 131072
EXDB 6213 0 4227072
EXDB 6216 0 1245184
EXDB 6373 0 3579904
EXDB 56085 0 4194304
EXDB 66849 0 438763520
EXDB 71493 0 5636096
EXDB 71497 0 1351680
EXDB 71573 0 2801664
EXDB 71775 0 1801412608
CellCLI> list flashcachecontent –
where dbUniqueName like 'EX.?.?' and hitcount > 100 -
and objectNumber like '.*775'
2356637742 6 71775
CellCLI> list flashcachecontent –
where dbUniqueName like '.*X.?.?' –
and objectNumber like '.*775' detail
cachedKeepSize: 0
cachedSize: 1801412608
dbID: 2356637742
dbUniqueName: EXDB
hitCount: 34994
missCount: 46315
objectNumber: 71775
tableSpaceNumber: 6
CellCLI> list flashcachecontent –
where dbUniqueName like 'EX.?.?' –
and hitcount > 100 -
and objectNumber like '.*775'
2356637742 6 71775
CellCLI> list flashcachecontent –
attributes objectNumber, hitCount, missCount –
where dbUniqueName like 'EX.?.?' –
and hitcount > 100 –
and objectNumber like '.*775'
71775 34994 46315
The DESCRIBE
verb works similarly to the way it does in SQL*Plus, but it must be fully spelled out; you can't use the familiar DESC
as an abbreciation. Notice that there are no headings for column-oriented output. As you can see, you can execute scripts that contain CellCLI commands using the @
character, and use SET ECHO ON
to display the commands in the scripts that you execute. Many of the LIST
commands were strung across multiple lines by using the continuation operator (-
). The LIST
commands look a lot like SQL, except for LIST
being used instead of SELECT
and the regex expressions for matching when using the LIKE
keyword. Also notice that in the last command a number was matched with a regex expression, implying a data type conversion, although all data may be treated as text. You can see that the ATTRIBUTES
and WHERE
keywords can be anywhere on the command line after the LIST
ObjectType
keywords. In other words, these two keywords are not positional; either one can be used first. Finally, the DETAIL
keyword turns the output sideways. Or as the help says, “Formats the display as an attribute on each line, with an attribute descriptor preceding each value.”
In addition to running CellCLI interactively as you've seen in these examples, you can specify the -e
option to pass in CellCLI commands from your operating system prompt. For example the following listing shows how the -e
option can be used to query the status of cellsrv
directly from the OS command line:
[exacel05:root] /root
> cellcli -e "list cell detail"
name: exacel05
bmcType: IPMI
cellVersion: OSS_11.2.0.3.0_LINUX.X64_110520
…
cellsrvStatus: running
msStatus: running
rsStatus: running
Among other things, the –e
option is helpful when you want to invoke CellCLI from within an operating system shell script.
CellCLI is also used in a number of ways for configuring everything from disk storage to cell alerts. You can also use CellCLI for management tasks such as startup and shutdown. Following are a few examples of how to use CellCLI to configure and manage the storage cell.
Cell Services can be shut down one at a time or all at once. The following commands are used to shut down cell services:
-- Shutdown cell services one at a time --
CellCLI> alter cell shutdown services cellsrv
CellCLI> alter cell shutdown services ms
CellCLI> alter cell shutdown services rs
-- Shutdown all cell services --
CellCLI> alter cell shutdown services all
Cell services may also be started up one by one, or all at once. Note that the RS
process must be started first or CellCLI will throw an error such as the following:
CellCLI> alter cell startup services cellsrv
Starting CELLSRV services…
CELL-01509: Restart Server (RS) not responding.
The following commands are used to start up cell services.
-- Startup cell services one at a time --
CellCLI> alter cell startup services rs
CellCLI> alter cell startup services ms
CellCLI> alter cell startup services cellsrv
-- Startup all cell services --
CellCLI> alter cell startup services all
The following commands shut down and restart the cell services.
-- Bounce cell services one at a time --
alter cell restart services cellsrv
alter cell restart services rs
alter cell restart services ms
-- Bounce all cell services --
alter cell restart services all
To show the current status of cellsrv
, use the LIST CELL DETAIL
command as follows:
CellCLI> list cell detail
name: exacel05
bmcType: IPMI
cellVersion: OSS_11.2.0.3.0_LINUX.X64_101206.2
cpuCount: 24
fanCount: 12/12
fanStatus: normal
id: 1105FMM0J5
interconnectCount: 3
interconnect1: bondib0
iormBoost: 0.0
ipaddress1: 192.168.12.9/24
kernelVersion: 2.6.18-194.3.1.0.3.el5
locatorLEDStatus: off
makeModel: SUN MICROSYSTEMS SUN FIRE X4270 M2 SERVER SATA
metricHistoryDays: 7
notificationMethod: mail
notificationPolicy: critical,warning,clear
offloadEfficiency: 139,935.6
powerCount: 2/2
powerStatus: normal
smtpFrom: "Exadata"
smtpFromAddr: [email protected]
smtpPort: 25
smtpPwd: ******
smtpServer: smtp.ourcompany.com
smtpToAddr: [email protected],[email protected]
smtpUser:
smtpUseSSL: FALSE
status: online
temperatureReading: 27.0
temperatureStatus: normal
upTime: 37 days, 18:22
cellsrvStatus: running
msStatus: running
rsStatus: running
Several of the settings you see in this listing can be set using the ALTER CELL
command. These settings may be configured one at a time or together by separating them with a comma. For example:
-- Configure notification level for alerts --
CellCLI> ALTER CELL notificationPolicy='critical,warning,clear'
-- Configure the cell for email notifications --
CellCLI> ALTER CELL smtpServer='smtp.enkitec.com', -
smtpFromAddr='[email protected]', -
smtpFrom='Exadata', -
smtpToAddr='[email protected],[email protected]', -
notificationPolicy='critical,warning,clear', -
notificationMethod='mail'
By the way, if you haven't already stumbled across this feature, CellCLI stores a command history similar to the Bash shell. You can scroll up and down through your history and edit commands using the arrow keys. And regex also provides a very powerful pattern-matching capability. The CellCLI syntax will be something new to system administrators and DBAs alike, but once you understand the syntax, it really isn't difficult to master.
dcli
is a tool by which you can execute a single command across all cells. Having worked on various clustered systems over the years, we've come to appreciate the importance of keeping scripts (and some configuration files) identical across all nodes. It's also very handy to have a facility for executing the same command consistently across all nodes of a cluster. Oracle provides the dcli
command to do just that. Among other things, the dcli
command allows you to:
dcli
uses SSH equivalency to authenticate your session on the remote servers. If you do not have SSH equivalency established across servers/cells, you can still use it, but it will prompt you for a password for each remote system before executing the command. dcli
executes all commands in parallel, aggregates the output from each server into a single list, and displays the output on the local machine. For example, the following listing shows the pmon
processes running on all database servers:
[enkdb02:root] /root
> /usr/local/bin/dcli -l root -g dbs_group ps -ef | grep ora_pmon | grep -v grep
enkdb01: oracle 4973 1 0 Jun09 ? 00:00:16 ora_pmon_DEMO1
enkdb01: oracle 9917 1 0 Jun06 ? 00:00:15 ora_pmon_BL16K1
enkdb01: oracle 9929 1 0 Jun06 ? 00:00:20 ora_pmon_TEST1
enkdb01: oracle 10325 1 0 Jun06 ? 00:00:15 ora_pmon_SANDBOX1
enkdb01: oracle 32630 1 0 Jun06 ? 00:00:17 ora_pmon_DBFS1
enkdb02: oracle 715 1 0 Jun09 ? 00:00:16 ora_pmon_DEMO2
enkdb02: oracle 3718 1 0 Jun06 ? 00:00:15 ora_pmon_DBFS2
enkdb02: oracle 9531 1 0 Jun06 ? 00:00:20 ora_pmon_EXDB2
enkdb02: oracle 10072 1 0 Jun06 ? 00:00:10 ora_pmon_SNIFF
enkdb02: oracle 10085 1 0 Jun06 ? 00:00:20 ora_pmon_TEST2
enkdb02: oracle 10087 1 0 Jun06 ? 00:00:13 ora_pmon_BL16K2
enkdb02: oracle 10136 1 0 Jun06 ? 00:00:14 ora_pmon_SANDBOX2
enkdb02: oracle 23494 1 0 Jun06 ? 00:00:10 ora_pmon_SCRATCH
dcli
is particularly usefull when you want to collect information from all storage cells using the CellCLI commands. The following example shows how dcli
and CellCLI commands can be used together to report the status of all storage cells in a half rack cluster.
[enkdb01:root] /root
> dcli -g /root/cell_group -l root cellcli -e "list cell"
enkcel01: enkcel01 online
enkcel02: enkcel02 online
enkcel03: enkcel03 online
enkcel04: enkcel04 online
enkcel05: enkcel05 online
enkcel06: enkcel06 online
enkcel07: enkcel07 online
You may have recognized the /root/cell_group
parameter in this example. This file is generated during the Exadata installation procedure (see Chapter 8 for more details). There are actually several of these “group files” that are useful for running distributed commands using dcli
. These files may be described as follows:
dbs_group
: This file contains the management hostnames for all database servers in your Exadata configuration. It provides a conventient way to executedcli
commands on the database servers.
cell_group
: This file contains the management hostnames for all storage cells in your Exadata configuration. It provides a convenient way to executedcli
commands limited to the storage cells.
all_group
: This file is a combination of thedbs_group
andcell_group
files and contains a list of the management hostnames for all database servers and storage cells in your Exadata configuration. Using this file, you can executedcli
commands on all database servers and storage cells.
Any of the CellCLI commands we've discussed in this appendix may be executed from a central location using dcli
. In fact the only restriction is that the command cannot be interactive (requiring user input during execution). For example, the following listing illustrates collecting all the current performance metrics from the storage cells.
dcli -l root -g /root/cell_group cellcli -e "LIST METRICCURRENT ATTRIBUTES name, objecttype,
metricObjectName, metricValue, collectionTime"
The output from this command is much too verbose to show here. But with a simple script it can be formatted and distributed as needed. In fact, we used this script in Chapter 7 to collect IORM performance metrics and report on them through an external table definition in the database. Scheduling a report such as this to run daily would be a very convenient way to monitor I/O at the storage-cell level.
There are many more uses for dcli
and CellCLI than we've covered here. System administrators will also find it useful for creating new user accounts on the database servers using the useradd
and groupadd
commands, for example. DBAs will find dcli
useful for distributing scripts and other files to other servers in the cluster. And using dcli
and CellCLI together provides a convenient way of managing, extracting, and reporting key performance metrics from the storage cells.