Viewing performance information
This chapter discusses the following topics:
2.1 Prerequisites for monitoring FlashSystem devices
Although setting up a Network Time Protocol (NTP) server and configuring all devices in the data center to have the time synchronized is relevant for all device monitoring, it is especially important and necessary when monitoring FlashSystem 840 or FlashSystem 900 devices with IBM Spectrum Control.
For most of the devices, IBM Spectrum Control uses the time stamp reported by the devices when saving the performance data in its database. For FlashSystem 840 and FlashSystem 900, performance data statistics are collected through SNMP. Because SNMP does not provide the date and time information of the device, the IBM Spectrum Control server’s time stamp is used.
To use the end to end view provided by IBM Spectrum Control to troubleshoot performance-related issues across your data center, it is essential to have all your devices including the IBM Spectrum Control server configured with an NTP server.
 
Reference: For more information about setting up an NTP server for your devices, see the Usage of an NTP Server in Storage Area Network environments white paper.
2.1.1 Adding FlashSystem devices to IBM Spectrum Control
Any device that you want to monitor with Spectrum Control must be added to the Spectrum Control database.
To add a FlashSystem device, complete these steps:
1. Click Add Storage System from the Block Storage Overview in the Spectrum Control main window, and select FlashSystem Family, as shown in Figure 2-1.
Figure 2-1 Add Storage System window
2. After selecting FlashSystem Family, select the appropriate radio button for the corresponding FlashSystem device that you want to add, as shown in Figure 2-2.
Figure 2-2 Select Flash System model window
Instead of using user name and password for authentication, an SSH key can be used. For more details, see Appendix A, “Using SSH key for device configuration” on page 103.
3. After devices are added to IBM Spectrum Control, the probe (daily collection of status, health, configuration, and capacity information (1)) and the performance monitoring (2) can be scheduled, as shown in Figure 2-3.
Figure 2-3 Schedule probe and Performance monitor
4. If you select the automatic probe schedule as shown in Figure 2-3 on page 7, IBM Spectrum Control automatically balances all device probes that are set to automatic probe schedule across the time frame configured in Settings/Automated Probe Schedule. See Figure 2-4.
Figure 2-4 Automated Probe Schedule
5. When you select manual probe schedule, you can set the time of the day for when the probe runs on that device. However, it is desirable to have the probes of the individual devices balanced as much as possible, as shown in Figure 2-5.
Figure 2-5 Manual probe schedule
6. The default monitoring interval is set to every minute. The default can be changed to a longer interval, as shown in Figure 2-6.
Figure 2-6 Performance monitor interval
One-minute performance data is kept for seven days, but they are also aggregated to five-minute samples, and hourly and daily data. The retention time settings for samples, and hourly and daily data can be configured in the History Retention time settings window in IBM Spectrum Control, as shown in Figure 2-7.
Figure 2-7 History Retention time settings
Because all performance monitoring data, like any other data that IBM Spectrum Control collects are kept in a DB2® database called TPCDB, a compromise between data accuracy, retention time settings, database size, CPU, and memory of the Spectrum Control server is necessary.
When collecting performance data, IBM Spectrum Control collects the device’s performance counters after each interval. After two intervals, the delta is calculated and divided by the interval. Therefore, the shorter the interval, the more likely peaks will be visible.
2.1.2 Considerations for Performance Monitoring with IBM Spectrum Control
After performance data is collected, it can be viewed in the IBM Spectrum Control GUI, at different levels, for the complete Storage System and for almost all of its internal resources.
Accessing performance data in IBM Spectrum Control is consistent across all levels. The examples in this chapter are for A9000 volumes, as shown in Figure 2-8.
You can access the performance charts using these methods:
•Selecting the Performance tab (1) opens the performance chart to display the top five volumes with the most total I/O. Depending on the number of volumes in Spectrum Control, this operation can take a few seconds.
•Using sorting and filtering, you can identify and highlight the volumes of interest. Click View Performance (2) to open a new window with the performance chart.
Figure 2-8 View performance.
in Figure 2-9, the performance chart shows the performance graphs for the individual volumes (1) and the selected metrics (2). The time frame can be either relative or absolute. The performance graphs can be opened in a new window (5). The URL of the chart being displayed can be bookmarked for future use. The metrics of the table underneath the chart can be selected by clicking Edit Table Metrics (6) and can be sorted (7). The selected and highlighted volumes are then shown in the chart.
Figure 2-9 Performance chart tangerine
The metrics to be displayed can be chosen as shown in Figure 2-10
Figure 2-10 Select Performance metrics
Up to 10 different volumes can be selected. Up to six different metrics with two different units can be displayed at the same time because only two y-axis can be displayed.
Although displaying each individual volume and displaying each individual metric can be toggled on and off, it might be confusing having that many graphs in one chart.
Figure 2-11 shows the performance for only one volume, although three were initially selected, because two of them were deselected (1). The same behavior applies to the performance metrics where only two of the six metrics were selected to be displayed (2).
Figure 2-11 Volume performance chart in IBM Spectrum Control
To show more metrics with different units, the performance data can also be displayed as a table by using the toggle chart/table function (1) and then adding additional metrics as columns to the table (2), as shown in Figure 2-12.
Figure 2-12 Table view of performance chart
Depending on the time frame that is selected (and the history retention time settings), the granularity of the performance chart changes automatically:
•One minute sample data, if collected, is shown for one hour, six hours, 12 hours, or one day range. Otherwise, the five minute sample data is used.
•Five minutes sample data is shown for one week.
•Hourly data is shown for one month.
•Daily data is shown for one year.
When you decrease the granularity, the data accuracy also decreases. For the time range of one day, the one-minute sample data is available, as well as aggregated hourly data. The granularity can be manually adjusted. Although the averages of Total I/O and Response Time are almost the same, the maximum values differ a lot, which can be seen in the table underneath the graphs but can already be assumed when looking at the graphs.
Comparison of sample versus hourly data can be seen in Figure 2-13 and in Figure 2-14.
Figure 2-13 Sample data
Figure 2-14 shows the hourly data for the same period.
Figure 2-14 Hourly data
For troubleshooting purposes, the displayed time frame should be small enough so that the smallest interval data can be shown.
Figure 2-15 illustrates the drill up and drill down capability available in IBM Spectrum Control.
Because in this example the V9000 write response time shows a peak at a certain time, this functionality can be used to show for the same time frame the V9000’s pools or V9000’s volume write response time to determine the root cause of those peaks.
Figure 2-15 Drill up and drill down functionality
The new performance charts open in separate windows and can be arranged for comparison purposes.
The date and time settings of multiple windows can be synchronized by using the Synchronize Time feature, as shown in Figure 2-16.
Figure 2-16 Synchronize time
2.1.3 Special considerations for FlashSystem devices
At the time of writing, IBM Spectrum Control version (v 5.2.15) is limited to showing Fibre Channel ports only. If the FlashSystem device is equipped with InfiniBand, iSCSI, or FCoE ports, these ports will not be displayed in IBM Spectrum Control. In this case, no port performance data will be collected.
Performance metrics that are only available in the native FlashSystem GUI can be accessed easily in IBM Spectrum Control by clicking Actions → Open Storage System GUI, as shown in Figure 2-17 on page 16.
Figure 2-17 Open Storage System GUI functionality in IBM Spectrum Control
Some FlashSystem devices like the V9000 also offer a direct link to the system’s performance graphs by clicking Action → View Real-Time Performance, as shown in Figure 2-18. This function can also be used to view real-time performance values if the one-minute interval data that can be shown in IBM Spectrum Control is not detailed enough.
Figure 2-18 View real-time performance
2.2 Monitoring performance for FlashSystem 840/ FlashSystem 900
This section discusses performance monitoring for FlashSystem 840 and FlashSystem 900, first using the system Storage Management native tools and then using Spectrum Control.
2.2.1 Monitoring FlashSystem 840 an FlashSystem 900 using native tools
You can monitor the performance by using the system native tools available from the FlashSystem built-in Storage Management GUI. This GUI is web-based and can be used with any browser pointed at the management IP address of the FlashSystem 840 or 900 to be monitored. Note that Java Script and cookies must be enabled in your browser.
Log in to the GUI
To log in and open the Monitoring Performance window, complete these steps:
1. Enter your login credentials in the login window, as shown in Figure 2-19. Because the performance monitor is just a monitoring tool, any user role can be used here, and specifically, the monitor role.
Figure 2-19 Log-in window for a Flash 840
2. After a successful login, you can open the Monitoring function by clicking the Monitoring icon, and then selecting Performance, as shown in Figure 2-20.
Figure 2-20 Opening the Monitoring Performance window
Performance Monitor
Figure 2-21 shows the Performance Monitor window. Labels were added to indicate specific areas of interest.
Figure 2-21 Performance Monitor Entry window
By default, the display shows the average latency for read and write operations at a system level. The x-axis shows the time frame (default is 10 minutes) and the y-axis shows selected value, such as latency in this example.
The time frame can be changed by clicking the Time buttons:
•now
•1 hour
•1 day
•1 week
•all
The “all” selection includes all the collected data, for up to 300 days.
At the bottom of the diagram, a slider can be used to display the targeted time frame for further analysis of this period. This slider has two parts:
•One part to set the starting point
•One part to set the endpoint to focus on a shorter time frame or one in the past.
The selected start and end time or date are also shown under the x-axis.
Graphs menu
The Graphs menu is accessed from the upper left of the Performance Monitoring window. The following different types of graph can be selected:
•System I/O
•System Latency
•System Bandwidth
•Interface Port Total IOPS
•Interface Port Total Queue Depth
Figure 2-22 shows the Graphs menu with five predefined graphs. Other graphs can be created as described in “Customize chart tab” on page 20. The different graphs can be chosen based on specific needs. This figure shows the menu for Fibre Channel ports. If there are other ports installed in the storage system, such as iSCSI or InfiniBand, then they are shown as well.
Figure 2-22 Graphs menu
The content and meaning of the selectable graphs is as follows:
•System I/O
The System I/O graph displays the average number of read, write, and total I/O operations per second (IOPS) over the sample period. Each request type (read, write, and total) is represented by a different color line.
•System Latency
The System Latency graph displays the average amount of time in milliseconds (ms) that each read and write I/O request takes over the specified sampling period. Each request type (read and write) is represented by a different color line. The user can see the average or the maximum latency, but not both at the same time.
•System Bandwidth
The System Bandwidth graph displays the average number of megabytes per second (MBps) of read, write, total, and rebuild bandwidth over the sample period. Each bandwidth type (read, write, total, and rebuild) is represented by a different color line. There is one line graph for each system that is selected.
•Average Port IOPS
The Average Port IOPS graph displays the average number of read, write, and total IOPS for all ports on all adapters over the sample period.
•Average Port Queue Depth
The Average Port Queue Depth graph displays the average number of operations of that type for all ports on all adapters over the sample period.
Customize chart tab
The default graphs can be edited and customized. Click below the graphs menu icon on the left side to select the Resources. Figure 2-23 shows some options to customize the graphs at a system level.
Figure 2-23 Customize graphs
The resources to be customized can be selected in the upper right corner of this menu. The following available resources are available for customization:
•System
•Flash Modules
•Interface ports
Based on the selected level, different set of resources can be selected. At the system level, configured systems can be selected. In our example, there is only one system. At the Flash module level, up to six modules can be selected. At the interface port level, up to six ports can be selected. If more than six are chosen, they are summarized as a single resource with a single line in the chart.
If the selected resource is an interface, the representation is particular: For each port, a corresponding line is shown, but the color is based on the adapter. So if three ports are selected, one from adapter 1 and two from adapter 2, three lines are shown in the graph (one for each adapter and port), but only two colors are shown (one color per adapter). This system can be confusing at first.
Figure 2-24 shows this situation in detail. In the upper right corner is the caption that lists only Host Adapter 1 Canister 1 and Host Adapter 2 Canister, each with its own color. Because three ports are selected in the customized chart, they are all shown in the chart. As visible in the detail box, there are three lines shown with the color based on the HBA.
Figure 2-24 Chart for Fibre Channel ports
By clicking the graph at a particular point in time, all values for the existing lines are shown in a single box. By using this function, exact values can be made visible.
The metrics shown in the chart depend on the selected resource. Not all metrics can be shown at the same time. Only compatible metrics can be shown within the same chart. See also Figure 2-42 on page 34. There is no option for a second y-axis.
For a complete table of the available metrics, see Table 2-1 on page 32 for the native or built-in area. For a detailed description of every metric, select Help Contents, as shown in Figure 2-25.
Figure 2-25 Performance Help
Equipped with all this information, users can now build their own customized graphs to monitor different metrics. If you want to compare different metrics that cannot be shown in a single chart, you can easily add a second chart to the view. For instance, if you want to compare IOPS with latencies, bring them into the same view by clicking the symbol shown in Figure 2-26.
Figure 2-26 Adding a second chart
In the customized chart, the resources and metrics can be selected. An illustration is shown in Figure 2-27.
Figure 2-27 View with two charts
Both graphs have the same scale in the x-axis because they are by default linked. To view different time frames or have a different scale on the x-axis, the charts can be unlinked by clicking the small symbol in the middle on the right side, as shown in Figure 2-28. You can click the same symbol link the charts again.
Figure 2-28 Unlink charts
You can create a bookmark to customized charts that are important or used often. This bookmark adds those charts to the Favorites in the Charts menu. A chart that is often used or shared by a team, it can be added to the predefined charts for easy access and sharing. To add a chart to the Charts menu, create your chart and then click the small yellow star on the right side of the chart. You also must give it a name. When a user adds a chart as a favorite, as shown in Figure 2-29, it becomes available to every other user.
Figure 2-29 Adding and naming the chart
The new favorite chart becomes accessible by using the Charts menu. Here the favorite chart can easily be selected, renamed, and deleted by selecting the appropriate action from the pull-down menu, as shown in Figure 2-30.
Figure 2-30 Chart menu with a new favorite
From the same menu, you can also set your favorite chart (or any predefined chart) as the default chart or pin it to the toolbar. When added to the toolbar, the chart can be opened with a single click from the toolbar. Figure 2-31 shows how the icons are placed.
Figure 2-31 Icon placement
When the mouse cursor is placed over the icon, it is magnified and the name of that chart is shown.
Finally, to download and export the values from the chart, click the Disk icon near the Favorites icon. As a result, a comma-separated values (csv) file is created with all the metrics included in the chart for the complete time frame.
A tool is available on IBM Techdocs that offers an extra way to obtain performance data in an csv file format that includes more metrics. It is called the IBM Flash System Performance Statistics Data Capture and Extraction Procedure.
With this procedure, all the data that is collected by the GUI can be downloaded and formatted as a csv file. The file can be used for further analysis in performance studies or in a proof of concept scenario. Just download the files and follow the documentation.
2.2.2 Monitoring FlashSystem 840 / 900 with IBM Spectrum Control
To monitor performance for the s FlashSystem 840 or FlashSystem 900 with IBM Spectrum Control, enable the SNMP agent on the FlashSystem device to allow Spectrum Control to collect performance statistics. The SNMP agent on the FlashSystem device is disabled by default.
 
Note: FlashSystem 840 and FlashSystem 900 only support SNMP v1, which does not have encryption capability.
SNMP service on the FlashSystem 840/900 can be easily turned on from the Management GUI, as illustrated in Figure 2-32.
Figure 2-32 FlashSystem 900 Web UI
Alternatively, you can enable SNMP from the FlashSystem CLI as shown in Example 2-1.
Example 2-1 Enabling SNMP from CLI
IBM_FlashSystem:FlashSystem-840-03:superuser>lssnmpagent
enabled no
community ibmpublic
IBM_FlashSystem:FlashSystem-840-03:superuser>chsnmpagent -enable yes
IBM_FlashSystem:FlashSystem-840-03:superuser>lssnmpagent
enabled yes
community ibmpublic
IBM_FlashSystem:FlashSystem-840-03:superuser>
If you try to schedule performance monitoring without having the SNMP agent enabled on FlashSystem 840 or FlashSystem 900, you are prompted to enable it. Figure 2-33 shows the Data Collection Schedule window.
Figure 2-33 Data Collection Schedule window
The SNMP community is automatically detected during the probe. If it is necessary to change the community, then the FlashSystem 840 or FlashSystem 900 might need to first be probed again to continue with the performance data collection.
 
Important: If your device is behind a firewall, port 161 must be open to get performance data. For more information about ports that need to be open, see the IBM Spectrum Control section of IBM Knowledge Center.
You can use the Hyperlink (1) to open the FlashSystem GUI to enable SNMP as shown in Figure 2-33.
Clicking Test SNMP (2) checks whether SNMP is turned on by using the read community to get a confirmation.
Either method opens the default Data Collection window, as shown in Figure 2-34.
Figure 2-34 Data Collection Schedule
Performance Monitoring for the FlashSystem can now be scheduled.
Because no individual volume performance data is available yet, the Overview window (Figure 2-35 on page 28) shows the following charts:
•Overall System Activity
 – I/ORate (Read/Write/Total)
 – Data Rate (Fed/Write/Total)
 – Response Time (Read/Write/Total) 1
•Most Active Nodes
 – I/O Rate (Total - per node)
 – Data Rate (Total - per node)
•Drive Activity
 – Back-end Data Rate (Total - per drive)
 – Flash Health Percentage (per drive)
•Most Active Ports
 – Bandwidth (Overall - per Port)
Figure 2-35 FlashSystem 840 / FlashSystem 900 Overview window
2.2.3 Performance metrics for IBM FlashSystem 840 / 900
To get performance metrics, the FlashSystem 840 or FlashSystem 900 needs to be at firmware version 1.4.4. or later. To get response time values, the firmware version needs to be 1.4.5 or later.
The FlashSystem 840 or FlashSystem 900 does not provide any performance statistics for volumes. Therefore, there is also no performance data for pools and RAID arrays.
Available performance statistics are primarily for ports. Node and overall FlashSystem 840 or FlashSystem 900 performance statistics are calculated or aggregated from port performance data.
For FlashSystem 840 or FlashSystem 900 that have no Fibre Channel adapters, performance data is only available for the drives.
For the FlashSystem 840 and FlashSystem 900, the Performance tab (1)/ the View Performance Button (2) and the View Performance action (3) are enabled for ports, drives, nodes and for the overall FlashSystem 840 or FlashSystem 900 system performance. Figure 2-36 shows how to select port performance for FlashSystem 840 / 900.
Figure 2-36 Selecting Port Performance for FlashSystem 840 / FlashSystem 900
Available Port Performance metrics
Figure 2-37 shows the selection window for Available Port Performance metrics for FlashSystem 840 and FlashSystem 900.
Figure 2-37 Port performance metrics for FlashSystem 840/FlashSystem 900
Available Flash module performance metrics
For the Flash modules, which are shown as drives in IBM Spectrum Control, a metric called Flash Health Percentage is available. That percentage decreases when your flash modules suffer degradation. The Flash Health Percentage is calculated based on the number of unusable blocks in the flash module.
Because access to the drives is not I/O based but more like a type of Direct Memory Access, no Back-end I/O Rate metrics are available for the flash modules. Only Data Rates can be reported.
IBM Spectrum Control supports a range of different storage systems that use different flash devices embedded in different kind of modules, such as MicroLatency modules. The generic term disks is used instead of MicroLatency modules.
Figure 2-38 shows the available Performance Metrics for the FlashSystem 840 / 900 Flash modules.
Figure 2-38 Flash Module Performance metrics for FlashSystem 840 / FlashSystem 900
Although the performance chart implies that the FlashSystem 840/FlashSystem 900 can report back-end response time for drives, this metric is not available.
Available Node Performance metrics
The Nodes’s Volume metrics are calculated from the Port metrics and show the same values as the Port metrics except the Transfer Size, which is not available in the Volume Metrics tab.
Figure 2-39 shows the selection window for Available Node Performance Metrics for the FlashSystem 840/FlashSystem 900.
Figure 2-39 Node Performance Metrics for the FlashSystem 840/FlashSystem 900
Although the performance chart implies that the FlashSystem 840/FlashSystem 900 can report on Node’s System CPU Utilization, this metric is not available.
Available overall FlashSystem 840 / 900 System Performance Metrics
Figure 2-40 shows the selection window for the overall FlashSystem 840/FlashSystem 900 Port Performance metrics.
Figure 2-40 Overall FlashSystem 840 / FlashSystem 900 Port Performance metrics
Figure 2-41 shows the window for Overall FlashSystem 840/FlashSystem 900 Disk Performance metrics.
Figure 2-41 Overall FlashSystem 840 / FlashSystem 900 Disk Performance metrics
The Volume’s Metrics tab shows the sum of the measured performance of all the ports in the storage system, and therefore shows the same values as the Port Metrics tab, except for the Transfer Size metrics.
2.2.4 Comparing native GUI and Spectrum Control
Table 2-1 shows an overview of available Performance Metrics and where they can be displayed, in the FlashSystem 840 / FlashSystem 900 GUI or in Spectrum Control.
Table 2-1 FlashSystem 840/900 Performance Metrics
Level
Metric
Native
with IBM Spectrum Control
Storage System
IO in ops/s: Read/Write/Total
X
X
Storage System
Bandwidth/Data Rate in MiB/s: Read/Write/Total
X
X
Storage System
Total Rebuild Bandwidth in MiB/s
X
 
Storage System
Latency/Response Time in ms/op: Read/Write
X
X
Storage System
Overall Latency/Response Time in ms/op
 
X
Storage System
Max Latency/Response Time in ms/op: Read/Write
X
 
Flash Module / Drives
Flash Health %
X
X
Flash Module / Drives
Power in Watts
X
 
Flash Module / Drives
Garbage Collection: User Data %: Hot/Warm/Cold
X
 
Flash Module / Drives
Garbage Collection: User Data %: Total Write Amplification Factor
X
 
Flash Module / Drives
Background Health Checker: Health Checker Operations: Iterations/Blocks Checked/Wear Leveling Moves
X
 
Flash Module / Drives
Background Health Checker: Total Completion Percentage
X
 
Flash Module / Drives
Data Rate (MiB/s): Read/Write/Total
 
X
Port
IO in ops/s: Read/Write/Total
X
X
Port
Total Read Modify Writes (Operations)
X
 
Port
Bandwidth/Data Rate in MiB/s: Read/Write/Total
X
X
Port
Bandwidth Percentage: Send/Receive/Overall
 
X
Port
Queue Depth Operations
X
 
Port
Aligned Command Size Operations: Small/Large
X
 
Port
Block Size Operations: Small/Large
X
 
Port
Response Time in ms/op: Send/Receive/Overall
 
X
Port
Transfer Size (KiB/op): Send/Receive/Overall
 
X
Node: Port metrics
IO in ops/s: Send/Receive/Overall
 
X
Node: Port metrics
Data Rate in MiB/s: Send/Receive/Overall
 
X
Node: Port metrics
Response Time in ms/op: Send/Receive/Overall
 
X
Node: Port metrics
Transfer Size (KiB/op): Send/Receive/Overall
 
X
Historical data can be seen with native tools for the last 300 days. With IBM Spectrum Control, historical performance data can be seen as configured in the retention time settings as shown in Figure 2-7 on page 9.
With the native FlashSystem 840/900 GUI, you can either select multiple resources or multiple metrics, as shown in Figure 2-42.
Figure 2-42 FlashSystem 840/900 metrics selection window
You can also select multiple resources and multiple metrics in IBM Spectrum Control, as shown in Figure 2-43.
Figure 2-43 Multiple resources and multiple metrics in IBM Spectrum Control
In the FlashSystem 840/900 native storage management GUI, you can easily open a split window to show different resources or different metrics. For example, you can show the systems performance in the upper part of the window and the port performance in the lower window, as illustrated in Figure 2-44.
Figure 2-44 FlashSystem 840/900 split Performance window
In IBM Spectrum Control, you can open different performance windows with the Open in new window function and align them to compare the data, as shown in Figure 2-45.
Figure 2-45 Arranging multiple Spectrum Control windows
An advantage of opening and arranging multiple windows with IBM Spectrum Control is that you can easily compare different time ranges. With IBM Spectrum Control, as many windows can be opened and arranged as necessary, whereas with the native FlashSystem 840/900 GUI, the performance chart can only be split into two windows.
The performance charts on the FlashSystem 840/900 are updated every five seconds, whereas in IBM Spectrum Control the minimum interval is one minute. To get real-time performance data, the native FlashSystem 840/900 GUI can be opened from IBM Spectrum Control as shown in Figure 2-17 on page 16.
Although the port performance charts in the FlashSystem 840/900 GUI use the same color for ports that are connected to the same adapter as shown in Figure 2-24 on page 21, IBM Spectrum Control shows each port with its own color, independently of which adapter it is connected to, as shown in Figure 2-30 on page 23.
2.3 Monitoring Performance for FlashSystem V9000
This section describes performance monitoring for FlashSystem V9000, first using the system storage management native tools, and then using Spectrum Control.
2.3.1 Monitoring FlashSystem V9000 with the native tools
FlashSystem V9000 can be managed and monitored from the system GUI or from the CLI. To use the web-based GUI, complete these steps:
1. Enter the system IP address in your browser.
2. Enter your credentials to get access, as shown in Figure 2-46. The examples are based on Version 7.x.
Figure 2-46 V9000 log-in window
3. From the main widow, select Monitoring → System → Performance to open the Performance monitor, as shown in Figure 2-47.
Figure 2-47 Menu to enter the Performance monitor
The Performance Monitor window, shown in Figure 2-48 on page 38, is divided into four sections of equal size that provides information about performance of the different areas. Their location and purpose is:
 – CPU Utilization (upper left). Here the usage is shown for these metrics:
 • System CPU %
 • Compression CPU % (only when compression is enabled)
 – Volumes (upper right). This view shows the overall volume data for these metrics:
 • Read - Total read traffic of the system
 • Write - Total write traffic for the system
 • Read latency - Average read time per volume
 • Write latency - Average write time per volume
 – Interfaces (lower left). The overall statistic for every available port is shown here divided into these metrics:
 • FC - All Fibre Channel traffic
 • iSCSI - All iSCSI traffic
 • SAS - All serial attached SCSI traffic, backend, and frontend
 • IP Remote Copy - Remote Copy traffic over IP connections
 – Mdisks (lower right). The overall statistics for MDisks for these categories:
 • Read - Total read traffic of the system
 • Write - Total write traffic for the system
 • Read latency - Average read time per volume
 • Write latency - Average write time per volume
Figure 2-48 V9000 Performance monitor
This view is fixed. All four charts are always displayed at the same place. The monitored time frame is 5 minutes for every chart in this window, giving you the same time frame view of all metrics. In the upper left of the window, two pull-down menus allow you to customize the views. The System Statistics menu can be used to select a single control enclosure (here called node) or the whole system. If only one control enclosure needs to be monitored, it can be selected here. The second pull-down menu allows you to display IOPS or MBps, as shown in Figure 2-49.
Figure 2-49 Changeable settings for the V9000 Performance monitor
For each of the four charts, you can select or clear the measurements to show. The choice depends on the resources being monitored (volume, MDisk, interfaces, or CPU). Because data is collected continuously, after you select a metric for display, you can immediately see the last 5 minutes of collected data. See Figure 2-50.
Figure 2-50 Volume performance
To get a detailed value for a single point in time, you can click the curve in every chart and see the detailed values along with the time when those items were measured. Figure 2-51 shows an example for the Interfaces chart.
Figure 2-51 Exact value in Performance Monitor
A general view for the complete system is always visible at the bottom of every window in the GUI. It displays as a dashboard and shows real-time values, as illustrated in Figure 2-52.
Figure 2-52 Performance summary visible on every GUI window on the V9000
You can also monitor the V9000 performance through the CLI. The CLI interface uses the same IP address as the GUI. To open a CLI session, complete these steps:
1. Open a Secure Shell (SSH) session.
2. Enter your user ID and password (alternatively, a key can be provided, which is more convenient for scripts and automation) to authenticate with the CLI session.
3. Use the lsnodestats and lssystemstats commands to get real-time performance data.
 
Note: From the CLI, the term node is often used instead of control enclosure. In any output, the term node really stands for control or expansion enclosure.
To get the overall system performance data, issue the lssystemstats command. This command displays the most recent values of all node statistics in the system. The command output is a list of all collected statistics for the last available point in time. This output can be formatted with the standard CLI delimiters and filters. A shortened output is shown in Example 2-2.
Example 2-2 Output of lssystemstats
IBM_FlashSystem:Cluster:superuser>lssystemstats
stat_name stat_current stat_peak stat_peak_time
compression_cpu_pc 0 0 170811015618
cpu_pc 2 2 170811015618
fc_mb 0 13 170811015353
fc_io 614 691 170811015353
sas_mb 0 0 170811015618
sas_io 0 0 170811015618
To get a history of a single statistic like the CPU usage (cpu_pc) in the above example, the -history parameter can be used, as shown in Example 2-3.
Example 2-3 Displaying a specific statistic
IBM_FlashSystem:Cluster:superuser>lssystemstats -history fc_mb
sample_time stat_name stat_value
170811015621 fc_mb 320
170811015626 fc_mb 411
170811015631 fc_mb 442
170811015636 fc_mb 404
170811015641 fc_mb 527
The lsnodestats command shows the same statistics, but based on a single node (that is, enclosure) in the system. That is why two additional columns are in use for the Node ID and Node name. The other columns are the same as for the complete system. In total, there are six columns:
node_id ID of the reporting node
node_name Name of the node
stat_name Name of the statistic that is listed
stat_current Current value of the statistic
stat_peak Peak value of the listed statistic during the last 5 minutes
stat_peak_time Timestamp of the peak value
The values are shown for each node separately. This command can be used if only a single enclosure or node is in focus or needs to be checked. The -history parameter can be used also, but must be used with the node ID or node name as additional parameter. Example 2-4 shows the output of the lsnodestats command.
Example 2-4 lsnodestats command output
IBM_FlashSystem:Cluster_Hotel:superuser>lsnodestats
node_id node_name stat_name stat_current stat_peak stat_peak_time
1 node1 compression_cpu_pc 0 0 170811023200
1 node1 cpu_pc 2 2 170811023200
1 node1 fc_mb 0 0 170811023200
1 node1 fc_io 306 315 170811023100
1 node1 sas_mb 0 0 170811023200
1 node1 sas_io 0 0 170811023200
The performance monitoring capability described here and included with the V9000 native storage management GUI is for real-time monitoring over a short period. Spectrum Control offers the advantage of collecting performance data over an extended period (longer history) and covering more aspects of the system.
The FlashSystem V9000 management GUI collects data for Mdisks, volumes, nodes, and SAS drives automatically. The data collected is part of a support file called snap, but the data can be downloaded without downloading the complete snap.
 
Restriction: The data for Flash modules is not included in the drive data collection.
The data is stored in a single file for every interval. Such an interval can be from 1 minute to up to 60 minutes. The default value is 5 minutes. The limitation here is that the data is stored for 16 intervals maximum. When the 17th interval is started, the oldest data set is deleted. Therefore, the interval time also sets the maximum monitored time frame. If the interval is set to two minutes, 32 minutes of data is stored in the files. However, the default value of five minutes brings 80 minutes of performance data, but the granularity is then five minutes. So, depending on what situation you need to analyze, it is important to decide upfront what data granularity is required.
The interval time can be set by using the startstats command. To get all the details and possible parameters, see the command help or consult IBM Knowledge Center.
Keep in mind that the last setting will remain in effect until it is changed again. If the interval was changed for a specific scenario or maybe just for testing, make sure that it is set back to your standard interval to avoid unexpected results in the next data collection.
 
Note: Do not use the startstats command to change the interval if you use an external monitoring software such as IBM Spectrum Control. Change the interval in the monitoring software instead.
The data is collected on a per node basis and not for the complete cluster at a single location. Even though the samples are taken at the same time on all nodes, the data files are stored on their respective collecting nodes. To get all the files for members in the cluster, the data must be copied to the config node before download.
The files are stored in the /dumps/iostats directory of each node. To collect the data on the config node, use the cpdumps command to consolidate before download. The file names have the following format:
•MDisk statistics:
Nm_stats_<node_serial_number>_<date>_<time>
•VDisk statistics:
Nv_stats_<node_serial_number>_<date>_<time>
•Node statistics:
Nn_stats_<node_serial_number>_<date>_<time>
•Disk drive statistics (not for IBM Flash System V9000 Flash Modules):
Nd_stats_<node_serial_number>_<date>_<time>
The <node_serial_number> gives the information about which node or control enclosure the data is collected, <date> is the day of collection in the form <yymmdd>, and the <time> shows the exact time stamp in the form <hhmmss>.
The filename Nv_stats_78AV320_170810_182416 is an example of a file for volume data, collected on the Node with the serial 78AV320, on Aug 10th at 06:24pm. To list all collected data, use the lsdumps command with the prefix iostat, as shown in Example 2-5.
Example 2-5 lsdump command
IBM_FlashSystem:V9000_tpcv9k1:superuser>lsdumps -prefix /dumps/iostats
id filename
0 Nv_stats_78AV320_170810_182416
1 Nm_stats_78AV320_170810_182416
2 Nd_stats_78AV320_170810_182416
3 Nn_stats_78AV320_170810_182416
4 Nd_stats_78AV400_170810_182416
5 Nn_stats_78AV400_170810_182416
6 Nv_stats_78AV400_170810_182416
 
Tip: To copy the files from the Flashsystem V9000 to a local workstation the pscp command, included with PuTTY, can be used. Open a command prompt and change to the directory where pscp.exe is located.
C:pscp>pscp -unsafe -load performancedata [email protected]:/dumps/iostats/* c:v9000data
Using keyboard-interactive authentication.
Password:
Nd_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%
d_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%
Nd_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%
Note the following about this output:
•The -unsafe parameter is needed because wildcards are in use
•A PuTTY session with the name behind the -load needs to be configured up front. If there is no key stored, pscp prompts for a password, as shown in the above example.
A complete list with a short explanation of all collected data can be found in IBM Knowledge Center.
2.3.2 Monitoring V9000 with IBM Spectrum Control
In terms of performance monitoring, IBM Spectrum Control basically captures the same information for FlashSystem V9000 as for any other IBM Storwize® family system.
A complete list of performance metrics available for V9000 is listed in the IBM Spectrum Control section of IBM Knowledge Center.
The flash modules can be seen in the disks window, as illustrated in Figure 2-53.
Figure 2-53 V9000 Flash modules in IBM Spectrum Control
Drive performance data
The V9000 Nd_stat file contains only statistics of SAS drives, not for Flash modules in a Flash storage enclosure. Therefore, drive performance data cannot be shown in IBM Spectrum Control. For more information, see the IBM FlashSystem V9000 section of IBM Knowledge Center.
Because you can also add enclosures with SAS drives to a V9000, the disks performance tab is available in IBM Spectrum Control for V9000 disks.
Zero Buffer Credit Percentage/Port Send Delay I/O Percentage
The hardware design of the 16 Gb HBAs has changed and they do longer provide zero buffer credit counters. In the past (up until version 5.2.13) Spectrum Control showed an incorrect value of 0.00 for Zero Buffer Credit Timer and Zero Buffer Credit Percentage (%) for 16 Gb ports (as reported by the Storwize systems). Starting with With IBM Spectrum Control v 5.2.14, N/A (rather than 0.00) is shown for these metrics for 16 Gb ports only.
Because the lack of the Zero Buffer credit counters means that there is no way to debug slow-draining device problems in the SAN, new counters (dtdt, dtdc, and dtdm) were introduced with Storwize version 7.8.1. These counters will attempt to measure sent I/O delay by retrieving a subset of traffic that can be measured by software. Therefore the new counters do not replace the zero buffer credit counters. Instead, they represent the delay expected in transmitting data calculated based on the amount of data queued in the port and the estimated port speed. These new counters are available in IBM Spectrum Control starting with version 5.2.14. They can be displayed using the Port Send Delay Time (ms/op), Port Send Delay I/O Percentage (%), or both.
These new counters and therefore the new metrics Port Send Delay Time (ms/op) and Port Send Delay I/O Percentage (%) are available for all ports.
Figure 2-54 shows a comparison of Zero Buffer Credit Percentage and Port Send Delay I/O Percentage.
Figure 2-54 Zero Buffer Credit Percentage versus Port Send Delay I/O Percentage
Compression savings %
When using compression on any of the Storwize or SAN Volume Controllers (SVC) devices, always evaluate whether compression savings are worth the effect of compression on the system performance. To get a quick view of the compression savings, sort the volumes in Spectrum Control by compression savings in descending order (1), as shown in Figure 2-55.
Figure 2-55 Volumes sorted by compression savings
For volumes with low compression savings (< 40%), it might be better to turn compression off. We have seen compressed volumes in the field that were bigger after compression than they were before.
As described in the IBM Spectrum Control section of IBM Knowledge Center, compression Savings is calculated by using this formula:
Compression savings % = (1- used space / written space) * 100
2.3.3 Comparing V9000 native tools with IBM Spectrum Control
In IBM Spectrum Control, a comprehensive list of performance metrics is available for monitoring V9000 performance, as shown in Table 2-2. The V9000 native GUI provides only a limited set of performance metrics and only on a cumulated/aggregated level per system or per node.
In Spectrum Control, you can see the performance metrics not only cumulated/aggregated for the complete storage system or per node, but also for each individual internal resource.
With IBM Spectrum Control, you also can see all the data historically, which is not possible in the V9000 GUI. The minimum interval in IBM Spectrum Control that can be set to collect performance data is one minute, whereas the performance chart on the V9000 GUI gets updated every five seconds. If you need real-time performance data, use the View Real-Time Performance function, as shown in Figure 2-18 on page 16.
Table 2-2 Performance metrics available in the V9000 GUI
Level
Metric
Available in Spectrum Control
System Statistics/Node
CPU Utilization: System %
X
System Statistics/Node
CPU Utilization: Compression %
X
System Statistics/Node
FC Ports : I/O Rate in ops/s /
X
System Statistics/Node
FC Ports : Data Rate in MiB/s
X
System Statistics/Node
iSCSI Ports : I/O Rate in ops/s /
 
System Statistics/Node
iSCSI Ports : Data Rate in MiB/s
 
System Statistics/Node
SAS Ports : I/O Rate in ops/s /
 
System Statistics/Node
SAS Ports : Data Rate in MiB/s
 
System Statistics/Node
ip Replication Ports : I/O Rate in ops/s /
 
System Statistics/Node
ip Replication Ports : Data Rate in MiB/s
 
System Statistics/Node
Volume Read / Write Data Rate in MiB/s
X
System Statistics/Node
Volume Read / Write I/O Rate in ops/s
X
System Statistics/Node
Volume Read / Write Latency in ms
X
System Statistics/Node
MDisk Read / Write I/O Rate in ops/s
X
System Statistics/Node
MDisk Read / Write Data Rate in MiB/s
X
System Statistics/Node
MDisk Read / Write Latency in ms
X
2.4 Monitoring performance for FlashSystem A9000 or A9000R
This section discusses performance monitoring for FlashSystem A9000/A9000R, first using system native tools and then using Spectrum Control.
2.4.1 Monitoring A9000 or A9000R with HyperScale Manager
FlashSystem A9000 and A9000R GUI uses the IBM Hyper-Scale Manager server. The management GUI is accessed by directing your web browser at the Hyper-Scale Manager server (http://<Hyper-ScaleManager server IP address>:8080 or for secure servers https://<Hyper-ScaleManager server IP address>:8443). A login window, as shown in Figure 2-56, prompts you to enter valid credentials.
Figure 2-56 IBM Hyper-Scale Manager Login window
Upon successful login, the dashboard is presented. The dashboard lists any Spectrum Accelerate systems, such as A9000, A9000R, or IBM XIV®, included in the inventory and that can be accessed through the same credentials. You can select to focus on just one particular system, as shown in Figure 2-57.
Figure 2-57 Menu to select the system for further actions
The selection of a single system changes the dashboard view to display information related to that particular system only. In the lower right corner, you can see an overview of the system total IOPS and maximum the latency. This information can be used as a first indication. For more detailed information, click the Statistics icon to open the Statistics Views menu. The first four options in that menu pertain to performance statistics:
•System & Ports Performance Statistics
•Volume Performance Statistics
•Host & Port Performance Statistics
•QoS Performance Statistics
The options are shown in Figure 2-58
Figure 2-58 Different options for performance statistics
If the Volume Performance Statistics is selected, a list of all volumes is created. If Host & Ports Performance is selected, a list with all hosts is shown. Below the list, there is one chart for the corresponding selected object and the current IOPS as a metric (see Figure 2-58). For QoS statistics, historic data is shown.
Based on the selection, all corresponding objects are preselected. For instance, if Volumes are the monitored object, all volumes belonging to the particular A9000 are selected.
If there is no object selected, no chart is shown. To enable the chart, select at least one object (Volume, Host, and so on) to get option to open a chart.
Figure 2-59 shows the view with the list of the possible objects that can be monitored and a single chart with the default settings. You can customize that view. Note that the figure is specific to Volumes, but the same actions and selections apply to other objects.
You can filter the view to restrict it to a single or small group of volumes. You can enter and combine different filters.
The columns can be used to have the volumes sorted by size, name, or properties.
The content icon in the top right can be used to customize the columns to be shown.
If the result needs to be saved, the list can be exported into a csv file format.
Figure 2-59 Statistics view for an A9000
The objects that are shown in the chart can now be selected. If there no volume is selected, no chart is shown. If this is the case, select at least one chart and click the Statistics field at the bottom of the list, as shown in Figure 2-60.
Figure 2-60 Open a chart for performance statistics
The chart with the real-time performance statistics is progressively drawn, and displays the total values for all selected volumes. This feature allows you to monitor a single volume or a group of volumes. For example, you can group all objects that belong to an application, server, or group of servers.
The chart can be further customized by using the pull-down menus in the middle left area of the window (Figure 2-59 on page 48).
The second menu from the left can be used to select the complete system or only one interface.
From the third menu, specific metrics can be selected. Because metrics are different at a System level or host level, the menus are different. The options are as listed here:
•Qos
 – Historic IOPS
 – Historic Latency
 – Historic Bandwidth
•Volumes
 – Current IOPS
 – Historic IOPS
 – Current Latency
 – Historic Latency
 – Current Bandwidth
 – Historic Bandwidth
•Host and Port same as Volumes plus:
 – Written by Host
 – Written by Host + Forecast
•System same as Volume plus
 – Physical usage
 – Physical usage + Forecast
 – Written / Allocated
 – Written / Projected
 – Written / Projected + Forecast
 – Projected / Actual
The current metrics are shown for a predefined time frame. For the historic values, there are many more options to choose from. If one of the historic options is selected, new menus are visible in this chart. One is the type of I/O, Read, Write, or Read+Write. The second one is Memory Hit, Memory Miss or Hit+Miss and the third is the size of the I/O or all together.
Any combination of the menu options is possible, like only reads, which have a Memory Miss and are bigger than 512 KB.
A last menu on the top right of the chart is the Range. The time scale of the chart can be selected here based on preselected ranges such as last hour or last month, or an individual time frame can be selected.
Other charts can be added by clicking the Add (+) sign. This action opens a second window allowing you to compare two different charts. When a chart is no longer needed, it can be removed with the Close (x) button.
To get more details about the real-time graphs (current) and the values, click in the chart to open a tooltip window that gives more details about the value at that time. A nice feature for several monitored volumes (or hosts etc) is the ordering. Depending on the number, the Total value as well as the max value and min values are shown. Sometimes the average is also included.
2.4.2 Monitoring A9000 with IBM Spectrum Control
In terms of performance monitoring with IBM Spectrum Control, Flashsystem A9000 and A9000R are showing basically the same information as for the XIV systems. A complete list of available performance metrics is available in the IBM Spectrum Control section of IBM Knowledge Center.
Available performance metrics
Performance values are available for the A9000/A9000R grid controllers (interface modules for XIV), but not for the individual flash modules (MicroLatency Modules) that are listed as drives or disks in IBM Spectrum Control. No performance metrics are available for the Flash enclosures that are shown as RAID Arrays in IBM Spectrum Control.
IBM FlashSystem A9000 and IBM FlashSystem A9000R do not track performance statistics for volumes that were never used. Because there are no performance statistics, performance metrics are not shown for these volumes and their related components.
Although the IBM Spectrum Control metrics selection offers the SSD Read Cache Hits % performance metric for the A9000/R as well, they only apply for XIV, which uses solid-state drives for caching only, as described in the hover help shown in Figure 2-61.
Figure 2-61 A9000 Volume Performance Metric selection
2.4.3 Comparing A9000 Hyperscale Manager GUI with IBM Spectrum Control
Table 2-3 lists the performance metrics available in the Hyperscale Manager UI and IBM Spectrum Control.
Table 2-3 Level of performance metrics
Level
Hyperscale Manager UI
IBM Spectrum Control
System (per interface/module)
X
X
Volumes (per interface/module)
X
X
Domain
X
 
Mapped Host
X
X
Mirrored Volumes
X
 
Snapshot
X
 
Pool
 
X
Quality of Service
X
 
Port
 
X
With the Hyperscale Manager, the I/O Rate for Memory Hit or Memory Miss can be displayed as shown on Figure 2-62 as well as the total I/O Rate.
Figure 2-62 Hyperscale Manager: Memory Hit versus Memory Miss I/O Rate
This chart can also be customized for individual blocksizes, or a combination of all blocksizes, as shown in Figure 2-63. It cannot be combined with other metrics.
Figure 2-63 Hyperscale Manager: I/O Rate for individual Block sizes
In Spectrum Control, the I/O Rate is shown on one y-axis and the Cache Hit Percentage on the other second y- axis, as illustrated in Figure 2-64.
Figure 2-64 Spectrum Control: I/O Rate versus Cache Hit Percentage
Because a third y-axis cannot be displayed, all three metrics (I/O Rate, Cache Hit Percentage, Transfer Size) cannot be shown in the same chart (see also Chapter 2.1.2, “Considerations for Performance Monitoring with IBM Spectrum Control” on page 10). However, it is possible to show all three metrics at the same time in the table view of the graph by using the table/chart toggle (1), as shown in Figure 2-65.
Figure 2-65 Spectrum Control table view of performance diagram
As shown in Figure 2-66 and in Figure 2-67 on page 54, two different charts can be displayed on top of each other with HyperScale Manager. In Spectrum Control, this can be done by arranging multiple charts accordingly, as described in Figure 2.3.3.
Depending on how many resources are being selected, HyperScale Manager shows individual performance graphs (Figure 2-66) or accumulated performance graphs (Figure 2-67 on page 54).
Figure 2-66 Hyper Scale Manager: Individual Performance graphs
Figure 2-67 shows the accumulated performance graphs.
Figure 2-67 Hyper Scale Manager: Accumulated Performance graphs
In Spectrum Control, up to 10 individual graphs can be displayed, as shown in Figure 2-68, and can be individually toggled on/off as described in 2.1.2, “Considerations for Performance Monitoring with IBM Spectrum Control” on page 10.
Figure 2-68 Spectrum Control: Up to 10 individual graphs are being displayed
Although IBM Spectrum Control does not show any Quality of Service (QoS) data, volumes belonging to a QoS could be added to an application as described in IBM Spectrum Family: IBM Spectrum Control Standard Edition, SG24-8321. Doing so allows performance to be tracked easily for volumes belonging to a certain QoS.
The minimum interval for collecting performance data with IBM Spectrum Control is one minute, whereas HyperScale Manager shows real time performance for every five seconds. Currently, IBM Spectrum Control does not provide the Open Storage System GUI and View Real Time Performance functionality is not available in IBM Spectrum Control for A9000/R.
2.5 IBM Spectrum Control reporting options
IBM Spectrum Control provides numerous ways of reporting about Flash performance data. This section provides an overview of the various options and their appropriate usage in ongoing performance management of IBM FlashSystem storage devices:
•Web GUI Reports
•Create Performance Support Package
•REST API
•IBM Cognos®
•Native SQL Reports
2.5.1 Web GUI reports
IBM Spectrum Control Web GUI provides the following export options (see Figure 2-69):
•To export the data used in the performance chart, click the Export icon located in the upper left of the chart (1).
•To export the summary table shown underneath the chart: from the Action pull-down (2), click More (3) → Export (4) → Select the format (5).
s
Figure 2-69 Performance data export options
2.5.2 Create Performance Support Package
With IBM Spectrum Control v5.2.10, the Create Performance Support Package function has been improved. The purpose of this function is to provide performance data to the IBM device support teams. This feature can also be used for other purposes, but it is not optimized as a generic report generator.
Supported devices are storage systems (both block and file), fabrics, and switches.
Create Performance Support Package using the Web GUI
With IBM Spectrum Control v5.2.10, the Create Performance Support Package function is implemented in the IBM Spectrum Control GUI. The function can create reports for a specific time range as a set of compressed CSV files for a single device at a time. Each report is a single CSV file and covers a single resource type. By default, all resource types supported by the specified device are included in the zip file.
To create a Performance Support Package for the FlashSystem storage, complete these steps:
1. Right-click the storage system in the Block Storage Systems window, as shown in Figure 2-70, and select Create Performance Support Package.
Figure 2-70 Create Performance Support Package in the Web UI
2. The Create Performance Support Package window is displayed, as shown in Figure 2-71. Specify a time range using one of these options:
 – One of the predefined ranges (4, 8, or 12 hours) (1)
 – Clicking the time range and entering a custom range (2)
For time ranges less than 12 hours, volume data and, if applicable, 1 min interval data are included in the export files. If you select a time range greater than 12 hours, volume data and 1 min interval data are included when you select the Advanced Package (3).
Figure 2-71 Specify time-range for the Performance Support Package
3. Click Next (4) and the Spectrum Control server estimates the expected file size, as shown in Figure 2-72.
Figure 2-72 Expected File size of the Performance Support Package is shown
4. Click Create, as shown in Figure 2-72. Depending on your browser, you might be prompted to specify where to save the zip file.
Using the Export Performance Package script
There is also a script available to export the performance data. The script, called exportPerformanceData.bat, is located in the scripts directory of the Spectrum Control installation directory, as shown in Example 2-6.
Example 2-6 Directory of the exportPerformanceData script
Directory of C:Program FilesIBMTPCscripts
 
07/21/2017 11:07 PM <DIR> .
07/21/2017 11:07 PM <DIR> ..
07/21/2017 10:45 PM 684 exportPerformanceData.bat
11/09/2016 05:48 PM <DIR> ldap
03/03/2017 05:47 PM <DIR> LTPA
06/02/2017 07:26 PM 4,013 sslCert.bat
07/21/2017 10:57 PM 890 startTPCAlert.bat
07/21/2017 10:55 PM 619 startTPCData.bat
07/21/2017 10:57 PM 850 startTPCDevice.bat
07/21/2017 11:07 PM 817 startTPCWeb.bat
07/21/2017 10:57 PM 1,449 stopTPCAlert.bat
07/21/2017 10:55 PM 616 stopTPCData.bat
07/21/2017 10:57 PM 1,419 stopTPCDevice.bat
07/21/2017 11:07 PM 1,564 stopTPCWeb.bat
04/02/2017 10:26 AM <DIR> TPCNew_Data_201704020126
04/02/2017 10:00 AM 1,289 TPCPerf.bat
11 File(s) 14,210 bytes
5 Dir(s) 341,685,374,976 bytes free
 
C:Program FilesIBMTPCscripts>
With the script, several parameters can be specified, as shown in Example 2-7.
Example 2-7 exportPerformanceData help
 
C:Program FilesIBMTPCscripts>exportPerformanceData.bat
Calling Spectrum Control Performance Exporter ...
 
Usage: exportPerformanceData.bat -user <user> -pwd <password> [options] -resNames <resource_names>
required:
-user The user name for logging into the IBM Spectrum Control server.
-pwd The password for the specified user name.
-resNames One or more storage system, switch, or fabric names as
displayed in the UI, delimited by spaces.
If a name contains spaces, it should be within quotes.
options:
-server Hostname or IP address of the IBM Spectrum Control server.
Default is localhost.
-port Spectrum Control Web server port.
Default is 9569.
-out The name of the exported zip file, or if -zip no is
specified, the directory to store the exported data files.
Default is the current directory.
-resTypes A comma-separated list of resource types for which to export
the performance data.
Valid values are:
storageSystem, nodes, modules, ioGroups, hostConnections
pools, raidarrays, mdisks, disks, volumes, filesystems,
switches, ports, interSwitchConnections
Default is all resource types supported by the specified resource.
-start Start time in the "yyyy-MM-dd HH:mm:ss" format, including quotes.
Overrides -last if both are specified.
-end End time in the "yyyy-MM-dd HH:mm:ss" format, including quotes.
Default is the current time.
-last Length of time range, in hours.
Ignored if -start is also specified. Default is 4 hours, 48 hours, or
720 hours if -summType sample, hourly, or daily is in effect, respectively.
-summType Performance data collection summarization type.
Valid values are: sample, hourly, daily
Default is sample.
-advPkg Include 1 minute samples and volume data if available.
Valid values are: yes, no
Ignored if the length of the specified time range is 12 hours or less.
Default is no unless time range is 12 hours or less.
-zip Compress the CSV files into a zip file.
Valid values are: yes, no
Default is yes.
 
Example 2-8 shows how to create performance packages including the 1 min interval data and the volume data for two Flash storage systems by specifying custom start and custom end dates.
Example 2-8 Create Export Performance Data Package
C:Program FilesIBMTPCscripts>exportPerformanceData.bat -user user -pwd password -resNames "FlashSystem V9000-9846-V9000_2BB_HS-IBM" "FlashSystem 900-9840-FS900_1-IBM" -start "2016-08-19 08:00:00" -end "2016-08-20 08:00:00" -advPkg yes
 
Note: Although multiple devices can be specified in the command, the export performance package is processed serially for each device.
Export Performance Package
The downloaded Performance Support Package zip file includes one zip file for each device and each zip file contains csv files for every internal resource type that is supported by the device, as shown in Figure 2-73.
Figure 2-73 Sample Content of Export Performance Package for a V9000 Storage System
If you open one of the csv files, such as for the resource type volumes, you will find columns identifying information for each volume, the time stamp, configuration data columns, and performance data columns, as shown in Figure 2-74.
Figure 2-74 Sample Output of a Volumes.csv File for a V9000 Storage System
The downloaded zip file also contains a log.txt file with many rows per resource type collected and downloaded. If there were any errors during the data collection or the download, they will be shown in the log.txt file, as illustrated in Example 2-9.
Example 2-9 log.txt file example
Performance report exported with 'sample' data for block storage system A9000 on August 18, 2017 10:40:34 AM PDT.
Time range of report: August 18, 2017 6:40:24 AM PDT with a duration of 4 hours 0 minutes.
 
Storage System data: 240 rows were processed.
Port data: 1,440 rows were processed.
Host Connection data: 240 rows were processed.
Pool data: 960 rows were processed.
Volume data: 61,696 rows were processed.
Module data: 723 rows were processed.
 
Note: If you create performance packages for multiple devices, you might run into NTFS restrictions while unzipping the files and receive a “path too long” error. In this situation, either move the files to a directory with a shorter path or rename the files to reduce the length of the path name.
2.5.3 REST API
Although the Representational State Transfer (REST) API implementation for Spectrum Control is not a reporting tool, it can be used to access information about resources in json format.
Using the browser:
The IBM Spectrum Control REST API is hosted here:
https://<hostname>:9569/srm/REST/api/v1/
To get a list of StorageSystems with their configuration data, specify the following URL:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems
A sample output result is shown in Example 2-10.
Example 2-10 Example output of a REST request for Storage Systems
[
{
"Acknowledged": "No",
"Allocated Space": "8,830.84",
"Assigned Volume Space": "100",
"Available Pool Space": "685,792.00",
....
"Firmware": "8.1.0.0 (build 137.3.1708160754000)",
"Last Successful Monitor": "Aug 17, 2017, 20:40:00",
"Last Successful Probe": "Aug 16, 2017, 21:21:03",
....
"Model": "DH8",
"Name": "ssd-svc-dh8-01",
....
"Type": "SAN Volume Controller - 2145",
"id": "7494"
},
{
"Acknowledged": "No",
"Allocated Space": "22,994.68",
....
"FlashCopy": "30",
"IP Address": "9.155.112.248",
....
"Raw Disk Capacity": "64,820.05",
"Read Cache": "452.56",
"Remote Relationships": "331",
"Serial Number": "75FAW81",
"Shortfall": "27",
"Status": "Normal",
....
"Type": "DS8000",
"Unassigned Volume Space": "5,555.81",
"Unprotected Volumes": "1567",
"Unreserved Pool Space": "24,245.00",
"Used Pool Space": "24,003.00",
"Used Space": "22,994.68",
"Vendor": "IBM",
"Virtual Allocation": "49.31",
"Volumes": "1,861",
"Write Cache": "32.00",
"id": "7540"
},
....
}
]
Adding the StorageSystem id and a resource type, for example Volumes, to the URL will display a list of volumes for that storage system:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem id>/Volumes
To get a list of available performance metrics for a StorageSystem, add Volume id /Performance in the URL:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem id>/Volumes/<Volume id>/Performance
You can also specify a specific metric id:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem id>/Volumes/<Volume id>/Performance/<metric id>
Example 2-11 shows the response times of a A9000 Volume in json format.
Example 2-11 Using RESTAPI to get performance data
https://9.155.122.36:9569/srm/REST/api/v1/StorageSystems/65100/Volumes/2160532/Performance/823
 
[
{
"metricDetails": {
"823": {
"description": "Average number of milliseconds that it took to service each write operation, for a particular component over a particular time interval.",
"name": "Write Response Time",
"units": "ms/op"
}
}
},
{
"componentId": 2160532,
"current": [
{
{
"x": 1503016447000,
"y": "0.00"
},
},
{
"x": 1503016747000,
"y": "16.07"
},
{
"x": 1503016807000,
"y": "16.07"
},
{
"x": 1503016867000,
"y": "16.73"
},
{
"x": 1503016928000,
"y": "16.48"
},
{
"x": 1503016987000,
"y": "16.69"
},
{
"x": 1503017047000,
"y": "16.01"
},
{
"x": 1503017107000,
"y": "16.25"
},
Multiple resources and multiple metrics can be queried at the same time by using the following syntax:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem id>/Volumes/Performance?<metric id1>,<metric id2>,<metric id3>&ids=<Volume id1>,Volume id3>,Volume id3>
A sample output is shown in Example 2-12.
Example 2-12 Querying multiple metrics and multiple resources
https://9.155.122.36:9569/srm/REST/api/v1/StorageSystems/65100/Volumes/Performance?metrics=823,822,821&ids=2160532,2160549,2160521
 
 
[
{
"metricDetails": {
"821": {
"description": "Average number of mebibytes (2^20 bytes) per second transferred for read and write operations.",
"name": "Total Data Rate",
"units": "MiB/s"
},
"822": {
"description": "Average number of milliseconds that it took to service each read operation, for a particular component over a particular time interval.",
"name": "Read Response Time",
"units": "ms/op"
},
"823": {
"description": "Average number of milliseconds that it took to service each write operation, for a particular component over a particular time interval.",
"name": "Write Response Time",
"units": "ms/op"
}
}
},
{
"componentId": 2160521,
"current": [
{
"x": 1502985600000,
"y": null
},
{
"x": 1502987827000,
"y": "0.32"
 
....
{
"x": 1503075600000,
"y": null
}
],
"deviceId": 65100,
"deviceName": "AIX_BIG_001<br /> A9000",
"endTime": 1503075600000,
"label": "Write Response Time",
"maxValue": 40.56,
"metricId": 823,
"minValue": 0.0,
"precision": 2,
"resourceID": 2160521,
"startTime": 1502989200000,
"units": "ms/op"
},
{
"componentId": 2160521,
"current": [
{
"x": 1502989200000,
"y": null
},
{
"x": 1502989327000,
"y": "0.20"
 
....
],
"deviceId": 65100,
"deviceName": "AIX_BIG_001<br /> A9000",
"endTime": 1503075600000,
"label": "Read Response Time",
"maxValue": 101.56,
"metricId": 822,
"minValue": 0.0,
"precision": 2,
"resourceID": 2160521,
"startTime": 1502989200000,
"units": "ms/op"
}, },
{
"componentId": 2160532,
"current": [
{
"x": 1502989200000,
"y": null
},
{
"x": 1502989327000,
"y": "0.00"
 
Tip: The timestamps are shown in epoch time. To convert the timestamps, use the EpochConverter.
The granularity as well as start and end time for the performance data can be specified as well:
https://<server>:9569/srm/REST/api/v1/StorageSystems/<id>/Volumes/<vol id>/Performance/824?granularity=sample&startTime=1471935241000&endTime=1472021568000
 
Note: The REST API syntax is case sensitive.
Reference: For more information regarding the implementation of the RESTful API in IBM Spectrum Control, see IBM Knowledge Center.
Using Wget
GNU Wget is a free software package that retrieves content from web servers using HTTP, HTTPS, and FTP and is part of the GNU Project.
 
Reference: wget can be downloaded from the GNU website.
For authentication, a security token can be stored in a file, such as cookies.txt for later queries, as shown in Example 2-13.
Example 2-13 Authentication token
wget --post-data "j_username=<username>&j_password=<password>" --no-check-certificate --keep-session-cookies --save-cookies cookies.txt https://<servername>:9569/srm/j_security_check
Now this security token can be used for queries against the IBM Spectrum Control’s REST API, as illustrated in Example 2-14.
Example 2-14 Retrieving the authentication token
wget --no-check-certificate --load-cookies cookies.txt -O Output.html https://<servername>:9569/srm/REST/api/v1/StorageSystems/<id>/Volumes
In Example 2-15, the previously saved security token in cookies.txt is being used for the authentication. The output of this request is then saved in the file Output.html, but still in the JSON format.
To parse the JSON output, a tool like JSON query (jq) can be used.
 
Reference: JSON Query can be obtained from the GitHub website.
Example 2-15 Parsing the json output file
C:UsersIBM_ADMIN>type C:SpectrumControlwgetOutput.html |jq -r ".[]|[.id,.Name,.Capacity,."Storage System"]|@csv"
"2160667","AIX_HS_027","45.63","A9000"
"2160664","AIX_HS_024","45.63","A9000"
"2160665","AIX_HS_025","45.63","A9000"
"2160666","AIX_HS_026","45.63","A9000"
"2160668","AIX_HS_028","45.63","A9000"
"2160669","AIX_HS_029","45.63","A9000"
"2160670","AIX_HS_030","45.63","A9000"
"2160671","AIX_HS_031","45.63","A9000"
"2160672","AIX_HS_032","45.63","A9000"
"2160673","AIX_HS_033","45.63","A9000"
"2160675","AIX_HS_035","45.63","A9000"
To get rid of the quotation marks and backslashes, the jq command can be combined with the sed utility, as illustrated in Example 2-16.
Example 2-16 The sed utility
C:UsersIBM_ADMIN>type C:SpectrumControlwgetOutput.html |jq -r ".[]|[.id,.Name,.Capacity,."Storage System"]|@csv" |sed -e "s/"*//g"
2160667,AIX_HS_027,45.63,A9000
2160664,AIX_HS_024,45.63,A9000
2160665,AIX_HS_025,45.63,A9000
2160666,AIX_HS_026,45.63,A9000
2160668,AIX_HS_028,45.63,A9000
2160669,AIX_HS_029,45.63,A9000
2160670,AIX_HS_030,45.63,A9000
2160671,AIX_HS_031,45.63,A9000
2160672,AIX_HS_032,45.63,A9000
2160673,AIX_HS_033,45.63,A9000
2160674,AIX_HS_034,45.63,A9000
2.5.4 IBM Cognos
In IBM Spectrum Control, it is possible to run and view predefined Cognos reports and create custom reports. The Cognos reporting engine is accessible using the Reports action in the top navigation bar.
IBM Spectrum Control provides over 70 predefined reports that display capacity and performance information collected by IBM Spectrum Control. Charts are automatically generated for most of the predefined reports. Depending on the type of resource, the charts show statistics for space usage, workload activity, bandwidth percentage, and other statistics. You can schedule reports and specify to create the report output in HTML, PDF, and other formats. You can also configure reports to save the report output to your local file system or to send reports as mail attachments.
FlashSystem 840/FlashSystem 900 Performance data are not included in the performance package. Therefore, it is not possible to get any performance data for the FlashSystem 840/ FlashSystem 900 using Predefined reports or using the Performance package for custom reports.
However, you can use native SQL for creating queries against the FlashSystem 840/ FlashSystem 900 performance tables. Native SQL Queries can be rendered with IBM Cognos Report Studio.
 
Reference: For more information regarding the usage of Cognos and its functions, see IBM Tivoli Storage Productivity Center V5.1 Technical Guide, SG24-8053, and IBM Spectrum Family - Spectrum Control Standard, SG24-8321.
2.5.5 Native SQL reports
IBM Spectrum Control stores all data in a DB2 database called TPCDB that you can query easily by using native SQL.
Connections can also be done using the ODBC interface, for example with Microsoft Excel, or you can render your native SQL Query in IBM Cognos by using IBM Cognos Report Studio.
 
Tip: Always specify with ur for read only in your SQL Queries. Otherwise your tables might get locked during the read operation and slow down the TPCDB performance.
For more information see: Locks and concurrency control and read-only-clause at IBM Knowledge Center.
Performance data for the FlashSystem 840 / FlashSystem 900 are stored in the following DB2 tables:
•Sample performance data for flash modules, nodes, ports, and the entire subsystem:
 – T_PRF_FLSH_MODULE
 – T_PRF_FLSH_NODE
 – T_PRF_FLSH_PORT
 – T_PRF_FLSH_SYSTEM
•Hourly and daily performance data for flash modules, nodes, ports, and the entire subsystem:
 – PRF_T_FLSH_MODULE
 – T_PRF_T_FLSH_NODE
 – T_PRF_T_FLSH_PORT
 – T_PRF_T_FLSH_SYSTEM
An example of querying the Spectrum Control database using native SQL with Microsoft Excel, and an example of how to render a native SQL Query in IBM Cognos is provided in Chapter 5 of IBM Spectrum Family - Spectrum Control Standard, SG24-8321.
2.5.6 TPCTOOL
You can use the TPCTOOL command-line interface (CLI) to extract data from the IBM Spectrum Control database. It requires no knowledge of the IBM Spectrum Control schema or SQL query skills, but you need to understand how to use the tool. For more information regarding the TPCTOOL, see these publications:
•IBM Tivoli Storage Productivity Center V5.2 Release Guide, SG24-8204
•Reporting with TPCTOOL, REDP-4230
To get performance data for FlashSystem 900,you can use TPCTOOL as well. Proceed as follows:
1. Start a command window (CMD).
2. From the <install>/cli directory enter tpctool -user db2admin -pwd password to get the tpctool prompt.
3. Enter lsdev -perf -l to get the GUID of devices for which performance data is collected as shown in Example 2-17.
Example 2-17 Using TPCTOOL
C:Program FilesIBMTPCcli>tpctool -user db2admin -pwd password
tpctool> lsdev -perf -l
GUID Name Label Type Status Timestamp
=================================================================================================================================================
100000051E0B2F42 100000051E0B2F42 Blueswitch-1 UNREACHABLE 2017.07.06:13:26:22
0000020060C14298+0 SVC-2145-svc_72_73-IBM SVC-2145-svc_72_73-IBM IBM SAN Volume Controller NORMAL 2017.08.18:19:48:29
00000200A1200082+0 Storwize V7000-2076-v7000_ctr_5-IBM v7000_ctr_5 IBM Storwize V7000 NORMAL 2017.08.17:21:38:16
2107.75ACA91+0 DS8000-2107-75ACA91-IBM 2107.75ACA91 IBM System Storage DS8000 NORMAL 2017.08.18:04:45:24
2810.1340008+0 XIV-2810-XIV_04_1340008-IBM XIV_04_1340008 IBM System Storage XIV NORMAL 2017.08.18:06:24:45
00000200A1C004EA+0 Storwize V7000-2076-v7000_ctr_04-IBM v7000_ctr_04 IBM Storwize V7000 NORMAL 2017.08.18:05:47:07
0000020063A20412+0 FlashSystem 840-9840-FlashSystem-840-03-IBM FlashSystem-840-03 IBM FlashSystem 840 NORMAL 2017.08.17:21:15:41
2810.1340010+0 XIV-2810-XIV_PFE02_1340010-IBM XIV_PFE02_1340010 IBM System Storage XIV NORMAL 2017.08.18:06:31:04
9835.1320902+0 FlashSystem A9000-9835-A9000R-IBM A9000R FlashSystem A9000R NORMAL 2017.08.18:12:59:26
9836.1322131+0 FlashSystem A9000-9836-A9000-IBM A9000 FlashSystem A9000 NORMAL 2017.08.17:21:44:36
00000203224045BC+0 FlashSystem V9000-9846-Cluster_Hotel-IBM PFE_V9000(Cluster_Hotel) IBM FlashSystem V9000 NORMAL 2017.08.17:22:03:21
tpctool>
4. With lstype, you get a list of available components, as shown in Example 2-18.
Example 2-18 Output of lstype command
tpctool> lstype
Name Type Description
====================================
subsystem 1 Subsystem
subsys_port 2 "HBA port"
controller 3 Controller
host_conn 4 "Host Connection"
svc_iogrp 5 "SVC I/O Group"
ds_rio 6 "RIO Loop"
stor_pool 7 "Storage Pool"
da 8 "Device Adapter"
ds_rank 9 Rank
array 10 Array
svc_mdisk 11 "SVC Managed Disk"
vol 12 Volume
switch 13 Switch
switch_port 14 "Switch Port"
svc_node 15 "SVC Node"
svc_disk 20 Disk
switch_trunk 27 "Switch Trunk"
5. Use lsmetrics shown in Example 2-19 to get the available performance metrics for the storage subsystem ports of FlashSystem 900.
Example 2-19 Output of lsmetrics command
tpctool> lsmetrics -subsys 0000020063A20412+0 -ctype subsys_port
Metric Value
==============================================
"Port Send Data Rate" 858
"Port Receive Data Rate" 859
"Port Send I/O Rate" 852
"Port Receive I/O Rate" 853
"Total Port I/O Rate" 854
"Total Port Data Rate" 860
"Port to Host Send I/O Rate" 901
"Port to Host Receive I/O Rate" 902
"Total Port to Host I/O Rate" 903
"Port to Disk Send I/O Rate" 904
"Port to Disk Receive I/O Rate" 905
"Total Port to Disk I/O Rate" 906
"Port to Local Node Send I/O Rate" 907
"Port to Local Node Receive I/O Rate" 908
"Total Port to Local Node I/O Rate" 909
"Port to Remote Node Send I/O Rate" 910
"Port to Remote Node Receive I/O Rate" 911
"Total Port to Remote Node I/O Rate" 912
"Port to Host Send Data Rate" 913
"Port to Host Receive Data Rate" 914
"Total Port to Host Data Rate" 915
"Port to Disk Send Data Rate" 916
Press Enter To Continue...
6. The getrpt command as shown in Example 2-20 lists the port sample performance data.
Example 2-20 Output of getrpt command
tpctool> getrpt -subsys 0000020063A20412+0 -ctype subsys_port -columns 852,853,854 -level sample -duration 3600 -start 2017.08.17:08:00:00 -fs ;
Timestamp;Interval;Device;Component;852;853;854
===============================================
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 1";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 2";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 1";1262;1264;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 2";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 1, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 1, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 2, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 2, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 1";1242;1250;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 2";1245;1246;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 1";1244;1247;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 2";1246;1245;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 2, Adapter 1, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 2, Adapter 1, Port 4";0;0;0
Press Enter To Continue...
1 If FlashSystem 840/FlashSystem 900 Firmware version is 1.4.5 or later.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset