Managing IBM Spectrum Scale with IBM Spectrum Control
IBM Spectrum Scale (Spectrum Scale) is a software-defined, scalable, high-performance data and file management solution (formerly IBM General Parallel File System (IBM GPFS™)).
 
Note: IBM Spectrum Scale Version 4.1 is the first version of Spectrum Scale and is based on GPFS. For versions before Version 4.1, the name GPFS is maintained.
Spectrum Scale integrated with IBM Spectrum Control (Spectrum Control) delivers scalable capacity and performance to handle the demanding data analytics. Storage administrators can combine flash, disk, cloud, and tape storage into a unified system that is high-performing and has a lower cost than traditional approaches. Integrating Spectrum Control with Spectrum Scale with business enterprises provides intelligent analytics for managing storage and the capability to monitor multiple Spectrum Scale installations (and other storage).
Spectrum Scale is a storage infrastructure that can adapt to any demanding applications across the enterprise. By including Spectrum Scale in the software-defined infrastructure (SDI), organizations can streamline data workflows, help improve service, reduce costs, manage risk, and deliver business results while positioning the enterprise for future growth.
This chapter describes the Spectrum Scale enhancements and practical troubleshooting scenarios of Spectrum Scale by using the Spectrum Control web-based GUI from a system and storage administrator perspective. In addition, this chapter describes the benefits that Spectrum Control offers beyond the management capabilities of the Spectrum Scale CLI and GUI.
7.1 Historical overview of Spectrum Scale support in Spectrum Control
Starting with Spectrum Control Version 5.2.3 (formerly known as IBM Tivoli Storage Productivity Center), when initial support was added, it gradually improved with each release. This section describes what is improved.
Enhancements in Version 5.2.3
This is the first version when support was added for GPFS. You can use the web-based GUI to add GPFS clusters and GPFS Storage Server (GSS) / Elastic Storage Server (ESS) systems for monitoring. These systems can be added as file storage systems (similar to Storwize V7000 Unified or Scale Out Network Attached Storage). The initial data collection and alerting function was enabled in this release.
The Spectrum Scale cluster can contain one of more servers, or a combination of servers and ESS systems.
The initial firmware level of Spectrum Scale supported in Spectrum Control was V4.1 and V4.1 TL1 (technical level 1 fix pack), where the cluster nodes can run on Linux Red Hat Enterprise 6.
Enhancements in Version 5.2.4
In this release, the following enhancements were made available:
•Support for all Linux platforms for Spectrum Scale nodes
•Spectrum Scale node correlation to Spectrum Control managed servers (Storage Resource Agent and agentless)
•Spectrum Scale NSD correlation to SAN storage volumes
•Status updates for Spectrum Scale nodes and NSDs
•Spectrum Scale snapshots for file systems and file sets
•Files system inode overview chart
•Capacity bars showing total/user/free space, including Spectrum Scale snapshots used space
•Alert configuration and viewing alert logs
Enhancements in Version 5.2.5
In this release, a series of enhancements in terms of usability of the Spectrum Scale and ESS were made available:
•Display for the largest and fullest pools in the overview window of a cluster
•Spectrum Scale volume mapping display enhancements
•Correlation of Spectrum Scale clusters with related fabrics and switches
•Show of file systems mounted per Spectrum Scale node
•Show quotas from Spectrum Scale file systems
•Server details window for Spectrum Scale NSDs
•Show the metadata and data space breakdown per Spectrum Scale pools
•Add inodes details
•Display file set comments
Enhancements in version 5.2.6
In this release, support for Spectrum Scale V4.1.1 was introduced with some further usability enhancements:
•New charts in the overview window showing file sets closest to running out of free inodes:
 – One view showing file sets with the least number of free inodes
 – One view showing file sets with the most number of total inodes (max inodes)
•Adding a Spectrum Scale cluster to Spectrum Control by using a non-root user.
•Spectrum Scale probe is extended to collect information about remote mounted file systems (three new columns: owning cluster, owning storage subsystem and creation name).
•Update of GPFS/GSS to Spectrum Scale (rebranding).
•AIX cluster node support.
Enhancements in Version 5.2.7
In this release, further usability enhancements were introduced:
•With Spectrum Scale V4.1.1, the node protocols information can be reviewed on the Nodes chart. A new column that is named Protocols was added, which contains comma-separated values of the enabled protocols, for example, NFS, Object, and SMB.
•Status failure is enhanced to show whether the cluster export service is unavailable on any given node, and the reason of the failure (suspended, network down, or failed).
Enhancements in Version 5.2.8
In this release, a series of high-value enhancements were introduced:
•Monitoring the performance at the cluster, node and file system level (with Spectrum Scale Version 4.x1):
 – Analyze the performance of multiple clusters with the performance troubleshooting charts to determine which clusters have the heaviest I/O loads, highest data rates, and longest response times.
 – View which node that mounts various file systems have the heaviest I/O loads and the highest CPU usage or memory usage.
 – View which file system on a cluster has the heaviest I/O loads and longest response time.
•Monitoring object storage systems:
 – View which GPFS file systems are associated with an object storage system, end to end troubleshooting by viewing the relationship between Spectrum Scale and the SAN-attached storage, and determining which object container is stored and on a file system and correlate it with back-end storage.
 – View the capacity and space usage of specific object accounts and containers.
 – Monitor object count and space quotas for containers and plan for capacity.
7.2 Troubleshooting scenarios
This section describes practical troubleshooting scenarios of Spectrum Scale by using the Spectrum Control web-based GUI. These general scenarios provide resolutions to a set of questions that an administrator of a Spectrum Scale typically must handle:
•Which clusters or nodes have health problems?
•Which clusters are running out of free space?
•Which file systems or pools are running out of capacity?
•Which file systems are mounted on which nodes?
•How much space is occupied by snapshots, and how obsolete are some of the snapshots?
•Which quotas are close to being exceeded or already are exceeded?
•Which file sets are close to running out of free inodes?
•Are all the volumes backing network-attached storage (NSDs) performing correctly?
•Which NSDs are at risk of becoming unavailable, or are unavailable?
•Is the performance of the file systems and cluster nodes satisfactory?
•Are all nodes that are fulfilling critical roles in the cluster up and running?
•Which object accounts are managing the most container and violating quotas?
•Which object container is violating the space quota?
•Send a notification when a node goes offline or file systems fill up beyond a threshold.
7.2.1 Adding Spectrum Scale files and objects to Spectrum Control
With Spectrum Control V5.2.8, there are many user interface changes. The dashboard is reorganized around the concept of storage providers and storage consumers (applications and departments). There also is a performance chart for Top Block Storage Performance, which is introduced (Figure 7-1).
Figure 7-1 Spectrum Control dashboard
The storage providers are split into three parts, mainly because of the addition of object storage support for Spectrum Scale to the existing block and file storage. The storage providers (Storage systems) section now contains the following systems:
•Block storage systems
•File storage systems
•Object storage system
The same sections can be found under the menu that is shown in Figure 7-2.
Figure 7-2 Block/File/Object storage systems
To have Spectrum Scale managed by Spectrum Control, click File, click the Object Storage System direct link, or use the top navigation menu.
If you want to add the Spectrum Scale file personality only, click File Storage Systems, click + Add Storage System, and select Spectrum Scale. Other options are Scale Out Network Attached Storage, Storwize V7000 Unified, and NetApp (Figure 7-3).
 
Note: Within the File Personality Add Device wizard, it is possible to add the object-related credentials for Spectrum Scale.
Figure 7-3 shows an example of adding the Spectrum Scale file personality.
Figure 7-3 Adding Spectrum Scale file personality
If you want to add Spectrum Scale Object personality, click Object Storage Systems, click + Add Storage System. The window where you enter the IP and Hostname / User Name / Password opens, and the Specify authentication credentials for object storage option is already selected. The Object personality requires an extra set of credentials, which are related to Keystone, which is the OpenStack identity service. For more information about the Keystone role, see the following website:
For examples about configuring OpenStack access to monitor the object storage system, and about user requirements, see the following website:
For Object Storage Systems, Spectrum Scale is the only type that is supported by Spectrum Control (Figure 7-4).
 
Note: It is not possible to add the Spectrum Scale as an Object storage system only without specifying the file side credentials.
Figure 7-4 Add Spectrum Scale object personality
This section does not cover how to add or collect asset, status, and performance data of Spectrum Scale because these tasks are similar to the other storage device types. One of the differences with Spectrum Scale is when it is added without root privileges. For more information about adding Spectrum Scale without using root privileges, see the Spectrum Control IBM Knowledge Center, found at:
Performance monitoring is available only on Spectrum Scale if the “Performance Monitoring tool” (also called ZiMon) is installed, configured, and running on the Spectrum Scale cluster, as of Spectrum Control V5.2.8. More details are available in the IBM Knowledge Center, found at:
 
Note: Performance monitoring of Spectrum Scale is supported only in Spectrum Control, starting with Version 4.1.1.
7.2.2 Usage scenarios for troubleshooting Spectrum Scale
This section adds details to the list of previously asked questions in 7.2, “Troubleshooting scenarios” on page 198. It showcases the advantages of using the Spectrum Control web-based GUI as an alternative to using the Spectrum Control command-line interface (CLI) or GUI, which are available for Spectrum Scale Version 4.2.
A storage administrator often must quickly handle a reported problem that is reported by Spectrum Control alerting (see Chapter 6, “Advanced alert management” on page 177) or by an application owner. This problem consumes Spectrum Scale resources, so determining the root cause of a problem within a short period is paramount in today’s dynamic environments.
One of the first advantages of using Spectrum Control is that a storage administrator can monitor many Spectrum Scale clusters and, when logging in to the dashboard, make an immediate assessment of the managed clusters (Figure 7-5 on page 202). The following sections describe items that the administrator should investigate.
Which clusters or nodes have health problem
Starting with Spectrum Control V5.2.8, the way the status is reported changed. The condition metric was introduced, which represents the propagated status, and the status metric represents the device status. For more information, see the Identifying problems on resources topic in the IBM Knowledge Center at the following website:
From the Storage System pane of the dashboard, you can easily tell that there are two file storage systems (out of eight) where the condition is “Error” (see Figure 7-5). Clicking the File Storage System link opens the details window of the file storage systems. One of the clusters is the one in the “Error” condition (Figure 7-6).
Figure 7-5 Spectrum Control dashboard pane of Storage Systems
Figure 7-6 Spectrum Scale cluster condition
This condition is a propagated value (which is the same case for the dashboard). So, there is a possibility that one of the internal components of the cluster has an issue. From the File Storage Systems window, you can drill into the GPFS_v41.storage.tucson.ibm.com cluster by either double-clicking the item from the table or by right-clicking and selecting View Details from the drop-down menu (Figure 7-7).
Figure 7-7 Spectrum Scale cluster nodes overlay window
As shown in Figure 7-7, the detail window of GPFS_v41.storage.tucson.ibm.com opens, with the Nodes overlay selected. In the left pane, you see a red struck circle, indicating that one of the Nodes statuses is in “Error”, and as seen in the right side table, maize8.storage.tucson.ibm.com node status is “Error”. This status shows the device status of maize8.storage.tucson.ibm.com node, which must be investigated further by logging in to one of the Spectrum Scale manager nodes.
The device status of the cluster itself is healthy, which is reported from the cluster, and despite one of the nodes being in “Error” status, the cluster continues to function because the node is not a critical node (Manager or Quorum).
Example 7-1 shows the cluster node error status.
Example 7-1 Investigate cluster node error status
[root@maize9 ~]# mmlscluster
 
GPFS cluster information
========================
GPFS cluster name: GPFS_v41.storage.tucson.ibm.com
GPFS cluster id: 9485674298140078727
GPFS UID domain: GPFS_v41.storage.tucson.ibm.com
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
Repository type: CCR
 
GPFS cluster configuration servers:
-----------------------------------
Primary server: sandman.storage.tucson.ibm.com (not in use)
Secondary server: (none)
 
Node Daemon node name IP address Admin node name Designation
-----------------------------------------------------------------------------------------------------------
8 sandman.storage.tucson.ibm.com 9.11.97.44 sandman.storage.tucson.ibm.com quorum-manager
15 maize8.storage.tucson.ibm.com 9.11.92.11 maize8.storage.tucson.ibm.com
20 tucsonx3.storage.tucson.ibm.com 9.11.97.51 tucsonx3.storage.tucson.ibm.com manager
21 maize9.storage.tucson.ibm.com 9.11.92.75 maize9.storage.tucson.ibm.com manager
 
[root@maize9 ~]# ssh [email protected]
ssh: connect to host maize8.storage.tucson.ibm.com port 22: Connection refused
As shown in Example 7-1, the mmlscluster command ran correctly, but maize8.storage.tucson.ibm.com is not reachable by SSH. This is the likely cause of why the node has an “Error” status.
 
Note: For the scenario in Example 7-1 to succeed, a connection to SSH is necessary for communication.
Now, look again at the Spectrum Scale GUI to see what the health of the cluster or node is.
 
Note: The Spectrum Scale GUI is only available starting with Version 4.2.
For more information about installing the Spectrum Scale GUI, see the IBM Knowledge Center, found at:
Figure 7-8 on page 205 shows the Spectrum Control V4.2 GUI dashboard.
Figure 7-8 Spectrum Scale V4.2 GUI dashboard
In the Spectrum Scale GUI interface dashboard section, you see that one of the nodes of the cluster is inactive (Figure 7-8 (Spectrum Control V4.2)). There is no dedicated pane for nodes as there is in Spectrum Control (Figure 7-7 on page 203 (Spectrum Control V4.1)) in the Events window. However, you can see the Critical error about the GPFS daemon on node “rice5.storage.tucson.ibm.com” being down. On this same window, you can run a fix procedure to start the daemon (Figure 7-9 (Spectrum Control V4.2)).
Figure 7-9 Spectrum Scale GUI - Events window with Run fix procedure
One of fundamental differences between the two GUIs is that you can use Spectrum Scale to change the configuration/run actions on the cluster, and Spectrum Control shows the cluster entities in a consistent manner with other storage devices that Spectrum Control manages.
Which clusters are running out of free space
Spectrum Control can manage many Spectrum Scale clusters. By looking at the File Storage System window, a storage administrator can immediately determine which cluster is running out of free space, as shown in Figure 7-10.
Figure 7-10 Spectrum Scale file system capacity
By using the File System Capacity stacked bar, you can easily grasp the usage level. Hovering your cursor over the bar shows a breakdown of the Used Space into Available Space and Total Capacity. This same information is available throughout the various columns of data from this table, which can be enabled by right-clicking the table header.
Which file systems or pools are running out of capacity
The File Storage System level File System Capacity stacked bar is an aggregated value of the File System Capacity(%) from the details window, which is shown in Figure 7-10, but by looking into the details of each File System or Storage Pool, you can determine which file system or pool is running out of capacity.
Figure 7-11 shows Spectrum Scale File Systems details window.
Figure 7-11 Spectrum Scale file systems capacities
Figure 7-11 shows that one of the file systems (gpfs2) is at 61% capacity usage, and this is likely the file system that will run out of capacity and needs attention.
Although there are no historical capacity charts for file storage system-based storage devices in Spectrum Control V5.2.8, you can access this information by opening the Overview window and reviewing the Total File System Space usage (Figure 7-12).
Figure 7-12 Spectrum Scale Overview window
The Overview window is customizable (this is a common function in the web-based GUI) and different pods with valuable information can be selected (Figure 7-12).
Switching to the Pools window (Figure 7-13), you see that there is a system pool in which all the file systems are placed.
Figure 7-13 Spectrum Scale Pools window
Spectrum Scale has two types of storage pools:
•Internal storage pools
•External storage pools
The internal storage pools are the pools that are managed by Spectrum Scale, and the external storage pools are managed by external applications such as IBM Spectrum Protect (formerly Tivoli Storage Manager).
.
Note: Spectrum Control supports only internal storage pools.
Using the Spectrum Scale CLI, you can view the same pools information by running the mmlspool command (Example 7-2).
Example 7-2 Spectrum Scale listing of file system and storage pool
[root@rice5 ~]# mmlspool gpfs2
Storage pools in file system at '/gpfs2':
Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB)
system 0 256 KB yes yes 20971520 8257792 ( 39%) 20971520 8270848 ( 39%)
However, it is not possible to list information about multiple storage pools at the same time; you can list information only for a specific file system.
The Spectrum Scale web-based GUI is organized slightly differently, that is, around capacity values. When you click Monitoring → Capacity, you can see the various entities (File System, Pools, Filesets, Use, and so on) to which capacity values can apply (Figure 7-14), although there is no dedicated window as in Spectrum Control for Pools, where some of the related details can be seen from the window itself, whether the pools contains Data, Metadata, and which NSD is being mounted.
Figure 7-14 Spectrum Scale capacity values for Pools
The Spectrum Scale GUI has the advantage of historical charting on the File System/Pools level, as shown in Figure 7-14. Historical charting is available in Spectrum Control V5.2.8 only for block-based devices.
Which file systems are mounted on which nodes
In “Which file systems or pools are running out of capacity” on page 206, you saw that the Spectrum Control web-based GUI has the overall advantage of showing related resources. For example, a storage administrator might be interested in which file systems are mounted on which nodes. You can use the Spectrum Control Nodes window to answer this question (Figure 7-15).
Figure 7-15 Spectrum Control Nodes - mounted file systems
Using the Spectrum Control CLI or operating system commands, you can determine the file systems that are mounted per node. You can also run the df command, as shown in Example 7-3.
Example 7-3 Spectrum Scale file systems mounted by node
[root@rice5 ~]# df -h
File system Size Used Avail Use% Mounted on
/dev/mapper/volGroup-volGroup--root 67G 17G 51G 24% /
devtmpfs 7.7G 0 7.7G 0% /dev
tmpfs 7.8G 64K 7.8G 1% /dev/shm
tmpfs 7.8G 7.7G 27M 100% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda2 497M 279M 219M 57% /boot
/dev/GPFS_NSD_pear_XIV_Vol10G_FS 10G 2.0G 8.1G 20% /MNTPoint_FS_GPFS_NSD_pear_XIV_Vol10G
/dev/gpfs1 32G 2.2G 30G 7% /gpfs1
/dev/gpfs2 20G 13G 7.9G 61% /gpfs2
If you use the Spectrum Scale mmdf or mmdsh commands, you must set the working collective (WCOLL) environment variable. With this command, you can see more details about a specific file system mount, as shown in Example 7-4.
Example 7-4 Spectrum Scale mmdf command
[root@rice5 ~]# mmdf gpfs1 --block-size=auto
disk disk size failure holds holds free free
name group metadata data in full blocks in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system (Maximum disk size allowed is 286 GB)
nsd1 32G 1 Yes Yes 29.81G ( 93%) 5.234M ( 0%)
------------- -------------------- -------------------
(pool total) 32G 29.81G ( 93%) 5.234M ( 0%)
 
============= ==================== ===================
(total) 32G 29.81G ( 93%) 5.234M ( 0%)
 
Inode Information
-----------------
Total number of used inodes in all Inode spaces: 5863
Total number of free inodes in all Inode spaces: 242905
Total number of allocated inodes in all Inode spaces: 248768
Total of Maximum number of inodes in all Inode spaces: 2796224
The Spectrum Scale GUI cannot show which file system is mounted per node.
How much space is occupied by snapshots and how obsolete are some of the snapshots
Spectrum Scale can create snapshots at the file system, file set, and file level to preserve the content at a single point. The snapshots at the entire file system level are known as global snapshots. These snapshots are read-only and can cause a significant impact in terms of storage space.
Some of the questions a storage administrator might have is “How much space is occupied by snapshots?” and “How obsolete are some of the snapshots?”. Spectrum Control has several windows that can answer these questions.
The first window that can help you answer these questions is the overall File Storage System window, which is found under the File System Capacity (%) stacked bar chart, as shown in Figure 7-16 on page 211.
Figure 7-16 Spectrum Control file storage system level snapshot used space
This is an aggregated value. For more information about which file systems have snapshots, the storage administrator must go to the details of the managed Spectrum Scale server, as shown in Figure 7-11 on page 206.
By viewing the details window of the managed Spectrum Scale snapshots view, the storage administrator can tell how obsolete the snapshots are either by looking at the snapshots‘ name (which contain @date as trailing characters) or by enabling the “CREATION_TIME” column, as shown in Figure 7-17.
Figure 7-17 Spectrum Control showing snapshots of Spectrum Scale
In contrast to Spectrum Control, the Spectrum Scale GUI has more capabilities and shows a richer set of information about snapshots. The snapshots table can be accessed by clicking Files → Snapshots, as shown in Figure 7-18.
Figure 7-18 Spectrum Scale web-based GUI snapshots table
Here, you can do snapshots at the file system, file set, and file level. Spectrum Control can show only file system and file set level snapshots. In terms of actions, the Spectrum Scale GUI can create snapshots and edit existing ones (Figure 7-18).
You can use the Spectrum Scale CLI to list details about one snapshot at a time, as shown in Example 7-5.
Example 7-5 Spectrum Scale mmls snapshot
[root@rice5 ~]# mmlssnapshot gpfs1
Snapshots in file system gpfs1:
Directory SnapId Status Created Fileset
[email protected] 1 Valid Fri Oct 30 11:26:00 2015 gpfs1_Fileset1
Which quotas are close to being exceeded or already are exceeded
Spectrum Scale allows definitions of quotas that can be applied to the following components:
•Individual users
•Groups of users
•Individual file sets
The quotas can be enabled either at the creation time of the file system or afterward by using the mmchfs command. You can enable quotas by using the mmquotaon command, display information about a quota by using the mmlsquota command, and edit quotas by using the mmedquota command with your default editor. You can create reports on quotas by using the mmrepquota command, as shown in Example 7-6 on page 213.
Example 7-6 Spectrum Scale mmrepquota command
[root@rice5 ~]# mmrepquota -u -v gpfs2
*** Report for USR quotas on gpfs2
Block Limits | File Limits
Name type KB quota limit in_doubt grace | files quota limit in_doubt grace entryType
root USR 4477696 0 0 0 none | 8 0 0 0 none default on
testuser USR 3004160 4194304 8388608 0 none | 3 0 0 0 none e
testuser2 USR 3004032 10485760 15728640 0 none | 3 0 0 0 none d_fsys
 
Note: The mmsetquota command can be used to set explicit quotas, for example, for specific users/groups.
For more information and examples, see the Spectrum Scale IBM Knowledge Center, found at:
For more information about quotas, see IBM Spectrum Scale (formerly GPFS), SG24-8254.
Spectrum Control also can show the quotas of Spectrum Scale through the Quotas view, as shown in Figure 7-19.
Figure 7-19 Spectrum Control quotas
A storage administrator can quickly determine whether the soft or hard limit is violated. Figure 7-19 shows the lines in the stacked bar chart, which denote the soft/hard limits on each quota. Sorting is possible on this column (by default, it is sorted on which quota limits are reached) and the capacity value can exceed 100% if quota limit checking is not activated.
In the Spectrum Scale web-based GUI, you can create or edit a quota. You can see whether a capacity value can exceed 100% by viewing the status of that quota, which is either “Enforced” or “Disabled”, and by viewing the small icon next to the Used columns for Soft % used and Hard % Used (Figure 7-20). The Quotas can be found by clicking Files → Quotas.
Figure 7-20 Spectrum Scale web-based GUI quotas
Which file sets are close to running out of free inodes
Spectrum Scale uses file sets that are defined as a subtree of a file system. In many respects, a file set behaves like an independent file system. You can use file sets to administer partitions of a file system at a granular level.
Spectrum Control can show details about the file sets, and a storage administrator can easily determine which file set is at risk of running out of inodes, as shown in Figure 7-21, by looking at the stacked bar chart, which is common through Spectrum Control table for User Inodes (%).
Figure 7-21 Spectrum Control file sets
The inodes are important when working with Spectrum Scale. An inode is a UNIX operating system data structure that contains important information about the files in the file system, which is known as metadata. Every time a file or directory is created on the file system, a new inode is allocated. If at the creation time of the file system the bytes/inode ratio is not correct, there is a risk of running out of inodes. A larger byte per ratio means that fewer inodes are created, but the typical impact of 1% of the size of the file system is higher. A smaller byte per ratio that is combined with a high number of files on the file system can lead to exhaustion of the inodes number.
Spectrum Scale also has the concept of a dependent file set, which has its inode space allocated from an independent file set, which means it many dependent file sets, which also can lead to depletion of the inodes. The advantages of using dependent file sets are that at snapshot time, a snapshot can be created for an independent file set that includes all the dependent file sets.
The Spectrum Control Overview window has a menu that you can use to determine which independent file sets have the fewest available inodes, as shown in Figure 7-22.
Figure 7-22 Spectrum Control Overview window - inodes by independent file set
Using the Spectrum Scale web-based GUI, you can see the same information presented by Spectrum Control in Figure 7-21 on page 214. The only difference is that you can use the Spectrum Scale web-based GUI to create and modify the file sets. You can access the file sets by using the left menu pane and clicking Files → Filesets, as shown in Figure 7-23.
Figure 7-23 Spectrum Scale GUI file sets
By using the Spectrum Scale CLI, you can run the mmlsfileset command (Example 7-7) to show a specific file set that is related to a file system; to show the inode usage, run the mmdf command.
Example 7-7 Spectrum Scale mmlsfileset command
[root@rice5 ~]# mmlsfileset gpfs1 -d --block-size auto
Collecting fileset usage information ...
Filesets in file system 'gpfs1':
Name Status Path Data
root Linked /gpfs1 1.112G
gpfs1_Fileset1 Linked /gpfs1/Fileset1 256K
gpfs1_Fileset1_Subfileset1 Linked /gpfs1/Fileset1/Subfileset1 256K
objectfset Linked /gpfs1/objectfset 2.016M
obj_test-policy Linked /gpfs1/obj_test-policy 1007M
Are the volumes that are backing up the network shared disks performing correctly
Figure 7-24 on page 217 shows the scenario when a performance problem must to be investigated and the Spectrum Scale topology has the nodes that are directly attached with SAN storage.
Figure 7-24 Spectrum Scale with direct attached storage (SAN)
Spectrum Scale can show details about the network shared disks (NSDs). These NSDs provide a block-level interface over TPC/IP network. The mounted file system looks the same to the users and applications (Spectrum Control handles the I/O request transparently), as shown in Figure 7-25.
Figure 7-25 Spectrum Control network shared disks
Spectrum Control is useful in Spectrum Scale environments because it has visibility into the SAN storage. The performance of the block storage backing the NSD volumes can be investigated through the related resources area (Figure 7-26) for cases where the back-end storage is monitored by Spectrum Control.
Figure 7-26 Spectrum Control related resources - backing volumes
Spectrum Control automatically can correlate the monitored storage devices with the LUNs that are assigned to be used by Spectrum Scale. To do so, use the web-based GUI and go to Related Resources (arrow 1 in Figure 7-26). Here, you can see details about the fabrics and switches through which the SAN storage is connected.
The performance charts can be displayed for either the Volume Mappings or for the related storage systems by using the Performance tab. These performance charts display the typical metrics that you can find for the block storage devices, which by default are “Overall I/O Rate” and “Response Time”.
The Storage System tab along with the “Condition” status can help you answer the next question a storage administrator wants to know: Which NSDs are at risk of becoming unavailable? Whenever the backing storage condition is in the “Error” state, this can indicate that the storage device is at risk of becoming unavailable and requires attention.
In addition to the block-level performance, Spectrum Control can collect performance information for the file side of Spectrum Scale, which is described in “Is the performance of the file systems and cluster nodes satisfactory” on page 219.
 
Note: The Spectrum Scale GUI cannot report on the SAN-attached storage performance.
NSDs are also at the risk of becoming unavailable whenever the mounting servers (nodes) lose connectivity with the NSD. A node typically serves multiple NSDs. In the Nodes window, the user can easily determine whether any of the NSDs are at risk by viewing the NSDs Served column. When you open the property notebook, you see the list of NSDs Served along with the “Connectivity” column, which shows “Connected” if the status of the connectivity is good (Figure 7-27 on page 219).
Figure 7-27 Status of NSDs served along with the Connectivity column
Is the performance of the file systems and cluster nodes satisfactory
As stated in “Enhancements in Version 5.2.8” on page 197, staring with Spectrum Control V5.2.8, monitoring support for performance is available for Spectrum Scale. The performance chart is available for file systems and nodes.
Using these performance charts, a storage administrator can easily determine whether things are out of balance, or compare the performance across multiple clusters. Using this information, an administrator can understand which of the clusters are doing most of the workloads, which helps them plan.
Figure 7-28 shows an example of a Spectrum Control performance chart.
Figure 7-28 Spectrum Control file stem performance chart
Figure 7-28 on page 219 shows the performance of the three file systems that are managed by the GPFS_v42 cluster. The metrics that are supported for file systems can be shown for Read/Write or Total/Overall and are as follows:
•Overall I/O Rate (ops/s)
•Data Rate (MiB/s)
•Response Time (ms/op)
Figure 7-29 shows the performance of the two nodes that make up the GPFS_v42 cluster. Here are the metrics that are supported for the nodes:
•System CPU Utilization (%), User CPU Utilization (%), and Total CPU Utilization (%)
•Memory Utilization (%), Cache and Buffer Utilization (%), and Total Memory Utilization (%)
Figure 7-29 Spectrum Control Nodes performance chart
In Figure 7-29, the CPU utilization for rice5 cluster node hovered around 50% and the pear cluster node is at 2-3% in the last 12 hours. This usage is similar to a longer period in the past. This metric suggests that either the rice5 node hardware is potentially becoming obsolete and performance is marginal, or the workload is not balanced across the cluster nodes and some configuration adjustment is necessary. Another possibility is that some user process on rice5 is consuming more processor than the system processes. This can be a sign of an application of which the storage administrator is unaware, but if you use the charts that are offered by Spectrum Control, this consumption is easy to detect.
The Spectrum Scale GUI offers performance charts, which can be viewed in the following ways:
•Viewing them on the dashboard
•Clicking Monitoring → Performance
On the dashboard, there is a chart with two views showing the throughput and the top node by system load (Figure 7-30 on page 221).
Figure 7-30 Spectrum Scale GUI performance charts
In the Monitoring → Performance window, the number of views is configurable. You can show either one or two views with various resources, the aggregation level, and metrics, as shown in Figure 7-31.
Figure 7-31 Spectrum Scale GUI performance monitoring charts
Resources types can be the following ones:
•Network (InfiniBand or Ethernet)
Aggregation level:
 – Node
 – Adapter per node
 – Adapter
 – Cluster
•System resources
Aggregation level:
 – Node
 – Cluster
•Spectrum Scale Server
Aggregation level:
 – NSD
 – File System
 – Pool
 – Cluster
•Spectrum Scale Client
Aggregation level:
 – File System
 – Cluster
The metrics that are available for each resource type are much richer than what Spectrum Control has available. You can find more information about the metrics in the Spectrum Scale IBM Knowledge Center, found at:
For more information about performance monitoring by using the Spectrum Scale GUI, see the Spectrum Scale IBM Knowledge Center, found at:
Are all nodes that fulfill critical roles in the cluster running
A node in the Spectrum Scale cluster is considered to be any server on which the product is installed with direct storage access or network access to another node. Depending on the type of access, each node can have different roles. Some of the nodes are essential t o the normal functioning of the cluster (cluster manager, quorum node, file system manager, or meta-node), and other nodes, such as the application node, have a smaller role, but as a storage administrator, you want to make sure that all nodes fulfilling critical roles have a good health status.
The Spectrum Control Nodes window is shown in Figure 7-32.
Figure 7-32 Spectrum Control nodes window
Figure 7-32 on page 222 shows the maize8 node in “Error” status, but all the rest of the nodes fulfilling critical roles (Manager, Quorum) are in “Normal” status, so the Spectrum Scale cluster continues to function correctly.
Which object accounts manage the most containers and are violating quotas
Starting with Spectrum Control V5.2.8, the Spectrum Scale object personality is supported and you have visibility into object accounts and containers. One task a storage administrator must deal with is which accounts are managing the most containers or which quotas are being violated.
 
Note: Object/Swift has its own set of quotas, that is, these are not GPFS quotas.
For more information about installing Object Storage (Swift) manually and quotas, see the IBM Cloud Manager with OpenStack 4.3.0 IBM Knowledge Center, found at:
In Figure 7-33, you can sort the table on the Containers column and concurrently see Space Quota (%), which is in violation of the administrator account.
You can use Spectrum Scale GUI to display Accounts-related information by going to the left pane and clicking Object → Accounts. The Object menu shows users and roles that are not being shown as views in Spectrum Control. Spectrum Scale can create or edit the users and roles.
Figure 7-33 Spectrum Control Object accounts
Which object containers are violating space quotas
After a storage administrator determines which accounts are violating the space quota, the next question is which containers are violating space quotas or object counts? Spectrum Control can display information about containers, as shown in Figure 7-34.
Figure 7-34 Spectrum Control displays information about containers
The Spectrum Scale GUI does not have a view around containers, but you can see them by using the Accounts view.
Being notified when a node goes offline or a file system violates a threshold
The alerting capabilities were improved in Spectrum Control V5.2.8. You can define alerts and be notified when configuration or capacity violations occur.
 
Note: As of Spectrum Control V5.2.8, threshold alerts are not supported for Spectrum Scale performance.
For more information about these alert improvements, see the IBM Knowledge Center, found at:
This section describes how to define a warning severity alert for when a Spectrum Scale node goes offline or the file system fills beyond 40%. Complete the following steps:
1. Open the Alerts definition window, where you can set alerts for a series of metrics. Click NAS Node and then the General tab, as shown in Figure 7-35 on page 225.
Figure 7-35 Spectrum Scale alerts definition for nodes
2. Select is error for the Status, and for severity, select the “error” icon, as shown in Figure 7-35.
3. Click File Systems and then the Capacity tab and look for the Used Space % metric. Change the alert to Enabled status by using the switch. Enter “>=” and “40%”, and for the severity, set the “warning” icon for an early alert. Select + to double the metric, enter “>=” and “80%”, and select the “error” icon for severity. Click Save, as shown in Figure 7-36.
Figure 7-36 Spectrum Control alert definition for file systems
When you run the next probe, check whether the alert is being triggered for any of your nodes or file systems, as shown in Figure 7-37.
Figure 7-37 Spectrum Control Alert for File System violation triggered
As shown in Figure 7-37, the alert for the File System Used Space Percent for the “>= 40%” setting was triggered and the severity of the alert is “warning”.
Spectrum Scale GUI has a limited event monitoring capability. It is not possible to define any alerts and triggering conditions, but you can use it to run fix procedures directly from the GUI, as shown in Figure 7-9 on page 205.
7.3 Advantages of using the Spectrum Control web-based GUI to manage Spectrum Scale clusters
Using the Spectrum Control web-based GUI to manage Spectrum Scale clusters has the following advantages:
•Monitoring of multiple Spectrum Scale clusters and keeping historical data for an extended period
•Comparing performance across all the managed Spectrum Scale clusters, allowing side-by-side comparison at various levels (node/file system) and taking advantage of all the charting features of Spectrum Control (time sync and table view)
•Correlating the SAN-attached storage and performance charting of the block level LUNs
•Showing the related resources, such as back-end storage systems, switches, and fabrics
•Comprehensive set of alerting capabilities
•Allows grouping of resources into applications and departments
•Health status information at the resource level
7.4 Advantages of using the Spectrum Scale GUI to manage Spectrum Scale
Using the Spectrum Scale GUI to manage an individual Spectrum Scale cluster has the following advantages:
•Monitoring only a single cluster at a time, but has a richer set of functions and details
•Monitoring performance on a larger set of metrics, including the networking layer for InfiniBand or Ethernet
•Visualizing the internal details and changing the existent configuration
•Display of events and running fix procedures in place
•Display of topological information
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset