Test scenarios
This chapter provides test scenarios that are performed in an example non-production environment.
The following topics are described in this chapter:
5.1 Test environment
Figure 5-1 shows the environment that is used for the test scenarios in this chapter.
Figure 5-1 Environment that is used for the test scenarios
The environment is composed of the following items:
•One KSYS
•Two IBM Power Systems S822 servers: S822-1 and S882-2
•Two Hardware Management Consoles (HMCs): HMC1 and HMC2
•One IBM Storwize V7000 that is used for shared storage
Example 5-1 shows the KSYS cluster that is created.
Example 5-1 Listing the KSYS cluster
p18ksys:/ # ksysmgr query cluster
Name: ksyscluster
State: Online
Type: HA
The HMCs that are used in this environment are listed in Example 5-2.
Example 5-2 Listing the HMCs
p18ksys:/ # ksysmgr query hmc
Name: hmc2
Ip: 129.40.180.25
Login: hscroot
Managed Host List:
Host Name Uuid
========= ====
Server-8284-22A-SN10EE85P fc68b423-a590-3db7-86e9-a7b7d26a07f0
============================================================================
Name: hmc1
Ip: 129.40.180.24
Login: hscroot
Managed Host List:
Host Name Uuid
========= ====
Server-8284-22A-SN101AFDR 07e502a4-5051-3867-bd64-11adc2fe8e68
============================================================================
The host group that is used in this scenario is shown in Example 5-3.
Example 5-3 Listing the host group
p18ksys:/ # ksysmgr query host_group
Name: HG_TCHU
Hosts: Server-8284-22A-SN101AFDR
Server-8284-22A-SN10EE85P
Memory_capacity: Priority Based Settings
low:100
medium:100
high:100
CPU_capacity: Priority Based Settings
low:100
medium:100
high:100
Skip_power_on: No
HA_monitor: enable
Restart_policy: auto
VM_failure_detection_speed: normal
Host_failure_detection_time: 90
SSP Cluster Attributes
Sspname: KSYS_ksyscluster_1
Sspstate: UP
Ssp_version: VIOS 3.1.0.11
VIOS: p18v02
p18v01
p18v04
p18v03
5.2 Linux virtual machine failures
This scenario simulates a Linux virtual machine (VM) failure that is managed by IBM VM Recovery Manager High Availability (HA).
Complete the following steps:
1. Check where the VM that is used in the scenario is allocated. Example 5-4 shows VM p18lnx02 is on host Server-8284-22A-SN10EE85P before you start the Linux VM failure.
Example 5-4 Listing the location of VM p818nx02
p18ksys:/ # ksysmgr query vm p18lnx02
Name: p18lnx02
UUID: 495778C3-80DB-4FE3-989D-E38C905450E8
State: VERIFY
Host: Server-8284-22A-SN101AFDR
Priority: High
VM_failure_detection_speed: fast
HA_monitor: enable
Homehost: Server-8284-22A-SN10EE85P
VM_status: NO_OPERATION_IN_PROGRESS
Version_conflict: No
2. List all VMs that are allocated to host Server-8284-22A-SN101AFDR, as shown in Example 5-5.
Example 5-5 Listing VMs that are allocated on host Server-8284-22A-SN101AFDR
hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
p18lnx02
3. List all VMs that are allocated to host Server-8284-22A-SN10EE85P, as shown in Example 5-6.
Example 5-6 Listing VMs that are allocated to host Server-8284-22A-SN10EE85P
hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01
p18v02
p18lnx01
4. Check the version of Linux in VM p18lnx02, as shown in Example 5-7.
Example 5-7 Checking the Linux version on VM p18lnx02
p18lnx02:/etc # cd ..
p18lnx02:/ # cat /etc/os-release
NAME="SLES"
VERSION="15"
VERSION_ID="15"
PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"
5. Simulate a Linux kernel crash, as shown in Example 5-8.
Example 5-8 Simulating a Linux crash
p18lnx02:/ # sh -x system_crash.sh
kernel.panic=0
+ echo c
sysrq: SysRq : Trigger a crash
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048
NUMA
pSeries
.
.
.
Kernel panic - not syncing: Fatal exception
---[ end Kernel panic - not syncing: Fatal exception
On HMC p18vhmc1, VM p18lnx02 changed the reference code, as shown in Example 5-9.
Example 5-9 VM p18lnx02 reference code after a Linux kernel crash
p18lnx02:B200A101 LP=00009
Example 5-10 shows that in HMC p18vhmc2 VM p18lnx02 restarted.
Example 5-10 VM p18lnx02 initializing on Server-8284-22A-SN10EE85P
p18lnx02:CA00E140
6. Monitor the KSYS to see the VM restart of p18lnx0, as shown in Example 5-11.
Example 5-11 Monitoring the VM restart
p18ksys:/ # ksysmgr query system status monitor=yes
Host_group HG_TCHU is currently in Ready state
Press Q to quit monitoring for activity
Restart in progress for Host_group HG_TCHU
Stopping HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 stopped
Shutdown has started for VM p18lnx02
Shutdown has completed for VM p18lnx02
Restart has started for VM p18lnx02
Starting HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18lnx02
7. Check the status of VM p18lnx02 on host Server-8284-22A-SN10EE85P, as shown in Example 5-12.
Example 5-12 Listing VM s on host Server-8284-22A-SN10EE85P
hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01
p18v02
p18lnx01
p18lnx02
After the final restart of VM p18lnx02, the reference on host Server-8284-22A-SN101ADFDR is cleared, as shown in Example 5-13.
Example 5-13 Listing VMs on host Server-8284-22A-SN101AFDR
scroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
Example 5-14 Listing VM p18lnx02
p18ksys:/ # ksysmgr query vm p18lnx02
Name: p18lnx02
UUID: 495778C3-80DB-4FE3-989D-E38C905450E8
State: READY
Host: Server-8284-22A-SN10EE85P
Priority: High
VM_failure_detection_speed: fast
HA_monitor: enable
Homehost: Server-8284-22A-SN10EE85P
VM_status: NO_OPERATION_IN_PROGRESS
Version_conflict: No
LPM Validation Status
LPM validation was successful for Hosts:
Server-8284-22A-SN10EE85P
5.3 AIX VM failures
This scenario simulates an AIX VM failure when managed by VM Recovery Manager HA.
Complete the following steps:
1. Check where the VM that is used for this scenario is allocated. Example 5-15 shows that VM p18aix02 is on host Server-8284-22A-SN10EE85P before you start the VM failure.
Example 5-15 Listing VM p18aix02
p18ksys:/ # ksysmgr query vm p18aix02
Name: p18aix02
UUID: 52138D9D-034C-4114-AA45-4E2DA66D8761
State: VERIFY
Host: Server-8284-22A-SN101AFDR
Priority: Medium
VM_failure_detection_speed: normal
HA_monitor: enable
Homehost: Server-8284-22A-SN10EE85P
VM_status: NO_OPERATION_IN_PROGRESS
Version_conflict:
2. List all VMs that are allocated on host Server-8284-22A-SN101AFDR, as shown in Example 5-16.
Example 5-16 Listing VMs that are allocated on host Server-8284-22A-SN101AFDR
hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
3. List all VMs that are allocated on host Server-8284-22A-SN10EE85P, as shown in Example 5-17.
Example 5-17 Listing VMs that are allocated on host Server-8284-22A-SN10EE85P
hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01:
p18v02:
p18lnx01:Linux ppc64le
p18lnx02:Linux ppc64le
4. Check the version of AIX from VM p18aix02, as shown in Example 5-18.
Example 5-18 Checking the AIX version on VM p18aix02
p18aix02:/ # oslevel -s
7200-03-02-1846
5. Simulate the AIX kernel crash, as shown in Example 5-19.
Example 5-19 Simulating an AIX kernel crash
p18aix02:/vm_crash # sh -x aix_crash.sh
+ + hostname -s
shost=p18aix02
+ PS1=p18aix02:$PWD #
+ alias __A=
+ alias __B=
+ alias __C=
+ alias __D=
+ alias __H=
.
.
.
+ ./crash_vm -c
Crash ..
Uhh.. I am crashing
crash
Illegal Trap Instruction Interrupt in Kernel
.panic_trap+000000 tweq r14,r14 r14=0000000000000002
KDB(0)> PuTTY
On HMC p18vhmc1, VM p18lnx02 changed the reference code, as shown in Example 5-20.
Example 5-20 VM p18aix02 reference code after a kernel crash
p18aix02:0c20
On HMC p18vhmc2, VM p18lnx02 restarted, as shown in Example 5-21.
Example 5-21 VM p18aix02 initializing on Server-8284-22A-SN10EE85P
p18aix02:CA00E140
6. Monitor the KSYS to see the restart of VM p18lnx02, as shown in Example 5-22.
Example 5-22 Monitoring the VM restart
p18ksys:/ # ksysmgr query system status monitor=yes
Host_group HG_TCHU is currently in Ready state
Press Q to quit monitoring for activity
Restart in progress for Host_group HG_TCHU
Stopping HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 stopped
Shutdown has started for VM p18aix02
Shutdown has completed for VM p18aix02
Restart has started for VM p18aix02
Starting HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix02
Configuration cleanup started for VM p18aix02
VM monitoring for VM p18aix02 started
Configuration cleanup completed for VM p18aix02
1 out of 1 VMs have been successfully restarted
After the final restart of VM p18aix02, the reference code on host Server-8284-22A-SN101ADFDR is cleared, as shown in Example 5-23.
Example 5-23 Listing the VMs on host Server-8284-22A-SN101AFDR
scroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix03
p18aix04
p18aix05
7. Check the home host of VM p18lnx02, as shown in Example 5-24.
Example 5-24 Listing VM s on host Server-8284-22A-SN10EE85P
scroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01:
p18v02:
p18aix02:
p18lnx01:Linux ppc64le
p18lnx02:Linux ppc64le
Example 5-25 Listing VM p18aix02
p18ksys:/ # ksysmgr query vm p18aix02
Name: p18aix02
UUID: 52138D9D-034C-4114-AA45-4E2DA66D8761
State: READY_TO_MOVE
Host: Server-8284-22A-SN10EE85P
Priority: Medium
VM_failure_detection_speed: normal
HA_monitor: enable
Homehost: Server-8284-22A-SN10EE85P
VM_status: NO_OPERATION_IN_PROGRESS
Version_conflict: No
LPM Validation Status
LPM validation was successful for Hosts:
Server-8284-22A-SN10EE85P
5.4 Host failures
This scenario simulates a host crash of host Server-8284-22A-SN101AFDR.
Complete the following steps:
1. List the VMs that are available on host Server-8284-22A-SN101AFDR, as shown in Example 5-26.
Example 5-26 Listing VMs that are available on host Server-8284-22A-SN101AFDR
hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name:refcode
p18v03:
p18ibmi01:00000000
p18v04:
p18aix02:
p18aix03:
p18aix04:
p18aix05:
p18lnx02:Linux ppc64le
2. List the VMs that are available on host Server-8284-22A-SN10EE85P, as shown in Example 5-27.
Example 5-27 Listing VMs that are available on host Server-8284-22A-SN10EE85P
hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name:refcode
p18v01:
p18v02:
p18lnx01:Linux ppc64le
3. Simulate a host crash by shut downing immediately both Virtual I/O Servers (VIOSs) from host Server-8284-22A-SN10EE85P, as shown in Example 5-28.
Example 5-28 Shutting down the VIOSs from host Server-8284-22A-SN10EE85P
chsyscfg chsyspwd chsysstate
hscroot@p18vhmc1:~> chsysstate -m Server-8284-22A-SN101AFDR -r lpar -n p18v03 -o shutdown --immed
hscroot@p18vhmc1:~> chsysstate -m Server-8284-22A-SN101AFDR -r lpar -n p18v04 -o shutdown --immed
All VMs from host Server-8284-22A-SN10EE85P have crashed, as shown in Example 5-29.
Example 5-29 VMs crashed from host Server-8284-22A-SN10EE85P
p18ibmi01:A6040266
p18v04:00000000
p18aix02:0c20
p18aix03:CA00E175
p18aix04:CA00E175
p18aix05:CA00E175
p18lnx02:00000000
4. Check the KSYS by running the ksysmgr command. All monitored VMs restarted in host Server-8284-22A-SN10EE85P, as shown in Example 5-30.
Example 5-30 VM restarting in host Server-8284-22A-SN10EE85P
p18ksys:/ # ksysmgr query system status monitor=yes
Host_group HG_TCHU is currently in Ready state
Press Q to quit monitoring for activity
Restart in progress for Host_group HG_TCHU
Stopping HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 stopped
Shutdown has started for VM p18lnx02
Shutdown has completed for VM p18lnx02
Restart has started for VM p18lnx02
Starting HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18lnx02
Stopping HA monitoring for VM p18aix03
Stopping HA monitoring for VM p18aix04
HA monitoring for VM p18aix04 stopped
Stopping HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 stopped
Stopping HA monitoring for VM p18aix05
Shutdown has started for VM p18ibmi01
Starting VM monitoring for VM p18lnx02
HA monitoring for VM p18aix03 stopped
HA monitoring for VM p18aix05 stopped
Shutdown has completed for VM p18ibmi01
Restart has started for VM p18ibmi01
Shutdown has started for VM p18aix04
Shutdown has started for VM p18aix02
Shutdown has started for VM p18aix05
Shutdown has started for VM p18aix03
Shutdown has completed for VM p18aix04
Shutdown has completed for VM p18aix02
Restart has started for VM p18aix04
Restart has started for VM p18aix02
Shutdown has completed for VM p18aix05
Restart has started for VM p18aix05
Starting HA monitoring for VM p18aix04
HA monitoring for VM p18aix04 started
Starting HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 started
Starting HA monitoring for VM p18aix05
HA monitoring for VM p18aix05 started
Shutdown has completed for VM p18aix03
Restart has started for VM p18aix03
Starting HA monitoring for VM p18aix03
HA monitoring for VM p18aix03 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18ibmi01
Configuration cleanup started for VM p18ibmi01
ERROR: Restart has encountered an error for VM p18ibmi01 during Configuration Cleanup
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix02
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix04
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix05
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix03
Configuration cleanup started for VM p18aix02
VM monitoring for VM p18aix02 started
Configuration cleanup started for VM p18lnx02
ERROR: Restart has encountered an error for VM p18lnx02 during Configuration Cleanup
ERROR: Restart has encountered an error for VM p18aix02 during Configuration Cleanup
Configuration cleanup started for VM p18aix04
VM monitoring for VM p18aix04 started
ERROR: Restart has encountered an error for VM p18aix04 during Configuration Cleanup
Configuration cleanup started for VM p18aix03
Configuration cleanup started for VM p18aix05
VM monitoring for VM p18aix03 started
VM monitoring for VM p18aix05 started
ERROR: Restart has encountered an error for VM p18aix03 during Configuration Cleanup
ERROR: Restart has encountered an error for VM p18aix05 during Configuration Cleanup
6 out of 6 VMs have been successfully restarted
5. List the VMs on host Server-8284-22A-SN10EE85P. You see that all partitions that are monitored from the KSYS from the crashed host Server-8284-22A-SN101AFDR restarted on host Server-8284-22A-SN10EE85P, as shown in Example 5-31.
Example 5-31 Listing VMs on host Server-8284-22A-SN10EE85P
hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name:refcode
p18v01:
p18ibmi01:00000000
p18v02:
p18aix02:
p18aix03:
p18aix04:
p18aix05:
p18lnx01:Linux ppc64le
p18lnx02:Linux ppc64le
6. List the VMs on host Server-8284-22A-SN101AFDR. You see that the referenced VMs restarted, as shown in Example 5-32.
Example 5-32 Listing the remaining referenced VMs
hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name:refcode
p18v03:00000000
p18ibmi01:00000000
p18v04:00000000
p18aix02:00000000
p18aix03:00000000
p18aix04:00000000
p18aix05:00000000
p18lnx02:00000000
7. Start the VIOS from host Server-8284-22A-SN101AFDR, as shown in Example 5-33.
Example 5-33 Starting the VIOS from host Server-8284-22A-SN101AFDR
hscroot@p18vhmc1:~> chsysstate -m Server-8284-22A-SN101AFDR -r lpar -n p18v03 -o on -f default_profile
hscroot@p18vhmc1:~> chsysstate -m Server-8284-22A-SN101AFDR -r lpar -n p18v04 -o on -f default_profile
8. Perform discovery and verification on host group HG_TCHU, as shown in Example 5-34.
Example 5-34 Performing discovery and verification on host group HG_TCHU
p18ksys:/ # ksysmgr discovery host_group HG_TCHU verify=yes
Running discovery on Host_group HG_TCHU, this may take few minutes...
Existing HA trunk adapter found for VIOS p18v01
Existing HA trunk adapter found for VIOS p18v02
Existing HA trunk adapter found for VIOS p18v04
Existing HA trunk adapter found for VIOS p18v03
Preparing VIOS in Server-8284-22A-SN101AFDR for HA management
VIOS in Server-8284-22A-SN101AFDR prepared for HA management
Preparing VIOS in Server-8284-22A-SN10EE85P for HA management
VIOS in Server-8284-22A-SN10EE85P prepared for HA management
Preparing VM p18aix03 in Host Server-8284-22A-SN10EE85P for HA management
VM p18aix03 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Preparing VM p18aix04 in Host Server-8284-22A-SN10EE85P for HA management
VM p18aix04 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Preparing VM p18lnx02 in Host Server-8284-22A-SN10EE85P for HA management
VM p18lnx02 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Preparing VM p18aix02 in Host Server-8284-22A-SN10EE85P for HA management
VM p18aix02 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Preparing VM p18aix05 in Host Server-8284-22A-SN10EE85P for HA management
VM p18aix05 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Existing first HA client adapter found for VM p18lnx02
Existing first HA client adapter found for VM p18aix03
Existing second HA client adapter found for VM p18aix03
Existing first HA client adapter found for VM p18aix04
Existing second HA client adapter found for VM p18aix04
Existing second HA client adapter found for VM p18lnx02
Existing first HA client adapter found for VM p18aix02
Existing second HA client adapter found for VM p18aix02
Existing first HA client adapter found for VM p18aix05
Existing second HA client adapter found for VM p18aix05
Skipping VM p18lnx02 on Host Server-8284-22A-SN10EE85P due to some other process in progress
Discovery has started for VM p18aix03
Configuration information retrieval started for VM p18aix03
Discovery has started for VM p18ibmi01
Configuration information retrieval started for VM p18ibmi01
Discovery has started for VM p18aix04
Configuration information retrieval started for VM p18aix04
Discovery has started for VM p18aix02
Configuration information retrieval started for VM p18aix02
Discovery has started for VM p18aix05
Configuration information retrieval started for VM p18aix05
Configuration information retrieval completed for VM p18aix03
Discovery for VM p18aix03 is complete
Configuration information retrieval completed for VM p18ibmi01
Discovery for VM p18ibmi01 is complete
Configuration information retrieval completed for VM p18aix04
Discovery for VM p18aix04 is complete
Configuration information retrieval completed for VM p18aix02
Discovery for VM p18aix02 is complete
Configuration information retrieval completed for VM p18aix05
Discovery for VM p18aix05 is complete
VM monitor state has moved to 'STARTED' for VM p18aix03
VM monitor state has moved to 'STARTED' for VM p18aix04
VM monitor state has moved to 'STARTED' for VM p18aix02
VM monitor state has moved to 'STARTED' for VM p18aix05
Discovery has finished for HG_TCHU
1 managed VM has been skipped for discovery
5 out of 6 managed VMs have been successfully discovered
Host_group verification started for HG_TCHU
p18aix03 verification has started
p18ibmi01 verification has started
p18aix04 verification has started
p18lnx02 verification has started
p18aix02 verification has started
p18aix05 verification has started
p18ibmi01 verification has completed
p18lnx02 verification has completed
p18aix05 verification has completed
p18aix04 verification has completed
p18aix02 verification has completed
p18aix03 verification has completed
Verification has finished for HG_TCHU
6 out of 6 VMs have been successfully verified
After the discovery and verification of host group HG_TCHU, the VMs that are referenced on host Server-8284-22A-SN101AFDR are cleared, as shown in Example 5-35.
Example 5-35 Listing VMs on host Server-8284-22A-SN101AFDR
hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name:refcode
p18v03:
p18v04:
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.