Automation
This appendix provides an overview of the automation capabilities of CSM session actions by using the CSM Scheduled Task feature.
We introduce the action types that can be defined in a CSM scheduled task and discuss how specific CSM 4-site session scenarios can be automated with a task, which can be started on-demand or scheduled.
This appendix includes the following topics:
A.1 CSM scheduled task
CSM 6.2.1 introduced the Scheduled Task feature to create tasks that can be run on-demand or at regular schedules. In the original release, only tasks with Flash command actions were defined for sessions that supported the Flash command. In CSM 6.2.2, the scheduled task feature was significantly enhanced to allow the following support:
•Multiple actions per task
•All session types
•All session actions for a command action
•New action type to wait for a specific session state
With those enhancements, much more complex session scenarios can be defined as a task and therefore, allow simplified runs of repetitive session sequences, which optionally can be scheduled for automated and regular run.
CSM scheduled tasks can be created and modified through the CSM GUI only. CSM 6.2.3 added CSMCLI support to list and run defined scheduled tasks by way of the lstasks and runtask commands.
|
Note: The CSM scheduled task capabilities might be further enhanced in future CSM releases to allow task run progress reporting and more action types. This chapter describes the scheduled task capabilities as available with CSM 6.2.3.
|
A scheduled task in CSM can be run against single or multiple sessions. The task can be scheduled to run at defined intervals, or if the scheduled task is deactivated, it can also be run on demand only. If you want to create a scheduled task that should be run on-demand only, you can specify a dummy interval if the task schedule is mandatory, or select the No Schedule option if available in the task (depends on CSM release). A created task runs automatically only after it was specifically enabled for execution.
For more information about how you can create scheduled tasks, see IBM Knowledge Center.
A single task can act on multiple CSM sessions, run multiple commands, and wait for specific states after each command before proceeding to the next step. The only limitation of scheduled tasks is that you cannot define conditional based execution. Therefore, you cannot define automated actions to be taken if certain steps of the task fail. In case of an error or timeout in a single step, the entire task execution is stopped with an error message.
The following action types can be defined within a task:
•Command action (CSM 6.2.1 or higher)
This action type runs the selected session command against the selected session. If the command fails, the task execution stops with an error message.
•Wait For State action (CSM 6.2.3 or higher)
This action type waits with further processing until the selected session reaches the selected state. You can specify a timeout in minutes for the maximum time to wait. If the state is not reached within the timeout, the action fails and the task execution stops with an error message.
A.1.1 Create a task with multiple actions
To create a scheduled task, click Settings → Scheduled Tasks. In the Scheduled Tasks window, click Create Task... The scheduled task wizard opens, as shown in Figure A-1.
Figure A-1 Create a scheduled task wizard
In the first task window, you can define the task name, enter a more detailed description about the purpose of the task, and optionally enable the creation of a CSM server PE package if the task fails at any step. Click Next to proceed to the schedule window, as shown in Figure A-2.
Figure A-2 Create a schedule for the task
Specify the required schedule for your task. This schedule can be a regular interval per day, or a defined time at specific weekdays. If your task should not run automatically, select No Schedule in this window. Click Next to proceed to the Actions window.
Figure A-3 Task Actions window
In the Actions window, you can add any number of actions for the task. You can define session Command actions, or Wait for State actions. To add an action, click Add Action, which opens the Add Action wizard.
To create a session command action, select type Command; then, select the session and the command that you want to run against the selected session. Only commands that are supported by the selected session type are listed in the command list. If you must define multiple actions to specific sessions only, you can use the filter option for your session name to shorten the session list. A selected session remains preselected on subsequent actions that are being created.
Figure A-4 shows an example to run a SuspendH1H3 command against our 4-site session.
Figure A-4 Add session command action
Click OK to create the action and return to the Actions window.
Next, we add an action to wait for the session that is being Suspended (Partial). Click Add Action to add a Wait for State action for the same session.
In each Wait for State action, you must specify a timeout in minutes. The timeout is the maximum time that this step waits for the wanted state that is being reached. If the timeout is exceeded, the step fails and the entire task stops with an error. Select a suitable wait timeout for a previous command action to be completed. Figure A-5 shows an example to wait up to 3 minutes for the session reaching the Suspended (Partial) state after the previous SuspendH1H3 command.
Figure A-5 Add Wait For State action
Click OK to add this second action to the task.
In the Actions window, you can use the Move Up and Move Down buttons to change the position of the selected action. You can also Modify or Remove the selected action again.
After you created and ordered all required actions for the task, click Next to view the task summary window, as shown in Figure A-6.
Figure A-6 Task summary window
You can always return to previous windows and modify the information you entered before you complete the task creation process.
Click Finish to create the task and confirm the task creation notification, as shown in Figure A-7.
Figure A-7 Task creation confirmation
In the Scheduled Tasks window, you can now see your created task, as shown in Figure A-8.
Figure A-8 Scheduled Tasks window
Per default, any newly created task shows a status of Disabled. A disabled task means it does not run automatically at the specified schedule.
If you want that a task runs automatically at the defined schedule, select the task and then, click Actions… → Enable Task. To run a selected task on-demand, click Actions… → Run Task. A task can always be run on-demand if its status is not running.
A.1.2 Modify a scheduled task
You can easily modify a scheduled task if you need to change its name, description, schedule, order of actions, or if you want to insert or remove any of the actions or modify any of the actions. To modify a scheduled task, select the task and click Actions… → Modify Task.
The Modify Task wizard opens and is similar to the Create Task wizard, but includes the information from the selected task.
Click through the windows and modify the wanted values. After you reach the Modify Actions window, perform necessary changes to individual actions, as required. For example, to change the timeout of our Wait for State action, select the action and click Modify Action as shown in Figure A-9.
Figure A-9 Modify selected action
In the Modify Action window, you can change any of the action properties. You can even switch it to a Command action type, if wanted. In the example that is shown in Figure A-10, we decrease the timeout from 3 to 2 minutes and click OK.
Figure A-10 Modify Action
After you are done with your task modifications, click Next to review the task summary and Finish to save your changes. You see get a final notification that your changes were saved, which you must confirm by clicking OK, as shown in Figure A-11.
Figure A-11 Task modification confirmation
A.1.3 Manage scheduled tasks
You can find all defined tasks in the GUI Settings → Scheduled Tasks window. To manage a task, select the task and click Actions… to see the available actions, as shown in Figure A-12.
Figure A-12 Available task actions
The following options are available for the selected task:
•Modify Task
Change the task, schedule, or actions within the task.
•Remove Task
Delete the task permanently.
•Run Task
Run the task on-demand.
•Enable Task
Enable the defined schedule for the task for automatic running. This option is available only if the task has a schedule that is defined and disabled.
•Disable Task
Disable the defined schedule for the task to disable automatic running. This option is available only if the task has a schedule that is defined and is enabled.
A.1.3.1 Scheduled execution
To determine whether tasks are scheduled to run automatically, look for an Enabled status in the table. Enabled tasks also show a time stamp in the Next Run Time column, as shown in Figure A-13.
Figure A-13 Enabled task run time
The table also shows the last time that a scheduled task was run and the last message. Click the message link for more information about the message.
A.1.3.2 On-demand execution
To run any task on-demand, select the task and then, click Actions… → Run Task. You must confirm that you want to run the selected task by clicking Yes, as shown in Figure A-14.
Figure A-14 Run task on-demand confirmation
Next, you see a confirmation that the task was started. Click OK to close the information pop-up window and return to the Scheduled Tasks window.
A.1.3.3 Task monitoring
Running tasks can be identified in the Scheduled Tasks window. The status column shows a Running status while a task is being run, as shown in Figure A-15.
Figure A-15 Running task
While the task is running, you can follow the detailed progress in the CSM Console log. Click Console in the CSM GUI main menu to open the console log window. In the upper right corner, click the small icon with the arrow in the window to open the console log in a new window and keep it open while navigating in the CSM GUI, as shown in Figure A-16.
Figure A-16 Open CSM console in new browser window
Separate messages mark the start and the completion of a task, as shown in Example A-1.
Example A-1 Scheduled task start and end messages
normal Nov 5, 2019 4:30:25 PM : csmadmin : IWNR2208I : Forcing the scheduled task ITSO_4-SITE H4 Practice to run now.
normal Nov 5, 2019 4:30:25 PM : Server : IWNR2211I : The scheduled task ITSO_4-SITE H4 Practice has started running.
...
normal Nov 5, 2019 4:30:27 PM : Server : IWNR2212I : The scheduled task ITSO_4-SITE H4 Practice has finished running.
When you look for task messages, you must be aware that CSM does not log specific step messages for Command actions. You only see the standard command output, as when the command is run interactively against the session. However, you see that the command was issued by the Server user instead of a normal CSM user name (see Example A-2).
Example A-2 Task Command action message
normal Nov 5, 2019 4:30:25 PM : Server : IWNR1028I : The SuspendH1H3 command in the ITSO_4-SITE session was issued.
However, the Wait for State actions log individual step messages after they are completed (see Example A-3).
Example A-3 Task Wait For State action message
normal Nov 5, 2019 4:30:27 PM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 2 found session ITSO_4-SITE moved to state Suspended (Partial) in 0 minutes.
A.1.4 Remove scheduled tasks
A task also can be permanently deleted in the Scheduled Tasks GUI window. To delete a task, select the task, and then, click Actions… → Remove Task. You must confirm that you want to remove the selected task, as shown in Figure A-17.
Figure A-17 Remove task confirmation
Click Yes to proceed. Next, you see a notification that the task was removed, as shown in Figure A-18.
Figure A-18 Task removed notification
Click OK to close the notification pop-up and return to the Scheduled Tasks window. The deleted task disappeared from the Scheduled Tasks table.
A.2 Scheduled task examples for 4-site session
This section presents two examples of scheduled tasks for a 4-site session.
A.2.1 Task to create DR practice copy on H4
This task example describes the required steps for the practice scenario, as discussed in section 4.3.4, “Practice failover at fourth site” on page 43:
1. Issue Suspend H1-H3.
2. If the H1-H3 role pair does not show Recoverable for all pairs, you must issue Failover H3 to restore last Global Mirror consistency group on H3. You can also verify whether the Global Mirror details show that the master was paused with consistency, as shown in Figure 4-19 on page 44. In that case, the H3 volumes are consistent and the Failover H3 can be skipped.
3. Issue Suspend H3-H4.
4. Issue StartGM H1->H3 to restart Global Mirror replication.
5. Issue Failover H4.
Because CSM does not yet provide an action type to validate consistency and recoverability of a role pair, the required task must include the Failover H3 command to ensure that H3 is consistent or recoverable before suspending H3-H4.
It is also good practice to define a Wait For State action as first step to ensure that the task is run only if the session is in a meaningful state for the task. For example, the practice copy creation on H4 should be started only if the session is fully prepared. You can use a Wait For State action with a timeout of 0 minutes for such an initial state verification.
Table A-1 lists the required actions to be defined for such a task. The timeouts are proposals and must be adopted to the environment.
Table A-1 Recommended steps to create a consistent DR practice copy on H4
|
Step
|
Action type
|
Command/State
|
Time-out (min)
|
|
1
|
Wait for State
|
Prepared
|
0 (initial check for proper starting state)
|
|
2
|
Command
|
SuspendH1H3
|
|
|
3
|
Wait for State
|
Suspended (Partial)
|
3 (need to form last Global Mirror CG)
|
|
4
|
Command
|
FailoverH3
|
|
|
5
|
Wait for State
|
Suspended (Partial)
|
3 (might need to do Global Mirror Fast Reverse Restore of last CG
|
|
6
|
Command
|
SuspendH3H4
|
|
|
7
|
Wait for State
|
Suspended (Partial)
|
3 (need to drain Global Copy and pause consistently)
|
|
8
|
Command
|
StartGM H1-H3
|
|
|
9
|
Wait for State
|
Prepared
|
5 (need to wait until Global Mirror is resumed and started forming new CGs)
|
|
10
|
Command
|
FailoverH4
|
|
|
11
|
Wait for State
|
Prepared
|
1
|
Figure A-19 shows the task overview in the GUI.
Figure A-19 Task to create consistent DR practice copy on H4
When the task is run, you can follow the progress in the CSM Console log. Example A-4 shows the output that is produced by this task. The command actions do not log step specific messages to the console, but only the standard output as issuing the command interactively in the session. Each step entry of the task is highlighted in bold.
Example A-4 Console output of Task to create consistent DR practice copy on H4
normal Nov 6, 2019 10:54:12 AM : csmadmin : IWNR2208I : Forcing the scheduled task ITSO_4-SITE H4 Practice to run now.
normal Nov 6, 2019 10:54:12 AM : Server : IWNR2211I : The scheduled task ITSO_4-SITE H4 Practice has started running.
normal Nov 6, 2019 10:54:12 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 1 found session ITSO_4-SITE moved to state Prepared in 0 minutes.
normal Nov 6, 2019 10:54:12 AM : Server : IWNR1028I : The SuspendH1H3 command in the ITSO_4-SITE session was issued.
normal Nov 6, 2019 10:54:12 AM : Server : IWNR6005I : Pausing all pairs in role pair H1-J3 ...
normal Nov 6, 2019 10:54:12 AM : Server : IWNR6006I : Waiting for all pairs in role pair H1-J3 to reach a state of Suspended. See help for list of actions if transition is taking too long...
normal Nov 6, 2019 10:54:12 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Prepared state to the Suspending state.
warning Nov 6, 2019 10:54:12 AM : Server : IWNR1959W : Session ITSO_4-SITE has changed from Normal status to Warning status.
normal Nov 6, 2019 10:54:12 AM : Server : IWNR1026I : The SuspendH1H3 command in the ITSO_4-SITE session completed.
normal Nov 6, 2019 10:54:13 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Suspending state to the Suspended (Partial) state.
warning Nov 6, 2019 10:54:13 AM : Server : IWNR1959W : Session ITSO_4-SITE has changed from Warning status to Severe status.
normal Nov 6, 2019 10:54:13 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 3 found session ITSO_4-SITE moved to state Suspended (Partial) in 0 minutes.
normal Nov 6, 2019 10:54:13 AM : Server : IWNR1028I : The FailoverH3 command in the ITSO_4-SITE session was issued.
normal Nov 6, 2019 10:54:13 AM : Server : IWNR6004I : Recovering all pairs in role pair H1-J3 ...
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Suspended (Partial) state to the Failing Over state.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1960I : Session ITSO_4-SITE has changed from Severe status to Normal status.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR6012I : Flashing all pairs in role pair H3-J3 ...
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1041I : The command flash was successfully issued to all pairs under role pair H3-J3 for session ITSO_4-SITE.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Failing Over state to the Suspended (Partial) state.
warning Nov 6, 2019 10:54:15 AM : Server : IWNR1959W : Session ITSO_4-SITE has changed from Normal status to Severe status.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1026I : The FailoverH3 command in the ITSO_4-SITE session completed.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 5 found session ITSO_4-SITE moved to state Suspended (Partial) in 0 minutes.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1028I : The SuspendH3H4 command in the ITSO_4-SITE session was issued.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR6015I : Waiting until all pairs in role pair H3-H4 have copied 100% of their data.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Suspended (Partial) state to the Draining state.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1960I : Session ITSO_4-SITE has changed from Severe status to Warning status.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR1026I : The SuspendH3H4 command in the ITSO_4-SITE session completed.
normal Nov 6, 2019 10:54:15 AM : Server : IWNR6002I : Suspending all pairs in role pair H3-H4 ...
normal Nov 6, 2019 10:54:16 AM : Server : IWNR1041I : The command freeze was successfully issued to all pairs under role pair H3-H4 for session ITSO_4-SITE.
normal Nov 6, 2019 10:54:16 AM : Server : IWNR6020I : Releasing I/O for all pairs in role pair H3-H4...
normal Nov 6, 2019 10:54:16 AM : Server : IWNR1041I : The command thaw was successfully issued to all pairs under role pair H3-H4 for session ITSO_4-SITE.
normal Nov 6, 2019 10:54:16 AM : Server : IWNR6019I : Verifying the consistency of role pair H3-H4...
normal Nov 6, 2019 10:54:19 AM : Server : IWNR1041I : The command checkConsistency was successfully issued to all pairs under role pair H3-H4 for session ITSO_4-SITE.
normal Nov 6, 2019 10:54:19 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Draining state to the Suspended (Partial) state.
warning Nov 6, 2019 10:54:19 AM : Server : IWNR1959W : Session ITSO_4-SITE has changed from Warning status to Severe status.
normal Nov 6, 2019 10:54:19 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 7 found session ITSO_4-SITE moved to state Suspended (Partial) in 0 minutes.
normal Nov 6, 2019 10:54:19 AM : Server : IWNR1028I : The StartGM H1->H3 command in the ITSO_4-SITE session was issued.
normal Nov 6, 2019 10:54:19 AM : Server : IWNR6003I : Terminating all pairs in role pair H1-J1 ...
normal Nov 6, 2019 10:54:19 AM : Server : IWNR6004I : Recovering all pairs in role pair H2-H3 ...
normal Nov 6, 2019 10:54:19 AM : Server : IWNR6004I : Recovering all pairs in role pair H2-J3 ...
normal Nov 6, 2019 10:54:19 AM : Server : IWNR6000I : Starting all pairs in role pair H1-H3 ...
warning Nov 6, 2019 10:54:20 AM : Server : IWNR6030W : The pairs in role pair H3-H4 are not replicating the latest data from role pair H1-H3.
normal Nov 6, 2019 10:54:20 AM : csmadmin : IWNR1950I : Session ITSO_4-SITE changed from the Suspended (Partial) state to the Preparing state.
normal Nov 6, 2019 10:54:20 AM : csmadmin : IWNR1960I : Session ITSO_4-SITE has changed from Severe status to Warning status.
normal Nov 6, 2019 10:54:20 AM : Server : IWNR6008I : Waiting for all pairs in role pair H1-H3 to complete their first phase in the Global Copy synchronization or resynchronization ...
normal Nov 6, 2019 10:54:20 AM : Server : IWNR1026I : The StartGM H1->H3 command in the ITSO_4-SITE session completed.
normal Nov 6, 2019 10:54:20 AM : Server : IWNR6000I : Starting all pairs in role pair H1-J3 ...
normal Nov 6, 2019 10:54:22 AM : Server : IWNR1041I : The command start was successfully issued to all pairs under role pair H1-J3 for session ITSO_4-SITE.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR1950I : Session ITSO_4-SITE changed from the Preparing state to the Prepared state.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 9 found session ITSO_4-SITE moved to state Prepared in 0 minutes.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR1028I : The FailoverH4 command in the ITSO_4-SITE session was issued.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR6004I : Recovering all pairs in role pair H3-H4 ...
warning Nov 6, 2019 10:54:27 AM : Server : IWNR6030W : The pairs in role pair H3-H4 are not replicating the latest data from role pair H1-H3.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR1026I : The FailoverH4 command in the ITSO_4-SITE session completed.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR2220I : The scheduled task ITSO_4-SITE H4 Practice in step 11 found session ITSO_4-SITE moved to state Prepared in 0 minutes.
normal Nov 6, 2019 10:54:27 AM : Server : IWNR2212I : The scheduled task ITSO_4-SITE H4 Practice has finished running.
A.2.2 Task to create regular Golden Copies on H4
This example describes the required steps for the scenario as discussed in section 4.4, “Creating a Golden Copy at fourth site” on page 46 in a way that regular golden copies can be created:
1. Issue StartGC H3-H4.
2. Issue Suspend H1-H3.
3. If the H1-H3 role pair does not show Recoverable for all pairs, you must issue Failover H3 to restore last Global Mirror consistency group on H3. You can also verify whether the Global Mirror details show that the master was paused with consistency, as shown in Figure 4-19 on page 44. In that case, the H3 volumes are consistent and the Failover H3 can be skipped.
4. Issue Suspend H3-H4.
5. Issue StartGM H1->H3 to restart Global Mirror replication.
6. If you must access the Golden Copy, issue Failover H4.
Because CSM 6.2.3 does not yet provide an action type to validate consistency and recoverability of a role pair, the required task must include the Failover H3 command to ensure that H3 is consistent and recoverable before issuing the Suspend H3-H4.
It is also good practice to define a Wait For State action as first step to ensure that the task is run only if the session is in a meaningful state for the task. For example, the refresh of a golden copy on H4 should be started only if the session is fully prepared and not while any testing or recovery situation exists. You can use a Wait For State action with a timeout of 0 minutes for such an initial state verification.
This scenario is similar to the scenario that is described in A.2.1, “Task to create DR practice copy on H4” on page 79, but the difference is that the task should be run automatically at a regular interval and H3-H4 role pairs might be left in a Suspended or Target Available state.
Even if the H3->H4 Global Copy can be restarted at the beginning of the task, in CSM 6.2.3 no action is available yet to validate that a role pair reached a specific progress percentage before proceeding with the next step. Therefore, the Suspend H3-H4 in the subsequent step might last some time until all changed tracks from last Golden Copy are drained and the role pair can pause consistently.
The timeout for the Suspend H3-H4 completion in Step 9 of Table A-2 on page 84 must consider this duration and be increased, depending on the task frequency and the expected number of changes during peer periods.
The optional Failover H4 is recommended in the scenario only if the Golden Copy is created on-demand for dedicated access on H4. In our example, we assume a regular task execution to automatically refresh the Golden Copy. We skip the Failover H4 step, which also leaves the Golden Copy volumes in a PPRC state that protects from accidental access.
Table A-2 lists the required actions to be defined for such a task. The timeouts are proposals and must be adopted to the environment.
Table A-2 Recommended steps to create regular golden copies on H4
|
Step
|
Action type
|
Command/State
|
Time-out (min)
|
|
1
|
Wait for State
|
Prepared
|
0 (initial check for proper starting state)
|
|
2
|
Command
|
StartGC H3->H4
|
|
|
3
|
Wait for State
|
Prepared
|
3 (need to restart Global Copy relations)
|
|
4
|
Command
|
SuspendH1H3
|
|
|
5
|
Wait for State
|
Suspended (Partial)
|
3 (need to form last Global Mirror CG)
|
|
6
|
Command
|
FailoverH3
|
|
|
7
|
Wait for State
|
Suspended (Partial)
|
3 (might need to do Global Mirror Fast Reverse Restore of last CG)
|
|
8
|
Command
|
SuspendH3-H4
|
|
|
9
|
Wait for State
|
Suspended (Partial)
|
10 (need to drain changed tracks from last Golden Copy and pause consistently)
|
|
10
|
Command
|
StartGM H1-H3
|
|
|
11
|
Wait for State
|
Prepared
|
5 (Ensure that Global Mirror is resumed and started forming new CGs)
|
Figure A-20 shows the task action overview in the GUI.
Figure A-20 Task to create Golden Copy on H4
To run the task automatically at regular intervals, you must define an appropriate schedule in the task and issue Action... → Enable Task in the Scheduled Task overview.
To verify proper task execution, you can also issue Action... → Run Task to run the task immediately instead of waiting for the next automatic task run.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.