Installing an IBM Watson Machine Learning Accelerator notebook
The examples in this appendix help you customize a notebook package, add a notebook package, create a Spark Instance Group (SIG) that includes the notebook package, start a notebook service, and test a notebook.
The base notebook package and some sample notebooks are available GitHub.
The following topics are covered in this chapter:
Customizing a notebook package
In this section, we customize a notebook to include Anaconda. That module is used in a Jupyter Notebook to connect to a Livy server running on a Hadoop cluster. Once connected, Spark jobs can be run remotely.
Start by downloading the PowerAI-1.5.4.1-Notebook-Base.tar.gz notebook package from Github to the workstation where you are running the IBM Watson Machine Learning Accelerator Cluster Management Console.
The PowerAI-1.5.4.1-Notebook-Base.tar.gz package includes the scripts that are used to start, stop, and monitor the notebook server.
This section shows how to update the PowerAI-1.5.4.1-Notebook-Base.tar.gz package to create a PowerAI-1.5.4.1-Notebook-HI.tar.gz package. A newer version of the notebook package might be available, such as Version 1.5.4.2. If so, use the newer version in the following instructions by replacing V1.5.4.1 with the latest version.
1. Create a custom working directory and extract the PowerAI-1.5.4.1-Notebook-Base.tar.gz package into it by running the following commands:
mkdir custom
tar -C custom -xzvf PowerAI-1.5.4.1-Notebook-Base.tar.gz
2. Download the Anaconda repo file by running the following commands:
cd custom/package
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
cd ..
3. (Optional) Update the build.version to include today’s date. This example shows the edits by using a vi editor.
vi build.version (or use an editor of your choice)
a. Enter insert mode (a) and edit the build data and number. For example:
Build Date: "Jan 17 2019"
Build Number: 20190117
b. Press Esc and save the update (:qw).
4. (Optional) Add more pip packages of interest for your custom notebook by updating the added_packs file:
vi scripts/added_packs
a. Enter insert mode (a) and scroll down to bottom to add additional packages (one per line):
theano
keras
pandas
sparkmagic
tensorvision
b. Press Esc and save the update (:qw).
5. Compress the updated custom package (HI = Hadoop Integration) by running the tar command:
tar -czvf PowerAI-1.5.4.1-Notebook-HI.tar.gz
In the next section, we add this notebook package to IBM Watson Machine Learning Accelerator.
Adding a notebook package
This section describes how to add the notebook package that was customized in “Customizing a notebook package” on page 122 into IBM Watson Machine Learning Accelerator.
To add the notebook package, complete the following steps:
1. Using the management console, go to the notebook management user interface by selecting Workload → Spark → Notebook Management, as shown in Figure B-1.
Figure B-1 Notebook Management menu
2. Click Add, as shown in Figure B-2.
Figure B-2 Notebook Management user interface
3. Complete the details for the PowerAI-1.5.4.1-Notebook-HI.tar.gz notebook and click Add, as shown in Figure B-3:
a. Give it a name (PowerAIHI) and a version (1.5.4.1 or match the version of the notebook .tar file).
b. Click Browse to find the tar.gz file that you downloaded or customized.
c. Select the Enable monitoring for the notebook and Enable collaboration for the notebook check boxes.
d. Complete the Start command, Stop command, and Job monitor command fields as follows:
 • ./scripts/start_jupyter.sh
 • ./scripts/stop_jupyter.sh
 • ./scripts/jobMonitor.sh
e. In the Longest update interval for job monitor field, specify the number of seconds (180 is a good default).
Figure B-3 Add Notebook wizard
Wait for the copying the process to complete, as shown in Figure B-4. The copy speed depends on the network speed and the package size (over 300 MB in this case).
Figure B-4 Notebook copy in progress
After the Add Notebook process is complete, you can add the notebook to an existing SIG by stopping it and updating its configuration, or you can create a SIG and add the notebook to it.
Creating a Spark Instance Group with a notebook
To create a SIG with a notebook enabled, complete the following steps in the IBM Watson Machine Learning Accelerator cluster management console:
1. Select Spark → Spark Instance Groups, as shown in Figure B-5.
Figure B-5 Spark Instance Groups menu
2. Click New, as shown in Figure B-6.
Figure B-6 Creating a Spark Instance Group
3. Complete the SIG details. Name the SIG, specify a deployment directory that is on the local storage, and specify the OS user for the SIG, as shown in Figure B-7.
Figure B-7 New Spark Instance Group: Part 1
We create demouser as the execution user for our tests. IBM Spectrum Conductor Deep Learning Impact template-based SIGs run as egoadmin. For the Spark deployment directory, we append a local directory, a run-as user, and the SIG name (/cwslocal + /demouser + /notebook.
4. Select the check box on the notebook to enable in the SIG, as shown in Figure B-8.
Figure B-8 New Spark Instance Group: Part 2
For the consumers, we used the default that is shown in Figure B-9.
Figure B-9 New Spark Instance Group: Part 3
For the Resources Groups, we specified the compute host resource group (RG) except for the GPU executors, as shown in Figure B-10.
Figure B-10 New Spark Instance Group: Part 4
Click Create and Deploy Instance Group to create and deploy the SIG.
5. Start the SIG when its status is Ready, as shown in Figure B-11.
Figure B-11 Starting a Spark Instance Group
After the SIG starts, notebooks can be created for the users.
Creating notebooks for users
To create notebooks for users, complete the following steps:
1. Go to the Notebooks tab in the SIG details and click Create Notebooks for Users, as shown in Figure B-12.
Figure B-12 The Creating Notebooks for Users menu option
2. Select the notebook and users to create and click Create, as shown in Figure B-13.
Figure B-13 Creating Notebooks for Users details
3. The notebook is created. You can access it by clicking My Notebooks, as shown in Figure B-14.
Figure B-14 My Notebooks window
4. A list of the notebook services are shown. Select the notebook in which you are interested. The notebook URL opens in a browser window.
Testing notebooks
To test a notebook, complete the following steps:
1. Log in to the notebook service, as shown in Figure B-15.
Figure B-15 Notebook server login
2. Create a Python 3 notebook by selecting New → Python 3, as shown in Figure B-16.
Figure B-16 Notebook server home page
A simple notebook test can output the artificial intelligence (AI) framework Python dictionary by using a code fragment similar to the following one:
import <framework>
print(framework.__version__)
print(framework.__dict__)
An example of the test output is shown in Figure B-17.
Figure B-17 Notebook test
Conclusion and additional information
This appendix covered the customization of a notebook package, adding the notebook package to IBM Watson Machine Learning Accelerator, creating a SIG that includes the notebook package, starting a notebook service, and a notebook test sample.
The IBM Redbooks GitHub website includes several sample notebooks that can be downloaded from GitHub and uploaded in to the notebook server by using the Jupyter upload button.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset