Defining the difference between batch processing and scripting

Batch processing and scripting have different purposes when it comes to working with AWS (or any other cloud-based application strategy for that matter). Here’s a basic summary of the differences between the two:

• Batch processing: Uses a data focus to manage data directly either online or offline. The purpose of batch processing is to manipulate the data in some way to make it easier to process. For example, updating the data with the latest numbers requires a batch process. You can also use batch processing for data shaping. Removing redundant records or filling in missing data also works well as a batch process. The point is that batch processing focuses on data manipulation using repetitive methodologies.

  Batch processing does have some specific characteristics. Developers often associate batch processes with highly variable usage patterns that have significant usage peaks. For example, end-of-the-month data processing falls into this category. The batch process is the same each month, but the pattern can vary significantly depending on the amount of data to process and any special requirements (such as the addition of year-end processing). A batch-processing scenario can also require a complex setup using multiple AWS services to keep track of jobs, provide job status updates, track job performance, and automate fault tolerance by automatically resubmitting failed jobs.

• Scripting: Uses a task focus to accomplish goals that may not directly change data. For example, measuring application efficiency or adding new application modules both work as scripted tasks. Most scripts run in real time, but you can also schedule scripts to run later. Scripts tend to provide greater flexibility than batch processes, but scripts also require more work to create. Scripts offer a task-specific approach to perform both application and data manipulation using flexible methodologies.

  Scripting solutions tend to handle uncertainty better than batch solutions. When performing a batch task, you know that certain data will require manipulation at given times to meet specific goals. A scripting scenario may handle uncertain data at unusual intervals to meet goals based on conditions, some of which are unforeseen. For example, you don’t create new users every day, and the data and requirements for adding a user changes over time; consequently, adding users is a task that you handle better using scripts.

• Overlapping and Combining: As with everything, no absolute rule exists saying that you must use a particular approach for a particular need. For example, it’s perfectly acceptable to perform ad hoc monitoring using a batch process or data manipulation using scripting approaches. Often, the best approach is the one that you’re most familiar with and that allows you to perform the task most quickly. In addition, you must consider the availability of tools and the costs involved in automating the task. You may also find that you need to combine scripting and batch processing, creating another type of overlap between the two. For example, you might create a script that calls on a batch file and then schedule that script to run at a particular time each day.

Understanding the batch-processing options

You have access to a wide assortment of batch-processing options when working with AWS. In fact, so many options are available that covering them all in a single book chapter isn’t possible. This book is written with the developer in mind, so the following sections address batch-processing options that involve various levels of complexity and flexibility. For example, using AWS Batch is simple, but it may lack some of the flexibility you need to address specific needs.

Understanding the scripting options

As with batch processing, too many script solutions exist to discuss in a single chapter. In fact, if anything, you have more scripting choices when working with AWS than you have batch solutions. The script solutions also tend to vary more in approach, functionality, and flexibility.

Scripts can also execute automatically based on events. Chapter 10 discusses one such option using Lambda. The following sections provide an overview of common scripting options to meet generalized scripting scenarios. You need to do your homework and view the wide variety of options before making a final choice. These sections help make you aware of the potential solutions so that you can wade through the vast number of options with greater ease.

Considering aws-shell features

The aws-shell utility (https://aws.amazon.com/about-aws/whats-new/2015/12/aws-shell-accelerates-productivity-for-aws-cli-users/) comes with a host of interesting features. The interactivity alone makes it worth using, but you also get these additions as part of the package (make sure to also check out the blog that is available from within the preceding article link):

• Auto-completion: As you type commands, you see a list of available options to complete the next step. For example, after you type aws, you see a list of services that you can add as the next step in the command process.
• Auto-suggestion: If the shell recognizes a pattern to the values you type, it gives you a completed command. Pressing Tab automatically enters the remaining text.
• Command history: Even though most command-prompt and terminal-window implementations retain a command history, the information is available for only the current session. The aws-shell utility stores this information in a file for later use.
• Dot (.) commands: The aws-shell gives you access to additional aws-shell–specific commands through the dot (.) prompt. Here are a few examples:
  • .edit: Provides the means for saving the commands you type as part of a shell script.
  • .profile: Modifies the profile used to execute commands so that you don’t have to include the --profile argument every time you type a command.
  • .cd: Changes the directory to the specified location on disk.
• Fuzzy searches: You may not remember an argument or other command component precisely. Typing a value that appears as part of the actual argument displays a list of suggestions that you can use in place of that part you remember.
• Inline documentation: Help is always available as part of the shell. As you type values, the shell automatically displays help for that value so that you can be sure you’re typing the right information.
• Server-side auto-complete: Typing a command component that requires a server-side value, such as an ARN, usually requires a lookup on your part. When using aws-shell, the shell performs the lookup for you and displays a list of acceptable values.
• Shell command access: If you need to access the underlying operating system commands, type an exclamation mark (!) before the command. For example, type ! dir and press Enter to obtain a directory listing on a Windows system.
• Shorthand auto-complete: You can use shorthand notation to define a specific longer sequence of commonly used commands. Typing the shorthand form is the same as typing the full sequence.
• Toolbar options: The status bar shows a list of function keys that you can press to obtain specific functionality from aws-shell. For example, you can turn fuzzy searches on or off as needed.

Getting aws-shell

Because you have Anaconda installed on your system, you also have an appropriate version of Python installed, and you can use the pip utility to perform the task. To install aws-shell, open a command prompt or terminal window, type pip install --upgrade aws-shell, and press Enter. You see a series of installation messages as pip performs the installation process, as shown in Figure 9-1.

To test your installation, type aws-shell and press Enter. You see the initial display, shown in Figure 9-2. Note that the status bar shows the function keys you can press to enable or disable specific aws-shell features. At this point, you might want to get a cup of coffee because the documentation must download; waiting for a few minutes for this process to complete makes using aws-shell easier.

Performing simple tasks

Most previous commands found in this book begin with aws or some other CLI command. Note the prompt in Figure 9-2. Every command you type assumes that you have preceded it with aws. Consequently, if you want to perform a task using Elastic Beanstalk, you type elas to display a list of commands. It’s then possible to press the down-arrow key to select one of these commands as shown in Figure 9-3.

Press the spacebar to continue the command. You now see a list of commands that you can type, along with help information, as shown in Figure 9-4. Note that the help information is actually getting in the way in this case. You can turn it off by pressing F5. (Unfortunately, pressing F5 right at this moment will cause the command to terminate, but you can start it again without a problem.)

The service name appears in red letters. Type des after the space. You see a listing of possible commands, as shown in Figure 9-5. As before, you press the down-arrow key to select a particular command. For this example, select describe-environments. When you select a particular command, the text turns turquoise.

The default setup for aws for this book is to output information in tabular form. Type -- to display a list of options for this command, as shown in Figure 9-6. Use the down-arrow key to select --output. Note that the options appear in dark green. The help screen also tells you what to type next. In this case, you type a string that contains the format you want to use for the output.

Press the spacebar to select the command. This time, you don’t see any help information; you must know that text is one of the options. Type text and press Enter. You see the command output, as shown in Figure 9-7.

Of course, you can just as easily see the output in JSON format. Press the up-arrow key and you see the command repeated. Press Backspace to remove text and type json. Press Enter. Figure 9-8 shows the new output. This technique relies on using the command history. Interestingly enough, even after you stop and restart aws-shell, you have access to this command simply by using the command history.

The display has gotten messy at this point. You can clean it up on most platforms using the clear screen (cls) command. Of course, this means accessing the underlying command processor. Type !cls and press Enter. If your system supports the cls command, you see the screen cleared.

Obtaining help

One of the commands that you don’t see listed when you work with aws-shell is help. The command is still there, but you have to remember that it is available. For example, to see the help associated with the ec2 service, type ec2 help and press Enter. You see output similar to that shown in Figure 9-9. The aws-shell utility automatically uses any paging utility available on your platform to display one page of help information at a time.

Editing your commands

One of the handiest features of aws-shell is the capability to save your commands in permanent form. After you experiment for a while and know you have the right set of commands in place, it’s time to create a script or batch file. To perform this task, type .edit and press Enter. What you see is the text file editor for your particular platform with the history of the commands you have typed, as shown in Figure 9-10.

Note that the file lacks the !cls command that you type in the “Performing simple tasks” section, earlier in this chapter. The only commands you see are those that you actually type for aws. You can edit this file and then save it as needed. For example, when working with Windows, you can save the file with a .bat file extension and repeat the commands as a script.

Of course, if you want to use aws-shell as a means for creating scripts, you really do need to clear the history before each session. No command exists for performing this task now. Consequently, you must delete the history file found in the .awsshell folder of your system. This folder also contains an awsshellrc file that you can use to change the default settings for your aws-shell setup.