This chapter covers

• Storing media files and other static content within serverless applications
• Maintaining and accessing your files with your serverless API
• Processing static files using your serverless function

In addition to requiring processing and database storage, applications often also need file storage for static files. Static files are media such as photos, audio or video files, and text files (for example, HTML, CSS, and JavaScript files).

Serverless applications need to store static files, too. Keeping your whole application serverless implies that you need a storage solution that follows the same principles. This chapter takes a dive into serverless file storage possibilities and examines how to create a separate file processing function that uses the storage and provides requested files to your other Lambda—your serverless API.

7.1 Storing static files in a serverless application

In the case of your pizza application, it wouldn’t be complete without the images of Aunt Maria’s delicious pizzas. Your cousin Michelangelo (also known as Mike) already took awesome photos of all the pizzas, so you just need to store and serve these static files. AWS has a service for that as well: the Simple Storage Service (S3), which allows you to store the files—up to 5 TB—in a serverless manner.

Amazon S3 stores files in buckets—folder-like structures owned by an AWS account. Each file, or object, stored in a bucket has a unique identification key. S3 buckets support triggers for Lambda functions that allow you to invoke a certain Lambda when something happens in the bucket.

In S3 everything starts with the bucket, so you’ll create one using the AWS web console, an API, or the AWS CLI, which is our preferred method. The mb command requires an S3 URI as an argument. An S3 URI is the name of your S3 bucket prefixed with s3://. If you want to specify the region, you can do that using the --region flag. In our example, we’re naming the S3 bucket aunt-marias-pizzeria and specifying the region.

Run the following command at the CLI prompt:

aws s3 mb s3://aunt-marias-pizzeria --region eu-central-1

The response after running the command should be make_bucket: aunt-marias-pizzeria. If your bucket name isn’t unique, you’ll receive the following error, and you’ll have to rerun the command with a different bucket name:

    make_bucket failed: s3://bucket-name An error occurred 
  (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is 
  not available. The bucket namespace is shared by all users of the system. Please select a 
  different name and try again.

Now that you have the bucket, you want to allow only certain users to upload files to it. But before that, you should think about the folder structure of your bucket.

As you can see in figure 7.1, you should upload your images to an images folder. Sometimes raw images can be too big for a mobile application, so you should also have a thumbnails folder that will contain smaller versions of your images. Also, because there will be only a single menu.pdf file at a time, it doesn’t need to be stored in the folder.

Now that you have the folder structure in place, you need to allow certain users to upload images to the bucket. The easiest way to do so is to generate a presigned URL that will be used for the image upload.

By default, all objects and buckets are private—only the user that created them can access them. A presigned URL allows a user that does not have access permissions to upload files to the bucket. This URL is generated by the user who has access to the bucket and who will grant temporary permissions to anyone that knows it.

Because this URL needs to stay secret, you’ll create a new route in the Pizza API that will generate and return that URL. This route should also be protected; in this example you’ll allow all authorized users to use this API endpoint, but in a real-world application you should have a special user group that can access certain API endpoints, such as an administrators group.

To generate a URL, you need to create a new handler. To do so, use the getSignedUrl method of the S3 class. This method accepts two arguments: the first is the name of the method that will be used via the signed URL (putObject), and the second is an options object. This options object requires the following parameters:

• The name of the bucket that this signed URL is accessing.
• The unique key that will be used for signing the URL. Because generating a unique key on your own isn’t easy, you should use the uuid module. You used this module earlier, in chapter 3; just remember to reinstall it if you removed it from your package.json file. (You only need the version 4 UUID, so you’ll directly require uuid/v4.)
• The access control list (ACL), which defines how the public can interact with the objects in your bucket. In your case you want everyone to be able to see the objects, so set it to public-read.
• The expiration time in seconds for the generated URL. Two minutes should be enough, so set it to 120 seconds.

After generating the options object, use the getSignedUrl method to sign the URL and then return it as a JSON object. Create a file named generate-presigned-url.js in the handlers folder of your Pizza API, as shown in the following listing.

Listing 7.1 Pizza API handler for presigned URL generation

'use strict'
const uuidv4 = require('uuid/v4')    ①  
const AWS = require('aws-sdk')
const s3 = new AWS.S3()    ②  

function generatePresignedUrl() {    ③  
  const params = {
    Bucket: process.env.bucketName,    ④  
    Key: uuidv4(),    ⑤  
    ACL: 'public-read',    ⑥  
    Expires: 120    ⑦  
  }

  s3.getSignedUrl('putObject', params).promise()    ⑧  
    .then(url => {
      return {    ⑨  
        url: url
      }
    })
}

module.exports = generatePresignedUrl

Now that your handler is ready, to get the signed URL, you need to add a new route in your api.js file. You can name it /upload-url. As mentioned previously, you should protect this route in the same way the /orders routes are protected—only users authorized through the Cognito authorizer called userAuthentication should be able to get this URL. Listing 7.2 shows the end of the api.js file. The rest is unchanged; just remember to require the getSignedUrl handler at the top of the api.js file by adding the const getSignedUrl = require('./handlers/generate-presigned-url.js') line.

Listing 7.2 The new /delivery and /upload-url routes in the api.js file

api.post('delivery', (request) => {
  return updateDeliveryStatus(request.body)
}, {
  success: 200,
  error: 400
}, {
  cognitoAuthorizer: 'userAuthentication'
})
api.get('upload-url', (request) => {    ①  
  return getSignedUrl()    ②  
},
{ error: 400 },    ③  
{ cognitoAuthorizer: 'userAuthentication' })    ④  

module.exports = api

If you now update your API using the claudia update command and then visit your new route with an authorization token (received from the web application, as explained in the previous chapter), you’ll receive the signed URL that can be used for uploading files to your bucket.

7.2 Generating thumbnails

Because each uploaded image can be quite large, and Aunt Maria also has a mobile application, you’ll need to resize all the photos and create thumbnails. You don’t want thumbnail creation to block your API in any way, so image processing is a perfect candidate for an independent microservice.

An independent service, in this case, represents a separate Lambda function that will trigger automatically when a new photo is uploaded to Amazon S3. The flow of the events (figure 7.2) proceeds as follows:

• User requests a new signed URL via /upload-url route of Pizza API.
• New photo is uploaded to the generated URL.
• Amazon S3 triggers your new Lambda function.
• Lambda function resizes the image and stores the thumbnail in the thumbnails folder.

The new Lambda function won’t be triggered via an HTTP request, so you don’t need Claudia API Builder. Instead, you’ll need to export a simple handler function that will get the new object from S3 and then resize it using ImageMagick. ImageMagick is available in AWS Lambda by default; you do not need to install it before using it.

The first step when creating a separate service is to create a new project. You need to

1. Create a new folder outside of your pizza-api folder (a good name for it would be pizza-image-processor).
2. Inside your new folder, initialize a new NPM package (npm init).

Your next step is creating a file that exports your service handler function. Because this is just an image processor and not an API, you don’t need to use Claudia API Builder.

This service will be small and could fit in a single file, but for easier maintenance and a more test-friendly approach, you’ll split it into two files: the first is an initial file that just extracts the data from a Lambda event, and the second is the actual converter.

Your initial file needs a handler function that receives three arguments:

• The event triggered by the Lambda function
• The context of the Lambda function
• A callback that allows you to respond back from the Lambda function

In your initial file, you’ll first check if a valid event record exists and, if so, if it is coming from Amazon S3. Because multiple services can trigger the same Lambda function, you also need to check if it’s from your S3 storage. Then you need to extract your S3 bucket name and the filename with a path or an object key using a proper S3 query. Its response will be an image that you’ll need to pass to the convert function.

The initial code is shown in the following listing.

Listing 7.3 The initial file for your new image processor Lambda function

'use strict'
const convert = require('./convert')    ①  

function handlerFunction(event, context, callback) {    ②  
  const eventRecord = event.Records && event.Records[0]    ③  

  if (eventRecord) {    ④  
    if (eventRecord.eventSource === 'aws:s3' && eventRecord.s3) {    ⑤  
      return convert(eventRecord.s3.bucket.name, eventRecord.s3.object.key)
        .then(response => {    ⑥  
          callback(null, response)
        })
        .catch(callback)    ⑦  
    }

    return callback('unsupported event source')    ⑧  
  }

  callback('no records in the event')    ⑨  
}

exports.handler = handlerFunction    ⑩  

Now that you have the initial file, it’s time to create the convert function. Because this service is small, there’s no need for a complicated folder structure: just create the convert.js file in the same project folder. As shown in figure 7.3, the flow of the convert function is as follows:

• S3 triggers your AWS Lambda and your initial file invokes the convert function.
• The convert function downloads the image from S3 and stores it locally in the /tmp folder.
• The image is converted using the convert command from ImageMagick, and a thumbnail is saved in the /tmp folder.
• The convert function then uploads the new thumbnail to the S3 bucket.
• The convert function resolves the promise that tells the initial file that the operation was successful.

As shown in figure 7.3, your convert function first needs to download the file from S3 using the getObject method of the S3 class. This method accepts a bucket name and S3 file path and returns a promise that will, when resolved, return a response that contains the file body as a buffer.

Your convert.js file should export a convert function, which is a Node function that accepts a bucket name and an S3 file path as parameters and returns a promise. For the actual functionality, you need to import three core Node.js modules:

• fs, for manipulating your filesystem
• path, for file path manipulation
• child_process, for invoking an ImageMagick command

In addition to those three modules, you also need to install two additional packages from NPM: mime, a package that determines the uploaded file’s MIME type, and aws-sdk. The AWS SDK is required for programmatic usage of the S3 service.

Your next step is saving the downloaded file. In your Lambda function, only the /tmp folder is writable. Therefore, you should create two folders inside your /tmp folder: one called images, where you’ll store the downloaded images, and another called thumbnails, where you’ll store generated thumbnails.

When you are sure that the /images folder exists inside /tmp, use the fs.writeFile command with the same downloaded file as its argument to save to that folder. This method is asynchronous, but it doesn’t return a promise, so you should wrap it in a promise.

Now that your file is saved locally, you can use ImageMagick to create a thumbnail. To do so, you need to use the convert command, which allows you to resize or convert image files. At the moment, you’ll keep the same file format, so the only thing you should do is resize the given image. To do so, you invoke the convert command with the following command-line arguments:

• The path to the full image
• The -resize flag, which tells the command to resize the image
• A 120x120> value, which means that the image should be scaled so its larger dimension has a maximum of 120 pixels. Note the > after the value, which tells the command to resize the image only if its larger dimension is greater than 120 px.
• The destination path

The complete command for creating an image named image.png as a 120 px thumbnail is the following:

convert /tmp/images/image.png -resize 120x120> /tmp/thumbnails/image.png

To execute the command from your Lambda, you need to use the exec method of the child_process module that you imported at the top of the file. Although exec is asynchronous, it’s not promise-based, so you’ll need to wrap this function call in a promise.

As a final part of the convert function, you’ll upload the file to the Amazon S3 bucket. You can do this using the putObject method of the S3 class. This method returns a promise and requires the following:

• An options object with the bucket name
• The S3 file path
• The file body as a buffer
• The ACL
• The content type of your file

Because your image processor service can work with multiple file types, require the mime package at the top of your file to get the MIME type of your original image and set it as the content type of your thumbnail. If you don’t provide this value, S3 will assume your file type is binary/octet-stream.

The following listing shows the full code of the convert.js file.

Listing 7.4 Convert images to thumbnails

'use strict'
const fs = require('fs')
const path = require('path')
const exec = require('child_process').exec
const mime = require('mime')
const aws = require('aws-sdk')
const s3 = new aws.S3()
function convert(bucket, filePath) {    ①  
  const fileName = path.basename(filePath)

  return s3.getObject({
    Bucket: bucket,
    Key: filePath
  }).promise()
    .then(response => {
      return new Promise((resolve, reject) => {    ②  
        if (!fs.existsSync('/tmp/images/'))    ③  
          fs.mkdirSync('/tmp/images/')

        if (!fs.existsSync('/tmp/thumbnails/'))    ③  
          fs.mkdirSync('/tmp/thumbnails/')

        const localFilePath = path.join('/tmp/images/', fileName)    ⑤  

        fs.writeFile(localFilePath, response.Body, (err, fileName) => {    ⑤  
          if (err)
            return reject(err)

          resolve(filePath)
        })
      })
    })
    .then(filePath => {
      return new Promise((resolve, reject) => {    ⑦  
        const localFilePath = path.join('/tmp/images/', fileName)
        const localThumbnailPath = path.join('/tmp/thumbnails/', fileName)

        exec(`convert ${localFilePath} -resize 120x120\> ${localThumbnailPath}`, (err, stdout, stderr) => {    ⑧  
          if (err)
            return reject(err)

          resolve(fileName)
        })
      })
    })
    .then(fileName => {
      const localThumbnailPath = path.join('/tmp/thumbnails/', fileName)

      return s3.putObject({    ⑨  
        Bucket: bucket,
        Key: `thumbnails/${fileName}`,
        Body: fs.readFileSync(localThumbnailPath),    ⑩  
        ContentType: mime.getType(localThumbnailPath),    ⑪  
        ACL: 'public-read'    ⑫  
      }).promise()
    })
}

module.exports = convert

7.2.1 Deploying your S3 function

Now that you’ve implemented your service, you need to use Claudia to deploy it. Interestingly, in this scenario you don’t have an API. You’ll invoke claudia create with a --region flag, just as you did in chapter 2 for your API, but instead of the --api-module flag, for a function without an API you’ll use the --handler flag. You can see the command in listing 7.5. The --handler flag expects the path to your handler with a .handler suffix. For example, if you are using the handler export in a index.js file your path will be index.handler; if your handler is exported in a lambda.js file, you’ll specify lambda.handler.

Listing 7.5 Deploying the image processing service using Claudia

claudia create     ①  
  --region eu-central-1     ②  
  --handler index.handler    ③  

This command returns information about your Lambda function similar to that shown in the following listing, and it creates a claudia.json file in the root of your project.

Listing 7.6 The response from the claudia create command upon successful execution

{
  "lambda": {
    "role": "pizza-image-processor-executor",    ①  
    "name": "pizza-image-processor",    ②  
    "region": "eu-central-1"    ③  
  }
}

Before trying your new service, there is one more step. You need to set a trigger for your Lambda function from your S3 bucket. Claudia has a command for that as well—claudia add-s3-event-source. This command has several flags, but you’ll use the following two:

• --bucket—A required flag specifying your bucket name
• --prefix—An optional flag that allows you to specify a folder

As shown in the following listing, you should specify images/ as a prefix, because then the command will accept triggers only from the images folder.

Listing 7.7 Adding the S3 trigger to the Lambda function

claudia add-s3-event-source     ①  
  --bucket aunt-marias-pizzeria     ②  
  --prefix images/    ③  

A successfully added trigger returns an empty object as a response; otherwise, an error is shown.

One of the easiest ways to see if your new service is working is to manually upload a file to your S3 bucket’s images folder. Try to do that, then wait a few seconds and check the thumbnails folder in your S3 bucket.

If you want to try out the complete flow of your API and the image processor, you can use the front-end application from https://github.com/effortless-serverless/pizzeria-web-app.

7.3 Taste it!

This chapter was pretty easy, but because it’s the end of part 1 of this book, we’ve made the exercise more complicated as a bonus.

7.3.1 Exercise

Some images that Michelangelo prepares are big—10 megapixels or more. Because large file sizes can cause slow loading in both the web and mobile applications, you need to resize those images with a height or width larger than 1,024 px.

Here are a few tips:

• Use the convert function one more time to resize the file.
• Be careful during the resizing, because you are modifying the file that you are using for generating the thumbnail—maybe you shouldn’t do both actions in parallel.
• Upload both files to S3 in the end.

7.3.2 Solution

As shown in listing 7.8, most of the code stays the same—you still need to download the image from S3 and store it in a /tmp folder, then you need to generate the thumbnail, and finally you need to upload it to S3.

But there are some differences. After you’ve downloaded the image from S3 and stored it on your local filesystem, you need to resize it before you create a thumbnail. Technically, you can resize after you create the thumbnail, but it’s better to create the thumbnail from the smaller image than to use the original size.

After resizing the image and generating the thumbnail, you need to upload both files to Amazon S3. You can do this operation in parallel, so you should use Promise.all to parallelize the upload process.

Listing 7.8 shows the full code example for the sake of readability. Run claudia upload and try to manually upload a large image to your bucket to test your solution.

Listing 7.8 Convert images to thumbnails and resize big image to a reasonable size

'use strict'
const fs = require('fs')
const path = require('path')
const exec = require('child_process').exec
const mime = require('mime')

const aws = require('aws-sdk')
const s3 = new aws.S3()
function convert(bucket, filePath) {
  const fileName = path.basename(filePath)
  return s3.getObject({    ①  
    Bucket: bucket,
    Key: filePath
  }).promise()
    .then(response => {    ②  
      return new Promise((resolve, reject) => {
        if (!fs.existsSync('/tmp/images/'))
          fs.mkdirSync('/tmp/images/')

        if (!fs.existsSync('/tmp/thumbnails/'))
          fs.mkdirSync('/tmp/thumbnails/')

        const localFilePath = path.join('/tmp/images/', fileName)

        fs.writeFile(localFilePath, response.Body, (err, fileName) => {
          if (err)
            return reject(err)

          resolve(filePath)
        })
      })
    })
    .then(filePath => {    ③  
      return new Promise((resolve, reject) => {    ④  
        const localFilePath = path.join('/tmp/images/', fileName)

        exec(`convert ${localFilePath} -resize 1024x1024\> ${localFilePath}`, (err, stdout, stderr) => {    ⑤  
          if (err)
            return reject(err)

          resolve(fileName)
        })
      })
    })
    .then(filePath => {    ⑥  
      return new Promise((resolve, reject) => {
        const localFilePath = path.join('/tmp/images/', fileName)
        const localThumbnailPath = path.join('/tmp/thumbnails/', fileName)

        exec(`convert ${localFilePath} -resize 120x120\> ${localThumbnailPath}`, (err, stdout, stderr) => {
          if (err)
            return reject(err)

          resolve(fileName)
        })
      })
    })
    .then(fileName => {    ⑦  
      const localThumbnailPath = path.join('/tmp/thumbnails/', fileName)
      const localImagePath = path.join('/tmp/images/', fileName)

      return Promise.all([    ⑧  
        s3.putObject({    ⑨  
            Bucket: bucket,
            Key: `thumbnails/${fileName}`,
            Body: fs.readFileSync(localThumbnailPath),
            ContentType: mime.getType(localThumbnailPath),
            ACL: 'public-read'
          }).promise(),
          s3.putObject({    ⑩  
            Bucket: bucket,
            Key: `images/${fileName}`,
            Body: fs.readFileSync(localImagePath),
            ContentType: mime.getType(localImagePath),
            ACL: 'public-read'
          }).promise()
      ])
    })
}

module.exports = convert

7.4 End of part 1: Special exercise

You’ve come to the end of part 1 of the book. You’ve learned the basics of creating a serverless API, and now it’s time to put them to the test. Each part of the book ends with a special exercise where you’ll test the skills you learned throughout that part. Each special exercise extensively challenges your gained knowledge and contains an advanced task for those who need an extra challenge.

This special exercise builds on what you’ve learned in this chapter—your goal is to create a new DynamoDB table pizzas that will keep your pizzas, add your static pizza list there, and then add a new API call for better pizza image handling. Contrary to the implementation you saw earlier in the chapter, this new API call should save the uploaded pizza image to S3, and then send the generated URL to your DynamoDB database to persist it as an extra column in your pizzas table. This means that each pizza will have its corresponding image URL.

7.4.1 Advanced task

If that’s too easy for you, this task brings another layer of complexity that is common in many applications. The task is to extend the pizza object to have more than one image assigned to it, and also allow the option to set one as the default.

Summary

• Serverless applications don’t require serverless storage but need it to be fully serverless.
• When using AWS, S3 is the serverless storage service you need.
• Always try to separate your serverless application into smaller microservices—for example, for image processing you should always have a separate serverless function.
• Claudia.js helps you easily connect your Lambda to your S3 storage events.
• You can use ImageMagick within your serverless function to process your images and then store them to S3.