This chapter covers

• Running Express.js applications in AWS Lambda and the serverless ecosystem
• Serving static content from an Express.js application
• Connecting to MongoDB from a serverless Express.js application
• Understanding the limitations and risks of Express.js apps in a serverless ecosystem

Express.js is the most important and most used Node.js framework. That’s not without reason: Express.js is easy to use and has a large ecosystem of middleware that can help you build an API or server-rendered web application. But using Express.js still requires a server that will host the application, which means we’re back to the problems this book tries to solve by using serverless technologies. Is there a way to keep your existing Express.js application and still have all the benefits of serverless?

The Express.js web application framework is basically an HTTP server. Serverless applications do not need HTTP servers, because HTTP requests are handled by API Gateway. But fortunately, there is a way to use an existing Express.js application in AWS Lambda with minor modifications. This chapter teaches you how to do that, and it also presents some of the most important limitations of serverless Express.js applications.

13.1 Uncle Roberto’s taxi application

During your big family reunion, Aunt Maria brags about her new online business. It’s better than ever, but she says best of all is that the new app just works—whether it needs to handle a single order or a few dozen at the same time, everything works.

Her brother, your Uncle Roberto, tells her she’s lucky. He has many problems with his taxi company’s app. The app itself is nice, but it crashes often when more ride requests than usual are coming in—for example, when it’s raining. Unfortunately, his IT team is not very responsive in those situations, and he’s losing customers and money.

Roberto asks how you performed your magic for Aunt Maria, and wonders if it would work for his app, too. You explain that it depends on the technology his app is using.

A few days later, you receive a message explaining that the taxi app is using Express.js and MongoDB. It’s hosted on some small virtual private server that serves the RESTful API for the mobile app, and it uses server-rendered HTML pages for the admin panel. Overall, it sounds like a typical Express.js application. You agree to do some research and let Uncle Roberto know in a few days if you can do something to help him out.

13.2 Running an Express.js application in AWS Lambda

Before you start your investigation, you need to create a simple Express.js app. You’ll use that app to test how Express.js works in AWS Lambda. To do so, create a new project folder and name it simple-express-app. Then initiate a new NPM project in it, and install Express.js as a dependency by running the npm i express -S command.

As a first test, you should create one file with one Express.js route, and try to run it in AWS Lambda. Create the file app.js in your simple-express-app folder.

In this file, require the express module and create a new Express app with it. Then add a GET / route that will return the text “Hello World.” Finally, define the port the application will use and start the server using the server.listen function.

At this point, your app.js file should look similar to the next listing.

Listing 13.1 Express.js app

'use strict'
const express = require('express')
const app = express()    ①  

app.get('/', (req, res) => res.send('Hello World'))    ②  

const port = process.env.PORT || 3000    ③  
app.listen(port, () => console.log(`App listening on port ${port}`))    ④  

Then run your simple Express.js app using the following command:

node app.js

Unless something else is running on port 3000, this command will start your local server. If you visit http://localhost:3000 in your web browser, you should see the “Hello World” text.

The easiest way to run the existing Express.js app in AWS Lambda is by using the aws-serverless-express Node.js module. This module requires only minor changes in the Express.js app you created.

To prepare your app for AWS Lambda and API Gateway, open your app.js file and replace the app.listen function with a simple export, as shown in the next listing. This export allows the Express.js wrapper in AWS Lambda to require your app.

Listing 13.2 Express.js app modified for AWS Lambda

'use strict'
const express = require('express')
const app = express()
app.get('/', (req, res) => res.send('Hello World'))
module.exports = app    ①  

But doing this will break your Express.js app on localhost; you’ll no longer be able to run a local version using the node app.js command.

To fix that issue, create another file in the project folder, and name it app.local.js. This file should require your Express.js app from the app.js file, and then invoke the app.listen function to start a local server on the port you provided.

Your app.local.js file should look like the following listing.

Listing 13.3 Running the wrapped Express.js app locally

'use strict'
const app = require('./app')    ①  

const port = process.env.PORT || 3000    ②  
app.listen(port, () => console.log(`App listening on port ${port}`))

To confirm that the local version of the Express.js app still works as expected, run the following command:

node app.local.js

This command should show the “Hello World” text on http://localhost:3000 (unless you specified another port).

Now that the local version works, it’s time to generate a wrapper for your Express.js app. The easiest way to generate the wrapper is by using the claudia generate-serverless-express-proxy command. This command requires the --express-module option with a path to your main file without the .js extension. For example, you should run the following command when your index file is app.js:

claudia generate-serverless-express-proxy --express-module app

This command generates a file named lambda.js and installs the aws-serverless-express module as a development dependency.

The file created by this command is a wrapper that runs your Express.js app in AWS Lambda. It is using the awsServerlessExpress.createServer function to start your Express.js app inside your Lambda function. Then, it uses the awsServerlessExpress.proxy function to transform an API Gateway request to an HTTP request and pass it to your Express.js app, and to transform and pass the response back to API Gateway.

The contents of the file are shown in the next listing.

Listing 13.4 AWS Lambda wrapper for Express.js apps

'use strict'
const awsServerlessExpress = require('aws-serverless-express')    ①  
const app = require('./app')    ②  
const binaryMimeTypes = [    ③  
  'application/octet-stream',
  'font/eot',
  'font/opentype',
  'font/otf',
  'image/jpeg',
  'image/png',
  'image/svg+xml'
]    ④  
const server = awsServerlessExpress.createServer(app, null, binaryMimeTypes)
exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)    ⑤  

The next step is to deploy your API to AWS Lambda and API Gateway. You can do that with the claudia create command, but with an important difference from the APIs used in the previous chapters: you need to invoke it with the --handler option instead of --api-module, and also with --deploy-proxy-api. This will set up a proxy integration, which means that all requests to API Gateway will be passed directly to your Lambda function.

To deploy your Express.js app, run the following command:

claudia create --handler lambda.handler --deploy-proxy-api --region eu-central-1

When the command executes successfully, the response should look similar to the next listing.

Listing 13.5 The deployment result

{
  "lambda": {
    "role": "simple-express-app-executor",
    "name": "simple-express-app",
    "region": "eu-central-1"
  },
  "api": {
    "id": "8qc6lgqcs5",
    "url": "https://8qc6lgqcs5.execute-api.eu-central-1.amazonaws.com/latest"    ①  
  }
}

As you can see, the response is extended with the url parameter. And if you visit the URL (in our case, it’s https://8qc6lgqcs5.execute-api.eu-central-1.amazonaws.com/latest), you should see the “Hello World” text.

13.2.1 Proxy integration

As you learned in chapter 2, API Gateway can be used in the following two modes:

• With models and mapped templates for requests and responses
• With proxy pass-through integration

The first mode is useful for typed languages, such as Java and .Net, but because Claudia is focused only on JavaScript, it always uses the second approach. With this approach, API Gateway passes any requests directly to your AWS Lambda function, which is responsible for routing and managing the requests.

When you deploy a proxy API for Express.js app, Claudia does the following things for you:

• Creates a proxy resource with a greedy path variable of {proxy+}
• Sets the ANY method on the proxy resource
• Integrates the resource and method with the Lambda function

To learn more about proxy integration, see https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html.

13.2.2 How serverless-express works

The Express.js app is a small local HTTP server inside your AWS Lambda function, and the serverless-express module acts as a proxy between an API Gateway event and that local HTTP server.

When the user sends an HTTP request, API Gateway passes that request to your AWS Lambda function. In your function, serverless-express spins up the Express.js server and caches it for repeated invocations, and then transforms the API Gateway event to an HTTP request passed to the local Express.js app.

Then your Express.js app goes through its regular flow—the router routes the request to the selected handler, and all middleware functions are applied. When Express.js sends the response, the serverless-express module transforms it to the format that is expected by API Gateway, which then sends the reply back to the user. The request flow is depicted in figure 13.1.

13.3 Serving static content

Another scenario you want to test is serving static content from an Express.js app, because Uncle Roberto’s admin panel works that way.

To do so, you need a simple static HTML page to test. Any page that includes at least one image and a simple CSS file will work for this test, because that will allow you to test a few different file types.

The first step is to create a new folder named static inside your Express.js project. Then create an index.html file that loads style.css, shows some title text, and shows an image such as the Claudia logo (claudiajs.png). Both the CSS and the logo image will be loaded from the static folder.

Your index.html file should look like the following listing.

Listing 13.6 The index.html file

<!doctype html>
<html>
  <head>
    <title>Static site</title>
    <link rel="stylesheet" href="style.css">    ①  
  </head>
  <body>
    <h1>Hello from serverless Express.js app</h1>    ②  
    <img src="claudiajs.png" alt="Claudia.js logo" />    ③  
  </body>
</html>

Next, add the Claudia logo to the static folder (you can find it in source code you got with this book or on the Claudia website), and create the style.css file in the same folder.

This CSS file doesn’t need to do anything specific, but feel free to be creative. As a simple example, you can style the title to be in Claudia’s blue color and to have a shadow, and center the logo below it. The next listing shows what your CSS file might look like.

Listing 13.7 The style.css file

body {
  margin: 0;
}
h1 {
  color: #71c8e7;
  font-family: sans-serif;
  text-align: center;
  text-shadow: 1px 2px 0px #00a3da;
}
img {
  display: block;
  margin: 40px auto;
  width: 80%;
  max-width: 400px;
}

Then, update the app.js file to serve the static content from the static folder. To do that, you should use the express.static middleware, and your code should look like the following listing.

Listing 13.8 Serving static content in the Express.js app

'use strict'
const express = require('express')
const app = express()
app.use('/static', express.static('static'))    ①  

app.get('/', (req, res) => res.send('Hello World'))

module.exports = app

Now that everything is ready, you can confirm that the Express.js app is working locally by running the command node app.local.js again, and visiting http://localhost:3000/static.

If everything is okay locally, update your app by running the following command:

claudia update

Wait for the command to finish, and load https://8qc6lgqcs5.execute-api.eu-central-1.amazonaws.com/latest/static/ in your browser. You should see your static HTML page with the Claudia logo, similar to figure 13.2.

13.4 Connecting to MongoDB

So far, everything seems to work just fine with only minor modifications. But can you connect AWS Lambda to MongoDB?

You can connect AWS Lambda to any database, but if the database is not serverless, you’ll run into problems when your function scales up and tries to establish too many database connections, because the database will not scale automatically.

To make sure your database can work with an AWS Lambda function, you have the following options:

• Make sure your database can scale quickly.
• Limit your AWS Lambda concurrency to something your database can handle.
• Use a managed database.

The first option requires DevOps and a good understanding of databases, both of which are beyond the scope of this book.

The second option works, but having more users than your concurrent execution limit would result in an error for each user after the limit is reached. If you want to learn more about managing AWS concurrency, visit https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html.

The last option is the easiest and probably the best one, so let’s take a look at it. For MongoDB, which Uncle Roberto’s app is using, you can use MongoDB Atlas, offered by MongoDB, Inc. It hosts the database on one of a few supported cloud providers, including AWS. For more information about MongoDB Atlas, see https://www.mongodb.com/cloud/atlas.

13.4.1 Using a managed MongoDB database with your serverless Express.js app

Your first step is to create a MongoDB Atlas account and create the database, as described in appendix C.

You need to modify the app.js file to connect to MongoDB. To do so, you’ll need to install the mongodb and body-parser NPM modules as dependencies of your Express.js projects. The first allows you to connect to your MongoDB database, and the second allows your Express.js app to parse POST requests.

After you install the modules, you’ll create a connection to the database. AWS Lambda functions are not really stateless, because the same container might be reused if your function is invoked again within the next few minutes. This means that everything outside of your handler function will be preserved, and you can reuse the same MongoDB connection.

For example, if you store your database connection outside of your handler function, you can check if the connection is still active with the following function:

cachedDb.serverConfig.isConnected()

If the connection is still active, you should reuse it. If the database connection is not active, you can create a new one using the MongoClient.connect function and cache it before returning the connection. Then you should activate the body-parser module using Express.js middleware.

Reusing the existing database connection is important, because each database has a maximum number of concurrent incoming connections. For example, a free MongoDB Atlas instance has a maximum of 100 concurrent connections, which means that having more than 100 Lambda functions connecting at approximately the same time will cause some failed requests. Reusing existing connections can help with this issue, and it can also lower the latency, because each database connection takes some time to be established.

The flow of the MongoDB connection from your Lambda function is shown in figure 13.3.

At this point, the beginning of your app.js file should look like the next listing.

Listing 13.9 Beginning of app.js file

const express = require('express')
const app = express()
const { MongoClient } = require('mongodb')    ①  
const bodyParser = require('body-parser')    ②  

let cachedDb = null    ③  

function connectToDatabase(uri) {    ④  
  if (cachedDb && cachedDb.serverConfig.isConnected()) {    ⑤  
    console.log('=> using cached database instance')
    return Promise.resolve(cachedDb)
  }

  return MongoClient.connect(uri)    ⑥  
    .then(client => {
      cachedDb = client.db('taxi')
      console.log('Not cached')
      return cachedDb
    })
}

app.use(bodyParser.json())    ⑦  

Now that you’ve connected your Express.js function to MongoDB, it’s time to test your database connection. The easiest way to test the connection is by writing something to a MongoDB collection and then reading the collection to confirm that the item was saved.

To do so, you can add two routes that will be connected to your MongoDB database: one for writing the data, and one for reading the collection. For example:

• A POST /orders route, which will add a new order
• A GET /orders route, which will return all the existing orders

With these two new routes, the flow for creating and immediately reading orders will work like this:

1. A POST /orders request is received by API Gateway and passed to your AWS Lambda function.
2. The Lambda function starts the Express.js app.
3. The Lambda function then transforms the API Gateway request into an HTTP request to your Express.js app.
4. The Express.js app checks if a MongoDB connection already exists, and if not creates a new connection.
5. Your Express.js handler function stores the order in MongoDB and returns the response.
6. The Lambda function transforms the Express.js reply into the format that API Gateway expects.
7. API Gateway returns the response to the user.
8. The user sends a GET /orders request immediately after, and API Gateway passes it to your Lambda function.
9. The Lambda function transforms the request and passes it to an existing instance of the Express.js app.
10. The Express.js app checks if a MongoDB connection exists and, because it does, uses it to get all the orders from the database.
11. The Lambda function receives a response from Express.js, transforms it, and passes it to API Gateway.
12. The user receives the response from API Gateway with a list of the orders.

This flow is illustrated in figure 13.4.

To connect new route handlers to the MongoDB database, you can use the connectToDatabase function you created in the previous step. Pass it the MongoDB connection string, which can be stored in an environment variable.

After the connection is established, in your GET /orders route you should use the db.collection('orders').find().toArray() function to get all the items from the orders collection and convert them to a plain JavaScript array. This command returns a promise, and when the promise is resolved, you can use the res.send function from Express.js to send a result or an error.

The only difference for the POST /orders route is that you should insert a new item into the database instead of getting an item from the database. To do that, use the db.collection('orders').insertOne function. An order can be a JSON object that contains an address only.

The routes to add to your app.js file are shown in the following listing.

Listing 13.10 Routes for getting and adding new taxi rides

app.get('/orders', (req, res) => {    ①  
  connectToDatabase(process.env.MONGODB_CONNECTION_STRING)    ②  
    .then((db) => {
      return db.collection('orders').find().toArray()    ③  
    })
    .then(result => {
      return res.send(result)    ④  
    })
    .catch(err => res.send(err).status(400))    ⑤  
})

app.post('/orders', (req, res) => {    ⑥  
  connectToDatabase(process.env.MONGODB_CONNECTION_STRING)    ②  
    .then((db) => {
      return db.collection('orders').insertOne({    ⑧  
        address: req.body.address
      })
    })
    .then(result => res.send(result).status(201))    ④  
    .catch(err => res.send(err).status(400))    ⑤  
})

Now that the MongoDB connection is ready, you can test it locally by running node app.local.js—but don’t forget to pass the MONGODB_CONNECTION_STRING environment variable. For example:

MONGODB_CONNECTION_STRING=mongodb://localhost:27017 node app.local.js

If everything works fine locally, run the claudia update command with the --set-env or --set-env-from-json option, and pass the MONGODB_CONNECTION_STRING. For example, your command might look like this:

claudia update --set-env MONGODB_CONNECTION_STRING=mongodb://<user>:
<password>@robertostaxicompany-shard-00-00-rs1m4.mongodb.net:27017,
robertostaxicompany-shard-00-01-rs1m4.mongodb.net:27017,robertostaxicompany
-shard-00-02-rs1m4.mongodb.net:27017/taxi?ssl=true&replicaSet=
RobertosTaxiCompany-shard-0&authSource=admin

After your app is deployed, you can try sending a POST request to https://8qc6lgqcs5.execute-api.eu-central-1.amazonaws.com/latest/orders to add a new order. You can also visit the same address in your browser to list all the orders: https://8qc6lgqcs5.execute-api.eu-central-1.amazonaws.com/latest/orders.

13.5 Limitations of serverless Express.js applications

Now that you’ve tested all the most important cases, you can let Uncle Roberto know that his Express.js application will work in AWS Lambda. He’ll be happy for sure, and you might end up with a lot of free taxi rides.

But before you do that, it’s important to be aware that there are some limitations for serverless Express.js apps. Let’s address the most important ones.

First, and probably most obvious, is that you can’t use WebSockets in serverless Express.js apps. If Uncle Roberto is using WebSockets for real-time communication between his mobile app and the back end, serverless Express.js will not work as expected. Some limited support for WebSockets in AWS Lambda can be achieved through AWS IoT MQTT over the WebSockets protocol. To read more about the MQTT protocol, see https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html#mqtt. For an example project using Claudia, visit https://github.com/claudiajs/serverless-chat.

Another limitation is related to the file upload functionality. First, if your application is trying to upload files to any folder except /tmp, the upload will fail, because the rest of the AWS Lambda disk space is read-only. Even if you’re saving uploaded files to the /tmp folder, they will exist for a short time. To make sure your upload feature is working, upload files to AWS S3.

The next limitation is authentication. You can implement authentication in serverless Express.js apps as you do in any other Express.js app, for example using the Passport.js library, but you need to make sure that the session is not saved on the local filesystem. Or, if you use native Node.js libraries, you’ll need to have them packaged into the static binary using an EC2 machine running Amazon Linux. To learn more about native libraries, also known as Addons, see https://nodejs.org/api/addons.html.

Also, API Gateway has stricter rules than traditionally hosted Express.js apps. For example, in Node.js and Express.js, you can send a body with a GET request; API Gateway will not allow you to do that.

Additionally, there are certain execution limits—for example, API Gateway has a timeout of 30 seconds, and AWS Lambda’s maximum execution time is 5 minutes. If your Express.js app needs more than 30 seconds to reply, the request will fail. Also, if your Express.js app needs to answer the HTTP request and continue the execution, that will not work in AWS Lambda because AWS Lambda execution will stop as soon as the HTTP response is sent. This behavior depends on a callbackWaitsForEmptyEventLoop property of Lambda context; the default value for this property is true, and it means that the callback will wait until the event loop is empty before freezing the process and returning the results to the caller. You can set this property to false to request AWS Lambda to freeze the process soon after the callback is called, even if there are events in the event loop.

As long as you have these limitations in mind, your Uncle Roberto’s taxi app will work fine in AWS Lambda.

13.6 Taste it!

It’s time for a small exercise with Express.js.

13.6.1 Exercise

As an exercise, add a DELETE /order/:id route that will delete a request using the request ID that is passed as a URL parameter.

Here are a few tips that should help you:

• URL parameters are defined in the Express.js way (using :id), not in the API Gateway and Claudia API Builder way (using {id}).
• You can delete an item from MongoDB using the collection.deleteOne function.
• Make sure you convert the order ID to a MongoDB ID using the new mongodb.ObjectID function.

If you need an additional challenge, try implementing authentication in your Express.js app. (You can also try running an existing Express.js app in AWS Lambda, if you have one. There are no tips or a solution for this additional challenge in the next section.)

13.6.2 Solution

The solution for this exercise is similar to implementing the POST /orders route.

You need to add a new DELETE route to your app.js file, using the app.delete method. Then you need to connect to the database and use the db.collection('orders').collection.deleteOne function to delete an item from the orders collection. Because the order ID is passed as a string, you need to convert it to a MongoDB ID using the new mongodb.ObjectID(req.params.id) function.

Your new route should look like the following listing.

Listing 13.11 The delete order route

app.delete('/orders/id', (req, res) => {    ①  
  connectToDatabase(process.env.MONGODB_CONNECTION_STRING)    ②  
    .then((db) => {
      return db.collection('orders').collection.deleteOne({    ③  
        _id: new mongodb.ObjectID(req.params.id)    ④  
      })
    })
    .then(result => res.send(result))    ⑤  
    .catch(err => res.send(err).status(400))    ⑥  
})

After you deploy the function using claudia update, you can test the delete method using curl or Postman.

Summary

• You can run Express.js apps in AWS Lambda using Claudia and the serverless-express module.
• You can serve static pages using serverless Express.js without additional modifications.
• For a MongoDB connection, use a managed MongoDB instance unless you want to manage the scaling by yourself.
• Cache a database connection in a variable outside of the handler function.
• There are certain limitations, such as when using WebSockets and for requests taking longer than 30 seconds.