AWS Lambda Benchmark
>

Comparing Java and Node.js on AWS Lambda

Mark West 
6. aug. 2017

In this blog I share some experiences implementing a set of AWS Lambda Functions in both Node.js 6.10 and Java 8. I’ll begin with some initial observations before moving on to focus on performance and cost.

The AWS Lambda Functions

Earlier on this year I created a set of Node.js AWS Lambda Functions as part of my Smart Security Camera project. This project added image analysis to a Pi Zero web camera, allowing it to distinguish between real threats (i.e. burglars) and false alarms (i.e. the neighbours cat).

Once the original Node.js version of this project was completed I decided to recreate the same AWS Lambda Functions with Java, with the aim of comparing how Node.js and Java work on AWS Lambda.

The AWS Lambda Functions are as follows:

Java AWS Lambda Function
Node.js AWS Lambda Function
Purpose
AWS Service Calls
s3-trigger-image-processing
s3-trigger-image-processing
Triggered when a new image is uploaded to a given s3 upload folder.
-
rekognition-image-assessment
rekognition-image-assessment
Uses AWS Rekognition to generate a list of labels describing each uploaded picture.
Rekognition
rekognition-evaluate-labels
rekognition-evaluate-labels
Evaluates labels to find out if an alarm email should be sent.
-
ses-send-notification
nodemailer-send-notification
Sends an alarm email via AWS SES and JavaMail when the smart security camera detects a person.
SES
s3-archive-image
s3-archive-image
Moves the processed image to the correct archive location in S3.
S3

Table 1 : Lambda Function Overview

From a functional perspective the two sets of Lambda Functions are identical. Some, but not all, make calls to one of the following AWS Services:

  • AWS S3 - Copy and deleting files on S3. Monitoring S3 for new Image Uploads.
  • AWS Rekognition - Image Analysis of a image uploaded to S3.
  • AWS SES - Sending an email with an Image attachment.

Initial Observations 

Differing Memory Requirements

When creating Lambda Functions you can choose how much memory the functions are allocated at runtime. The more memory you allocate, the higher the cost per CPU second.

My Node.js AWS Lambda Functions were happy with the minimum memory allowance of 128MB. However my Java based Functions tended to terminate with OutOfMemoryError exceptions unless they were allocated at least 192MB and preferably 256MB. This resulted in a higher base cost per CPU second when using Java.

We'll look more closely at the relative costs of using Java vs. Node.js later on in this blog entry.

Size of Created Artefacts 

Another difference between Node.js and Java was the size of the created artefacts.

Java AWS Lambda Function Java Function Code Size Node.js Code Size Node.js AWS Lambda Function
s3-trigger-image-processing 8.2MB 954 bytes s3-trigger-image-processing
rekognition-image-assessment 7.9MB 813 bytes rekognition-image-assessment
rekognition-evaluate-labels 7.6MB 677 bytes rekognition-evaluate-labels
ses-send-notification 8.7MB 4.8MB nodemailer-send-notification
s3-archive-image 7.6MB 943 bytes s3-archive-image

Table 2 : Relative sizes of Node.js and Java artefacts

As you can see the Node.js artefacts are substantially smaller than their Java counterparts. There are a couple of reasons for this.

Reason 1: The AWS Toolkit for Eclipse

When creating the Java version of my Smart Security Camera project I used the AWS Toolkit for Eclipse, which provides support for creating, packaging and uploading AWS Lambda Functions. Functions are packaged into ZIP files which also include a set of dependancy JAR files (including the AWS SDK).  

Unfortunately the AWS Toolkit isn't selective about which JAR's it adds. A quick analysis of a deployed ZIP file revealed that it included a set of AWS Client JAR's that were not required for that specific Lambda Function to run. 

Amending the Maven POM file to remove these unneeded JAR's helped reduce the size of the Java Functions by up to 2MB.

Reason 2: Writing Node.js Code Directly in the Browser

If your Node.js based AWS Lambda Function doesn't require any 3rd party libraries, you can choose to write your code directly in the browser and avoid the packaging and upload steps. If you do this, the AWS SDK is implicitly available (no need to upload it).

In the above table, the four "smallest" functions were all written in this way, which explains their small size.

The "largest" Node based AWS Lambda Function was packaged as a ZIP file due to it's dependancy on the Nodemailer library.

Comparing Performance and Cost

AWS Lambda Pricing Model

Simply put, AWS Lambda is billed by Requests and Duration.

Request

The first million requests per month are free. After this you will be charged $0.20 per 1 million requests thereafter.

Duration

Duration is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 100ms.

Both price and the free usage limit depend on the amount of memory you allocate to your function, as shown in the table below.

Memory (MB) Free tier seconds per month Price per 100ms ($)
128 3,200,000 0.000000208
192 2,133,333 0.000000313
256 1,600,000 0.000000417
1536 266,667 0.000002501

Table 3 : Duration pricing examples

You can find more detailed information about AWS Lambda pricing at  https://aws.amazon.com/lambda/pricing.

Testing AWS Lambda Performance

Method 

I created a test battery that would simulate approximately 1500 alerts through my Smart Security Camera. This would result in all the above Lambda Functions being called - once for each alert.

To simulate the behaviour of my camera, alerts were triggered in batches of 30, with a 60 second pause between them.

As mentioned earlier in this blog, the Java based AWS Lambda Functions required 256MB memory to ensure that no OutOfMemoryException errors were thrown. To be fair I therefore allocated the same amount of memory to the Node.js Functions.

Before running the test batteries the Lambda Functions were allowed to be inactive for a period of 90 minutes.

Finally I used a dashboard that I created via AWS CloudWatch to view and analyse the results.

Results 

The initial results showed a marked difference in average performance between Java and Node.js.

Java AWS Lambda Function Java Average Duration Node.js Average Duration Node.js AWS Lambda Function
s3-trigger-image-processing 970 ms 419 ms s3-trigger-image-processing
rekognition-image-assessment 2.25 sec 1.72 sec rekognition-image-assessment
rekognition-evaluate-labels 6.57 ms 1.61 ms rekognition-evaluate-labels
ses-send-notification 3.04 sec 996 ms nodemailer-send-notification
s3-archive-image 1.05 sec 364 ms s3-archive-image

Table 4 : Duration comparison based on 256MB memory allocation

The following two charts, generated by AWS CloudWatch, show how the average duration varied over the course of the test. Note that the vertical axis (representing duration) has different scales for Node.js and Java.

Java Chart 3

Figure 1 : Chart showing average duration for Java functions during the test. The vertical axis represents the average function execution duration in milliseconds, while the horizontal axis represents time

Nodejs Chart 3

Figure 2 : Chart showing average duration for Node.js functions during the test. The vertical axis represents the average function execution duration in milliseconds, while the horizontal axis represents time

Observations 
Observation 1 : Java costs more than Node.js on AWS Lambda 

Based on the results of this test (and pricing information from AWS) I was able to extrapolate a total duration cost for these Lambda Functions processing 1,500,000 alerts in a month. When making this extrapolation I did not factor in the Free Tier allowance.

Type

Duration

Duration Cost 
Java 10,815,680 seconds $45.10
Node.js 5,152,370 seconds $21.49

Table 5 : Cost extrapolation

While these figures look pretty definitive, lets remember that they are based on a test with only 1500 iterations. The Java figures will therefore be strongly affected by the time it takes to warm up the JVM.

Observation 2 : Cold starts with the Java container take longer

AWS Lambda works by spinning up runtime containers on demand. While there is increasing demand, more runtime containers will be provisioned. When demand shrinks, idle runtime containers are removed.

The process of spinning up a new runtime container is known as a cold start. If you take a look back at figures 1 and 2 you can see these cold starts at the beginning of the test, where the average duration used to execute the functions is longer. Once the containers are warmed up the average execution times drop down and plateau.

Figures 1 and 2 show that both Node.js and Java require cold starts and that both are nicely warmed up after 5 minutes or so. But the cost (i.e. average duration) of the Java cold starts is much higher for Java than for Node.js.

The difference in size between the Node.js and Java artefacts are most likely also a factor in the amount of time to cold start the respective containers.

Observation 3 : The AWS Lambda JVM is effectively a black box

You can tweak the amount of memory available to the JVM, but that seems to be about it. It may possible that further tweaking (for example around Garbage Collection) may improve the Java results from this test.

Observation 4 : A good use case for Project JigSaw and Java 9

Project JigSaw refers to the modularisation of the JDK that is coming in Java 9. This will reduce the JDK's footprint and hopefully also help improve JVM performance. With a Java 9 release date of September 2017 it could take a while before a modular JDK appears on AWS Lambda.

Observation 5 : Although the two sets of Lambda Functions are functionally identical, the underlying implementations are very different

My two Lambda Function stacks use different third party implementations to solve common problems such as: 

  1. Sending Emails: For my Java implementation I use JavaMail and for my Node.js implementation I use NodeMailer
  2. Marshalling / Unmarshalling JSON: AWS Lambda Functions using JSON for communicating input and output parameters. In the JavaScript world JSON is a first class citizen, with no third party libraries required. In the Java world, one will generally use the Jackson library for parsing JSON objects.

What this all means is that both the size and performance of my AWS Lambda Functions will be influenced by the implementations and sub-dependancies of these third party libraries. 

Observation 6 : Calls to the AWS SDK slow things down

For both Node.js and Java the fastest Function calls were those that did not make a subsequent call to other AWS services:

  • rekognition-evaluate-labels: average 6.57 ms on Java, 1.61 ms on Node.js.
  • s3-trigger-image-processing: average 970 ms on Java, 419 ms on Node.js.

This is not really surprising as functions making calls to other AWS services (i.e. Recognition, SES or S3) will need to wait for the underlying function call to complete, therefore upping their execution time. The lesson here would therefore be to avoid synchronous calls where possible.

Summary

This article sums up my experiences with measuring performance of Node.js 6.10 and Java 8 on AWS Lambda.

That Node.js outperformed Java can likely be attributed to the following:

  • The relatively larger size of the Java ZIP files.
  • A longer cold start for Java containers in AWS Lambda.
  • A lack of options for tuning the JVM in AWS Lambda.
  • Fundamental differences in the implementations of the Java and Node.js AWS Lambda Functions.

As AWS Lambda is partially billed by execution duration, one can also argue that Java AWS Lambda Functions are more expensive than Node.js equivalents. However this argument is only relevant if you plan on exceeding the free tier limits.

Does all this mean that Java is a no-go on AWS Lambda? Not at all. Strict typing and the ecosystem around Java make it a very effective platform for creating software. Eclipse plugins and Maven archetypes exist to help one quickly get up and running with Java based AWS Lambda Functions. 

There is definitely a need for further testing. Perhaps allocating more memory might help Java catch up with or even beat JavaScript - perhaps even to the point when Java becomes cheaper than Node.js on AWS Lambda?

Finally thanks for reading this blogpost! If you have any comments or questions, please share them in the comments below! 

More information about the Smart Security Camera Project is available via the following links: