Build a serverless URL shortener with AWS Lambda and API Gateway services
Let's work together!
I'm also available for DevOps project work. Are you fighting fires instead of serving your customers? Hire me to identify the roadblocks and make improvements so you can get back to focusing on your customers.
I’ve had my eye on Amazon’s Lambda and API Gateway services for a few months. Running a web application without the costs or headaches of maintaining servers is attractive. Of course, no platform is without tradeoffs. So to get myself familiar with the finer points of serverless apps I decided to launch a simple project: a URL shortener service.
With a few hours of work and a few dollars per month in cost I got a decent prototype working capable of handling millions of requests per month. This post walks through the setup along with the hitches I hit. There’s also an accompanying code repo that you can use to try this out yourself.
Here were the basic requirements for the project:
- One HTTP endpoint to accept a JSON
POSTcontaining a short token and destination URL. These values would need to be stored somewhere.
- A second HTTP endpoint to take a short token via
GET, lookup a corresponding destination URL, and return a 301 redirect.
- The redirection endpoint should be accessible without authentication while the
POSTshould require authentication.
- Define everything via code as much as possible to make it reproducible.
- Put the service on a custom domain.
To accomplish this I needed a few components:
- A front-end to take in HTTP requests.
- A back-end to do something with the requests and generate responses.
- A datastore to keep all the associated short tokens and destination URL’s.
Let’s take a closer look at each of these components. At the end of the post I’ll layout some gotchas and how much this all costs.
Front-end: Amazon API Gateway
At the outset I wasn’t sure Amazon API Gateway would handle everything needed for the front-end. It would need to support both
text/html content as well as setting custom response codes and headers. Different content types are supported. And support for mapping Lambda function results to response headers was added at the end of 2015. API Gateway looked like it would fit the bill.
Creating an API
An API Gateway service is composed of resources and methods. Resources are essentially URL endpoints. Methods correspond to HTTP methods like
The URL shortener’s endpoints are limited. There’s one
GET to turn a provided short URL into a redirect. And a
POST to take in short token and destination URL associations for storage.
Each API Gateway method has four components. This is an example for the
GET method that accepts a token and returns a redirect:
- Method Request defines the incoming request including any parameters that get pulled from the request path, query string, and headers. It also supports enabling authorization for the particular method.
- Integration Request maps the request to a backend. In this case I’m using Lambda Function but it also supports proxying to another HTTP service or mocking responses for development purposes. Here you pick which Lambda function will handle a single request and map request parameters to a backend data payload.
- Method Response defines a collection of response status codes and headers that your service supports.
- Integration Response maps return data from Lambda function execution to appropriate response codes. This can be done using regex matchers. More detail to follow on that. It also allows you to set custom response headers and setup templates to transform Lambda results.
Mapping Lambda Results to Response Headers
Part of the magic for the URL shortener is mapping Lambda function results to a response header. I needed to set the
Location header to a destination URL provided by a Lambda function based on a short token lookup.
Below is the Integration Response settings for the
GET method. Note the
Location header mapping to a value in the Lambda function’s return JSON.
The url shortener’s
POST method requires authentication to control who can post entries. One of the authentication options API Gateway offers is an API key. API keys can be generated and associated with applications. When enabled on a method, all requests to that method must include a valid API key header to execute:
This approach allowed me to require authentication on the URL
POST and leave the
GET endpoint open to the world.
Managing API as Code
One of my goals was to manage this project in code instead of clicking through the Amazon control panel. API Gateway offers the ability to export and import application configuration in Swagger format.
Swagger is a specification aimed at describing and documenting RESTful API’s in JSON or YAML format. A variety of services can consume Swagger files to visualize, test, or implement an API service.
I have to concede that getting started with Amazon’s API Gateway service was much easier using the web based control panel. I ultimately was able to export the service definition into a YAML file. I was also able to apply changes and clone the API using the aws-apigateway-importer tool provided by Amazon.
All that said, I created the service first by clicking through the web panel. It greatly helped with my understanding of how API Gateway applications are structured. I recommend using the web panel as a starting point.
Back-end: Amazon Lambda
Function Inputs and Outputs
Lambda functions are executed on demand. Unlike EC2, AWS ensures the compute environment whenever the function is executed. There’s no server to maintain and capacity scales as needed.
Each function can take data in the form of parameters, execute functions, and return data. Functions can import external libraries for things like connecting to other AWS services.
Lambda handler functions receive two parameters when invoked:
context. When executed by an API Gateway
event includes parameters translated from the incoming request. In the case of a token lookup, the token value from the URL is mapped to a JSON property named
context argument provides runtime information about the function’s execution environment.
URL Shortener Functions
The URL shortener has two simple functions:
redir_lookup_token Takes in a token value and returns a corresponding destination URL.
redir_post_token Takes in a short URL and destination URL association and stores it.
Handling Errors with API Gateway
The Lambda function’s
context argument also includes methods for generating a response. It offers
done methods. The last method is a helper wrapper for the first two. It takes two parameters and if the first is anything other than
null it triggers a
API Gateway offers regex pattern matching for mapping
fail error messages to appropriate response status codes. This allows for translating backend failures to meaningful client responses. Various response status codes can be defined and associated with a regex pattern. If the Lambda function returns an error message matching one of the patterns, the client response will return the matching status code.
Each response code has its own template mapping. This allows response content to be tailored individually to each response condition. For the purposes of the short URL lookup, a 301 is the default response code for a successful request. If someone attempts a lookup for a token that doesn’t exist then a 404 should be returned.
The following shows an error condition in the token lookup Lambda function (line 7) and the corresponding 404 message response from the API Gateway.
API Gateway response statuses
Lambda function error result handling
Managing Lambda Functions with Apex
Lambda functions can be created directly in the AWS web panel or uploaded via zip file package. There are frameworks that help manage the coding and deployment of Lambda applications from a local development setup.
As you write code you can deploy changes using the
apex deploy command.
> apex deploy • deploying function=post_token • deploying function=lookup_token • created build (1.1 kB) function=lookup_token • created build (1.2 kB) function=post_token • config unchanged function=post_token • code unchanged function=post_token • config unchanged function=lookup_token • code unchanged function=lookup_token
Once uploaded, you can execute functions from your laptop with sample input using the
apex invoke command.
For small projects Apex’s structure is easy to get working quickly and it doesn’t get in the way. The project mentions plans to include API Gateway management in the future.
In keeping with the serverless theme, DynamoDB was a natural fit for storing the short and destination URLs. With DynamoDB instead of standing up a server you specify allotments of read and write capacity. Amazon ensures the allotted read and write capacities are available. You can scale up or down capacity units as needed.
DynamoDB tables are schemaless. Primary indexes can be a single value or a composite of two fields. An important limitation of composite keys is that they are hierarchical. This means that the second field is selectable in the context of the primary field. Selecting it on its own results in expensive table scans. Since the URL shortener primary index is a single short token string this was not a concern.
How much does this cost?
So how much does this all cost? The following examples assume a working example of roughly 1 million hits over a month. Prices do not include data transfer costs which will vary.
|API Gateway||per 1 million requests per month||$3.50|
|Lambda||(hits * typical execution seconds) * (memory/1024) * $.00001667||$1.04|
|DynamoDB||1 read, 1 write per second||$0.58|
|Under the AWS Free Tier this would be free.|
What are the gotchas?
- API Gateway response times can vary greatly. There is support for response caching though which greatly improves latency.
- Importing API Gateway from Swagger doesn’t create required Lambda permissions. I had to manually reselect each Lambda function through the web panel to apply the appropriate permissions for a service newly cloned from Swagger.
- API Gateway Swagger exports don’t seem to include any model schemas you create. The web panel seems to show an error message if you have no models.
- Quick followup post on creating a model: Forgot about Amazon API Gateway models
- API Gateway custom domains require SSL. There’s no connection with AWS Certificate Manager of Route 53 at this point. Certificates must be manually posted.
- Custom domains are pointed at API Gateway using a CNAME record. This means using a subdomain which isn’t ideal for short URLs.
- I wrote a followup post explaining how to do this: Using custom domains with Amazon’s API Gateway
February 24, 2016