Invoking AWS Lambda Functions from API Gateway and SQS
• AWS, API Gateway, SQS, Lambda
I’ve noticed people trying to return HTTP status codes from AWS Lambda functions when those functions are set up to receive events from SQS. That’s not necessarily “wrong”, but it really isn’t necessary. What is wrong is returning an object that tries to set the response status code to 500 and expecting SQS to retry the message. That is not how this works, you’re not working with API Gateway.
AWS Lambda with API Gateway
I imagine integrating with API Gateway is one of the first places a lot of developers interact with AWS Lambda functions. Things are straightforward, receive an event with details about the HTTP request, return an HTTP response with a status code, a body, and maybe some headers.
Here is what an event from API Gateway might look like. This example comes straight from AWS’s Lambda documentation.
{
"resource": "/",
"path": "/",
"httpMethod": "GET",
"requestContext": {
"resourcePath": "/",
"httpMethod": "GET",
"path": "/Prod/"
},
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding": "gzip, deflate, br",
"Host": "70ixmpl4fl.execute-api.us-east-2.amazonaws.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-5e66d96f-7491f09xmpl79d18acf3d050"
},
"multiValueHeaders": {
"accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"accept-encoding": ["gzip, deflate, br"]
},
"queryStringParameters": null,
"multiValueQueryStringParameters": null,
"pathParameters": null,
"stageVariables": null,
"body": null,
"isBase64Encoded": false
}Responses expected by API Gateway have a few of the same properties, below is an example of a response from AWS’s documentation.
{
"statusCode": 200,
"headers": {
"Content-Type": "application/json"
},
"isBase64Encoded": false,
"multiValueHeaders": {
"X-Custom-Header": ["My value", "My other value"]
},
"body": "{\n \"TotalCodeSize\": 104330022,\n \"FunctionCount\": 26\n}"
}For most who have been working with Lambda functions, I expect this is mostly a review.
AWS Lambda with SQS
When Lambda functions receive an event from SQS it can have multiple messages, called records, in the event. The maximum number of records to receive in a single event can be configured in the SQS integration to Lambda.
Below is an example of an event with two records from AWS’s Lambda documentation. These records are fairly simple, the body is plain text and there are no message attributes.
{
"Records": [
{
"messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
"receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1545082649183",
"SenderId": "AIDAIENQZJOLO23YVJ4VO",
"ApproximateFirstReceiveTimestamp": "1545082649185"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
"awsRegion": "us-east-2"
},
{
"messageId": "2e1424d4-f796-459a-8184-9c92662be6da",
"receiptHandle": "AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1545082650636",
"SenderId": "AIDAIENQZJOLO23YVJ4VO",
"ApproximateFirstReceiveTimestamp": "1545082650649"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
"awsRegion": "us-east-2"
}
]
}If you look at AWS’s documentation you don’t see anything about the expected response from the Lambda function. That’s because this is a pass/fail situation. If your Lambda function handles the event cleanly, i.e., without throwing an exception out of the handler, then the message was processed and SQS deletes the message. If your Lambda function exits with an error, then SQS does not consider the message processed. If SQS is configured to do so, and it really should be, it will try the message again and eventually place the message on a configured Dead-Letter Queue (DLQ).
You can stop reading here if you want. You’ve got the basics; different services expect different responses from Lambda functions. Remember that and check the documentation and you’ll be fine. Keep going and it’s a bit more of a deep dive into Lambda function invocations.
Lambda Invocations
There are two ways to invoke a Lambda function: synchronously and asynchronously. When you invoke a Lambda function synchronously you are waiting for the Lambda function to complete and return a response. When you invoke a Lambda function asynchronously you don’t wait for it to finish and respond. You can read about the details of synchronous and asynchronous invocation in the AWS documentation.
AWS builds everything to have an API, it’s ultimately how we end up interacting with everything. This includes invoking Lambda functions. The HTTP response from the API when we invoke our function is what the function returns, right? It only makes sense, and it is exactly what happens. The important thing to think about is Lambda does not care what you return. There is no magic here. The magic comes from the services that invoke your Lambas.
A successful synchronous call to the invoke API is going to return an HTTP status code of 200 and whatever your Lambda returns. If that payload you return indicates an HTTP 200 status code it is only because HTTP status codes are a standard. Lambda does not care.
If API Gateway calls Lambda and your function returns a JSON object indicating a status code of 500 it is API Gateway that cares about that. The invoke API that API Gateway used is still returning that HTTP 200 response from the synchronous call. API Gateway takes that payload and maps it to the response your API will return. That’s where the returned HTTP 500 status code ends up.
When your Lambda function is connected to an SQS queue it gets invoked synchronously via the API too. In this situation, you have the Lambda service polling the queue and invoking the Lambda function. Lambda doesn’t care about the payload from the API call. It only cares about the HTTP status code and a couple headers; the rest of the information is likely thrown away. Let’s say you throw an error in your Lambda function and never catch it, know what HTTP status code the invoke API returns? 200. It invoked your function, that went fine. The API did its job and it didn’t have an error. Your function had an error. When that happens there is a custom HTTP header set for the function error. That header is what will cause the Lambda service to not consider your processing of the messages from the queue a success.
Some services can invoke Lambda functions asynchronously, like SNS and S3. These services use asynchronous invocations because they don’t need to do anything with the response from your Lambda function. API Gateway uses synchronous invocation because it needs to process the results. SQS uses synchronous invocation because it needs to know if the message batch was successfully processed. SNS and S3 don’t have any of that. Once the Lambda service has your message their job is done. Internally the Lambda service drops asynchronous messages into a queue and passes them to your Lambda function. The invoke API, in this case, returns an HTTP 202 status code to the service sending the event. That HTTP 202 status code means the Lambda service has accepted the invocation and placed it on the queue.
Lambda functions make it easy to break up an application into smaller pieces, but you don’t want to find out a month after release that you have lost events because you didn’t exit your Lambdas correctly. Knowing a little more about how Lambda functions are invoked and why they are invoked in certain ways can help make sure you find these issues before people are testing things out.