This article explains how to use AWS Lambda with gRPC Protobufs. Before diving deep into the implementation details, here are brief descriptions to AWS Lambda and gRPC.
AWS Lambda - Lambda is also known as Function as a Service(FAAS) or simply “Serverless”. This service by AWS lets user define a sequence of computation as a function. Lambda saves a lot of time for developers, as they only need to write the core logic, and everything else is taken care by the Cloud provider. This logic will automatically execute whenever a trigger event happens.
gRPC - Google’s implementation of RPC protocol. Unlike traditional RPC(Remote Procedure Call), gRPC is language independent. To attain language independence, gRPC uses Protocol Buffers or Protobufs. Protobufs takes care of translating the data to programming language specific data types. Also, gRPC is multi-folds faster than REST API. Important reason is gRPC does not have to deal with JSON packaging and parsing during communication, which skips a significant overhead time especially in micro-service architecture settings.
Is it possible to trigger a Lambda function using gRPC?
On an abstract level, both gRPC functions and Lambda are “Remote Methods”. End user/Client tries to invoke a method defined in a remote computer. But there are differences in the way gRPC and Lambda are invoked.To proceed further into this question, its vital to understand the internals of gRPC and Lambda.
gRPC implementation is Client-Server architecture. If a Computer-B wants to execute a method on Computer A, here gRPC-Client(Computer-B) requests the gRPC-Server(Computer-A) over HTTP/2 to execute the remote method. So, the server will always be running waiting for requests from clients to be served.
Whereas, Lambda is based on Serverless architecture. There will be no active servers listening for requests. Instead, we can configure Lambda to execute on certain events. On such events, AWS will spawn up a light weight Virtual Machine, loads and executes the user defined Lambda function, returns the computed response and kills the VM.
While gRPC needs an active server, Lambda is a serverless function. It is possible to make Lambda a gRPC client, but not as a gRPC server. So currently on the time of this writing, it is not possible to invoke a Lambda function using gRPC by default.
Workarounds
The major challenge here was to deal with the design differences. gRPC requires a Server, while Lambda is designed to be serverless.
It is still possible to write a Lambda function that can act as a gRPC server listening to the requests from clients(using some workarounds obviously). Once when the Lambda is triggered, the gRPC server defined in the lambda starts and it keeps on running. This may seem as a valid workaround, but is not a viable solution.
Cost of AWS Lambda is computed based on the memory it consumes and the execution time. gRPC server on Lambda function essentially means, it keeps on executing without termination. It will eventually be very expensive than setting up a gRPC server on a standalone EC2 machine itself.
There are other people who also tried to combine gRPC and Lambda, and there is no direct way of achieving this. This article from Coinbase Blog is well documented, and shares different approaches they tried to achieve this. I’m skipping my failed attempts, as the Coinbase article itself is very extensive.
What if we write gRPC server code on Lambda? We just make sure it shuts down after the execution.
Combining the Pro’s of gRPC and Lambda:
gRPC’s main advantage is the Protobuffs. Protobuffs provie gRPC the language independence, so the client and server can be written in different programming languages, but still they can communicate with each other. Protobufs provide an internal representation for the methods and datatypes. It basically acts as a translator, and helps translate different data types across programming languages.
What if we make use of this Protobufs to communicate with the Lambda function. It dosent have a practical use case, but it’s just fun to experiment! So went ahead with this approach.
Implementation design:
Client:
My Client is built with Scala.
- Receive input from the User
- Validate the input. On failure, prompt user to provide input again.
- Create a Protobuf message object using the user defined inputs.
- Convert the object to string, then encode to bytes, and send it over to AWS Lambda API Gateway endpoint.
- Display the response from Lambda to the user.
Lambda Server:
- Setup API Gateway endpoint as a trigger function of Lambda invocation.
- On invocation, read the request body from the client.
- Decode the encoded request body, and use the resulting string to recreate Protobuf message object.
- Using the information from the Protobuf message object, perform computation, and return the result to the client.
We don’t use the full functionality of gRPC, but only the Protobuf. This basic idea can be extended to work with any programming languages or applications. But we need to be careful of overhead time to encode and decode protobuf messages.
Example Source code:
Find the github repo for the project that I built to invoke Lambda function using gRPC. Client requests a timestamp, and Lambda checks the log files for existence of logs for that timestamp.
https://github.com/laxmena/AWS-Lambda-with-gRPC
More readings:
How to write and deploy AWS Lambda Function (Will be published soon)
How to install python dependent libraries in AWS Lambda Environment (Will be published soon)
Getting started with gRPC in Scala
If you like the articles, have comments or feedbacks, do write an email to ConnectWith@laxmena.com or connect with me in LinkedIn.
Subscribe to my email newsletter for more articles and informations like this one!