Automating routine tasks is essential in today’s dynamic and agile cloud environments. Traditionally, cron jobs have been the backbone of these tasks, handling activities like data aggregation, system updates, patch applications, and cache cleaning. However, with serverless computing, AWS Lambda provides a scalable and cost-effective alternative to conventional cron job setups that require an “always-on” server.
In this article, we will explore best practices for building event-driven cron jobs using AWS Lambda and best practices to consider when choosing this solution approach.
Introduction
Cron jobs have been a staple for automating recurring tasks in traditional IT environments. However, maintaining servers and ensuring high availability can be time-consuming and expensive. With its event-driven model and serverless nature, AWS Lambda is a game changer, allowing you to run code without provisioning or managing servers.
We will explore how to leverage AWS Lambda to create robust and scalable cron jobs. We will also cover the integration with Amazon EventBridge for scheduling, provide a sample implementation, and discuss best practices for keeping our solution resilient and cost-effective.
Event-Driven Architecture (EDA) and Cron Jobs
Event-driven architecture (EDA) is a paradigm in which applications respond to events or changes in state. Instead of sequentially executing tasks, components in an EDA system react to events, making the overall system more scalable, maintainable, and decoupled.
Cron jobs traditionally run at fixed intervals, but when integrated into an event-driven system, they can trigger downstream events or processes. For instance, a scheduled Lambda function might:
Invoke APIs to pull in fresh data from a data source
Initiate data aggregation tasks
Trigger a system update
Start batch processes or workflows
This approach aligns with microservices architectures and modern cloud-native designs where loosely coupled components communicate through events.
Why AWS Lambda?
AWS Lambda is the backbone of serverless computing on AWS, offering several benefits that make it an ideal choice for scheduled tasks:
Serverless Model: No need to manage or provision servers
Cost-Effective: Pay-per-use model, eliminating idle server costs.
Event-Driven Integration: Seamless integration with Amazon EventBridge (and CloudWatch Events) allows you to trigger Lambda functions on a schedule.
Scalability: Lambda functions automatically scale in response to incoming events.
Ease of Maintainance: Simplified deployment and updates make it easier to iterate on your scheduled tasks.
These benefits make AWS Lambda a natural fit for running cron jobs in a modern, event-driven environment.
Setting Up Scheduled Lambda Functions with EventBridge Scheduler
Amazon EventBridge Scheduler is a fully managed service that enables you to schedule tasks based on rate or cron expressions for recurring tasks or set up tasks for one-time invocations. Unlike traditional Cloudwatch Event rules, EventBridge Scheduler offers additional flexibility, such as flexible time windows, timezone support, and a dedicated scheduling interface.
Schedule Expression Types: Cron vs. Rate vs. At
EventBridge scheduler supports three types of schedule expressions, each serving a different purpose:
cronExpressions: This is used if your task(s) is a recurring event that needs to run at specific times.rateExpressions: Used to define a simple, recurring interval without specifying an exact time. It is set up with the syntax:rate(value), wherevaluecan be minutes, hours, or days. For example, the schedule expressionrate(6 hours)triggers an event every six hours. This expression time is best for straightforward periodic tasks that need to run every fixed interval.atExpressions: This is used to schedule one-off events, such as triggering a task at the end of a project or on a specific future date. It is set up with the syntaxat(YYYY-MM-DDThh:mm:ss)following the ISO 8601 format.
The cron Expression Syntax
The cron expression allows you to set up a recurring schedule that runs precisely when you want. With EventBridge Scheduler, you can configure these schedules in UTC or any time zone of your choice. This approach gives you precise control over when and how often your tasks execute.
In simple terms, a cron expression is a string made up of five required parts: minutes, hours, day-of-month, month, and day-of-week—and one optional part for the year , syntactically typed—cron(Minutes Hours Day-of-month Month Day-of-week Year).These parts are separated by spaces, allowing you to define a detailed, custom schedule that fits your needs.
The cron expression below outlines an example expression used with the EventBridge Scheduler to trigger a job that runs every data at 10:00 AM UTC:
cron(0 10 * * ? *)
AWS EventBridge Scheduler uses a ? character in the Day-of-month or Day-of-week field when one is not specified. To learn more about the cron syntax that is valid with AWS EventBridge, check out the official documentation.
Event-Driven Lambda Functions in Action
The following diagram illustrates the high-level architecture of the serverless cron job solution using Amazon EventBrige Scheduler:
Following the diagram, we will set up an implementation using AWS SAM. This implementation will deploy a Lambda function with an EventBrigde Scheduler schedule that triggers the function at the desired time.
Prerequisites
To reproduce this solution, you must have the following:
AWS Account
AWS SAM CLI—Checkout the official installation guide here
Yeah, a computer 😉
Copy the below command and pull the solution from my GitHub for a fast-tracked spin-up.
git clone https://github.com/NwekeChidi/serverless_labs.git && cd cronJobLambda
Otherwise, create a new project on your IDE and create a new file template.yml in the project root. In the file, we will define the resources needed to implement this solution using AWS SAM.
template.yml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
AWS Lambda function triggered on a schedule using Amazon EventBridge Scheduler
Resources:
ScheduledLambdaFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs20.x
CodeUri: src/
Timeout: 10 # seconds
Policies:
- AWSLambdaBasicExecutionRole
SchedulerRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: scheduler.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: SchedulerInvokeLambda
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "lambda:InvokeFunction"
Resource: !GetAtt ScheduledLambdaFunction.Arn
EventBridgeSchedule:
Type: AWS::Scheduler::Schedule
Properties:
Name: DailyLambdaTrigger
Description: "Trigger Lambda every day at 10:00 AM UTC using a cron expression"
ScheduleExpression: "cron(0 10 * * ? *)" # Change to "rate(...)" or "at(...)" as needed.
FlexibleTimeWindow:
Mode: "OFF"
Target:
Arn: !GetAtt ScheduledLambdaFunction.Arn
RoleArn: !GetAtt SchedulerRole.Arn
State: ENABLED
The SAM template includes three key resources:
ScheduledLambdaFunction: The lambda function that executes the task.
SchdulerRole: An IAM role that grants EventBridge Scheduler permissions to invoke the Lambda function
EventBridgeSchedule: An
AWS::Scheduler::Scheduleresource that triggers the Lambda function.
The AWSLambdaBasicExecutionRole will allow the lambda function to create logs in Cloudwatch.
Lambda Function
Create a folder src/ and create a file named index.js in the folder with the following content
module.exports.handler = async (event) => {
console.log(`Lambda function triggered by EventBridge Scheduler on: ${new Date()}`);
try {
const response = await downstreamProcess();
console.log(`Downstream process completed with response: ${response}`);
} catch (error) {
console.log(`An Error Occurred 😕: ${error}`);
throw new Error("LambdaException");
}
return {
statusCode: 200,
body: JSON.stringify({
message: "Task Completed!🎉",
}),
};
};
const downstreamProcess = async () => {
// Function to simulate a downstream process
return `A New Hope on ${new Date()}!`;
};
Deployment
To deploy the application using AWS SAM:
- Build the application:
sam buld
- Deploy to AWS:
sam deploy --guided --stack-name cron-job-lambda
Follow the on-screen prompts to configure your deployment. Once deployed, EventBridge Scheduler will automatically trigger your Lambda function based on the specified schedule expression (cron, rate, or at).
Navigating to the Amazon EventBridge and Schedules console, you should see the schedule we just deployed.
Best Practices and Considerations
When implementing event-driven scheduled jobs with AWS Lambda and EventBridge Scheduler, consider the following best practices:
Idempotency: Ensure your lambda function is idempotent to avoid duplicate processing in case of retries.
Error Handling and Retries: Leverage Lambda’s built-in error handling and the EventBridge Schedule’s retry policy, and configure Dead Letter Queues (DLQs) to capture failures for troubleshooting.
Cost Management: Monitor Lambda usage and execution times to control costs, especially for high-frequency schedules.
Security: Follow the principle of least privilege when assigning IAM roles. Ensure that roles used by EventBridge Scheduler and Lamdba only have the necessary permissions.
Monitoring and Logging: Consider using CloudWatch to monitor function metrics and logs. Set up alarms to detect failures or performance issues
Conclusion
Integrating scheduled tasks into an event-driven architecture using AWS Lambda and Amazon EventBridge Scheduler offers a modern, scalable, and cost-effective solution for automating recurring jobs. By leveraging EventBridge Scheduler’s flexible scheduling capabilities, whether through cron, rate, or at expressions—and AWS Lambda’s serverless execution model, you can focus on your business logic without worrying about the underlying infrastructure.
This solution provides a robust framework for meeting the demands of modern cloud-native applications, such as automating data processing, streamlining maintenance tasks, or triggering complex workflows.
Follow the best practices discussed to build resilient, secure, and maintainable systems.
Don’t forget to clean up after you’re done with testing. Running the magic command to keep your account nice and rich💰
sam delete --stack-name cron-job-lambda
Happy coding! May the force be with your scheduled tasks!
Sayonara 😊