A serverless, data-driven Git metrics dashboard using Amazon Quick Sight AWS published a serverless Git metrics dashboard that automates collection from GitHub and GitLab APIs and visualizes the data in Amazon Quick Sight, using six core AWS services including EventBridge Scheduler, Step Functions Map states, and Amazon S3. The pipeline performs a full load on first execution, incremental loads thereafter, and an automatic full refresh every 24 hours, partitioning workloads into parallel chunks for organizations managing more than 20 repositories. AWS positions the solution against its AI-Driven Development Lifecycle (AI-DLC) framework, which requires teams using AI coding tools to set a baseline and track whether those tools speed up development or merely inflate commit counts. Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ A serverless, data-driven Git metrics dashboard using Amazon Quick Sight Git activity is one of the richest signals engineering teams produce that can provide continuous observability into development analytics. The challenge is extracting these Git metrics at scale, which has traditionally required hand-rolled extract, transform, and load ETL jobs, dedicated infrastructure, and ongoing maintenance. Further, with modern development tools becoming more prevalent, teams need a clear way to measure whether these tools are making developers faster, or if the investment is not paying off. The AWS AI-Driven Development Lifecycle https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-accelerate-software-dev-lifecycle-gen-ai/introduction.html AI-DLC framework makes a simple point: If you’re using AI coding tools, you need numbers to back it up. Set a baseline, track what changes, and keep watching. Otherwise, you’re flying blind. You won’t know whether AI is speeding things up, only padding the commit counts, or quietly introducing quality issues you did not expect. In this post, we introduce a serverless approach that automates Git metrics collection from GitHub https://github.com/ and GitLab https://about.gitlab.com/ to surface live analytics using Amazon Quick Sight https://aws.amazon.com/quick/quicksight/ . Whether you’re tracking sprint velocity, assessing release readiness, or you need visibility into team patterns, this solution provides near-real-time insights into your Git platform activities. The serverless design abstracts infrastructure management while staying low-cost at scale. Solution overview This solution implements an automated, event-driven pipeline that collects repository metrics from GitHub and GitLab APIs automatically on schedule. It then processes them through a serverless orchestration workflow and persists the results in Amazon Simple Storage Service https://aws.amazon.com/pm/serv-s3 Amazon S3 for graphical visualization using interactive Amazon Quick Sight https://aws.amazon.com/quick/quicksight/ dashboards. The key capabilities include: - Intelligent change detection – A dedicated detector function monitors GitHub events API and GitLab activity feeds to determine whether changes like commits, pull requests, issues, repository creation or deletion occurred since the last collection. If no changes are detected, the processing is skipped entirely. - Adaptive chunking – For organizations managing more than 20 repositories, the solution automatically partitions the workload into parallel chunks and processes them concurrently using AWS Step Functions https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html Map states https://docs.aws.amazon.com/step-functions/latest/dg/state-map.html . - Full and incremental loads – On first execution, the system performs a full load of all repository metadata. Subsequent runs use incremental logic, collecting only data that has changed. A full refresh is automatically triggered every 24 hours to maintain data accuracy. - Configurable scheduling – How often metrics get collected is up to you. It’s an AWS CloudFormation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html parameter that accepts both rate and cron expressions, so you can set whatever schedule fits. The solution aligns with the AI-DLC framework’s observability pillar by providing continuous, automated measurement of development activity. You can establish velocity baselines before adopting AI tools, track changes during rollout, and quantify improvements over time. Architecture The architecture uses six core AWS services to build a fully managed, event-driven pipeline: - Amazon EventBridge Scheduler https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html : A scheduled rule starts the workflow at the interval you configured. It runs the collection cycle automatically without any external triggers or manual intervention. - AWS Step Functions https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html : A Step Functions state machine orchestrates the entire collection workflow. It first invokes the change detector to determine whether new activity has occurred. Based on the result, it decides between a full load and an incremental load. A full load collects all repository metadata from scratch on the first run and every 24 hours thereafter. An incremental load collects only data that changed since the last run, which reduces API calls and execution time. The workflow then evaluates the number of active repositories. If the count exceeds the chunking threshold default: 20 , it partitions the repositories into equal-sized chunks and processes them concurrently using a Step Functions Map state. Each chunk is handled by a separate AWS Lambda invocation. If the count is below the threshold, a single direct invocation collects all repositories. The workflow includes comprehensive retry logic with exponential backoff for transient API failures. - AWS Lambda https://aws.amazon.com/lambda/ : 1. Change detector – github-change-detector : This Lambda function queries the activity feeds to determine the load type. It monitors six categories of events: push events, pull requests, issue creation and updates, repository creation, repository deletion, and contributor changes. If no relevant events are found since the last check, the workflow terminates early. 2. Metrics collector – github-metrics-collector : This Lambda function performs the actual data collection. It operates in three modes: collect all for direct processing, collect chunk for processing a subset of repositories during parallel execution, and aggregate for combining the chunk results. - Change detector – - AWS Secrets Manager https://aws.amazon.com/secrets-manager/ : Git tokens are stored securely in Secrets Manager. Both Lambda functions retrieve tokens at runtime, so that credentials are never hard coded or passed as environment variables. - Amazon S3 https://aws.amazon.com/pm/serv-s3 : Collected metrics are stored on Amazon S3 with versioning and server-side encryption enabled. The output includes a structured JSON file containing the full API response with nested repository details, and a flattened CSV file optimized for analytics. Both files include fields such as repository name, total commits, open and closed pull requests, open and closed issues, number of contributors, primary language, last activity timestamp, and repository creation date. S3 serves as the durable, cost-effective data store for all historical metrics. - Amazon Quick Sight https://aws.amazon.com/quick/quicksight/ : Reads the collected CSV data from the S3 data store or through Amazon Athena https://aws.amazon.com/athena/ for SQL-based queries. It loads the data into SPICE Super-fast, Parallel, In-memory Calculation Engine , a fast in-memory layer, to build interactive dashboards. You can build visualizations for summary metrics, pull request PR trends, development activity over time, per-repository drilldowns, and contributor analysis. The following figure provides an overview of the solution architecture using the listed services. Before moving on to the step-by-step walkthrough, let us look at the prerequisites required for you to follow along. Prerequisites - An active AWS account with permissions to create CloudFormation stacks, Lambda functions, S3 buckets, AWS Identity and Access Management IAM roles, Step Functions state machines, Amazon EventBridge rules, and Secrets Manager secrets. - AWS Command Line Interface https://aws.amazon.com/cli/ AWS CLI v2. - A GitHub or GitLab account with repositories to monitor, and the ability to generate Personal Access Tokens https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens . - An active Amazon Quick Sight subscription https://docs.aws.amazon.com/quick/latest/userguide/managing-qbs-subscriptions.html on either Standard or Enterprise edition for dashboard creation. - Least-privilege permissions for deploying the required resources: cloudformation: , s3: , lambda {CreateFunction,UpdateFunctionCode, UpdateFunctionConfiguration,GetFunction, PutFunctionConcurrency}, states:{CreateStateMachine,UpdateStateMachine,DescribeStateMachine}, events:{PutRule,PutTargets,DescribeRule}, sqs:{CreateQueue,GetQueueAttributes, SetQueueAttributes}, sns:{CreateTopic,GetTopicAttributes,SetTopicAttributes}, kms:{CreateKey, CreateAlias, PutKeyPolicy, EnableKeyRotation}, secretsmanager:{CreateSecret,GetSecretValue}, and iam:{CreateRole, PutRolePolicy, AttachRolePolicy, PassRole}. - Clone the solution from this GitHub repository https://github.com/aws-samples/aws-aj-BlogPosts/tree/main/Git-QS-Dashboard . Navigate to the cloned directory, subsequent steps reference file paths relative to this root. Walkthrough In this section, we elaborate the steps that you can follow to deploy a sample dashboard in your test environment. Step 1: Create personal access tokens Before deploying the infrastructure, you need API tokens for each Git platform you plan to monitor. For GitHub: - Navigate to GitHub, Settings, Developer settings, Personal access tokens and choose Tokens classic . - Choose Generate new token classic . Under scopes, select only the minimum required permissions: repo read-only access to repository metadata, commits, and pull requests and read:org read-only access to organization membership . Copy the generated token and store it securely. For GitLab: - Navigate to GitLab, Settings and Access Tokens. - Create a new token with scopes and copy the generated token. The following image shows the GitHub developer settings page where you can generate the new tokens: Step 2: Store tokens in AWS Secrets Manager Store each token as a secret in AWS Secrets Manager using the AWS CLI. See the following sample commands: Step 3: Deploy the CloudFormation stack Download this sample CloudFormation template https://github.com/aws-samples/aws-aj-BlogPosts/blob/main/Git-QS-Dashboard/starter-kit-gitdashboard/infrastructure/template.yaml to follow along and deploy the solution in your non-production environment. Option 1: Deploy using the AWS CLI. Replace placeholder values enclosed in angle brackets <… with your actual AWS account ID, Region, and the secret ARNs from Step 2: Option 2: Deploy using the AWS Management Console https://aws.amazon.com/console/ . 1. Navigate to the AWS CloudFormation console https://console.aws.amazon.com/cloudformation/home . 2. Choose Create stack , With new resources standard . 3. Upload the template, fill in parameters, and choose Submit . Stack reaches CREATE COMPLETE status in 3-5 minutes. It creates the following resources: an S3 bucket with versioning and encryption enabled, two Lambda functions Python 3.13 runtime , an IAM execution role with least-privilege permissions, a Step Functions state machine, an Amazon EventBridge schedule rule, and associated IAM roles. The following image shows the sample Create stack parameters and configuration: Step 4: Upload the Lambda deployment package The CloudFormation template deploys placeholder code for the Lambda functions. Update them with the actual logic using the pre-built deployment package from the cloned repository. The file lambda-package.zip is located in the deployment/ directory of the repository you cloned in the prerequisites. Replace