This blog explains how to build an AI-powered content moderation system for text, images, and video while keeping human review in the loop.
A social product does not fail only when the database is slow. It also fails when the feed becomes a place users no longer trust.
We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.
The design we shipped is deliberately split. Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition. Both pipelines write the same status model, store the AI evidence, and leave gray-area content for the admin panel.
Moderation is not a feature bolted onto the feed. It is a gate the feed has to pass through.
As content volume grows, moderation becomes less about identifying whether a single piece of content is safe and more about deciding where each piece should go.
Sending every piece of content to a human does not scale, while automatically approving or rejecting everything based on a single model score introduces its own risks.
The system therefore treats moderation as a triage pipeline: clear cases are automated, ambiguous cases are escalated, and the feed only exposes content that has reached an approved state.
User-generated content is the product. It is also a risk. Without a consistent review path, unsafe or policy-breaking material can reach other users in seconds.
Manual review of every caption and every frame does not scale.A fully automated approve-or-reject model also creates risk: models are confident on extremes and uncertain in the middle. The system is built around that fact.
Every piece of moderated content starts as pending. Background work then produces a decision: approved, under_review, or rejected.
Approved content can appear in feeds and listings. Rejected content stays out of the public product. Under-review content stays out of the public product until a moderator confirms or overrides the AI decision. That is the contract.
We did not force text and media through the same worker. Text is small, frequent, and already in PostgreSQL at the moment it is written. Media is large, arrives in S3, and is expensive to inspect. Each path uses the infrastructure that fits it.
Feeds that list posts, ads, stories, or media filter on content_moderation_status = approved. The existing status columns (active, trash, uploaded, and so on) still mean what they always meant. Moderation is a separate lifecycle on purpose.
Do not overload "active" to mean "safe." Safety needs its own column.
When a user writes text, the row is saved first. The HTTP request does not wait for OpenAI. Hasura fires an event; NestJS creates the content record and enqueues a job; a worker calls the model.
The content-moderation processor runs with limited concurrency and a rate limit so OpenAI traffic stays controlled. Empty text is approved without a call.
Failed jobs go to a dead-letter queue. Hasura retries the webhook (three attempts, ten-second interval, sixty-second timeout) if Nest is briefly unavailable.
OpenAI returns a flag plus a category_scores map, each value between 0 and 1. We do not treat the boolean flag as the decision. We convert the highest category score into a 0–100 risk score and classify from that number.
OpenAI category_scores
↓
Take the maximum score
↓
riskScore = round(max × 100)
↓
Compare to thresholds
↓
approved / under_review / rejected
Only categories that meet or exceed the approved threshold are stored as labels, as percentages. The admin later sees the violated categories, not a wall of near-zero scores. Thresholds are configurable through MODERATION_APPROVED_THRESHOLD and MODERATION_REJECTED_THRESHOLD; the production defaults are 40 and 75.
Scores of 40, 75, or 90 only become production policy after you measure what those cutoffs do to false positives, false negatives, and human workload. The value of these thresholds comes from the calibration loop that produced and maintains them.
We began with provider semantics and product risk posture, then validated against labelled samples rather than shipping intuition.
Before locking production defaults, we built a small labelled set covering three buckets: clearly safe, borderline / ambiguous, and clearly violating. For each item we stored the model's raw scores, the would-be decision at candidate cutoffs, and a human gold label.
Text settled at 40 / 75 after sweeps showed that scores below ~40 almost never carried human-confirmed violations in our sample, while scores at or above ~75 almost always did. The middle band produced the majority of human overrides, which is exactly where we want people in the loop.
Media settled at 75 / 90 based on the same calibration process but with a different cost curve: Rekognition confidences in the mid-70s were often ambiguous on a single frame, so we refuse to auto-approve there. Sustained or high-confidence hits (including the video frame policy) justify auto-reject without waiting for a moderator.
Thresholds are hypotheses. Production disagreement rates are the experiment.
We revisit cutoffs when operational signals move: rising appeals after auto-reject, reports against content that was auto-approved, or a review queue that ages past SLA. Calibration is continuous, not a one-time spreadsheet.
The media never waits in the Nest process for Rekognition. The user uploads to S3 under users/uploads/. An ObjectCreated notification is published to SNS, and the media path continues asynchronously.
SQS is a competing-consumer queue: each message is delivered to exactly one consumer. If two Lambda event-source mappings share a single queue and each "filters" by message shape, a message can land on the wrong function. That function may skip and still succeed—deleting the message—or fail and retry forever. Either way, delivery semantics are not safe for mixed payload types.
We therefore do not rely on two independently filtering consumers on one queue. The media pipeline is a two-stage design: an ingest queue owned by the scorer Lambda, and a results queue owned by the worker Lambda. SNS fans events into the correct stage; it is not used as a substitute for competing consumers.
SQS is the buffer. Lambda is the computer. Rekognition is the eyes. PostgreSQL is the record.
Rekognition returns a list of moderation labels, each with a confidence from 0 to 100. We keep every label name mapped to a rounded confidence, then take the maximum confidence as the risk score.
The defaults are MODERATION_REVIEW_THRESHOLD = 75 and MODERATION_REJECT_THRESHOLD = 90. They use a stricter review boundary than the text moderation thresholds because a false approval on a photo is more damaging in a feed than an unnecessary review.
The policy also calls out hard categories Rekognition can surface—explicit material, sexual activity, violence, hate symbols, drugs and tobacco, visually disturbing content. High confidence in those labels is what the 90% reject line is designed to catch without a human delay.
A video cannot be scored from a single still. Rekognition runs a content-moderation job across the file, publishes completion to SNS, and that notification arrives on the same SQS queue the image path already uses.
Each label is tied to a timestamp. We still compute riskScore as the maximum confidence, but the decision also depends on how often and how long the violation appears.
One risky frame is a question. Five frames or three seconds is an answer.
A post, ad, or story can have several media items plus text. The parent must not appear "approved" while one of its images is still pending or rejected.
→ Media A approved
→ Media B under_review
→ Caption approved
→ Parent = under_review
Statuses are ordered pending < approved < under_review < rejected. The database function apply_aggregate_moderation_status applies that order to user_post, hey_ads, and users_story. If any child is rejected, the parent is rejected.
List APIs for the product surface follow one rule: show content only when content_moderation_status is approved. Under-review and rejected items are not a client-side filter. They never qualify for the public listing query.
When content_moderation_status becomes rejected, a Hasura event on the content table calls POST /v1/event/content-rejected-notify. That queues an email through SendGrid and an in-app/push notification through Firebase, including the content type and rejection reason. The user is told the content did not pass review. They are not left guessing why it vanished.
Automation is the first reviewer, not the last. Anything in under_review—text, image, or video—is a case a person has to close. That is the admin panel's job.
On approval, both content_moderation_status and final_decision are set to approve. On rejection, the status becomes rejected, the existing rejection notification path fires, and the item still never reaches the feed. The AI decision remains on the row as ai_decision so we can measure how often humans agree with the model.
under_review is not a soft approve. It is a closed door until a person opens it.
Shipping thresholds and queues is not the end of moderation work. The system has to be operated. We treat the following signals as the minimum set for running AI moderation in production—and as the inputs that decide when to retune cutoffs.
Storing ai_decision alongside final_decision and reviewed_by is what makes this loop possible. Without that pair, you can run a model, but you cannot prove whether the model and the product still agree.
A moderation system without operational signals is a deployment, not a practice.
The media Lambdas are not updated by hand in the console as a normal workflow. Each function has a GitHub Actions workflow on the develop branch, scoped to its own directory.
Text moderation ships with the NestJS service: the Hasura trigger metadata, the webhook, the BullMQ processor, and the OpenAI client. Changing a threshold is a configuration change, not a rewrite of the decision engine.
The important change was not simply adding another moderation API. It was moving moderation from an isolated check into a product lifecycle. Content can return from the upload or post request without waiting for model inference, while remaining outside public listings until the moderation lifecycle resolves. Clear cases move automatically, uncertain cases reach the admin queue, and the final decision is recorded alongside the machine evidence.
This also creates a feedback loop: the AI decision remains stored alongside the human final decision, allowing the team to measure where automated decisions and human decisions agree or differ.
A moderation system that only auto-rejects the obvious still leaks. A system that auto-approves anything the model did not scream about still leaks. The useful design is a three-way split.
We used OpenAI for language because it already classifies the categories we care about in text. We used Rekognition for pixels because it already understands moderation labels, video timestamps, and celebrity faces. We used Lambda and SQS because media events are bursty and should not live inside the API process. We used Hasura and BullMQ because text events are already database events. We used an admin queue because a score of 62 is not a moral fact. It is a request for a person.
The goal of AI moderation is not to remove humans from the loop. It is to make sure humans spend their time where human judgment matters most.
So before a caption, a photo, or a video is allowed to represent the product, it has to survive the pipeline: pending, scored, decided, and—if the model is unsure—reviewed. That is how content moderation works here: as a path every piece of user content has to follow before it hits the feed, rather than as a slogan in a policy document.
The bigger lesson goes beyond content moderation. Production AI features need more than an accurate model. They need clear decision boundaries, asynchronous processing, human review paths, auditability, monitoring, and a way to improve as real-world behavior changes.
The same principle applies whether AI is moderating content, evaluating documents, prioritizing cases, or assisting users inside a product. Automate decisions where confidence is high, route ambiguous cases to people, and retain enough evidence to understand why each decision was made.
That is what turns an AI capability into a system a product team can operate at scale.
At GeekyAnts, our AI Engineering teams design and build production-ready AI systems around real product workflows, combining model integration with scalable architecture, observability, deployment, and human-in-the-loop controls.
Building an AI-powered feature that needs to move from model API to production system? Talk to our AI engineering team.