{"slug": "building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and", "title": "Building a Production RAG Pipeline on AWS: Bedrock Knowledge Bases, OpenSearch Serverless, and Terraform", "summary": "A developer built a production RAG pipeline on AWS that indexes 2,400+ internal documents (Confluence pages, runbooks, post-mortems, ADRs) to answer engineering questions with source citations. The stack uses Amazon Titan Text Embed v2 for embeddings, OpenSearch Serverless for retrieval, Claude 3.5 Sonnet via Bedrock for generation, and a Lambda query API behind API Gateway, all provisioned entirely through Terraform. The pipeline reports p50 latency of 1.8 seconds, p99 of 4.2 seconds, and roughly $380 monthly AWS cost at about 500 queries per day.", "body_md": "*November 2026 | ~18 min read*\n\nEvery engineering org has the same problem. New engineer joins, asks \"how does the deployment pipeline handle database migrations?\" The answer exists — buried in a Confluence page from 2023, a Slack thread from last April, and a post-mortem that three people remember reading. Nobody can find it. So the question gets asked again. And again.\n\nI built a production-ready RAG pipeline on AWS to solve this. Not a notebook demo. Not a \"just call the API\" tutorial. A real pipeline — document ingestion from S3, vector embeddings via Amazon Titan, retrieval from OpenSearch Serverless, generation via Claude 3.5 Sonnet through Bedrock, a Lambda query API behind API Gateway, and CloudWatch observability across the whole stack. All deployed with Terraform. Zero console clicks.\n\nThe goal: any engineer can hit an API endpoint (or a Slack bot wired to it) with a natural language question and get an accurate answer with citations back to the source documents.\n\n```\n┌──────────────────────────────────────────────────────────────────┐\n│                    PRODUCTION RAG PIPELINE                        │\n├──────────────────────────────────┬───────────────────────────────┤\n│ Metric                           │ Value                         │\n├──────────────────────────────────┼───────────────────────────────┤\n│ Documents indexed                │ 2,400+ (Confluence, runbooks, │\n│                                  │ post-mortems, ADRs)           │\n│ Avg query latency (p50)         │ 1.8 seconds                   │\n│ Avg query latency (p99)         │ 4.2 seconds                   │\n│ Monthly AWS cost                 │ ~$380 (at ~500 queries/day)   │\n│ Embedding model                  │ Amazon Titan Text Embed v2    │\n│ Generation model                 │ Claude 3.5 Sonnet via Bedrock │\n│ Infrastructure                   │ 100% Terraform, zero console  │\n└──────────────────────────────────┴───────────────────────────────┘\n```\n\nBefore writing any Terraform, I had to answer the first question: why RAG instead of fine-tuning a model on our internal docs?\n\n```\n┌──────────────────────────┬───────────────────────┬───────────────────────┐\n│ Factor                   │ RAG                   │ Fine-Tuning           │\n├──────────────────────────┼───────────────────────┼───────────────────────┤\n│ Data freshness           │ Real-time (re-ingest) │ Stale (retrain cycle) │\n│ Citation support         │ Native (source docs)  │ None                  │\n│ Cost to update           │ ~$0 (S3 sync)         │ $100s–$1000s/retrain  │\n│ Hallucination control    │ Grounded in sources   │ No guarantees         │\n│ Setup complexity         │ Medium                │ High                  │\n│ Latency                  │ Higher (~2s)          │ Lower (~500ms)        │\n│ Domain-specific language │ Depends on retrieval  │ Strong                │\n│ Data privacy             │ Data stays in your VPC│ Sent to training API  │\n└──────────────────────────┴───────────────────────┴───────────────────────┘\n```\n\nRAG wins when:\n\nFine-tuning wins when you need specialized domain language (medical terminology, legal jargon) or sub-second latency. For an internal knowledge base where accuracy and freshness matter more than speed, RAG is the clear choice.\n\n```\n┌─────────────────────────────────────────────────────────────────────────────┐\n│                         PRODUCTION RAG PIPELINE                             │\n│                                                                             │\n│  ┌──────────┐    ┌─────────────────┐    ┌──────────────┐                   │\n│  │          │    │  Bedrock Data    │    │   Amazon     │                   │\n│  │  S3      │───▶│  Source          │───▶│   Titan      │                   │\n│  │ (docs)   │    │  (chunking)     │    │   Embed v2   │                   │\n│  └──────────┘    └─────────────────┘    └──────┬───────┘                   │\n│                                                 │                           │\n│                                                 ▼                           │\n│                                    ┌────────────────────┐                  │\n│                                    │   OpenSearch        │                  │\n│                                    │   Serverless        │                  │\n│                                    │   (vector store)    │                  │\n│                                    └────────┬───────────┘                  │\n│                                             │                               │\n│  ┌──────────┐    ┌─────────────────┐        │                              │\n│  │  API     │    │   Lambda        │        │                              │\n│  │ Gateway  │───▶│   (query API)   │───▶ Bedrock KB ───▶ Claude 3.5       │\n│  └──────────┘    └─────────────────┘    (retrieve +      Sonnet            │\n│                                          generate)       (answer)          │\n│                                                                             │\n│  ┌─────────────────────────────────────────────────────────────┐           │\n│  │  CloudWatch: latency, tokens, errors, cost anomaly alarms  │           │\n│  └─────────────────────────────────────────────────────────────┘           │\n└─────────────────────────────────────────────────────────────────────────────┘\n```\n\n**Why OpenSearch Serverless over the alternatives?**\n\n**Component cost breakdown:**\n\n```\n┌──────────────────────────────────┬───────────────────────────────┐\n│ Component                        │ Monthly Cost                  │\n├──────────────────────────────────┼───────────────────────────────┤\n│ OpenSearch Serverless (2 OCUs)   │ ~$350                         │\n│ Bedrock - Titan Embed v2         │ ~$4 (ingestion)               │\n│ Bedrock - Claude 3.5 Sonnet      │ ~$18 (500 queries/day)        │\n│ Lambda                           │ ~$2                           │\n│ API Gateway                      │ ~$2                           │\n│ S3 (document storage)            │ <$1                           │\n│ CloudWatch                       │ ~$3                           │\n├──────────────────────────────────┼───────────────────────────────┤\n│ Total                            │ ~$380/month                   │\n└──────────────────────────────────┴───────────────────────────────┘\n```\n\nYes, OpenSearch Serverless is 92% of the bill. We'll talk about that.\n\nOpenSearch Serverless for vector search requires three things: an encryption policy, a network policy, and a data access policy. Without all three, the collection won't create. This trips up everyone the first time.\n\n```\n# opensearch-serverless.tf\n\nresource \"aws_opensearchserverless_security_policy\" \"encryption\" {\n  name = \"rag-kb-encryption\"\n  type = \"encryption\"\n  policy = jsonencode({\n    Rules = [\n      {\n        Resource     = [\"collection/rag-knowledge-base\"]\n        ResourceType = \"collection\"\n      }\n    ]\n    AWSOwnedKey = true\n  })\n}\n\nresource \"aws_opensearchserverless_security_policy\" \"network\" {\n  name = \"rag-kb-network\"\n  type = \"network\"\n  policy = jsonencode([\n    {\n      Rules = [\n        {\n          Resource     = [\"collection/rag-knowledge-base\"]\n          ResourceType = \"collection\"\n        },\n        {\n          Resource     = [\"collection/rag-knowledge-base\"]\n          ResourceType = \"dashboard\"\n        }\n      ]\n      AllowFromPublic = true\n    }\n  ])\n}\n\nresource \"aws_opensearchserverless_collection\" \"kb\" {\n  name        = \"rag-knowledge-base\"\n  description = \"Vector store for RAG knowledge base\"\n  type        = \"VECTORSEARCH\"\n\n  depends_on = [\n    aws_opensearchserverless_security_policy.encryption,\n    aws_opensearchserverless_security_policy.network,\n  ]\n}\n\nresource \"aws_opensearchserverless_access_policy\" \"kb\" {\n  name = \"rag-kb-access\"\n  type = \"data\"\n  policy = jsonencode([\n    {\n      Rules = [\n        {\n          Resource     = [\"collection/rag-knowledge-base\"]\n          ResourceType = \"collection\"\n          Permission = [\n            \"aoss:CreateCollectionItems\",\n            \"aoss:DeleteCollectionItems\",\n            \"aoss:UpdateCollectionItems\",\n            \"aoss:DescribeCollectionItems\"\n          ]\n        },\n        {\n          Resource     = [\"index/rag-knowledge-base/*\"]\n          ResourceType = \"index\"\n          Permission = [\n            \"aoss:CreateIndex\",\n            \"aoss:DeleteIndex\",\n            \"aoss:UpdateIndex\",\n            \"aoss:DescribeIndex\",\n            \"aoss:ReadDocument\",\n            \"aoss:WriteDocument\"\n          ]\n        }\n      ]\n      Principal = [\n        aws_iam_role.bedrock_kb.arn,\n        data.aws_caller_identity.current.arn\n      ]\n    }\n  ])\n}\n```\n\nA few things to note:\n\n`AllowFromPublic = true`` depends_on` block is required.\nBedrock Knowledge Bases require a specific index schema. You can't create this index through Terraform — the OpenSearch Serverless API doesn't expose index management. Use a Python script:\n\n``` python\n# scripts/create_vector_index.py\n\nimport boto3\nimport json\nfrom opensearchpy import OpenSearch, RequestsAWSV4SignerAuth, AWSV4SignerAuth\nfrom requests_aws4auth import AWS4Auth\n\nCOLLECTION_ENDPOINT = \"https://your-collection-id.us-east-1.aoss.amazonaws.com\"\nINDEX_NAME = \"bedrock-knowledge-base-default-index\"\nREGION = \"us-east-1\"\n\ncredentials = boto3.Session().get_credentials()\nauth = AWS4Auth(\n    credentials.access_key,\n    credentials.secret_key,\n    REGION,\n    \"aoss\",\n    session_token=credentials.token,\n)\n\nclient = OpenSearch(\n    hosts=[{\"host\": COLLECTION_ENDPOINT.replace(\"https://\", \"\"), \"port\": 443}],\n    http_auth=auth,\n    use_ssl=True,\n    verify_certs=True,\n    connection_class=\"RequestsHttpConnection\",\n)\n\nindex_body = {\n    \"settings\": {\n        \"index\": {\n            \"knn\": True,\n            \"knn.algo_param.ef_search\": 512,\n        }\n    },\n    \"mappings\": {\n        \"properties\": {\n            \"bedrock-knowledge-base-default-vector\": {\n                \"type\": \"knn_vector\",\n                \"dimension\": 1024,\n                \"method\": {\n                    \"engine\": \"faiss\",\n                    \"space_type\": \"l2\",\n                    \"name\": \"hnsw\",\n                    \"parameters\": {\n                        \"ef_construction\": 512,\n                        \"m\": 16,\n                    },\n                },\n            },\n            \"AMAZON_BEDROCK_METADATA\": {\n                \"type\": \"text\",\n                \"index\": False,\n            },\n            \"AMAZON_BEDROCK_TEXT_CHUNK\": {\n                \"type\": \"text\",\n            },\n        }\n    },\n}\n\nresponse = client.indices.create(index=INDEX_NAME, body=index_body)\nprint(json.dumps(response, indent=2))\n```\n\n**Why the field names matter:** Bedrock Knowledge Bases expect specific field names — `bedrock-knowledge-base-default-vector`, `AMAZON_BEDROCK_TEXT_CHUNK`, and `AMAZON_BEDROCK_METADATA`. If you use custom field names, you'll need to specify the mapping in the knowledge base configuration. Using the defaults saves a layer of configuration.\n\n**Dimension 1024** matches Amazon Titan Text Embeddings v2. If you switch to a different embedding model, this number changes (Cohere Embed = 1024, Titan v1 = 1536).\n\nThe knowledge base ties everything together: it connects S3 (documents) to the embedding model (Titan) to the vector store (OpenSearch Serverless).\n\nThe Bedrock service needs permissions to read from S3, generate embeddings, and write to OpenSearch Serverless:\n\n```\n# iam.tf\n\ndata \"aws_caller_identity\" \"current\" {}\n\nresource \"aws_iam_role\" \"bedrock_kb\" {\n  name = \"bedrock-knowledge-base-role\"\n\n  assume_role_policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Action = \"sts:AssumeRole\"\n        Effect = \"Allow\"\n        Principal = {\n          Service = \"bedrock.amazonaws.com\"\n        }\n        Condition = {\n          StringEquals = {\n            \"aws:SourceAccount\" = data.aws_caller_identity.current.account_id\n          }\n        }\n      }\n    ]\n  })\n}\n\nresource \"aws_iam_role_policy\" \"bedrock_kb\" {\n  name = \"bedrock-knowledge-base-policy\"\n  role = aws_iam_role.bedrock_kb.id\n\n  policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Sid    = \"BedrockInvokeModel\"\n        Effect = \"Allow\"\n        Action = [\"bedrock:InvokeModel\"]\n        Resource = [\n          \"arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0\"\n        ]\n      },\n      {\n        Sid    = \"S3ReadDocuments\"\n        Effect = \"Allow\"\n        Action = [\n          \"s3:GetObject\",\n          \"s3:ListBucket\"\n        ]\n        Resource = [\n          aws_s3_bucket.documents.arn,\n          \"${aws_s3_bucket.documents.arn}/*\"\n        ]\n      },\n      {\n        Sid    = \"OpenSearchServerlessAccess\"\n        Effect = \"Allow\"\n        Action = [\"aoss:APIAccessAll\"]\n        Resource = [\n          aws_opensearchserverless_collection.kb.arn\n        ]\n      }\n    ]\n  })\n}\n```\n\nThe `aws:SourceAccount` condition in the trust policy prevents confused deputy attacks — without it, any AWS account could potentially assume your Bedrock role.\n\n```\n# knowledge-base.tf\n\nresource \"aws_s3_bucket\" \"documents\" {\n  bucket = \"rag-knowledge-base-documents-${data.aws_caller_identity.current.account_id}\"\n}\n\nresource \"aws_s3_bucket_versioning\" \"documents\" {\n  bucket = aws_s3_bucket.documents.id\n  versioning_configuration {\n    status = \"Enabled\"\n  }\n}\n\nresource \"aws_bedrockagent_knowledge_base\" \"main\" {\n  name     = \"internal-knowledge-base\"\n  role_arn = aws_iam_role.bedrock_kb.arn\n\n  knowledge_base_configuration {\n    type = \"VECTOR\"\n    vector_knowledge_base_configuration {\n      embedding_model_arn = \"arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0\"\n    }\n  }\n\n  storage_configuration {\n    type = \"OPENSEARCH_SERVERLESS\"\n    opensearch_serverless_configuration {\n      collection_arn    = aws_opensearchserverless_collection.kb.arn\n      vector_index_name = \"bedrock-knowledge-base-default-index\"\n      field_mapping {\n        vector_field   = \"bedrock-knowledge-base-default-vector\"\n        text_field     = \"AMAZON_BEDROCK_TEXT_CHUNK\"\n        metadata_field = \"AMAZON_BEDROCK_METADATA\"\n      }\n    }\n  }\n}\n\nresource \"aws_bedrockagent_data_source\" \"s3\" {\n  name                 = \"s3-documents\"\n  knowledge_base_id    = aws_bedrockagent_knowledge_base.main.id\n  data_deletion_policy = \"RETAIN\"\n\n  data_source_configuration {\n    type = \"S3\"\n    s3_configuration {\n      bucket_arn = aws_s3_bucket.documents.arn\n    }\n  }\n\n  vector_ingestion_configuration {\n    chunking_configuration {\n      chunking_strategy = \"FIXED_SIZE\"\n      fixed_size_chunking_configuration {\n        max_tokens         = 512\n        overlap_percentage = 20\n      }\n    }\n  }\n}\n```\n\nBedrock Knowledge Bases support three chunking strategies:\n\nI chose **fixed-size at 512 tokens with 20% overlap** for three reasons:\n\nThe 20% overlap (102 tokens) ensures that if a concept spans a chunk boundary, at least part of it appears in both chunks. Without overlap, you get answers that start mid-sentence.\n\nAfter deploying the infrastructure and uploading documents to S3, trigger the initial sync:\n\n``` bash\n#!/bin/bash\n# scripts/trigger_ingestion.sh\n\nKNOWLEDGE_BASE_ID=$(terraform output -raw knowledge_base_id)\nDATA_SOURCE_ID=$(terraform output -raw data_source_id)\n\necho \"Starting ingestion job...\"\nINGESTION_JOB=$(aws bedrock-agent start-ingestion-job \\\n  --knowledge-base-id \"$KNOWLEDGE_BASE_ID\" \\\n  --data-source-id \"$DATA_SOURCE_ID\" \\\n  --output json)\n\nJOB_ID=$(echo \"$INGESTION_JOB\" | jq -r '.ingestionJob.ingestionJobId')\necho \"Job ID: $JOB_ID\"\n\n# Poll until complete\nwhile true; do\n  STATUS=$(aws bedrock-agent get-ingestion-job \\\n    --knowledge-base-id \"$KNOWLEDGE_BASE_ID\" \\\n    --data-source-id \"$DATA_SOURCE_ID\" \\\n    --ingestion-job-id \"$JOB_ID\" \\\n    --query 'ingestionJob.status' \\\n    --output text)\n\n  echo \"Status: $STATUS\"\n\n  if [ \"$STATUS\" = \"COMPLETE\" ] || [ \"$STATUS\" = \"FAILED\" ]; then\n    break\n  fi\n\n  sleep 10\ndone\n\n# Show statistics\naws bedrock-agent get-ingestion-job \\\n  --knowledge-base-id \"$KNOWLEDGE_BASE_ID\" \\\n  --data-source-id \"$DATA_SOURCE_ID\" \\\n  --ingestion-job-id \"$JOB_ID\" \\\n  --query 'ingestionJob.statistics'\n```\n\nFor 2,400 documents, initial ingestion takes about 15 minutes. Subsequent incremental syncs process only changed/new files and finish in under a minute.\n\nThe query layer is a Lambda function behind API Gateway. It takes a natural language question, calls Bedrock's `RetrieveAndGenerate` API, and returns a structured response with citations.\n\n``` python\n# lambda/query_handler.py\n\nimport json\nimport os\nimport logging\nimport boto3\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\nbedrock_agent = boto3.client(\"bedrock-agent-runtime\")\n\nKNOWLEDGE_BASE_ID = os.environ[\"KNOWLEDGE_BASE_ID\"]\nMODEL_ARN = os.environ.get(\n    \"MODEL_ARN\",\n    \"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0\",\n)\n\nSYSTEM_PROMPT = \"\"\"You are an internal knowledge assistant for an engineering team.\nAnswer questions using ONLY the provided search results. If the search results\ndon't contain enough information to answer the question, say so clearly —\ndo not make up information.\n\nRules:\n- Cite your sources by referencing the document title and section\n- If multiple documents provide relevant information, synthesize them\n- For code examples, include the full snippet from the source document\n- If the question is ambiguous, ask for clarification\n- Keep answers concise but complete\"\"\"\n\ndef handler(event, context):\n    try:\n        body = json.loads(event.get(\"body\", \"{}\"))\n        question = body.get(\"question\", \"\").strip()\n\n        if not question:\n            return response(400, {\"error\": \"Missing 'question' in request body\"})\n\n        if len(question) > 1000:\n            return response(400, {\"error\": \"Question exceeds 1000 character limit\"})\n\n        logger.info(f\"Query: {question[:100]}...\")\n\n        result = bedrock_agent.retrieve_and_generate(\n            input={\"text\": question},\n            retrieveAndGenerateConfiguration={\n                \"type\": \"KNOWLEDGE_BASE\",\n                \"knowledgeBaseConfiguration\": {\n                    \"knowledgeBaseId\": KNOWLEDGE_BASE_ID,\n                    \"modelArn\": MODEL_ARN,\n                    \"generationConfiguration\": {\n                        \"promptTemplate\": {\n                            \"textPromptTemplate\": (\n                                f\"{SYSTEM_PROMPT}\\n\\n\"\n                                \"Search results:\\n$search_results$\\n\\n\"\n                                \"Question: $query$\\n\\n\"\n                                \"Answer:\"\n                            )\n                        }\n                    },\n                    \"retrievalConfiguration\": {\n                        \"vectorSearchConfiguration\": {\n                            \"numberOfResults\": 5,\n                            \"overrideSearchType\": \"HYBRID\",\n                        }\n                    },\n                },\n            },\n        )\n\n        answer = result[\"output\"][\"text\"]\n        citations = []\n\n        for citation in result.get(\"citations\", []):\n            for ref in citation.get(\"retrievedReferences\", []):\n                location = ref.get(\"location\", {})\n                s3_uri = location.get(\"s3Location\", {}).get(\"uri\", \"\")\n                content_text = ref.get(\"content\", {}).get(\"text\", \"\")[:200]\n                citations.append(\n                    {\n                        \"source\": s3_uri.split(\"/\")[-1] if s3_uri else \"unknown\",\n                        \"excerpt\": content_text,\n                    }\n                )\n\n        return response(\n            200,\n            {\n                \"answer\": answer,\n                \"citations\": citations,\n                \"query\": question,\n            },\n        )\n\n    except bedrock_agent.exceptions.ThrottlingException:\n        logger.warning(\"Bedrock throttled\")\n        return response(429, {\"error\": \"Rate limited. Try again in a few seconds.\"})\n    except Exception as e:\n        logger.error(f\"Query failed: {str(e)}\", exc_info=True)\n        return response(500, {\"error\": \"Internal server error\"})\n\ndef response(status_code, body):\n    return {\n        \"statusCode\": status_code,\n        \"headers\": {\n            \"Content-Type\": \"application/json\",\n            \"Access-Control-Allow-Origin\": \"*\",\n        },\n        \"body\": json.dumps(body),\n    }\n```\n\nKey design decisions:\n\n`overrideSearchType: \"HYBRID\"`` numberOfResults: 5`\n\n```\n# lambda.tf\n\ndata \"archive_file\" \"query_handler\" {\n  type        = \"zip\"\n  source_file = \"${path.module}/lambda/query_handler.py\"\n  output_path = \"${path.module}/build/query_handler.zip\"\n}\n\nresource \"aws_lambda_function\" \"query\" {\n  function_name    = \"rag-query-handler\"\n  filename         = data.archive_file.query_handler.output_path\n  source_code_hash = data.archive_file.query_handler.output_base64sha256\n  handler          = \"query_handler.handler\"\n  runtime          = \"python3.12\"\n  timeout          = 30\n  memory_size      = 256\n\n  role = aws_iam_role.lambda_query.arn\n\n  environment {\n    variables = {\n      KNOWLEDGE_BASE_ID = aws_bedrockagent_knowledge_base.main.id\n      MODEL_ARN         = \"arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0\"\n    }\n  }\n}\n\nresource \"aws_iam_role\" \"lambda_query\" {\n  name = \"rag-query-lambda-role\"\n\n  assume_role_policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Action = \"sts:AssumeRole\"\n        Effect = \"Allow\"\n        Principal = {\n          Service = \"lambda.amazonaws.com\"\n        }\n      }\n    ]\n  })\n}\n\nresource \"aws_iam_role_policy\" \"lambda_query\" {\n  name = \"rag-query-lambda-policy\"\n  role = aws_iam_role.lambda_query.id\n\n  policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Sid    = \"BedrockRetrieveAndGenerate\"\n        Effect = \"Allow\"\n        Action = [\n          \"bedrock:RetrieveAndGenerate\",\n          \"bedrock:Retrieve\",\n          \"bedrock:InvokeModel\"\n        ]\n        Resource = \"*\"\n      },\n      {\n        Sid    = \"CloudWatchLogs\"\n        Effect = \"Allow\"\n        Action = [\n          \"logs:CreateLogGroup\",\n          \"logs:CreateLogStream\",\n          \"logs:PutLogEvents\"\n        ]\n        Resource = \"arn:aws:logs:*:*:*\"\n      }\n    ]\n  })\n}\n\n# API Gateway\n\nresource \"aws_api_gateway_rest_api\" \"rag\" {\n  name        = \"rag-query-api\"\n  description = \"RAG Knowledge Base Query API\"\n\n  endpoint_configuration {\n    types = [\"REGIONAL\"]\n  }\n}\n\nresource \"aws_api_gateway_resource\" \"query\" {\n  rest_api_id = aws_api_gateway_rest_api.rag.id\n  parent_id   = aws_api_gateway_rest_api.rag.root_resource_id\n  path_part   = \"query\"\n}\n\nresource \"aws_api_gateway_method\" \"query_post\" {\n  rest_api_id   = aws_api_gateway_rest_api.rag.id\n  resource_id   = aws_api_gateway_resource.query.id\n  http_method   = \"POST\"\n  authorization = \"NONE\"\n}\n\nresource \"aws_api_gateway_integration\" \"query_lambda\" {\n  rest_api_id             = aws_api_gateway_rest_api.rag.id\n  resource_id             = aws_api_gateway_resource.query.id\n  http_method             = aws_api_gateway_method.query_post.http_method\n  integration_http_method = \"POST\"\n  type                    = \"AWS_PROXY\"\n  uri                     = aws_lambda_function.query.invoke_arn\n}\n\nresource \"aws_lambda_permission\" \"api_gateway\" {\n  statement_id  = \"AllowAPIGatewayInvoke\"\n  action        = \"lambda:InvokeFunction\"\n  function_name = aws_lambda_function.query.function_name\n  principal     = \"apigateway.amazonaws.com\"\n  source_arn    = \"${aws_api_gateway_rest_api.rag.execution_arn}/*/*\"\n}\n\nresource \"aws_api_gateway_deployment\" \"rag\" {\n  rest_api_id = aws_api_gateway_rest_api.rag.id\n\n  depends_on = [\n    aws_api_gateway_integration.query_lambda,\n  ]\n\n  lifecycle {\n    create_before_destroy = true\n  }\n}\n\nresource \"aws_api_gateway_stage\" \"prod\" {\n  rest_api_id   = aws_api_gateway_rest_api.rag.id\n  deployment_id = aws_api_gateway_deployment.rag.id\n  stage_name    = \"prod\"\n}\ncurl -X POST https://your-api-id.execute-api.us-east-1.amazonaws.com/prod/query \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"question\": \"How does our deployment pipeline handle database migrations?\"}'\n```\n\nResponse:\n\n```\n{\n  \"answer\": \"The deployment pipeline handles database migrations through a two-phase process. First, the CI/CD pipeline runs `alembic upgrade head` in a pre-deploy Lambda that executes against the staging database. If migrations succeed, the pipeline proceeds to the blue-green deployment swap. Rollback migrations are required for every forward migration — the pipeline validates that `alembic downgrade` works before approving the PR.\\n\\nFor breaking schema changes, the team uses the expand-contract pattern: deploy the new schema alongside the old one, migrate data, then remove the old schema in a subsequent release.\",\n  \"citations\": [\n    {\n      \"source\": \"deployment-pipeline-runbook.md\",\n      \"excerpt\": \"Database migrations run as a pre-deploy step using Alembic. The migration Lambda has a 5-minute timeout and connects via RDS Proxy...\"\n    },\n    {\n      \"source\": \"adr-024-schema-migration-strategy.md\",\n      \"excerpt\": \"We adopt the expand-contract pattern for all breaking schema changes to enable zero-downtime deployments...\"\n    }\n  ],\n  \"query\": \"How does our deployment pipeline handle database migrations?\"\n}\n```\n\nThat's the power of RAG with citations. The answer synthesizes two source documents and tells you exactly where to go for the full details.\n\nA RAG pipeline without observability is a black box. You don't know if answers are degrading, costs are spiking, or queries are failing silently.\n\n```\n# observability.tf\n\nresource \"aws_cloudwatch_dashboard\" \"rag\" {\n  dashboard_name = \"rag-pipeline\"\n\n  dashboard_body = jsonencode({\n    widgets = [\n      {\n        type   = \"metric\"\n        x      = 0\n        y      = 0\n        width  = 12\n        height = 6\n        properties = {\n          title   = \"Query Latency (ms)\"\n          metrics = [\n            [\"AWS/Lambda\", \"Duration\", \"FunctionName\", \"rag-query-handler\", { stat = \"p50\" }],\n            [\"AWS/Lambda\", \"Duration\", \"FunctionName\", \"rag-query-handler\", { stat = \"p99\" }],\n          ]\n          period = 300\n          view   = \"timeSeries\"\n        }\n      },\n      {\n        type   = \"metric\"\n        x      = 12\n        y      = 0\n        width  = 12\n        height = 6\n        properties = {\n          title   = \"Query Volume & Errors\"\n          metrics = [\n            [\"AWS/Lambda\", \"Invocations\", \"FunctionName\", \"rag-query-handler\", { stat = \"Sum\" }],\n            [\"AWS/Lambda\", \"Errors\", \"FunctionName\", \"rag-query-handler\", { stat = \"Sum\" }],\n            [\"AWS/Lambda\", \"Throttles\", \"FunctionName\", \"rag-query-handler\", { stat = \"Sum\" }],\n          ]\n          period = 300\n          view   = \"timeSeries\"\n        }\n      },\n      {\n        type   = \"metric\"\n        x      = 0\n        y      = 6\n        width  = 12\n        height = 6\n        properties = {\n          title   = \"Bedrock Token Usage\"\n          metrics = [\n            [\"AWS/Bedrock\", \"InputTokenCount\", \"ModelId\", \"anthropic.claude-3-5-sonnet-20241022-v2:0\", { stat = \"Sum\" }],\n            [\"AWS/Bedrock\", \"OutputTokenCount\", \"ModelId\", \"anthropic.claude-3-5-sonnet-20241022-v2:0\", { stat = \"Sum\" }],\n          ]\n          period = 3600\n          view   = \"timeSeries\"\n        }\n      },\n      {\n        type   = \"metric\"\n        x      = 12\n        y      = 6\n        width  = 12\n        height = 6\n        properties = {\n          title   = \"OpenSearch Serverless\"\n          metrics = [\n            [\"AWS/AOSS\", \"SearchOCU\", \"CollectionId\", aws_opensearchserverless_collection.kb.id],\n            [\"AWS/AOSS\", \"IndexingOCU\", \"CollectionId\", aws_opensearchserverless_collection.kb.id],\n          ]\n          period = 3600\n          view   = \"timeSeries\"\n        }\n      }\n    ]\n  })\n}\nresource \"aws_cloudwatch_metric_alarm\" \"high_latency\" {\n  alarm_name          = \"rag-query-high-latency\"\n  comparison_operator = \"GreaterThanThreshold\"\n  evaluation_periods  = 3\n  metric_name         = \"Duration\"\n  namespace           = \"AWS/Lambda\"\n  period              = 300\n  statistic           = \"p99\"\n  threshold           = 10000\n  alarm_description   = \"RAG query p99 latency exceeded 10 seconds for 15 minutes\"\n  alarm_actions       = [aws_sns_topic.alerts.arn]\n\n  dimensions = {\n    FunctionName = aws_lambda_function.query.function_name\n  }\n}\n\nresource \"aws_cloudwatch_metric_alarm\" \"high_error_rate\" {\n  alarm_name          = \"rag-query-high-error-rate\"\n  comparison_operator = \"GreaterThanThreshold\"\n  evaluation_periods  = 2\n  threshold           = 5\n\n  metric_query {\n    id          = \"error_rate\"\n    expression  = \"(errors / invocations) * 100\"\n    label       = \"Error Rate %\"\n    return_data = true\n  }\n\n  metric_query {\n    id = \"errors\"\n    metric {\n      metric_name = \"Errors\"\n      namespace   = \"AWS/Lambda\"\n      period      = 300\n      stat        = \"Sum\"\n      dimensions = {\n        FunctionName = aws_lambda_function.query.function_name\n      }\n    }\n  }\n\n  metric_query {\n    id = \"invocations\"\n    metric {\n      metric_name = \"Invocations\"\n      namespace   = \"AWS/Lambda\"\n      period      = 300\n      stat        = \"Sum\"\n      dimensions = {\n        FunctionName = aws_lambda_function.query.function_name\n      }\n    }\n  }\n\n  alarm_description = \"RAG query error rate exceeded 5% for 10 minutes\"\n  alarm_actions     = [aws_sns_topic.alerts.arn]\n}\n\nresource \"aws_sns_topic\" \"alerts\" {\n  name = \"rag-pipeline-alerts\"\n}\n```\n\nBedrock can log every model invocation — input, output, latency, token counts — to S3 and/or CloudWatch. This is essential for debugging bad answers and tracking costs:\n\n```\nresource \"aws_s3_bucket\" \"bedrock_logs\" {\n  bucket = \"rag-bedrock-invocation-logs-${data.aws_caller_identity.current.account_id}\"\n}\n\nresource \"aws_bedrock_model_invocation_logging_configuration\" \"main\" {\n  logging_config {\n    embedding_data_delivery_enabled = true\n\n    s3_config {\n      bucket_name = aws_s3_bucket.bedrock_logs.id\n      key_prefix  = \"invocation-logs/\"\n    }\n  }\n}\n```\n\nWith invocation logging enabled, you can query the S3 logs with Athena to answer questions like \"which queries had the highest token counts this week?\" or \"what questions produced empty retrieval results?\"\n\nThe $350/month OpenSearch Serverless minimum is the elephant in the room. Here's how to minimize everything else:\n\nThe initial sync is manual — upload docs to S3, run the ingestion script. For production, you want an event-driven pipeline that automatically ingests new or updated documents.\n\n```\nS3 PutObject Event → EventBridge Rule → Lambda → Bedrock StartIngestionJob\n# ingestion-pipeline.tf\n\nresource \"aws_lambda_function\" \"ingest_trigger\" {\n  function_name    = \"rag-ingest-trigger\"\n  filename         = data.archive_file.ingest_trigger.output_path\n  source_code_hash = data.archive_file.ingest_trigger.output_base64sha256\n  handler          = \"ingest_trigger.handler\"\n  runtime          = \"python3.12\"\n  timeout          = 60\n  memory_size      = 128\n\n  role = aws_iam_role.lambda_ingest.arn\n\n  environment {\n    variables = {\n      KNOWLEDGE_BASE_ID = aws_bedrockagent_knowledge_base.main.id\n      DATA_SOURCE_ID    = aws_bedrockagent_data_source.s3.data_source_id\n    }\n  }\n}\n\nresource \"aws_iam_role\" \"lambda_ingest\" {\n  name = \"rag-ingest-lambda-role\"\n\n  assume_role_policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Action = \"sts:AssumeRole\"\n        Effect = \"Allow\"\n        Principal = {\n          Service = \"lambda.amazonaws.com\"\n        }\n      }\n    ]\n  })\n}\n\nresource \"aws_iam_role_policy\" \"lambda_ingest\" {\n  name = \"rag-ingest-lambda-policy\"\n  role = aws_iam_role.lambda_ingest.id\n\n  policy = jsonencode({\n    Version = \"2012-10-17\"\n    Statement = [\n      {\n        Effect = \"Allow\"\n        Action = [\"bedrock:StartIngestionJob\"]\n        Resource = [\n          aws_bedrockagent_knowledge_base.main.arn\n        ]\n      },\n      {\n        Effect = \"Allow\"\n        Action = [\n          \"logs:CreateLogGroup\",\n          \"logs:CreateLogStream\",\n          \"logs:PutLogEvents\"\n        ]\n        Resource = \"arn:aws:logs:*:*:*\"\n      }\n    ]\n  })\n}\n\n# EventBridge rule for S3 events\nresource \"aws_s3_bucket_notification\" \"documents\" {\n  bucket      = aws_s3_bucket.documents.id\n  eventbridge = true\n}\n\nresource \"aws_cloudwatch_event_rule\" \"s3_upload\" {\n  name        = \"rag-document-upload\"\n  description = \"Trigger ingestion when documents are uploaded to S3\"\n\n  event_pattern = jsonencode({\n    source      = [\"aws.s3\"]\n    detail-type = [\"Object Created\"]\n    detail = {\n      bucket = {\n        name = [aws_s3_bucket.documents.id]\n      }\n      object = {\n        key = [{\n          suffix = \".md\"\n        }, {\n          suffix = \".pdf\"\n        }, {\n          suffix = \".html\"\n        }, {\n          suffix = \".txt\"\n        }]\n      }\n    }\n  })\n}\n\nresource \"aws_cloudwatch_event_target\" \"ingest_lambda\" {\n  rule = aws_cloudwatch_event_rule.s3_upload.name\n  arn  = aws_lambda_function.ingest_trigger.arn\n}\n\nresource \"aws_lambda_permission\" \"eventbridge\" {\n  statement_id  = \"AllowEventBridgeInvoke\"\n  action        = \"lambda:InvokeFunction\"\n  function_name = aws_lambda_function.ingest_trigger.function_name\n  principal     = \"events.amazonaws.com\"\n  source_arn    = aws_cloudwatch_event_rule.s3_upload.arn\n}\npython\n# lambda/ingest_trigger.py\n\nimport json\nimport os\nimport logging\nimport time\nimport boto3\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n\nbedrock_agent = boto3.client(\"bedrock-agent\")\n\nKNOWLEDGE_BASE_ID = os.environ[\"KNOWLEDGE_BASE_ID\"]\nDATA_SOURCE_ID = os.environ[\"DATA_SOURCE_ID\"]\n\n# Debounce: don't start a new ingestion if one is already running\ndef handler(event, context):\n    logger.info(f\"S3 event received: {json.dumps(event)}\")\n\n    # Check if an ingestion job is already running\n    jobs = bedrock_agent.list_ingestion_jobs(\n        knowledgeBaseId=KNOWLEDGE_BASE_ID,\n        dataSourceId=DATA_SOURCE_ID,\n        sortBy={\"attribute\": \"STARTED_AT\", \"order\": \"DESCENDING\"},\n        maxResults=1,\n    )\n\n    if jobs.get(\"ingestionJobSummaries\"):\n        latest = jobs[\"ingestionJobSummaries\"][0]\n        if latest[\"status\"] in [\"STARTING\", \"IN_PROGRESS\"]:\n            logger.info(\n                f\"Ingestion job {latest['ingestionJobId']} already running. Skipping.\"\n            )\n            return {\"statusCode\": 200, \"body\": \"Ingestion already in progress\"}\n\n    # Start new ingestion\n    response = bedrock_agent.start_ingestion_job(\n        knowledgeBaseId=KNOWLEDGE_BASE_ID,\n        dataSourceId=DATA_SOURCE_ID,\n    )\n\n    job_id = response[\"ingestionJob\"][\"ingestionJobId\"]\n    logger.info(f\"Started ingestion job: {job_id}\")\n\n    return {\"statusCode\": 200, \"body\": json.dumps({\"ingestionJobId\": job_id})}\n```\n\nThe debounce check is important. If someone uploads 50 documents in quick succession, you don't want 50 ingestion jobs — one sync will pick up all the new files. The Lambda checks for a running job before starting a new one.\n\nBedrock Knowledge Bases natively handle:\n\nFor Confluence, I export pages as HTML (Confluence's REST API supports this) and drop them in S3. A small Lambda handles the export on a nightly schedule.\n\n**1. OpenSearch Serverless minimum cost is the biggest surprise.**\n\n$350/month for 2 OCUs before you index a single document. This is the minimum — you can't go lower. If your RAG pipeline is for a small team or a proof of concept, this cost is hard to justify. Watch for S3 Vectors support in Bedrock Knowledge Bases — when it ships, it'll drop the storage cost to pennies.\n\n**2. Chunking strategy matters more than model choice.**\n\nI spent a week trying different generation models (Haiku, Sonnet, Llama) before realizing the answer quality was determined by what the retriever returned, not which model summarized it. Bad chunks → bad retrieval → bad answers, regardless of model. Fix your chunking before upgrading your model.\n\n**3. Bedrock Knowledge Bases handle 80% of the RAG plumbing — don't rebuild it.**\n\nThe temptation to build a custom pipeline with LangChain or LlamaIndex is strong. Resist it unless you have a specific reason. Bedrock Knowledge Bases handle document parsing, chunking, embedding, indexing, retrieval, and citation generation out of the box. That's months of engineering you're not doing.\n\n**4. Citation quality depends on metadata, not just content.**\n\nThe difference between \"According to a document...\" and \"According to the Q3 2026 Post-Mortem for the Payment Service (last updated October 15)\" is metadata. Store descriptive filenames and folder structures in S3. The metadata is included in the vector store and returned with citations.\n\n**5. Start with Bedrock's managed RAG, escape to custom only when you hit limits.**\n\nYou'll hit limits — Bedrock Knowledge Bases don't support custom reranking, multi-modal retrieval, or query decomposition out of the box. But you'll discover which limits matter by running the managed version first. Don't prematurely optimize.\n\n`Resource: \"*\"` to specific ARNs", "url": "https://wpnews.pro/news/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and", "canonical_source": "https://dev.to/garrett_yan/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-serverless-and-1gpd", "published_at": "2026-11-02 21:35:17+00:00", "updated_at": "2026-11-02 21:53:04.331655+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["AWS", "Amazon Bedrock", "OpenSearch Serverless", "Amazon Titan", "Claude 3.5 Sonnet", "Terraform", "AWS Lambda", "API Gateway"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and", "markdown": "https://wpnews.pro/news/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and.md", "text": "https://wpnews.pro/news/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and.txt", "jsonld": "https://wpnews.pro/news/building-a-production-rag-pipeline-on-aws-bedrock-knowledge-bases-opensearch-and.jsonld"}}