RAG in Laravel: Embeddings and pgvector for a Knowledge-Base Bot

In the last post we streamed AI responses over SSE. Now let's fix the problem every chatbot hits in week one: the model doesn't know your data. Ask it about your refund policy or your API docs and it either hallucinates or shrugs.

RAG — retrieval-augmented generation — fixes that. Instead of hoping the model knows your content, you store your documents as embeddings, retrieve the few chunks relevant to each question, and hand only those to the model. Here's the whole pipeline in Laravel 11 with PostgreSQL and pgvector — no vector-database SaaS required.

I recently shipped a chatbot for my own portfolio site. Total knowledge base: about 2,000 tokens of facts and FAQ. I didn't build RAG for it — I put the entire corpus in the system prompt and let prompt caching make it cheap. Simpler code, zero retrieval bugs, and the model sees everything on every question.

The rule of thumb I use with clients:

If you're in the first bucket, close this tab and go ship. Still here? Let's build.

pgvector

extensionopenai-php/laravel

for embeddings + chatpgvector/pgvector-php

for the Eloquent cast

composer require openai-php/laravel pgvector/pgvector
php artisan vendor:publish --provider="OpenAI\Laravel\ServiceProvider"

On most managed Postgres (RDS, Supabase, Neon, Laravel Forge boxes) pgvector is one statement away:

CREATE EXTENSION IF NOT EXISTS vector;

One table stores chunks of your documents plus their embedding vectors. text-embedding-3-small

produces 1536 dimensions — good quality, cheap ($0.02 per million tokens).

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        DB::statement('CREATE EXTENSION IF NOT EXISTS vector');

        Schema::create('document_chunks', function ($table) {
            $table->id();
            $table->string('source');          // e.g. "refund-policy.md"
            $table->unsignedInteger('position'); // chunk order within the source
            $table->text('content');
            $table->timestamps();
        });

        DB::statement('ALTER TABLE document_chunks ADD COLUMN embedding vector(1536)');

        // HNSW index makes similarity search fast past ~10k rows
        DB::statement('CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)');
    }
};

The model:

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Pgvector\Laravel\Vector;

class DocumentChunk extends Model
{
    protected $fillable = ['source', 'position', 'content', 'embedding'];

    protected $casts = ['embedding' => Vector::class];
}

Chunking strategy matters more than people admit. My default: split on headings/paragraphs, target ~500 tokens per chunk, and never split mid-sentence. Fancy overlap windows can wait until you have evidence you need them.

namespace App\Console\Commands;

use App\Models\DocumentChunk;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;
use OpenAI\Laravel\Facades\OpenAI;

class IngestDocs extends Command
{
    protected $signature = 'rag:ingest {path : Directory of markdown files}';

    public function handle(): int
    {
        foreach (File::files($this->argument('path')) as $file) {
            $chunks = $this->chunk($file->getContents());

            // One API call per file, not per chunk — batch input is supported
            $response = OpenAI::embeddings()->create([
                'model' => 'text-embedding-3-small',
                'input' => $chunks,
            ]);

            DocumentChunk::where('source', $file->getFilename())->delete();

            foreach ($response->embeddings as $i => $embedding) {
                DocumentChunk::create([
                    'source' => $file->getFilename(),
                    'position' => $i,
                    'content' => $chunks[$i],
                    'embedding' => $embedding->embedding,
                ]);
            }

            $this->info("{$file->getFilename()}: " . count($chunks) . ' chunks');
        }

        return self::SUCCESS;
    }

    /** @return string[] */
    private function chunk(string $text, int $targetChars = 2000): array
    {
        $paragraphs = preg_split('/\n{2,}/', $text);
        $chunks = [''];

        foreach ($paragraphs as $p) {
            $current = array_key_last($chunks);
            if (strlen($chunks[$current]) + strlen($p) > $targetChars && $chunks[$current] !== '') {
                $chunks[] = $p;
            } else {
                $chunks[$current] .= "\n\n" . $p;
            }
        }

        return array_map('trim', $chunks);
    }
}

Run it: php artisan rag:ingest storage/docs

. Re-running replaces a file's chunks, so ingestion is idempotent — wire it to your deploy or a scheduled job and content stays fresh.

pgvector's <=>

operator is cosine distance. Lower = more similar. Embed the user's question, sort by distance, take the top handful:

namespace App\Services;

use App\Models\DocumentChunk;
use OpenAI\Laravel\Facades\OpenAI;

class Retriever
{
    /** @return array{content: string, source: string}[] */
    public function search(string $question, int $limit = 5): array
    {
        $embedding = OpenAI::embeddings()->create([
            'model' => 'text-embedding-3-small',
            'input' => $question,
        ])->embeddings[0]->embedding;

        return DocumentChunk::query()
            ->selectRaw('content, source, embedding <=> ? AS distance', [json_encode($embedding)])
            ->orderBy('distance')
            ->limit($limit)
            ->get()
            ->filter(fn ($c) => $c->distance < 0.55) // relevance floor — tune on real queries
            ->map(fn ($c) => ['content' => $c->content, 'source' => $c->source])
            ->all();
    }
}

That distance floor is your hallucination guard. If nothing scores under the threshold, the honest answer is "I don't know" — and you should say exactly that instead of feeding the model junk context.

namespace App\Services;

use OpenAI\Laravel\Facades\OpenAI;

class KnowledgeBot
{
    public function __construct(private Retriever $retriever) {}

    public function answer(string $question): string
    {
        $chunks = $this->retriever->search($question);

        if ($chunks === []) {
            return "I couldn't find that in our documentation — try rephrasing, or contact support.";
        }

        $context = collect($chunks)
            ->map(fn ($c) => "[{$c['source']}]\n{$c['content']}")
            ->implode("\n\n---\n\n");

        $response = OpenAI::chat()->create([
            'model' => 'gpt-4o-mini',
            'max_tokens' => 500,
            'messages' => [
                ['role' => 'system', 'content' => <<<PROMPT
                    Answer using ONLY the context below. If the context doesn't
                    contain the answer, say you don't know — never invent facts.
                    Cite the source file name when you answer.

                    Context:
                    {$context}
                    PROMPT],
                ['role' => 'user', 'content' => $question],
            ],
        ]);

        return $response->choices[0]->message->content;
    }
}

Wire KnowledgeBot::answer()

into the streaming controller from part 2 of this series and you have a knowledge-base bot that streams grounded answers.

Retrieval makes the model knowledgeable. The next step makes it capable: in part 4 we'll build AI agents in PHP — giving the model tools it can call (search orders, create tickets, query your database) and handling the tool-calling loop safely in Laravel. That's where chatbots turn into products.

I'm Aditya Kumar ( adityakdevin) — Tech Lead & full-stack developer building AI-powered web products with Laravel, Vue, and LLM APIs. Find me at adityadev.in.