{"slug": "how-does-retrieval-augmented-generation-change-the-way-we-search-websites", "title": "How does Retrieval-Augmented Generation change the way we search websites?", "summary": "Proctors, a digital agency, has detailed an architecture for building retrieval-augmented generation (RAG) search as a standalone service, separate from the website itself. The design uses PostgreSQL with pgvector for storage, n8n for content ingestion, and AWS Lambda for the API endpoint, enabling semantic search and conversational answers over website content.", "body_md": "Most website searches have a simple job: match the words entered by a visitor with the words appearing on a page. That works really well when someone knows exactly what they’re looking for. But it’s less useful when they ask a detailed question, use different terminology from your website, or want information spread across several pages.\n\nRAG offers a different approach.\n\nInstead of returning a list of pages containing matching keywords, a RAG-powered search tool can understand the meaning behind a visitor’s question, find the most relevant information within your website and use a large language model (LLM) to produce a useful, conversational answer.\n\nMore importantly, it can be built as a standalone service.\n\nThat means the RAG/search system can be kept separate from the website. If the website is rebuilt, moved to a different CMS or replaced with another technology, you only need to change the process that takes the website’s content and puts it into the search system.\n\nTraditional website search is usually based on keywords. For example, a user searching for ‘support for an existing website’ may receive poor results if the relevant page uses phrases such as ‘ongoing maintenance’, ‘technical consultancy’ or ‘managed digital services’.\n\nA semantic search system looks beyond exact wording. It attempts to understand that these phrases may describe related needs.\n\nThis makes RAG particularly useful for websites containing:\n\nLarge numbers of articles, services or product pages\n\nComplex technical information\n\nMultiple sectors or audience types\n\nDocumentation and support content\n\nInformation spread across several pages\n\nSpecialist terminology and acronyms\n\nContent created by different teams over many years\n\nRAG can also answer more detailed questions. Rather than searching for “cloud services”, a visitor could ask:\n\n“Can you help us migrate an existing business application to the cloud without replacing the whole system?”\n\nThe search tool can retrieve relevant sections from several pages, combine the information and provide a concise answer with links for further reading.\n\nAt Proctors, we deliberately design our RAG search as a separate service rather than building it into the website itself and only requires an interface capable of sending a question to an API endpoint (a way for the website to communicate with the RAG service) and displaying the response.\n\nOur suggested architecture can be separated into two main processes:\n\nContent ingestion – getting the information from the website and preparing it for search.\n\nSearch and answer – retrieving a question or prompt and returning with an answer.\n\nThere are many technologies capable of supporting a RAG application. A practical stack could use PostgreSQL, pgvector, n8n and AWS Lambda, combined with either Ollama or a commercial LLM service such as OpenAI or Anthropic. Here’s what we’d recommend.\n\n**PostgreSQL and pgvector for storing website and search data**\n\nPostgreSQL provides the main data store, while the pgvector extension adds support for storing embeddings and performing vector-similarity searches.\n\nThis means the system can store the original text, page metadata and vector representations together, rather than operating a completely separate specialist database.\n\n**n8n for content ingestion**\n\nn8n can provide the workflow used to receive, process and index website content. For example, an n8n workflow could be triggered by:\n\nA CMS webhook\n\nA scheduled website crawl\n\nAn API request\n\nA sitemap import\n\nA content export\n\nA manual publishing process\n\nThe workflow can clean the content, divide it into chunks, create embeddings and insert the resulting records into PostgreSQL.\n\n**AWS Lambda for public API endpoint**\n\nAWS Lambda can provide the endpoint used by the website’s front end. When a visitor submits a question, the Lambda function can:\n\nValidate and sanitise the request\n\nApply rate limits or security rules\n\nGenerate the query embedding\n\nSearch PostgreSQLl\n\nSelect the most relevant content\n\nConstruct the language-model prompt\n\nSend the prompt to the chosen language model or LLM API\n\nReturn the answer and source links\n\nBecause the website communicates with a standard API, the same endpoint can be used by any flavour of website, application or custom-built platform.\n\nWhen it comes to choosing an LLM to go for, our solution can use a model hosted by the organisation through Ollama, or a managed commercial LLM accessed through an API.\n\nOllama is useful when an organisation wants to run supported generative and embedding models within infrastructure it controls. It can keep more of the processing in-house, but the organisation must provide enough computing power to run the models, as well as manage model updates, scaling, monitoring and security.\n\nAlternatively, the search service can send the retrieved context to a commercial provider such as OpenAI, whose models power ChatGPT, Anthropic’s Claude, Google’s Gemini or another suitable LLM service. Managed services can reduce infrastructure work and provide access to high-performing models, but they introduce usage charges, provider-specific limits, data-processing terms and dependency on an external service.\n\nIt is independent of the website platform.\n\nIt supports natural-language questions.\n\nIt is grounded in approved content (only the retrieved website data is referenced).\n\nIt supports different levels of control over data and models (an LLM can be chosen to fit your needs).\n\nIt can grow beyond website search (separate interfaces for more specialised searching of product documentation, case studies etc. can be added).\n\nAnswers are only as good as the indexed content.\n\nChunking affects retrieval quality.\n\nContent must stay synchronised.\n\nLLMs can still make mistakes.\n\nSecurity considerations (ensuring private data is not included in results).\n\nUtilising RAG for site search allows visitors to ask complete questions and receive relevant, conversational answers grounded in your approved content. This can reduce friction, improve content discovery and help prospective customers understand your complex services more quickly.\n\nBuilt as a standalone service, the same search capability can continue working as your website changes and can later be extended to support documentation, product information, customer service or internal knowledge.\n\nHave you experimented with RAG for website search? What was your experience?", "url": "https://wpnews.pro/news/how-does-retrieval-augmented-generation-change-the-way-we-search-websites", "canonical_source": "https://dev.to/proctorstech/how-does-retrieval-augmented-generation-change-the-way-we-search-websites-44mc", "published_at": "2026-09-02 13:32:31+00:00", "updated_at": "2026-09-02 13:54:46.728342+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Proctors", "PostgreSQL", "pgvector", "n8n", "AWS Lambda", "Ollama", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/how-does-retrieval-augmented-generation-change-the-way-we-search-websites", "markdown": "https://wpnews.pro/news/how-does-retrieval-augmented-generation-change-the-way-we-search-websites.md", "text": "https://wpnews.pro/news/how-does-retrieval-augmented-generation-change-the-way-we-search-websites.txt", "jsonld": "https://wpnews.pro/news/how-does-retrieval-augmented-generation-change-the-way-we-search-websites.jsonld"}}