How to Build a Simple AI Web Scraper with Python

A new Python tutorial demonstrates how to build an AI web scraper that cleans HTML, converts it to Markdown, and uses OpenAI's gpt-5.4-nano model to answer user queries, reducing token usage by sending only useful page content. The guide, published for Jupyter Notebook users, installs packages including requests, BeautifulSoup, markdownify, and openai, and requires an OpenAI API key with billing enabled. The scraper fetches webpages with a custom User-Agent header and returns focused Markdown answers instead of full page dumps.

How to Build a Simple AI Web Scraper with Python Turn any webpage into a lightweight LLM-powered QA engine by cleaning HTML, converting content to Markdown, and returning focused answers while reducing token usage. Web scraping is the process of collecting information from websites automatically. A normal scraper usually extracts raw text, HTML elements, or the full page content. But when you are building AI agents or large language model LLM applications, sending the entire webpage to the model is not always the best approach. A better way is to first clean the page, convert it into Markdown, and then use an LLM to understand the content and return only the answer the user needs. This makes the output cleaner, easier to read, and easier to use in another workflow. It also helps reduce token usage. Instead of passing a messy webpage full of navigation links, buttons, scripts, footers, and repeated content, we only send the useful page content to the model. The LLM then returns a focused answer in Markdown instead of dumping the whole page back to the user. In this guide, we will build a simple AI web scraper in Python using Jupyter Notebook. It will fetch a webpage, clean the HTML, convert it into Markdown, accept a user query, and return a clear Markdown answer based on the page content. Setting Up We will use Jupyter Notebook for this project. It makes it easier to test each step first before turning the scraper into a proper application programming interface API or application. Start by installing the required Python packages: pip install requests beautifulsoup4 markdownify openai ftfy python-dotenv We will use: to fetch the webpage. requests https://pypi.org/project/requests/ to remove noisy HTML elements. BeautifulSoup https://www.crummy.com/software/BeautifulSoup/ to convert HTML into Markdown. markdownify https://pypi.org/project/markdownify/ to answer the user query. OpenAI https://pypi.org/project/openai/ to fix broken or messy text. ftfy https://pypi.org/project/ftfy/ to load the API key safely. python-dotenv https://pypi.org/project/python-dotenv/ In the next cell, import the required libraries: python import os import re import requests from bs4 import BeautifulSoup, Comment from ftfy import fix text from markdownify import markdownify as markdownify html from openai import OpenAI from dotenv import load dotenv from IPython.display import Markdown, display Next, make sure your OpenAI API key is available as an environment variable. The safer way is to create a .env file in the same folder as your notebook and add your key there: OPENAI API KEY=your api key here Then load it inside the notebook: load dotenv client = OpenAI api key=os.getenv "OPENAI API KEY" You can also check that the key was loaded correctly: if not os.getenv "OPENAI API KEY" : raise ValueError "OPENAI API KEY is missing. Add it to your .env file first." Also make sure your OpenAI platform account has billing set up. For new API accounts, you may need to add prepaid credits before you can run API calls. If a model is not available in your account, use another model from your OpenAI dashboard. Now define the model name: MODEL NAME = "gpt-5.4-nano" We are using a smaller model here because this task does not need a large reasoning model. The goal is simple: read the cleaned webpage content, understand the user query, and return a focused Markdown answer. Fetching the Webpage Now we will create the first function. This function will fetch the webpage using the requests package and return the raw HTML. php def fetch page url: str - str: """ Download the HTML content from a webpage. """ headers = { "User-Agent": "SimpleAIScraper/1.0" } response = requests.get url, headers=headers, timeout=15 response.raise for status return response.text The User-Agent header tells the website that the request is coming from our scraper. Some websites block requests that do not include a user agent, so adding one makes the request a bit more reliable. We also use timeout to avoid waiting indefinitely if the website does not respond. The raise for status call will stop the code if the request fails — for example, if the page returns a 404 or 500 error. Now let's test the function with a real website: raw = fetch page "https://www.olostep.com/" print raw :500 This will download the raw HTML from the webpage and print the first 500 characters. Raw HTML output | Image by Author At this stage, the output will still look messy because it contains the full page HTML, including tags, scripts, layout elements, and other content we do not need. Cleaning the HTML The raw HTML from a webpage usually contains a lot of content we do not need. It can include scripts, styling, navigation menus, buttons, forms, headers, footers, popups, and other layout elements. Before sending the page content to the LLM, we need to clean the HTML. This helps reduce noise and makes the final Markdown much easier for the model to understand. We will use BeautifulSoup to parse the HTML and remove unnecessary elements. python def clean html html : html = fix text html soup = BeautifulSoup html, "html.parser" Remove obvious noisy tags for tag in soup "script", "style", "noscript", "svg", "img", "iframe", "nav", "header", "footer", "aside", "form", "button" : tag.decompose noise words = "cursor", "modal", "popup", "floating", "signup", "login", "cookie", "banner", "navbar", "menu", "footer", "header", "subscribe", "newsletter", "loading", "wait", "success", "auth", "w-nav", "w-form" First collect noisy tags tags to remove = for tag in soup.find all True : if tag.attrs is None: continue class value = tag.get "class", id value = tag.get "id", "" if isinstance class value, list : class text = " ".join class value .lower else: class text = str class value .lower id text = str id value .lower if any word in class text or word in id text for word in noise words : tags to remove.append tag Then remove them safely for tag in tags to remove: tag.decompose body = soup.body if soup.body else soup return str body First, we use fix text to clean any broken or strange text encoding issues. Then BeautifulSoup parses the HTML so we can remove the parts we do not need. We remove obvious noisy tags like script , style , nav , header , footer , form , and button . These sections usually do not help answer the user query and can waste tokens. After that, we look for noisy class names and IDs. Many websites use words like popup , cookie , navbar , newsletter , or modal inside their HTML. If a tag contains those words, we collect it and remove it safely. Now let's run the function on the raw HTML: clean = clean html raw print clean :500 As you can see, the webpage is now much cleaner. It still contains useful HTML tags and text, but most of the noisy layout, scripts, navigation, and popups have been removed. Cleaned HTML output | Image by Author Converting HTML to Markdown Now we will convert the cleaned HTML into Markdown. Markdown is easier to read, easier to save, and easier for the LLM to understand compared to raw HTML. This step also helps reduce input tokens because we remove unnecessary formatting, images, blank lines, and repeated text. For the conversion, we will use markdownify. python def html to markdown html : markdown text = markdownify html html, heading style="ATX", bullets="-" markdown text = fix text markdown text Remove image markdown markdown text = re.sub r" \ . ?\ \ . ?\ ", "", markdown text Remove extra spaces and blank lines markdown text = re.sub r" \t +", " ", markdown text markdown text = re.sub r"\n{3,}", "\n\n", markdown text lines = skip lines = "click to try", "wait...", "you've successfully reserved your spot.", "thank you your submission has been received ", "oops something went wrong while submitting the form.", "product", "resources", "company" for line in markdown text.splitlines : line = line.strip if not line: continue if line.lower in skip lines: continue lines.append line return "\n".join lines First, we use markdownify to convert the cleaned HTML into Markdown. We set the heading style to ATX , which means headings will use standard Markdown syntax with , , and . Then we run fix text again to clean any remaining encoding issues. After that, we remove image Markdown because image links are usually not useful for answering text-based questions. We also remove extra spaces and blank lines so the final content is compact. This makes the page easier to inspect and helps reduce the number of tokens sent to the model. The skip lines list removes repeated website text such as form messages, navigation labels, and small call-to-action text. You can update this list based on the website you are scraping. Now let's run the function: md = html to markdown clean print md :500 As you can see, the text is now much cleaner and closer to the format we want. Instead of raw HTML, we now have readable Markdown with useful headings, paragraphs, and bullet points. Markdown output | Image by Author Asking a User Query Against the Page Now we will create the function that sends the cleaned Markdown content to the LLM. This function takes two inputs: the webpage content in Markdown and the user query. Instead of asking the model to summarize the whole page, we ask it to answer a specific question using only the page content. This makes the response more focused and useful. python def answer query from page markdown text, user query : prompt = f""" You are an AI web scraping assistant. You will receive Markdown extracted from a webpage. Your task is to answer the user's query using only the useful page content. User query: {user query} Webpage Markdown: {markdown text} Instructions: - Return only clean Markdown. - Use only information from the webpage Markdown. - Do not invent missing details. - Ignore navigation links, buttons, CTAs, popups, decorative labels, image captions, and repeated marketing fragments. - Ignore lines like "Start for free", "Contact Sales", "Your AI Agent", and decorative workflow examples unless they directly answer the query. - Focus on headings, paragraphs, product descriptions, feature sections, pricing details, documentation text, and factual claims. - If the page does not contain the answer, say: "The page does not contain this information." - Keep the answer short, clear, and focused. """ response = client.responses.create model=MODEL NAME, input=prompt return response.output text The prompt is the most important part of this step. It tells the model what role it should play, what content it can use, and what kind of answer it should return. We also tell the model to use only the provided Markdown. This is important because we do not want the model to guess or add information that is not present on the webpage. The instruction to return only clean Markdown makes the output easier to display in a notebook, save to a file, or pass into another AI workflow. This function is where the AI web scraper becomes genuinely useful. We are no longer just extracting page text — we are asking the LLM to understand the cleaned page and return the exact answer the user is looking for. Creating the Full AI Web Scraper Now we will create the final function that connects everything together. This function will take the URL and the user query as inputs. It will then fetch the webpage, clean the HTML, convert the content into Markdown, and return the answer using the gpt-5.4-nano model. python def ai web scraper url, user query : raw html = fetch page url cleaned html = clean html raw html markdown text = html to markdown cleaned html answer = answer query from page markdown text, user query return answer This is our complete AI web scraper pipeline. Instead of manually running each step one by one, we can now call a single function and get a clean Markdown answer from any webpage. The flow is simple: - Fetch the webpage. - Clean the HTML. - Convert it into Markdown. - Ask the LLM a question. - Return the final answer. This keeps the code simple and easy to reuse later in an API, chatbot, or agent workflow. Testing the AI Web Scraper Now let's test our AI web scraper. We will provide it with a website URL and ask what the company does. url = "https://www.olostep.com/" user query = "What does this company do?" result = ai web scraper url, user query display Markdown result In return, we get a proper Markdown response about the company and its product. This is much better than returning the full webpage content because the answer is focused, readable, and directly related to the user query. Scraper output for a company overview query | Image by Author Now let's try a different page and ask about pricing. url = "https://www.olostep.com/pricing" user query = "Help me understand the pricing" result = ai web scraper url, user query display Markdown result In a few seconds, we get a clean response that is easy to understand. Instead of manually visiting the pricing page and trying to find the relevant information, the scraper extracts the page, cleans it, and asks the LLM to explain only what matters. Scraper output for a pricing query | Image by Author We can also save the final response as a Markdown file. with open "ai scraper result.md", "w", encoding="utf-8" as file: file.write result print "Markdown saved to ai scraper result.md" Output: Markdown saved to ai scraper result.md Now the result is saved as a Markdown file, which you can open, edit, share, or use in another workflow. Final Thoughts Building your own AI tools is much easier now. With a few lines of Python and an LLM, we turned a normal webpage into a simple question-answering engine that can read the page, understand the user query, and return a clean Markdown answer. This is powerful because you do not always need a complex system to solve a specific problem. Sometimes, a small specialized solution is enough. But it is also important to remember that everything has a cost. Running the app on a server costs money. Calling an LLM costs money. Maintaining the scraper, fixing broken pages, handling errors, and improving the system over time also costs time and money. So before building your own custom solution, it is worth looking at existing tools like Olostep , , or Firecrawl https://www.firecrawl.dev/ . In some cases, paying for a ready-made scraping or web intelligence API may make more sense. In other cases — especially if the task is small, local, or very specific — building your own lightweight solution can be the better option. Exa https://exa.ai/ Abid Ali Awan https://abid.work @1abidaliawan https://www.linkedin.com/in/1abidaliawan is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in technology management and a bachelor's degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.