{"slug": "how-to-convert-arxiv-pdfs-to-markdown-with-python", "title": "How to convert arXiv PDFs to Markdown with Python", "summary": "A Spidra tutorial demonstrates converting the arXiv paper \"Attention Is All You Need\" (identifier 1706.03762, version 7) from PDF to Markdown using Python 3.9 or later, the Spidra Python SDK and the arxiv package. The code calls client.parse() on the PDF URL with pages=\"1,4,8\", mode=\"ocr\", include_pages=True and page_markers=True to extract three pages while preserving equations, tables and page references, and the tutorial also saves the paper's metadata for reuse in research tools, AI agents or RAG applications.", "body_md": "Research papers on arXiv contain valuable information for AI applications, but working directly with PDFs can make it difficult to extract text, preserve mathematical equations, and organize content for retrieval.\n\nConverting these papers to Markdown gives you structured text that is easier to search, store, and process. With Spidra, you can parse an arXiv PDF directly from its URL, extract selected pages, and preserve equations, tables, and page references.\n\nIn this tutorial, you'll use Python and [Spidra Parse](https://spidra.io/parse) to convert the *Attention Is All You Need* research paper into Markdown. You'll also save the paper's metadata and prepare the extracted content for reuse in research tools, AI agents, or RAG applications.\n\n## Get an arXiv research paper\n\nEvery arXiv paper has a unique identifier that you can use to access its abstract page and PDF.\n\nFor example, the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762), which introduced the Transformer architecture, has the identifier `1706.03762`.\n\nIts abstract page is available at:\n\n```\nhttps://arxiv.org/abs/1706.03762\n```\n\nThe PDF is available at:\n\n```\nhttps://arxiv.org/pdf/1706.03762\n```\n\nWe'll use version 7 of this paper, `1706.03762v7`, so the conversion refers to a specific revision.\n\nThis paper is a useful example because it includes mathematical equations, scientific tables, multi-column text, and multiple authors. We'll extract three PDF pages containing the title and abstract, the scaled dot-product attention equation, and the model evaluation results.\n\nYou don't need to download the PDF manually. Spidra accepts the document URL and returns its content as Markdown.\n\n## Set up Python and Spidra\n\nYou'll need Python 3.9 or later and a [Spidra account](https://app.spidra.io/) with an API key.\n\nInstall the Spidra Python SDK and the `arxiv` package:\n\n```\npip install spidra arxiv\n```\n\nThe Spidra SDK handles document parsing, while the `arxiv` package retrieves bibliographic information such as the paper's title, authors, publication date, and version from the arXiv API.\n\nNext, create an API key in your Spidra dashboard and save it as an environment variable:\n\n```\nexport SPIDRA_API_KEY=\"spd_YOUR_API_KEY\"\n```\n\nYou can now submit PDF links to Spidra using Python.\n\n## Convert the arXiv PDF to Markdown\n\nCreate a file named `parse_arxiv.py` and add the following code:\n\n``` python\n# parse_arxiv.py\nimport os\nfrom spidra import Spidra\n\nclient = Spidra(api_key=os.environ[\"SPIDRA_API_KEY\"])\n\ndoc = client.parse(\n    \"https://arxiv.org/pdf/1706.03762v7\",\n    pages=\"1,4,8\",\n    mode=\"ocr\",\n    include_pages=True,\n    page_markers=True,\n)\n```\n\nThe `parse()` method accepts the paper's URL and returns the extracted content as Markdown. The Python SDK handles job submission and polling, so you don't need to implement those steps yourself.\n\nA few options control the result:\n\n- `pages=\"1,4,8\"` selects three PDF pages instead of processing the entire document. You can also specify ranges such as`\"1-5\"` or`\"10-\"` .\n- `mode=\"ocr\"` uses Spidra's AI-powered reading mode for every selected page. This is useful for research papers containing mathematical notation, tables, and complex layouts.\n- `include_pages=True` returns each page separately with its page number and extracted Markdown.\n- `page_markers=True` inserts page markers into the combined Markdown, making it easier to trace content back to the original document.\n\nSpidra also offers `auto` mode, which uses a PDF's embedded text when available and reads scanned pages with AI. For this paper, we'll use OCR mode because it preserves the mathematical notation and table values particularly well.\n\n### What the extracted Markdown looks like\n\nThe conversion returns the paper's headings, paragraphs, author information, equations, and tables as Markdown.\n\nFor example, the extracted content includes the section:\n\n```\n### 3.2.1 Scaled Dot-Product Attention\n\nWe call our particular attention \"Scaled\nDot-Product Attention\" (Figure 2).\n```\n\nThe mathematical equation from the original PDF is represented using LaTeX:\n\n```\n$$\n\\text{Attention}(Q, K, V) =\n\\text{softmax}\\left(\n\\frac{QK^T}{\\sqrt{d_k}}\n\\right)V\n$$\n```\n\nThis represents the scaled dot-product attention equation:\n\n$$\n\n\\text{Attention}(Q,K,V)=\n\n\\text{softmax}\\left(\n\n\\frac{QK^T}{\\sqrt{d_k}}\n\n\\right)V\n\n$$\n\nPreserving the mathematical notation is useful when you want to index research papers, explain scientific concepts through an AI assistant, or render the extracted material in a Markdown viewer that supports LaTeX.\n\nThe results table is also converted into Markdown. Here is an excerpt from the paper's model comparison table:\n\n```\n| Model | BLEU | | Training Cost (FLOPs) | |\n| :--- | :---: | :---: | :---: | :---: |\n| | EN-DE | EN-FR | EN-DE | EN-FR |\n| Transformer (base model) | 27.3 | 38.1 | $3.3 \\cdot 10^{18}$ | |\n| Transformer (big) | 28.4 | 41.8 | $2.3 \\cdot 10^{19}$ | |\n```\n\nThe model names, BLEU scores, and training-cost values are available as text rather than being locked inside the PDF.\n\nYou can access the complete result through `doc.markdown`, or work with individual pages through `doc.pages`.\n\nEach page record includes its page number, Markdown content, and reading method. This lets you retain document context when building search or retrieval applications.\n\n## Retrieve the paper's metadata from arXiv\n\nThe Markdown contains the paper's text, but a useful research workflow should also preserve information about its source.\n\nFor example, if an AI assistant retrieves a passage from a research paper, you may want to display its title, authors, publication date, and the PDF page containing the answer.\n\nThe `arxiv` Python package provides this information through arXiv's API.\n\nAdd the following code:\n\n``` python\nimport arxiv\n\narxiv_client = arxiv.Client()\n\nsearch = arxiv.Search(\n    id_list=[\"1706.03762v7\"]\n)\n\npaper = next(arxiv_client.results(search))\n```\n\nThe `paper` object contains bibliographic information, including:\n\n- `paper.title` : The research paper's title.\n- `paper.authors` : The authors associated with the paper.\n- `paper.published` : The original submission date.\n- `paper.updated` : The revision date.\n- `paper.entry_id` : The URL identifying the paper on arXiv.\n- `paper.get_short_id()` : The arXiv identifier, including its version.\n\nThe `get_short_id()` method is especially useful for keeping track of revisions. A paper may be updated after its initial submission, and its version identifier lets you associate the extracted text with the correct revision.\n\nWe'll use this metadata when saving the Markdown to disk.\n\n## Save the Markdown and paper metadata using Python\n\nLet's combine document conversion and metadata retrieval into one Python script.\n\nThe script will:\n\n1. Retrieve a paper's bibliographic information from arXiv.\n2. Convert selected PDF pages into Markdown using Spidra.\n3. Save the Markdown with the original source information.\n4. Write the paper's metadata to a separate JSON file.\n\nCreate a new file named `arxiv_to_markdown.py`:\n\n``` python\n# arxiv_to_markdown.py\nimport json\nimport os\nfrom pathlib import Path\n\nimport arxiv\nfrom spidra import Spidra\n\nARXIV_ID = \"1706.03762v7\"\nSELECTED_PAGES = \"1,4,8\"\n\n# Retrieve bibliographic metadata.\npaper = next(\n    arxiv.Client().results(\n        arxiv.Search(id_list=[ARXIV_ID])\n    )\n)\n\npaper_id = paper.get_short_id()\npdf_url = f\"https://arxiv.org/pdf/{paper_id}\"\n\n# Convert the PDF to Markdown.\nclient = Spidra(\n    api_key=os.environ[\"SPIDRA_API_KEY\"]\n)\n\ndoc = client.parse(\n    pdf_url,\n    pages=SELECTED_PAGES,\n    mode=\"ocr\",\n    include_pages=True,\n    page_markers=True,\n)\n\n# Preserve source and page information.\nmetadata = {\n    \"arxiv_id\": paper_id,\n    \"title\": paper.title,\n    \"authors\": [author.name for author in paper.authors],\n    \"published\": paper.published.date().isoformat(),\n    \"updated\": paper.updated.date().isoformat(),\n    \"source_url\": paper.entry_id,\n    \"pdf_url\": pdf_url,\n    \"pages\": [page.page_number for page in doc.pages],\n    \"total_pages\": doc.total_pages,\n}\n\n# Create the output directory.\noutput_dir = Path(\"papers\")\noutput_dir.mkdir(exist_ok=True)\nbase_name = paper_id.replace(\"/\", \"_\")\n\n# Save Markdown with source attribution.\nmarkdown = (\n    f\"# {paper.title}\\n\\n\"\n    f\"**Authors:** {', '.join(metadata['authors'])}\\n\\n\"\n    f\"**Source:** {paper.entry_id}\\n\\n\"\n    f\"**PDF:** {pdf_url}\\n\\n\"\n    f\"{doc.markdown}\\n\"\n)\n\n(output_dir / f\"{base_name}.md\").write_text(\n    markdown,\n    encoding=\"utf-8\",\n)\n\n# Save structured paper metadata.\n(output_dir / f\"{base_name}.metadata.json\").write_text(\n    json.dumps(metadata, ensure_ascii=False, indent=2),\n    encoding=\"utf-8\",\n)\n```\n\nRun the script:\n\n```\npython arxiv_to_markdown.py\n```\n\nIt creates a `papers` directory containing two files:\n\n```\npapers/\n├── 1706.03762v7.md\n└── 1706.03762v7.metadata.json\n```\n\nThe Markdown file contains the paper's extracted content, together with its title, authors, source URL, and PDF link.\n\nBecause `page_markers=True` is enabled, the combined Markdown also contains HTML comments identifying the source pages:\n\n``` php\n<!-- page 1 -->\n\n<!-- page 4 -->\n\n<!-- page 8 -->\n```\n\nThese markers are inserted at the corresponding page boundaries, rather than being presented as a separate list. They remain available in the Markdown source even when a renderer hides HTML comments.\n\nThe JSON file provides the same paper identity in a format your application can read directly. Some of its fields look like this:\n\n```\n{\n  \"arxiv_id\": \"1706.03762v7\",\n  \"title\": \"Attention Is All You Need\",\n  \"pages\": [1, 4, 8],\n  \"total_pages\": 15\n}\n```\n\nThe complete JSON also includes the author list, publication and revision dates, and original source URLs.\n\nTo process a different research paper, change `ARXIV_ID`. You can also change `SELECTED_PAGES` to extract another range or remove the `pages` argument from `client.parse()` to convert the entire PDF.\n\nThe same script will retrieve the new paper's metadata and save its results under the corresponding arXiv identifier.\n\n## Prepare the Markdown for a research or RAG application\n\nOnce a research paper is available as Markdown, you can use the content in document search, summarization pipelines, or retrieval-augmented generation applications.\n\nFor RAG, preserving the source URL and page number alongside each passage is useful. It allows an application to identify where a retrieved answer came from rather than returning an unattributed block of text.\n\nSpidra's `include_pages=True` option makes it straightforward to build page-aware records.\n\nFor example, add this code to the end of `arxiv_to_markdown.py`:\n\n```\n# Save individual pages as JSONL records.\npages_path = output_dir / f\"{base_name}.pages.jsonl\"\n\nwith pages_path.open(\"w\", encoding=\"utf-8\") as file:\n    for page in doc.pages:\n        record = {\n            \"text\": page.markdown,\n            \"arxiv_id\": paper_id,\n            \"page\": page.page_number,\n            \"source_url\": pdf_url,\n        }\n        file.write(\n            json.dumps(record, ensure_ascii=False) + \"\\n\"\n        )\n```\n\nThis creates a third file, `1706.03762v7.pages.jsonl`, containing one JSON record per extracted page.\n\nEach record connects the extracted text to the paper's arXiv identifier, the original PDF, and the source page.\n\nFrom here, you can split longer pages into smaller chunks, generate embeddings, and store them in a vector database. Your retrieval layer can carry those metadata fields forward when presenting search results or generating citations.\n\nFor scientific content, retaining the original PDF link is also useful when an application needs to display the source equation, figure, or table alongside the extracted text.\n\nIf you're building a broader ingestion workflow, you can also use [Spidra Search](https://docs.spidra.io/search) with the `research` source to discover papers from arXiv and other research repositories before passing their PDF links to Parse.\n\n## Process more arXiv papers\n\nThe example above processes one paper, but the same approach works for a collection of research papers.\n\nFor each paper, you can retrieve its metadata, submit its PDF URL to Spidra Parse, and save the Markdown under its arXiv identifier. Keeping the version in the filename also makes it easier to distinguish revisions in a recurring ingestion workflow.\n\nIf you already have multiple PDF links, Spidra's `parse_batch()` method can process up to 50 documents per batch. For workflows that need to process many documents, see our guide on [batch processing PDFs with Python](https://spidra.io/blog/batch-process-pdfs-python).\n\nYou can also estimate the pages and credits required with `estimate_parse()` before submitting a document. For plain Markdown extraction, Spidra charges one credit per parsed PDF page, including pages read using OCR.\n\nFor more information about PDF inputs, uploads, and reusing documents, see our guide to [PDF URLs, uploads, and file IDs](https://spidra.io/blog/pdf-url-vs-upload-vs-file-id).\n\nWith Markdown files and source metadata stored together, you have the foundation for a repeatable research ingestion workflow that can support literature search, document analysis, and AI-powered research tools.\n\n## Convert your own arXiv papers with Spidra\n\nYou've now built a Python workflow that takes an arXiv paper, converts selected pages to Markdown, preserves mathematical expressions and scientific tables, and saves the result with its bibliographic metadata and page references.\n\nYou can use the same approach for other research papers or extend it into a larger document-processing pipeline.\n\n[Create a free Spidra account](https://app.spidra.io/), get your API key, and run `arxiv_to_markdown.py` with a paper you want to process.", "url": "https://wpnews.pro/news/how-to-convert-arxiv-pdfs-to-markdown-with-python", "canonical_source": "https://spidra.io/blog/arxiv-pdf-to-markdown-python", "published_at": "2026-10-27 00:00:00+00:00", "updated_at": "2026-10-27 17:41:35.568714+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-research", "large-language-models"], "entities": ["Spidra", "arXiv", "Attention Is All You Need", "Spidra Parse", "Spidra Python SDK", "arxiv Python package"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-convert-arxiv-pdfs-to-markdown-with-python", "markdown": "https://wpnews.pro/news/how-to-convert-arxiv-pdfs-to-markdown-with-python.md", "text": "https://wpnews.pro/news/how-to-convert-arxiv-pdfs-to-markdown-with-python.txt", "jsonld": "https://wpnews.pro/news/how-to-convert-arxiv-pdfs-to-markdown-with-python.jsonld"}}