How to convert arXiv PDFs to Markdown with Python

A Spidra tutorial demonstrates converting the arXiv paper "Attention Is All You Need" (identifier 1706.03762, version 7) from PDF to Markdown using Python 3.9 or later, the Spidra Python SDK and the arxiv package. The code calls client.parse() on the PDF URL with pages="1,4,8", mode="ocr", include_pages=True and page_markers=True to extract three pages while preserving equations, tables and page references, and the tutorial also saves the paper's metadata for reuse in research tools, AI agents or RAG applications.

Research papers on arXiv contain valuable information for AI applications, but working directly with PDFs can make it difficult to extract text, preserve mathematical equations, and organize content for retrieval. Converting these papers to Markdown gives you structured text that is easier to search, store, and process. With Spidra, you can parse an arXiv PDF directly from its URL, extract selected pages, and preserve equations, tables, and page references. In this tutorial, you'll use Python and Spidra Parse https://spidra.io/parse to convert the Attention Is All You Need research paper into Markdown. You'll also save the paper's metadata and prepare the extracted content for reuse in research tools, AI agents, or RAG applications. Get an arXiv research paper Every arXiv paper has a unique identifier that you can use to access its abstract page and PDF. For example, the paper Attention Is All You Need https://arxiv.org/abs/1706.03762 , which introduced the Transformer architecture, has the identifier 1706.03762 . Its abstract page is available at: https://arxiv.org/abs/1706.03762 The PDF is available at: https://arxiv.org/pdf/1706.03762 We'll use version 7 of this paper, 1706.03762v7 , so the conversion refers to a specific revision. This paper is a useful example because it includes mathematical equations, scientific tables, multi-column text, and multiple authors. We'll extract three PDF pages containing the title and abstract, the scaled dot-product attention equation, and the model evaluation results. You don't need to download the PDF manually. Spidra accepts the document URL and returns its content as Markdown. Set up Python and Spidra You'll need Python 3.9 or later and a Spidra account https://app.spidra.io/ with an API key. Install the Spidra Python SDK and the arxiv package: pip install spidra arxiv The Spidra SDK handles document parsing, while the arxiv package retrieves bibliographic information such as the paper's title, authors, publication date, and version from the arXiv API. Next, create an API key in your Spidra dashboard and save it as an environment variable: export SPIDRA API KEY="spd YOUR API KEY" You can now submit PDF links to Spidra using Python. Convert the arXiv PDF to Markdown Create a file named parse arxiv.py and add the following code: python parse arxiv.py import os from spidra import Spidra client = Spidra api key=os.environ "SPIDRA API KEY" doc = client.parse "https://arxiv.org/pdf/1706.03762v7", pages="1,4,8", mode="ocr", include pages=True, page markers=True, The parse method accepts the paper's URL and returns the extracted content as Markdown. The Python SDK handles job submission and polling, so you don't need to implement those steps yourself. A few options control the result: - pages="1,4,8" selects three PDF pages instead of processing the entire document. You can also specify ranges such as "1-5" or "10-" . - mode="ocr" uses Spidra's AI-powered reading mode for every selected page. This is useful for research papers containing mathematical notation, tables, and complex layouts. - include pages=True returns each page separately with its page number and extracted Markdown. - page markers=True inserts page markers into the combined Markdown, making it easier to trace content back to the original document. Spidra also offers auto mode, which uses a PDF's embedded text when available and reads scanned pages with AI. For this paper, we'll use OCR mode because it preserves the mathematical notation and table values particularly well. What the extracted Markdown looks like The conversion returns the paper's headings, paragraphs, author information, equations, and tables as Markdown. For example, the extracted content includes the section: 3.2.1 Scaled Dot-Product Attention We call our particular attention "Scaled Dot-Product Attention" Figure 2 . The mathematical equation from the original PDF is represented using LaTeX: $$ \text{Attention} Q, K, V = \text{softmax}\left \frac{QK^T}{\sqrt{d k}} \right V $$ This represents the scaled dot-product attention equation: $$ \text{Attention} Q,K,V = \text{softmax}\left \frac{QK^T}{\sqrt{d k}} \right V $$ Preserving the mathematical notation is useful when you want to index research papers, explain scientific concepts through an AI assistant, or render the extracted material in a Markdown viewer that supports LaTeX. The results table is also converted into Markdown. Here is an excerpt from the paper's model comparison table: | Model | BLEU | | Training Cost FLOPs | | | :--- | :---: | :---: | :---: | :---: | | | EN-DE | EN-FR | EN-DE | EN-FR | | Transformer base model | 27.3 | 38.1 | $3.3 \cdot 10^{18}$ | | | Transformer big | 28.4 | 41.8 | $2.3 \cdot 10^{19}$ | | The model names, BLEU scores, and training-cost values are available as text rather than being locked inside the PDF. You can access the complete result through doc.markdown , or work with individual pages through doc.pages . Each page record includes its page number, Markdown content, and reading method. This lets you retain document context when building search or retrieval applications. Retrieve the paper's metadata from arXiv The Markdown contains the paper's text, but a useful research workflow should also preserve information about its source. For example, if an AI assistant retrieves a passage from a research paper, you may want to display its title, authors, publication date, and the PDF page containing the answer. The arxiv Python package provides this information through arXiv's API. Add the following code: python import arxiv arxiv client = arxiv.Client search = arxiv.Search id list= "1706.03762v7" paper = next arxiv client.results search The paper object contains bibliographic information, including: - paper.title : The research paper's title. - paper.authors : The authors associated with the paper. - paper.published : The original submission date. - paper.updated : The revision date. - paper.entry id : The URL identifying the paper on arXiv. - paper.get short id : The arXiv identifier, including its version. The get short id method is especially useful for keeping track of revisions. A paper may be updated after its initial submission, and its version identifier lets you associate the extracted text with the correct revision. We'll use this metadata when saving the Markdown to disk. Save the Markdown and paper metadata using Python Let's combine document conversion and metadata retrieval into one Python script. The script will: 1. Retrieve a paper's bibliographic information from arXiv. 2. Convert selected PDF pages into Markdown using Spidra. 3. Save the Markdown with the original source information. 4. Write the paper's metadata to a separate JSON file. Create a new file named arxiv to markdown.py : python arxiv to markdown.py import json import os from pathlib import Path import arxiv from spidra import Spidra ARXIV ID = "1706.03762v7" SELECTED PAGES = "1,4,8" Retrieve bibliographic metadata. paper = next arxiv.Client .results arxiv.Search id list= ARXIV ID paper id = paper.get short id pdf url = f"https://arxiv.org/pdf/{paper id}" Convert the PDF to Markdown. client = Spidra api key=os.environ "SPIDRA API KEY" doc = client.parse pdf url, pages=SELECTED PAGES, mode="ocr", include pages=True, page markers=True, Preserve source and page information. metadata = { "arxiv id": paper id, "title": paper.title, "authors": author.name for author in paper.authors , "published": paper.published.date .isoformat , "updated": paper.updated.date .isoformat , "source url": paper.entry id, "pdf url": pdf url, "pages": page.page number for page in doc.pages , "total pages": doc.total pages, } Create the output directory. output dir = Path "papers" output dir.mkdir exist ok=True base name = paper id.replace "/", " " Save Markdown with source attribution. markdown = f" {paper.title}\n\n" f" Authors: {', '.join metadata 'authors' }\n\n" f" Source: {paper.entry id}\n\n" f" PDF: {pdf url}\n\n" f"{doc.markdown}\n" output dir / f"{base name}.md" .write text markdown, encoding="utf-8", Save structured paper metadata. output dir / f"{base name}.metadata.json" .write text json.dumps metadata, ensure ascii=False, indent=2 , encoding="utf-8", Run the script: python arxiv to markdown.py It creates a papers directory containing two files: papers/ ├── 1706.03762v7.md └── 1706.03762v7.metadata.json The Markdown file contains the paper's extracted content, together with its title, authors, source URL, and PDF link. Because page markers=True is enabled, the combined Markdown also contains HTML comments identifying the source pages: php < -- page 1 -- < -- page 4 -- < -- page 8 -- These markers are inserted at the corresponding page boundaries, rather than being presented as a separate list. They remain available in the Markdown source even when a renderer hides HTML comments. The JSON file provides the same paper identity in a format your application can read directly. Some of its fields look like this: { "arxiv id": "1706.03762v7", "title": "Attention Is All You Need", "pages": 1, 4, 8 , "total pages": 15 } The complete JSON also includes the author list, publication and revision dates, and original source URLs. To process a different research paper, change ARXIV ID . You can also change SELECTED PAGES to extract another range or remove the pages argument from client.parse to convert the entire PDF. The same script will retrieve the new paper's metadata and save its results under the corresponding arXiv identifier. Prepare the Markdown for a research or RAG application Once a research paper is available as Markdown, you can use the content in document search, summarization pipelines, or retrieval-augmented generation applications. For RAG, preserving the source URL and page number alongside each passage is useful. It allows an application to identify where a retrieved answer came from rather than returning an unattributed block of text. Spidra's include pages=True option makes it straightforward to build page-aware records. For example, add this code to the end of arxiv to markdown.py : Save individual pages as JSONL records. pages path = output dir / f"{base name}.pages.jsonl" with pages path.open "w", encoding="utf-8" as file: for page in doc.pages: record = { "text": page.markdown, "arxiv id": paper id, "page": page.page number, "source url": pdf url, } file.write json.dumps record, ensure ascii=False + "\n" This creates a third file, 1706.03762v7.pages.jsonl , containing one JSON record per extracted page. Each record connects the extracted text to the paper's arXiv identifier, the original PDF, and the source page. From here, you can split longer pages into smaller chunks, generate embeddings, and store them in a vector database. Your retrieval layer can carry those metadata fields forward when presenting search results or generating citations. For scientific content, retaining the original PDF link is also useful when an application needs to display the source equation, figure, or table alongside the extracted text. If you're building a broader ingestion workflow, you can also use Spidra Search https://docs.spidra.io/search with the research source to discover papers from arXiv and other research repositories before passing their PDF links to Parse. Process more arXiv papers The example above processes one paper, but the same approach works for a collection of research papers. For each paper, you can retrieve its metadata, submit its PDF URL to Spidra Parse, and save the Markdown under its arXiv identifier. Keeping the version in the filename also makes it easier to distinguish revisions in a recurring ingestion workflow. If you already have multiple PDF links, Spidra's parse batch method can process up to 50 documents per batch. For workflows that need to process many documents, see our guide on batch processing PDFs with Python https://spidra.io/blog/batch-process-pdfs-python . You can also estimate the pages and credits required with estimate parse before submitting a document. For plain Markdown extraction, Spidra charges one credit per parsed PDF page, including pages read using OCR. For more information about PDF inputs, uploads, and reusing documents, see our guide to PDF URLs, uploads, and file IDs https://spidra.io/blog/pdf-url-vs-upload-vs-file-id . With Markdown files and source metadata stored together, you have the foundation for a repeatable research ingestion workflow that can support literature search, document analysis, and AI-powered research tools. Convert your own arXiv papers with Spidra You've now built a Python workflow that takes an arXiv paper, converts selected pages to Markdown, preserves mathematical expressions and scientific tables, and saves the result with its bibliographic metadata and page references. You can use the same approach for other research papers or extend it into a larger document-processing pipeline. Create a free Spidra account https://app.spidra.io/ , get your API key, and run arxiv to markdown.py with a paper you want to process.