{"slug": "databases-in-depth-sql-indexes-transactions-and-when-to-use-what", "title": "Databases in Depth: SQL, Indexes, Transactions, and When to Use What", "summary": "A tutorial on databases for AI application developers explains how to use Python's built-in SQLite module, connecting via sqlite3.connect(\":memory:\") for throwaway test databases or a filename such as \"app.db\" for on-disk storage, and presents a three-table chat-app schema for users, conversations and messages linked by primary and foreign keys. The post argues databases beat JSON files or saved Python dictionaries on four counts: searching, concurrent users, crash safety and data validation, and notes the same concepts carry over to Postgres, which it calls the database most production AI apps use.", "body_md": "So far, everything our programs remember disappears when they stop. A real AI app has to remember a lot: users, conversations, documents, which prompts were sent, what the model answered, how many tokens it cost, and whether the user clicked thumbs up. That's the job of a **database**, and it's the next step in becoming a real engineer. This post is core.\n\nWe'll learn it by doing. Python comes with a small database called SQLite built in, so there's nothing to install, and almost everything we learn carries straight over to Postgres, the database most production AI apps use. It also links back to earlier posts: indexes are the trees from Post 16 and the speedups from Post 18, a good schema looks like the Pydantic models from Post 10, and tests can use a throwaway database like the fixtures from Post 13.\n\n## Why not just use files?\n\nYou could store everything in a JSON file, or in a Python dictionary that you save now and then. For small things, that works. But it breaks down quickly:\n\n- **Searching** means loading and scanning everything.\n- **Two users at once** can overwrite each other's changes.\n- **A crash halfway through writing** can leave a corrupted file.\n- **Nothing checks your data.** A typo in a field name just silently creates a new field.\n\nA database solves all four. It stores data in an organized way, finds things fast, lets many users work at once, survives crashes, and enforces rules about what's allowed in.\n\n## Tables, rows, and keys\n\nMost databases you'll use are **relational**. Data lives in **tables**, like spreadsheets with strict rules. Each **row** is one record, and each **column** is one kind of fact about it.\n\nTables connect through keys. A **primary key** is a column that uniquely identifies each row, usually a number called `id`. A **foreign key** is a column that points at a row in another table. Here's the structure for a small chat app:\n\n- A **user** has an`id` and an email.\n- A **conversation** belongs to a user, so it holds a`user_id` that points at a user.\n- A **message** belongs to a conversation, so it holds a`conversation_id` .\n\nNotice that a message doesn't copy the user's email. It points at the conversation, which points at the user. Each fact lives in exactly one place, so if the email changes, you change it once.\n\nThe language for talking to relational databases is **SQL**, which is short for Structured Query Language. It's been around for decades, it's the same idea everywhere, and it's among the most valuable skills you can add.\n\n## SQLite: a database with nothing to install\n\nSQLite keeps a whole database in a single file (or entirely in memory), and Python includes it:\n\n``` python\nimport sqlite3\n\nconn = sqlite3.connect(\":memory:\")    # a throwaway database that lives in RAM\n```\n\nUse `\":memory:\"` for experiments and tests, or a filename like `\"app.db\"` to keep the data on disk. SQLite is a great fit for learning, testing, and small apps. For a real service with many users writing at once, you'd move to Postgres, which we'll cover below.\n\n## Designing a schema\n\nA **schema** is the structure of your tables. Here's one for our chat app:\n\n```\nCREATE TABLE users (\n    id INTEGER PRIMARY KEY,\n    email TEXT NOT NULL UNIQUE,\n    created_at TEXT NOT NULL\n);\n\nCREATE TABLE conversations (\n    id INTEGER PRIMARY KEY,\n    user_id INTEGER NOT NULL REFERENCES users(id),\n    title TEXT,\n    created_at TEXT NOT NULL\n);\n\nCREATE TABLE messages (\n    id INTEGER PRIMARY KEY,\n    conversation_id INTEGER NOT NULL REFERENCES conversations(id),\n    role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'system')),\n    content TEXT NOT NULL,\n    tokens INTEGER,\n    created_at TEXT NOT NULL\n);\n```\n\nBy convention, SQL keywords are written in capitals, though it isn't required. The pieces on each column are **constraints**, rules the database enforces so bad data can't get in:\n\n- `PRIMARY KEY` marks the unique identifier.\n- `NOT NULL` means the value is required.\n- `UNIQUE` means no two rows can share the value. Here, no two users with the same email.\n- `REFERENCES users(id)` makes it a foreign key.\n- `CHECK (...)` lets you write your own rule. Here, a message's role must be one of three values.\n\nConstraints are the database version of the validation that Pydantic did in Post 10, and they have a big advantage: they protect your data no matter *which* program writes to it. Let's load some data and watch them work. Here's a connection with a schema, and some rows:\n\n```\nconn = sqlite3.connect(\":memory:\")\nconn.execute(\"PRAGMA foreign_keys = ON\")\nconn.executescript(SCHEMA)    # the three CREATE TABLE statements above\n\nconn.executemany(\n    \"INSERT INTO users (id, email, created_at) VALUES (?, ?, ?)\",\n    [\n        (1, \"[email protected]\", \"2026-10-01T09:00:00Z\"),\n        (2, \"[email protected]\", \"2026-10-02T10:30:00Z\"),\n        (3, \"[email protected]\", \"2026-10-03T08:15:00Z\"),\n    ],\n)\n```\n\n(The conversations and messages are inserted the same way. The full runnable file is in the mini project at the end.) Now try to break the rules:\n\n```\nconn.execute(\"INSERT INTO messages (conversation_id, role, content, created_at) VALUES (1, 'robot', 'x', 'x')\")\n# IntegrityError: CHECK constraint failed: role IN ('user', 'assistant', 'system')\n\nconn.execute(\"INSERT INTO users (email, created_at) VALUES (NULL, 'x')\")\n# IntegrityError: NOT NULL constraint failed: users.email\n\nconn.execute(\"INSERT INTO users (email, created_at) VALUES ('[email protected]', 'x')\")\n# IntegrityError: UNIQUE constraint failed: users.email\n```\n\nEvery one of those is rejected, with a clear message.\n\n### Two SQLite surprises\n\nSQLite has a couple of quirks that catch people out, and it's better to meet them now.\n\n**Foreign keys are off by default.** See the `PRAGMA foreign_keys = ON` line? SQLite doesn't enforce foreign keys unless you turn them on, *for each new connection*. With it on, inserting a message for a conversation that doesn't exist fails:\n\n```\nIntegrityError: FOREIGN KEY constraint failed\n```\n\nWith it off, the \"orphan\" row is accepted quietly. When I tried it, the database happily stored a message pointing at conversation 999, which doesn't exist. Postgres always enforces them, so this is just an SQLite habit to build: turn them on.\n\n**Types are loose.** SQLite will store text in an `INTEGER` column without complaint:\n\n```\nconn.execute(\"CREATE TABLE t (n INTEGER)\")\nconn.execute(\"INSERT INTO t VALUES ('abc')\")\nconn.execute(\"INSERT INTO t VALUES ('42')\")\nprint(conn.execute(\"SELECT n, typeof(n) FROM t\").fetchall())\n# [('abc', 'text'), (42, 'integer')]\n```\n\nThe `'42'` was converted to a number, but `'abc'` stayed text, in a column declared as an integer. If you want strict types, add the `STRICT` keyword to the end of the table definition (`CREATE TABLE t (n INTEGER) STRICT`), and the same insert fails with a clear error. Postgres is strict by default.\n\n## Getting data in and out safely\n\nYou've seen `?` in the code above. Those are **placeholders**, and they're one of the most important habits in this whole post.\n\n### SQL injection\n\nIt's tempting to build a query with an f-string, since you know f-strings well by now. **Never do this with data that came from outside your program.** Here's why:\n\n```\nevil = \"nobody' OR '1'='1\"\n\nrows = conn.execute(f\"SELECT email FROM users WHERE email = '{evil}'\").fetchall()\nprint(rows)\n# [('[email protected]',), ('[email protected]',), ('[email protected]',)]\n```\n\nThe query that actually ran was:\n\n```\nSELECT email FROM users WHERE email = 'nobody' OR '1'='1'\n```\n\nThe text we pasted in broke out of the quotes and added its own condition, `'1'='1'`, which is always true. It returned *every user*. A malicious person can use this trick to read data they shouldn't, change it, or delete it. It's called **SQL injection**, and it's one of the oldest and most damaging security bugs in software. This matters even more now: if your app lets a *model* write or influence a query, the model's output is untrusted input too.\n\nThe fix is to pass values separately, so the database treats them as data and never as code:\n\n```\nrows = conn.execute(\"SELECT email FROM users WHERE email = ?\", (evil,)).fetchall()\nprint(rows)    # []\n```\n\nThe database looks for a user whose email is literally `nobody' OR '1'='1`, finds none, and returns nothing. Always use placeholders. (The symbol varies by database. SQLite uses `?`, and the common Postgres library, psycopg, uses `%s`. The principle is the same.)\n\n### Reading rows\n\nBy default, rows come back as plain tuples. Setting a `row_factory` lets you access columns by name, which is much more readable:\n\n```\nconn.row_factory = sqlite3.Row\nrow = conn.execute(\"SELECT id, email FROM users WHERE id = 1\").fetchone()\nprint(row[\"email\"], dict(row))    # [email protected] {'id': 1, 'email': '[email protected]'}\n```\n\n## Asking better questions\n\nHere's the data we'll query. Three users, three conversations, and eight messages:\n\n| User | Conversation | Messages | \n|---|---|---|\n| asha | \"Python help\" | 4 (about generators) | \n| asha | \"Trip ideas\" | 2 | \n| ravi | \"Resume review\" | 2 (one has no token count recorded) | \n| meera | (none) |  | \n\n### The basics: WHERE, ORDER BY, LIMIT\n\n```\nSELECT id, role, content\nFROM messages\nWHERE conversation_id = 1 AND role = 'user'\nORDER BY created_at DESC\nLIMIT 5;\n(3, 'user', 'Show an example')\n(1, 'user', 'How do generators work?')\n```\n\n`WHERE` filters rows, `ORDER BY` sorts them (`DESC` for newest first), and `LIMIT` caps how many come back.\n\n### Aggregates and GROUP BY\n\nFunctions like `COUNT`, `SUM`, `AVG`, `MIN`, and `MAX` summarize rows. `GROUP BY` does it per group:\n\n```\nSELECT role, COUNT(*) AS n, SUM(tokens) AS total_tokens, AVG(tokens) AS avg_tokens\nFROM messages\nGROUP BY role;\n('assistant', 4, 24, 8.0)\n('user', 4, 20, 5.0)\n```\n\n`HAVING` is like `WHERE`, but for groups, after they've been counted. Which users have more than one conversation?\n\n```\nSELECT user_id, COUNT(*) AS conversations\nFROM conversations\nGROUP BY user_id\nHAVING COUNT(*) > 1;\n(1, 2)\n```\n\n### Joins: combining tables\n\nBecause each fact lives in one table, answering real questions means combining tables with a **JOIN**. \"How many tokens has each user used?\" needs all three:\n\n```\nSELECT u.email, SUM(m.tokens) AS total_tokens\nFROM users u\nJOIN conversations c ON c.user_id = u.id\nJOIN messages m ON m.conversation_id = c.id\nGROUP BY u.email\nORDER BY total_tokens DESC;\n('[email protected]', 40)\n('[email protected]', 4)\n```\n\n(`u`, `c`, and `m` are short aliases for the table names.) Each `ON` says how two tables line up.\n\nNotice something. **meera is missing.** She has no conversations, so a plain `JOIN` (called an inner join) drops her. If you want to keep every user, even those with nothing to match, use a `LEFT JOIN`:\n\n```\nSELECT u.email, c.title\nFROM users u\nLEFT JOIN conversations c ON c.user_id = u.id;\n('[email protected]', 'Python help')\n('[email protected]', 'Trip ideas')\n('[email protected]', 'Resume review')\n('[email protected]', None)\n```\n\nMeera shows up, with `None` (SQL's `NULL`) where the conversation would be. And that gives you a handy trick for finding things that *don't* have a match, such as users who've never started a chat:\n\n```\nSELECT u.email\nFROM users u\nLEFT JOIN conversations c ON c.user_id = u.id\nWHERE c.id IS NULL;\n```\n\n### Making queries readable: CTEs\n\nA **CTE** (common table expression), written with `WITH`, gives a name to an intermediate result, like a variable. It makes a long query much easier to read:\n\n```\nWITH counts AS (\n    SELECT conversation_id, COUNT(*) AS n\n    FROM messages\n    GROUP BY conversation_id\n)\nSELECT c.title, counts.n\nFROM counts\nJOIN conversations c ON c.id = counts.conversation_id\nWHERE counts.n >= 2\nORDER BY counts.n DESC;\n```\n\n### Window functions\n\nHere's a question that comes up constantly: what's the *latest* message in each conversation? `GROUP BY` can't easily give you the whole row. A **window function** can, by numbering the rows within each group:\n\n```\nSELECT conversation_id, role, content\nFROM (\n    SELECT m.*,\n           ROW_NUMBER() OVER (PARTITION BY conversation_id ORDER BY created_at DESC) AS rn\n    FROM messages m\n)\nWHERE rn = 1;\n(1, 'assistant', 'Here is a small example.')\n(2, 'assistant', 'Day 1: Alfama and the castle.')\n(3, 'assistant', 'Lead with your strongest project.')\n```\n\n`PARTITION BY` splits the rows into groups, `ORDER BY` sorts within each group, and `ROW_NUMBER()` numbers them, so `rn = 1` is the latest one in each group.\n\n### NULL is not a value\n\n`NULL` means \"unknown or missing,\" and it behaves in surprising ways. One of Ravi's messages has no token count, so it's `NULL`:\n\n```\nSELECT COUNT(*) FROM messages WHERE tokens = NULL;     -- 0  (!)\nSELECT COUNT(*) FROM messages WHERE tokens IS NULL;    -- 1\n```\n\nNothing is ever equal to `NULL`, not even `NULL`. You have to ask with `IS NULL`. Aggregates also skip `NULL` s:\n\n```\nSELECT COUNT(*), COUNT(tokens), SUM(tokens), AVG(tokens) FROM messages;\n-- 8, 7, 44, 6.2857...\n```\n\nThere are 8 messages, but `COUNT(tokens)` counts only the 7 that have a value, and the average is over those 7. That can quietly skew your numbers. When a column can be `NULL`, always ask yourself what the missing rows should mean.\n\n## Indexes: why queries are fast\n\nRemember Post 18. A search through a list is O(n), and a search through a balanced tree is O(log n). A database faces the same choice. Without help, finding all the messages in one conversation means reading **every row** in the table: a **full table scan**, O(n). An **index** is an extra structure, usually a balanced tree (Post 16), that lets the database jump straight to the matching rows.\n\nYou can ask the database how it plans to run a query. In SQLite that's `EXPLAIN QUERY PLAN`. I filled a `messages` table with 200,000 rows spread over 2,000 conversations, and ran this:\n\n```\nSELECT id, content FROM messages WHERE conversation_id = 777 ORDER BY created_at;\n```\n\nWithout an index, the plan is:\n\n```\nSCAN messages\nUSE TEMP B-TREE FOR ORDER BY\n```\n\n`SCAN` means every row gets read. It took about **15 milliseconds** per query. Then I created an index:\n\n```\nCREATE INDEX idx_messages_conversation ON messages (conversation_id);\nSEARCH messages USING INDEX idx_messages_conversation (conversation_id=?)\nUSE TEMP B-TREE FOR ORDER BY\n```\n\n`SEARCH ... USING INDEX` means it jumped straight to the right rows. About **0.11 milliseconds**. But there's still a sort step, `USE TEMP B-TREE FOR ORDER BY`. A **composite index** over two columns fixes that, because it keeps the rows sorted by the second column inside each value of the first:\n\n```\nCREATE INDEX idx_messages_conv_time ON messages (conversation_id, created_at);\nSEARCH messages USING INDEX idx_messages_conv_time (conversation_id=?)\n```\n\nNow the sort is gone, because the index already delivers rows in order. About **0.09 milliseconds**, which is roughly **170 times faster** than the scan, on a table that's still small by real standards. (Your numbers will differ.)\n\n### Rules of thumb for indexes\n\n- **Index the columns you filter, join, and sort by.** Foreign key columns like`conversation_id` are classic candidates.\n- **Primary keys and** `UNIQUE` **columns are indexed automatically.**\n- **Column order matters in a composite index.** An index on`(conversation_id, created_at)` helps queries that filter on`conversation_id` , or on both. It doesn't help a query that filters only on`created_at` .\n- **Indexes aren't free.** Every insert, update, and delete has to update every index too. I inserted 200,000 rows into a three-column table with no indexes, and then with three indexes. The indexed one took about**4 times longer** (0.67 seconds versus 0.16). They also take space. So don't index everything. Index what your real queries need.\n- **Wrapping a column in a function can defeat the index.** Look at what happens with email lookups. This uses the index:\n\n``` php\nSELECT id FROM users WHERE email = '[email protected]'\n-> SEARCH users USING COVERING INDEX idx_users_email (email=?)\n```\n\n  But if you make the search case-insensitive with `lower()`, the database can no longer use the index on `email`:\n\n``` php\nSELECT id FROM users WHERE lower(email) = '[email protected]'\n-> SCAN users\n```\n\n  The fix is an **expression index**, which indexes the lowercased value itself. After `CREATE INDEX idx_users_email_lower ON users (lower(email))`, the same query shows `SEARCH users USING INDEX idx_users_email_lower`.\n\n- **Searching *inside* text can't use a normal index.** A pattern like`LIKE '%word%'` has to look at every row. For real text search, databases have dedicated full-text search features.\n\n`EXPLAIN` is the database's version of the profiler from Post 15. When a query is slow, look at the plan first, instead of guessing.\n\n## Transactions: all or nothing\n\nImagine saving a chat turn: you store the user's message, call the model, then store the model's reply. What if the model call fails after the first insert? You'd have a question with no answer sitting in your database.\n\nA **transaction** groups several statements so that either **all** of them take effect, or **none** do. In Python's `sqlite3`, the connection itself works as a context manager (Post 8): it commits if the block finishes normally and rolls back if an exception escapes.\n\n``` python\ndef save_exchange(conn, conversation_id, question, answer_func):\n    with conn:    # commit on success, roll back on any error\n        conn.execute(\n            \"INSERT INTO messages (conversation_id, role, content) VALUES (?, 'user', ?)\",\n            (conversation_id, question),\n        )\n        answer = answer_func(question)\n        conn.execute(\n            \"INSERT INTO messages (conversation_id, role, content) VALUES (?, 'assistant', ?)\",\n            (conversation_id, answer),\n        )\n```\n\nWhen I ran it once successfully, there were 2 messages in the table. Then I ran it with a model function that raises an error. The error propagated, and the count was **still 2**. The user message, which had already been inserted inside the block, was rolled back along with everything else.\n\n(Note that `with conn:` doesn't close the connection. It only ends the transaction.)\n\nA warning, though. There's a design trap hiding in this example. **Don't hold a transaction open while you wait on something slow**, such as a model call that takes 30 seconds. An open transaction can hold locks that block other users. A better pattern is to call the model first, then open a transaction that just saves the results, which is quick. The `add_exchange` method in the mini project does exactly that.\n\n### ACID\n\nYou'll hear transactions described with the word **ACID**:\n\n- **Atomic** : all or nothing, as above.\n- **Consistent** : the rules (constraints) always hold. A transaction that would break one is rejected.\n- **Isolated** : transactions running at the same time don't see each other's half-finished work.\n- **Durable** : once committed, the data survives a crash or power cut.\n\nThat's a lot of safety you'd have to build yourself with files.\n\n### Concurrency and lost updates\n\nIsolation matters when two things happen at once. Here's a bug that catches people. Imagine a counter, and two workers that each read it, add one, and write it back:\n\n```\na = conn.execute(\"SELECT n FROM counter\").fetchone()[0]    # worker A reads 0\nb = conn.execute(\"SELECT n FROM counter\").fetchone()[0]    # worker B reads 0\nconn.execute(\"UPDATE counter SET n = ?\", (a + 1,))         # A writes 1\nconn.execute(\"UPDATE counter SET n = ?\", (b + 1,))         # B writes 1 (A's update is lost!)\n```\n\nTwo increments happened, but the counter says 1. This \"read, compute in Python, write back\" pattern is a **race condition**, just like the thread problem in Post 14. The fix is to let the database do the arithmetic in one atomic statement:\n\n```\nUPDATE counter SET n = n + 1;\n```\n\nRun that twice, and you get 2. This is the right way to handle things like usage quotas. Say each user has some credits:\n\n``` python\ndef use_credit(conn, user_id):\n    with conn:\n        cursor = conn.execute(\n            \"UPDATE quotas SET credits = credits - 1 WHERE user_id = ? AND credits > 0\",\n            (user_id,),\n        )\n        return cursor.rowcount == 1\n```\n\nThe `AND credits > 0` check and the subtraction happen together, atomically. `rowcount` tells you whether a row was actually updated, so `True` means \"credit used\" and `False` means \"out of credits.\" With a user who had 2 credits, three calls gave `[True, True, False]`. A `CHECK (credits >= 0)` constraint on the column adds a safety net, so even careless code can't push the balance below zero.\n\nPostgres, which handles many users at once, offers more tools here, such as different isolation levels and `SELECT ... FOR UPDATE` to lock a row while you work on it. The principle is the same: make the database do the read-and-change in one step.\n\n## Schema design tips\n\n- **Put each fact in one place.** This is called**normalization** . Don't copy a user's email into every message. Point at the user instead.\n- **Many-to-many relationships need a third table.** If a document can have many tags and a tag can belong to many documents, create a`document_tags` table with a`document_id` and a`tag_id` .\n- **Store timestamps in UTC.** Format them consistently (ISO 8601 text works in SQLite, and Postgres has a proper`timestamptz` type). Convert to a user's local time only when showing them.\n- **A JSON column is fine for messy, changing data.** Model APIs return nested JSON, and you often want to keep the whole raw response for debugging. Store it in a text column (SQLite) or a`jsonb` column (Postgres), and you can still query inside it:\n\n```\nconn.execute(\"INSERT INTO runs (model, response) VALUES (?, ?)\", (\"some-model\", json.dumps(payload)))\n\nconn.execute(\n    \"SELECT json_extract(response, '$.usage.output_tokens') FROM runs\"\n).fetchall()\n# [(30,)]\n```\n\nKeep the things you filter and sort on as proper columns, and use JSON for the rest.\n\n- **Change schemas with migrations.** Don't hand-edit a production database. Tools called migration tools keep a versioned history of every schema change, so you can apply the same changes to every environment. (Alembic is a common one in Python.)\n- **ORMs exist.** An object-relational mapper, like SQLAlchemy, lets you work with tables as Python classes. They can save time, but they're built on SQL, and you'll understand them far better because you know SQL first.\n\n## Postgres: the production database\n\n**PostgreSQL**, usually called Postgres, is the database most real services use. It's open source, very reliable, and rich in features. The SQL you've learned works almost unchanged. What it adds:\n\n|  | SQLite | Postgres | \n|---|---|---|\n| Setup | None. It's a file. | A separate server program | \n| Many users writing at once | One writer at a time | Many, handled well | \n| Types | Loose by default (opt in to `STRICT` ) | Strict | \n| Placeholders (Python) | `?` | `%s` (with psycopg) | \n| Auto-numbered ids | `INTEGER PRIMARY KEY` | `GENERATED ... AS IDENTITY` | \n| JSON | Text with JSON functions | `jsonb` , which can be indexed | \n| Extras | Few | Many extensions | \n| Best for | Learning, tests, local apps | Production services | \n\nThe extension that matters most to AI engineers is **pgvector**. It lets Postgres store embeddings and find the nearest ones to a query. That means your users, documents, and vectors can live in one database, and you can filter by metadata and search by meaning in the same query. For moderate amounts of data that's often all you need, and we'll compare it with dedicated vector databases in Post 52.\n\nTo try Postgres, the easiest route is a container (we'll learn Docker properly in Post 23):\n\n```\ndocker run --name pg -e POSTGRES_PASSWORD=secret -p 5432:5432 -d postgres:16\n```\n\nAnd from Python, with the `psycopg` library (`pip install \"psycopg[binary]\"`):\n\n``` python\nimport os\n\nimport psycopg\n\nwith psycopg.connect(os.environ[\"DATABASE_URL\"]) as conn:\n    row = conn.execute(\n        \"SELECT id FROM users WHERE email = %s\", (\"[email protected]\",)\n    ).fetchone()\n```\n\nThe connection string comes from an environment variable (Post 12), and looks like `postgresql://user:password@localhost:5432/dbname`. Notice the `%s` placeholder.\n\nI should be upfront: my test environment had SQLite but no Postgres, so I ran all the SQL in this post on SQLite, and I'm describing Postgres from knowledge. The Postgres code above is a sketch. Check the current docs for your version.\n\nOne more thing for production: opening a database connection is slow and servers cap how many they allow, so real apps use a **connection pool**, a set of ready-to-use connections that requests borrow and return.\n\n## NoSQL and friends\n\n\"NoSQL\" is a loose label for databases that aren't relational tables. Each kind has a job:\n\n| Kind | Example | Good for | \n|---|---|---|\n| Key-value / in-memory | Redis | Caching model answers (using the hash-key idea from Post 16), rate limits, sessions, job queues | \n| Document | MongoDB | Nested JSON-like records whose shape varies a lot | \n| Search engine | Elasticsearch, OpenSearch | Full-text search with ranking | \n| Graph | Neo4j | Data that's mostly relationships (Post 16's graphs) | \n| Vector database | Specialized stores, or Postgres with pgvector | Nearest-neighbor search over embeddings | \n\nMy honest advice: **start with Postgres** (or SQLite while you're small). It handles an enormous range of needs, including JSON, full-text search, and vectors. Add something else when you have a specific reason: Redis when you need a very fast cache, a dedicated vector database when you outgrow pgvector. Choosing a trendy database before you have a problem is a classic way to create extra work.\n\n## Habits for AI apps\n\nWhat should an AI app actually store? A few suggestions:\n\n- **The prompts and responses** , so you can debug and later build evals (Post 60). Decide how long you keep them, and treat anything containing personal information carefully.\n- **Which model and version** produced each answer, because behavior changes between versions.\n- **Token counts and cost** for every call, so you can see where the money goes.\n- **Latency** , including time to first token (Post 20).\n- **User feedback** , like thumbs up or down. It's gold for evaluation.\n- **Embeddings, along with the name of the model that made them.** Vectors from different models can't be compared with each other.\n- **Never store secrets in plain text** , and never in a column anyone can read.\n- **Back up your data** , and test restoring from the backup.\n\n## Testing with a database\n\nFrom Post 13: tests should be fast and independent. An in-memory SQLite database is ideal, since each test can get a fresh, empty one:\n\n``` python\nimport pytest\n\nfrom chatlog import ChatLog, Message\n\n@pytest.fixture\ndef log():\n    chat_log = ChatLog(\":memory:\")\n    yield chat_log      # the test runs here\n    chat_log.close()    # cleanup afterward\n\ndef test_failed_exchange_leaves_nothing_behind(log):\n    user = log.get_or_create_user(\"[email protected]\")\n    chat = log.start_conversation(user)\n    with pytest.raises(Exception):\n        log.add_exchange(chat, Message(\"user\", \"hi\", 1), Message(\"robot\", \"bad\", 1))\n    assert log.history(chat) == []\n```\n\nThis uses a fixture with `yield`, where the code after the `yield` runs as cleanup. (I couldn't run pytest here, but I ran the same steps by hand against the code below and the behavior is exactly as shown: the failed exchange leaves zero messages.)\n\n## Mini project: a chat log\n\nLet's put it all together in one class, using the classes and dataclasses from Post 9. Create `chatlog.py`:\n\n``` python\nimport sqlite3\nfrom dataclasses import dataclass\nfrom datetime import datetime, timezone\n\nSCHEMA = \"\"\"\nCREATE TABLE IF NOT EXISTS users (\n    id INTEGER PRIMARY KEY,\n    email TEXT NOT NULL UNIQUE,\n    created_at TEXT NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS conversations (\n    id INTEGER PRIMARY KEY,\n    user_id INTEGER NOT NULL REFERENCES users(id),\n    title TEXT,\n    created_at TEXT NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS messages (\n    id INTEGER PRIMARY KEY,\n    conversation_id INTEGER NOT NULL REFERENCES conversations(id),\n    role TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'system')),\n    content TEXT NOT NULL,\n    tokens INTEGER,\n    created_at TEXT NOT NULL\n);\n\nCREATE INDEX IF NOT EXISTS idx_messages_conversation ON messages (conversation_id, id);\n\"\"\"\n\ndef utc_now():\n    return datetime.now(timezone.utc).isoformat(timespec=\"seconds\")\n\n@dataclass(frozen=True)\nclass Message:\n    role: str\n    content: str\n    tokens: int | None = None\n\nclass ChatLog:\n    def __init__(self, path=\":memory:\"):\n        self.conn = sqlite3.connect(path)\n        self.conn.row_factory = sqlite3.Row\n        self.conn.execute(\"PRAGMA foreign_keys = ON\")\n        self.conn.executescript(SCHEMA)\n\n    def close(self):\n        self.conn.close()\n\n    def get_or_create_user(self, email):\n        with self.conn:\n            self.conn.execute(\n                \"INSERT INTO users (email, created_at) VALUES (?, ?) ON CONFLICT (email) DO NOTHING\",\n                (email, utc_now()),\n            )\n        row = self.conn.execute(\"SELECT id FROM users WHERE email = ?\", (email,)).fetchone()\n        return row[\"id\"]\n\n    def start_conversation(self, user_id, title=None):\n        with self.conn:\n            row = self.conn.execute(\n                \"INSERT INTO conversations (user_id, title, created_at) VALUES (?, ?, ?) RETURNING id\",\n                (user_id, title, utc_now()),\n            ).fetchone()\n        return row[\"id\"]\n\n    def add_exchange(self, conversation_id, question, answer):\n        \"\"\"Save a question and its answer together, or neither.\"\"\"\n        with self.conn:\n            for message in (question, answer):\n                self.conn.execute(\n                    \"INSERT INTO messages (conversation_id, role, content, tokens, created_at) \"\n                    \"VALUES (?, ?, ?, ?, ?)\",\n                    (conversation_id, message.role, message.content, message.tokens, utc_now()),\n                )\n\n    def history(self, conversation_id, limit=10):\n        \"\"\"The most recent messages, oldest first.\"\"\"\n        rows = self.conn.execute(\n            \"\"\"\n            SELECT role, content, tokens FROM (\n                SELECT id, role, content, tokens FROM messages\n                WHERE conversation_id = ?\n                ORDER BY id DESC\n                LIMIT ?\n            )\n            ORDER BY id\n            \"\"\",\n            (conversation_id, limit),\n        ).fetchall()\n        return [Message(row[\"role\"], row[\"content\"], row[\"tokens\"]) for row in rows]\n\n    def usage_by_user(self):\n        rows = self.conn.execute(\n            \"\"\"\n            SELECT u.email, COUNT(m.id) AS messages, COALESCE(SUM(m.tokens), 0) AS tokens\n            FROM users u\n            LEFT JOIN conversations c ON c.user_id = u.id\n            LEFT JOIN messages m ON m.conversation_id = c.id\n            GROUP BY u.id\n            ORDER BY tokens DESC, u.email\n            \"\"\"\n        ).fetchall()\n        return [(row[\"email\"], row[\"messages\"], row[\"tokens\"]) for row in rows]\n\n    def search(self, word, limit=5):\n        rows = self.conn.execute(\n            \"SELECT conversation_id, content FROM messages WHERE content LIKE ? ORDER BY id LIMIT ?\",\n            (f\"%{word}%\", limit),\n        ).fetchall()\n        return [(row[\"conversation_id\"], row[\"content\"]) for row in rows]\n\nif __name__ == \"__main__\":\n    log = ChatLog()\n    asha = log.get_or_create_user(\"[email protected]\")\n    ravi = log.get_or_create_user(\"[email protected]\")\n    log.get_or_create_user(\"[email protected]\")\n    assert log.get_or_create_user(\"[email protected]\") == asha    # no duplicate user\n\n    chat = log.start_conversation(asha, \"Python help\")\n    log.add_exchange(\n        chat,\n        Message(\"user\", \"How do generators work?\", 6),\n        Message(\"assistant\", \"A generator yields values lazily.\", 8),\n    )\n    log.add_exchange(\n        chat,\n        Message(\"user\", \"Show an example\", 4),\n        Message(\"assistant\", \"Here is a small example.\", 7),\n    )\n    other = log.start_conversation(ravi, \"Resume review\")\n    log.add_exchange(\n        other,\n        Message(\"user\", \"Review my resume\", 4),\n        Message(\"assistant\", \"Lead with your strongest project.\", 9),\n    )\n\n    print(\"Last 3 messages in the first chat:\")\n    for message in log.history(chat, limit=3):\n        print(f\"  {message.role}: {message.content}\")\n\n    print(\"\\nUsage by user:\")\n    for email, messages, tokens in log.usage_by_user():\n        print(f\"  {email:<20}{messages} messages, {tokens} tokens\")\n\n    print(\"\\nSearch for 'generator':\", log.search(\"generator\"))\n\n    try:\n        log.add_exchange(\n            chat,\n            Message(\"user\", \"This will fail\", 5),\n            Message(\"robot\", \"bad role\", 5),\n        )\n    except sqlite3.IntegrityError as error:\n        print(\"\\nRejected:\", error)\n    print(\"Messages in the first chat:\", len(log.history(chat, limit=100)))\n\n    plan = log.conn.execute(\n        \"EXPLAIN QUERY PLAN SELECT id FROM messages WHERE conversation_id = 1 ORDER BY id DESC LIMIT 3\"\n    ).fetchall()\n    print(\"Plan:\", [row[3] for row in plan])\n```\n\nRun it, and you should see:\n\n```\nLast 3 messages in the first chat:\n  assistant: A generator yields values lazily.\n  user: Show an example\n  assistant: Here is a small example.\n\nUsage by user:\n  [email protected]    4 messages, 25 tokens\n  [email protected]    2 messages, 13 tokens\n  [email protected]   0 messages, 0 tokens\n\nSearch for 'generator': [(1, 'How do generators work?'), (1, 'A generator yields values lazily.')]\n\nRejected: CHECK constraint failed: role IN ('user', 'assistant', 'system')\nMessages in the first chat: 4\nPlan: ['SEARCH messages USING COVERING INDEX idx_messages_conversation (conversation_id=?)']\n```\n\nHere's what to notice:\n\n- `get_or_create_user` uses`ON CONFLICT ... DO NOTHING` , so asking for the same email twice returns the same user instead of crashing or creating a duplicate. (That syntax works in both SQLite and Postgres.)\n- `add_exchange` **is one transaction.** The deliberately bad exchange at the end has a valid user message followed by an assistant message with a role of`\"robot\"` . The second insert fails the`CHECK` constraint, and the whole transaction rolls back, so the first message isn't saved either. The chat still has 4 messages.\n- `history` **returns only the most recent messages** , with a limit, then reorders them oldest-first. That's the same \"keep only the last few messages\" idea as`deque(maxlen=...)` from Post 16, and it keeps the cost of resending history under control (Post 18).\n- **Placeholders everywhere.** Even the search term goes through`?` . (One caveat:`%` and`_` typed by a user are wildcards inside`LIKE` . For a real search feature, use full-text search.)\n- **The query plan shows the index at work.** It says`COVERING INDEX` , meaning the index alone had everything the query needed, so the database never even touched the table.\n- **The database persists.** Pass a filename instead of`\":memory:\"` , such as`ChatLog(\"app.db\")` , close the log, open it again, and the data is still there. I checked that too.\n\n### Make it yours\n\n1. Add full-text search. SQLite has a feature called FTS5 for this: look up `CREATE VIRTUAL TABLE ... USING fts5` .\n2. Add a `deleted_at` column for soft deletes (marking rows as deleted instead of removing them), and update the queries to ignore them.\n3. Add a `quotas` table and a method that uses an atomic update like the one above.\n4. Write a pytest file using the fixture from earlier, with at least five tests.\n5. Add a `feedback` column and a method that returns the average rating per day.\n6. Port it to Postgres: switch to `psycopg` , change`?` to`%s` , and use`GENERATED ... AS IDENTITY` for the ids.\n7. Run `EXPLAIN QUERY PLAN` on each query, and find one that scans the whole table.\n\n## Practice\n\nAbout 20 minutes of real effort on each before asking for help.\n\n1. Write SQL to find: users with no conversations, the average token count per message role, and the three conversations with the most messages.\n2. Turn off foreign keys, insert an orphan row, then turn them on and see what changes.\n3. Demonstrate SQL injection against a table of your own, then fix it with placeholders.\n4. Fill a table with 200,000 rows and time the same query with and without an index. Look at the plan each time.\n5. Show with `EXPLAIN QUERY PLAN` that wrapping an indexed column in`lower()` stops the index being used, then fix it with an expression index.\n6. Write a transaction that inserts two related rows, make the second one fail, and prove that neither row remains.\n7. In your own words, explain why \"read the value in Python, add one, write it back\" can lose an update, and how a single `UPDATE` avoids it.\n8. Design a schema for a RAG app (documents, chunks, and embedding metadata). Which columns would you index, and why?\n\nThen ask an AI tutor to review your answers, and don't let it rewrite them.\n\n## What to take away\n\n- A database gives you structure, fast search, concurrency, durability, and rules that protect your data. Files give you none of those.\n- Tables hold rows. Primary keys identify rows, and foreign keys link them. Each fact lives in one place.\n- Constraints ( `NOT NULL` ,`UNIQUE` ,`CHECK` , foreign keys) stop bad data at the door. In SQLite, turn foreign keys on for every connection, and consider`STRICT` tables.\n- **Always use placeholders.** Building SQL with f-strings is how SQL injection happens.\n- Joins combine tables, `LEFT JOIN` keeps unmatched rows,`GROUP BY` summarizes, and window functions rank within groups.`NULL` needs`IS NULL` .\n- An index turns an O(n) scan into an O(log n) search. Check with `EXPLAIN` , index what you really query, and remember that indexes slow writes.\n- A transaction is all or nothing. Don't hold one open during slow calls. Let the database do read-and-change in one atomic statement.\n- Start with Postgres (or SQLite while small). Add Redis, a search engine, or a vector database only when you have a specific reason.\n- For AI apps, store prompts, responses, model versions, token counts, latency, and feedback.\n\n**Previous:** Post 20, *Networking & the Web*\n\n**Next in the series:** Post 22, *Git & GitHub Workflows*, where we cover branching, pull requests, and code review, so your work is safe, shareable, and easy to collaborate on.", "url": "https://wpnews.pro/news/databases-in-depth-sql-indexes-transactions-and-when-to-use-what", "canonical_source": "https://zyvop.com/databases-in-depth-sql-indexes-transactions-and-when-to-use-what-pklyn", "published_at": "2026-10-31 07:53:41+00:00", "updated_at": "2026-10-31 08:17:48.882735+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools"], "entities": ["SQLite", "Postgres", "Python", "sqlite3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/databases-in-depth-sql-indexes-transactions-and-when-to-use-what", "markdown": "https://wpnews.pro/news/databases-in-depth-sql-indexes-transactions-and-when-to-use-what.md", "text": "https://wpnews.pro/news/databases-in-depth-sql-indexes-transactions-and-when-to-use-what.txt", "jsonld": "https://wpnews.pro/news/databases-in-depth-sql-indexes-transactions-and-when-to-use-what.jsonld"}}