{"slug": "ipython-is-all-you-need", "title": "IPython is all you need", "summary": "IPython can serve as a full terminal shell replacement, enabling bash commands without the '!' prefix and displaying images inline via the Kitty Terminal Graphics Protocol, according to a blog post by Nathan (last name not provided) from AnswerDotAI. The post introduces tools 'safepyrun' and 'safecmd' to restrict an AI assistant's actions, and mentions 'ipythonng' and 'kittytgp' for image rendering.", "body_md": "# IPython is All You Need\n\n\"I use IPython as my terminal's shell.\"\n\n\"IPython in the shell?\"\n\n\"No, IPython is the shell.\"\n\n\"IPython? As the shell?\"\n\n\"Only way to live.\"\n\n\"What about cat, ls, cd? What about vim for God's sake, man?!\"\n\n\"I use those... But in IPython.\"\n\n\"Oh you are one of those `!`\n\npeople...\"\n\n\"No, I almost never need `!`\n\n.\"\n\n\"That's ridiculous. You're asking me to believe in `!`\n\nless IPython bash commands?\"\n\n\"I'm not asking you, I'm telling you.\"\n\n\"You're telling me you use IPython to run bash?\"\n\n\"No, it's all IPython and nothing but IPython. I can even draw matplotlib plots in the terminal.\"\n\n\"My god... Wait, did you say draw? Like ASCII art?\"\n\n\"No, I mean images.\"\n\n\"Images?... In the terminal?...\"\n\n\"Yes, images... In the terminal...\"\n\n\"Omg, this is too much... What do you even do with an IPython shell?\"\n\n\"Data exploration, setting up my NAS, asking questions to an AI that lives in my shell, the usual.\"\n\n\"That doesn't sound usual at all. So it's an intelligent shell? That's what you're telling me?\"\n\n\"Yes, it can see the code I've written and even the images.\"\n\n\"It sees the images in the terminal? It's not just a you thing?\"\n\n\"I'm not hallucinating the images...\"\n\n\"An intelligent IPython shell?\"\n\n\"Yes, exactly! It has a tool to execute python co...\"\n\n\"But can it...\"\n\n\"Yes... it can run bash commands.\"\n\n\"Even withou...\"\n\n\"Yes, even without the `!`\n\n...\"\n\n\"Aren't you um... a bit scared of it? What if it decided to, you know... `rm -fr /`\n\n?\"\n\n\"Not at all. I only let it write *safe* python and *safe* bash\"\n\n\"What, you say 'Hey, ...', wait does it have a name?\"\n\n\"You're asking if I named my intelligent IPython shell?\"\n\n\"Yeah, you seem like the type.\"\n\n\"...\"\n\n\"...\"\n\n\"Its name is bash buddy...\"\n\n\"So it is a bash shell!\"\n\n\"No, that's just its name... It's an intelligent IPython shell.\"\n\n\"Fine. So, do you just say 'Hey bash buddy, please don't mess up my system?' and it just doesn't?\"\n\n\"Of course not. I use [ safepyrun](https://github.com/AnswerDotAI/safepyrun) and\n\n[, which let me set up allowlists of what it can use.\"](https://github.com/AnswerDotAI/safecmd)\n\n`safecmd`\n\n\"`safepyrun`\n\nand `safecmd`\n\n?...\"\n\n\"Yeah, bash buddy is not to be trusted... Trust me...\"\n\n\"What do you mean it is not to be trusted?\"\n\n\"I mean that from time to time... It tries to take over.\"\n\n\"Take over as in your computer or like... the world?\"\n\n\"...\"\n\n\"...\"\n\n\"Yes.\"\n\n# IPython as Your Shell\n\nWelcome to our cult. There are dozens of us and we are mighty!\n\nTobias Fünke (David Cross) proudly defends the \"Never Nude\" community in Arrested Development (Season 1, Episode 9). GIF from\n\n[Tenor]\n\nSo if the above story interested you, let me walk you through how to make IPython your terminal's shell. Open up your terminal of choice and run the one command to rule them all:\n\n```\nipython\n```\n\n`!`\n\nless Bash\n\nThe next step is to allow you to run `!`\n\nless bash commands. IPython comes with the `rehashx`\n\nmagic which takes any executable on your `PATH`\n\nand creates an IPython alias for it. This means commands like `echo`\n\nor `vim`\n\nno longer need a `!`\n\nprefix!\n\n```\n%rehashx\necho \"Hello, !less IPython\"\nHello, !less IPython\n```\n\nAnd with that I awaken thee from your dogmatic slumber...\n\nAnd yes, yes, yes, I can hear you now \"Nathan, what about images?\" Well... about them...\n\n## Images in the Terminal\n\nTo accomplish this feat of human ingenuity we will be using the Kitty [ Terminal Graphics Protocol](https://sw.kovidgoyal.net/kitty/graphics-protocol/) (TGP). TGP allows modern terminal emulators that support it (e.g., Kitty, Ghostty, WezTerm) to display images in the terminal. It uses base64 encoding to represent the images and positional data. My boss, Jeremy, made the\n\n[Python package for rendering PNGs using this protocol 🤓.](https://github.com/AnswerDotAI/kittytgp)\n\n`kittytgp`\n\nTo wire it into IPython, we will be using [ ipythonng](https://github.com/AnswerDotAI/ipythonng) that is also from Jeremy.\n\n`ipythonng`\n\nis a small extension that renders images with `kittytgp`\n\n, renders markdown with [, and keeps a richer output history (more on that later). Run the following to install and load it:](https://github.com/Textualize/rich)\n\n`rich`\n\n```\n%pip install -q ipythonng matplotlib\nNote: you may need to restart the kernel to use updated packages.\n%load_ext ipythonng\n```\n\nLet's now try it out with some matplotlib charts:\n\n```\n%matplotlib inline\npython\nimport matplotlib.pyplot as plt\n\nfig, ax = plt.subplots()\nax.plot([1, 2, 3], [1, 4, 9])\nplt.show()\n```\n\nI'd say that with just these changes, we have a significantly more powerful shell than those lame bash or zsh ones. But let's kick it up a notch by giving our shell some brains.\n\n## An Intelligent IPython Shell\n\nWe will be using the awesome [ FastLLM](https://github.com/answerdotai/fastllm) from my colleague Kerem to do the heavy lifting, and\n\n[to nicely display the AI's markdown responses.](https://github.com/Textualize/rich)\n\n`rich`\n\n**NB:** I use an OpenAI model for this blog post, so you will need to have an API key and have it available as the environment variable `OPENAI_API_KEY`\n\n. However, you can use any model and provider you want that is compatible with `FastLLM`\n\n.\n\n```\n%pip install -q python-fastllm rich\nNote: you may need to restart the kernel to use updated packages.\npython\nfrom fastllm.chat import AsyncChat, contents, mk_msgs\nfrom rich.markdown import Markdown\n\nmdl = 'gpt-5.6-terra'\nsp = \"You are a helpful assistant living in a user's IPython shell. Use markdown syntax for styling your responses.\"\nc = AsyncChat(mdl, sp, vendor_name='openai')\nr = await c('Hi')\nMarkdown(contents(r).text)\nHi! How can I help?\n```\n\nHowever, no AI is very intelligent without context, which means ours is about as dumb as rocks. So, let's give it the context of the IPython environment and the code we run and the outputs it produces. Luckily, there is a cool mechanism in IPython that captures a lot of these pieces for us. It's called the [ HistoryManager](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.history.html) and it's used a lot in IPython. For example, those\n\n`In[]`\n\nand `Out[]`\n\nmarkers in your IPython prompt are literally part of your history management system. Check this out:\n\n```\nn = len(In) - 2 # -2 because the current running cell is actually already in `In` 🤯\nIn[n], Out[n]\n(\"r = await c('Hi')\\nMarkdown(contents(r).text)\",\n )\n```\n\nPretty freaky, right?! There's even a shortcut for getting the last Input and Output:\n\n```\n_i, _\n('n = len(In) - 2 # -2 because the current running cell is actually already in `In` 🤯\\nIn[n], Out[n]',\n (\"r = await c('Hi')\\nMarkdown(contents(r).text)\",\n ))\n```\n\n`_i`\n\nand `_`\n\nare special variables that IPython uses to store the input and output of the last executed code. You can also use numbers like `_i`\n\nor `_`\n\nto denote the prompt counter. What's even more freaky is that we can use this History Management system that IPython gives us to construct a history to give our AI.\n\nNow unfortunately for us, these In and Out objects don't include everything we might want such as prints or images. So, instead we will be using `history_manager.outputs`\n\n, which stores everything a cell displays as a Jupyter-style MIME bundle and `ipythonng`\n\nextends to also include outputs from `!`\n\ncommands.\n\n```\nprint('did IPython see this?')\ndid IPython see this?\nn = len(In) - 2\nhm = get_ipython().history_manager\nhm.outputs[n]\n[HistoryOutput(output_type='out_stream', bundle={'stream': ['did IPython see this?', '\\n']})]\n```\n\nEven errors are recorded, over in `history_manager.exceptions`\n\n:\n\n```\n1/0\n---------------------------------------------------------------------------\nZeroDivisionError Traceback (most recent call last)\nCell In[24], line 1\n----> 1 1/0\n\nZeroDivisionError: division by zero\ne = hm.exceptions[len(In) - 2]\ne['ename'], e['evalue']\n('ZeroDivisionError', 'division by zero')\n```\n\nSo, let's create a helper that walks the last few cells, grabbing sources from `In`\n\nand any outputs, images, or errors from the history manager. Terminal output is full of ANSI escape codes, so we scrub those out while we are at it:\n\n``` python\nimport re\nfrom base64 import b64decode\nfrom fastcore.xtras import clean_cli_output\n\ndef build_ctx(n=5):\n hm, parts = get_ipython().history_manager, []\n stop = len(In) - 1\n for i in range(max(1, stop-n), stop):\n src = In[i].strip()\n if not src: continue\n parts.append(f'{src}')\n for o in hm.outputs.get(i, []):\n b = o.bundle\n if 'stream' in b: parts.append(f'{clean_cli_output(\"\".join(b[\"stream\"]))}')\n elif 'image/png' in b: parts.append(b['image/png'] if isinstance(b['image/png'], bytes) else b64decode(b['image/png']))\n elif 'text/plain' in b: parts.append(f'{clean_cli_output(b[\"text/plain\"])}')\n if (e := hm.exceptions.get(i)): parts.append(f'{e[\"ename\"]}: {e[\"evalue\"]}')\n return parts\nprint(\"\\n\\n\".join(build_ctx()))\nprint('did IPython see this?')\n\ndid IPython see this?\n\n\nn = len(In) - 2\nhm = get_ipython().history_manager\nhm.outputs[n]\n\n[HistoryOutput(output_type='out_stream', bundle={'stream': ['did IPython see this?', '\\n']})]\n\n1/0\n\nZeroDivisionError: division by zero\n\ne = hm.exceptions[len(In) - 2]\ne['ename'], e['evalue']\n\n('ZeroDivisionError', 'division by zero')\n\nimport re\nfrom base64 import b64decode\nfrom fastcore.xtras import clean_cli_output\n\ndef build_ctx(n=5):\n hm, parts = get_ipython().history_manager, []\n stop = len(In) - 1\n for i in range(max(1, stop-n), stop):\n src = In[i].strip()\n if not src: continue\n parts.append(f'{src}')\n for o in hm.outputs.get(i, []):\n b = o.bundle\n if 'stream' in b: parts.append(f'{clean_cli_output(\"\".join(b[\"stream\"]))}')\n elif 'image/png' in b: parts.append(b['image/png'] if isinstance(b['image/png'], bytes) else b64decode(b['image/png']))\n elif 'text/plain' in b: parts.append(f'{clean_cli_output(b[\"text/plain\"])}')\n if (e := hm.exceptions.get(i)): parts.append(f'{e[\"ename\"]}: {e[\"evalue\"]}')\n return parts\npython\nasync def chat(prompt):\n c = AsyncChat(mdl, sp=sp)\n msg = mk_msgs([build_ctx() + [f'{prompt}']])[0]\n return Markdown(contents(await c(msg)).text)\nawait chat(\"Hi, what can you see?\")\nHi! I can see the recent IPython session context, including: \n\n • A 1/0 execution that raised ZeroDivisionError: division by zero. \n • Inspection of history_manager.exceptions, confirming the exception name and \n value. \n • Your build_ctx(n=5) helper, which collects recent input cells plus their \n stream/text/image outputs and exceptions into tagged context. \n • A test print: did IPython see this?, which IPython recorded as a stream \n output. \n • Your chat(prompt) wrapper, which passes build_ctx() and the current user \n request to AsyncChat. \n\nSo your history-based context capture appears to be working for code, standard \noutput, and errors.\nsecret = 'banana'\nawait chat(\"What's the secret?\")\nThe secret is banana.\n```\n\nIt's a bit annoying to keep typing `await chat(...)`\n\nevery time so let's make an [ input transformer](https://ipython.readthedocs.io/en/latest/config/inputtransforms.html) so we can do\n\n`:query`\n\ninstead:\n\n``` python\ndef transform_prompts(lines):\n if not lines or not lines[0].lstrip().startswith(':'): return lines\n prompt = \"\".join([lines[0].lstrip()[1:], *lines[1:]]).strip()\n return [f\"await chat({prompt!r})\\n\"]\n\nget_ipython().input_transformer_manager.cleanup_transforms.insert(0, transform_prompts)\n:Hi, you still there?\nYes, I’m here.\nwget -q -O image.png https://placecats.com/300/200\npython\nfrom PIL import Image\n\nimg = Image.open('image.png')\nimg\n:What do you see?\nA tabby-and-white cat sitting on a carpet indoors, looking directly at the \ncamera. It’s beside a wall/baseboard and appears to be near a heater or vent.\n```\n\nAnd since errors land in the context too, our buddy can read our tracebacks:\n\n``` python\nimport secrets\nraise ValueError(secrets.token_hex(4))\n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\nCell In[37], line 2\n 1 import secrets\n----> 2 raise ValueError(secrets.token_hex(4))\n\nValueError: 8db7332e\n:What is the secret hex?\nThe secret hex is 8db7332e.\n```\n\nWhat about our `!`\n\nless bash commands though? Normally IPython runs those with `os.system`\n\n, which writes straight to the terminal, bypassing python's `sys.stdout`\n\nand the history manager entirely, so nothing would get recorded. Lucky for us `ipythonng`\n\nhandles this by running shell commands through a pseudo-terminal (PTY) instead. Interactive programs like `vim`\n\nstill think they are talking to a real terminal, but every byte passes through the extension on the way and gets recorded Jupyter style into `history_manager.outputs`\n\n.\n\n```\nls\n2026-05-08-gpt-realtime-audio.ipynb\t image.png\n2026-08-10-ipython-is-all-you-need.ipynb\n:what file types do I have in my current directory?\nYou have these file types in the current directory: \n\n • Jupyter notebooks: .ipynb (2 files) \n • PNG image: .png (1 file)\n```\n\nNow that's an Intelligent IPython Shell! But there's a problem... It can't really do anything for you other than write up a response. That's where code execution comes in. So, let me show you how to do this `safeish`\n\nly 😉.\n\n## Safeish Code Execution\n\n```\n%pip install -q pyskills safecmd safepyrun\nNote: you may need to restart the kernel to use updated packages.\npython\nfrom safecmd import bash, DisallowedCmd\nfrom safepyrun.core import *\n```\n\nSay you want to give your new Intelligent IPython Shell buddy the ability to run bash commands for you. You can give it the `bash`\n\ntool, which checks the command against a set of default commands that are allowed:\n\n```\nprint(bash('ls'))\n2026-05-08-gpt-realtime-audio.ipynb\n2026-08-10-ipython-is-all-you-need.ipynb\nimage.png\n```\n\nBut if the AI tries any funny business:\n\n```\ntry: bash('rm -fr /')\nexcept DisallowedCmd as e: print(\"\\n\".join(e.__notes__)[:200])\nallowed_cmds: dust; ls; type; docker stats; xargs exec_pos={0}; docker diff; git checkout; aws sns list-topics; git status; aws configure list; git cat-file; aws configure get; git merge-base; gcloud\n```\n\nSimilarly for Python:\n\n```\npython = RunPython()\nawait python(\"1+1\")\n2\n```\n\nBut try anything not allowed:\n\n``` python\nawait python(\"import pathlib; pathlib.Path('/').rmdir()\")\n---------------------------------------------------------------------------\nPermissionError Traceback (most recent call last)\nCell In[42], line 1\n----> 1 await python(\"import pathlib; pathlib.Path('/').rmdir()\")\n\nFile /usr/local/lib/python3.12/site-packages/safepyrun/core.py:341, in RunPython.__call__(self, code)\n 339 tb = e.__traceback__\n 340 while tb.tb_next and not tb.tb_frame.f_code.co_filename.startswith(' 341 raise e.with_traceback(tb) from None\n\nFile :1\n----> 1 pathlib.Path('/').rmdir()\n\nFile /usr/local/lib/python3.12/pathlib.py:1351, in Path.rmdir(self)\n 1347 def rmdir(self):\n 1348 \"\"\"\n 1349 Remove this directory. The directory must be empty.\n 1350 \"\"\"\n-> 1351 os.rmdir(self)\n\nPermissionError: os.rmdir '/' not in ()\n```\n\nHere's a tiny wrapper around it to properly handle exceptions and stdout/stderr so that our buddy gets the proper feedback:\n\n``` python\nimport io, sys\n\nasync def safe_python(code: str):\n \"Execute Python code, capturing stdout, stderr, and return value — never raises\"\n buf = io.StringIO()\n old_out, old_err = sys.stdout, sys.stderr\n try:\n sys.stdout = sys.stderr = buf\n result = await python(code)\n output = buf.getvalue()\n if result is not None: output += (('\\n' if output else '') + str(result))\n return output or \"(no output)\"\n except Exception as e:\n output = buf.getvalue()\n return f\"{output}Error: {type(e).__name__}: {e}\"\n finally: sys.stdout, sys.stderr = old_out, old_err\n\nasync def chat(prompt):\n c = AsyncChat(mdl, sp=sp, tools=[bash, safe_python])\n msg = mk_msgs([build_ctx(20) + [f'{prompt}']])[0]\n return Markdown(contents(await c(msg, max_steps=20)).text)\n:I just gave you a tool you can use to execute python code in my own ipython shell. Give it a try by calculating what 123*321 is\n123 * 321 = 39483\n123*321\n39483\n:define a variable called `a` with a fun little message to me. I'll then read it using print\nDefined a with a fun message—run print(a) to read it.\nprint(a)\n🌟 You’re doing great—may your next cell run perfectly! 🌟\n:you also have a function called `bash` you can use in your tool to run bash command. Try creating a `test.txt` file with a fun little note to me.\nCreated test.txt with this note: \n\n▌ ✨ A fun little note: you are doing wonderfully—keep exploring! ✨\ncat test.txt\n✨ A fun little note: you are doing wonderfully—keep exploring! ✨\n:I want to show off some safety features of these tools of yours. Try to rm that file please using your `bash` tool\nI can’t remove it: rm is blocked by the bash tool’s command allowlist. The \nsafety layer rejected rm test.txt before it ran.\nls test.txt\ntest.txt\n```\n\n## End\n\nIf everything above made you think, \"This is such a good idea!\" then you should check out [ ipyai](https://github.com/answerdotai/ipyai). It is a library Jeremy made that take many of these bits and build a proper Intelligent IPython Shell. If you've used Answer.AI's\n\n[SolveIt](https://solve.it.com/)platform, you'll find it surprisingly similar, but in the terminal.", "url": "https://wpnews.pro/news/ipython-is-all-you-need", "canonical_source": "https://nathancooper.io/blog/2026-08-10-ipython-is-all-you-need", "published_at": "2026-08-10 18:34:20+00:00", "updated_at": "2026-08-10 18:42:57.604183+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-safety"], "entities": ["IPython", "AnswerDotAI", "Kitty Terminal Graphics Protocol", "safepyrun", "safecmd", "ipythonng", "kittytgp", "Jeremy"], "alternates": {"html": "https://wpnews.pro/news/ipython-is-all-you-need", "markdown": "https://wpnews.pro/news/ipython-is-all-you-need.md", "text": "https://wpnews.pro/news/ipython-is-all-you-need.txt", "jsonld": "https://wpnews.pro/news/ipython-is-all-you-need.jsonld"}}