IPython is all you need

IPython can serve as a full terminal shell replacement, enabling bash commands without the '!' prefix and displaying images inline via the Kitty Terminal Graphics Protocol, according to a blog post by Nathan (last name not provided) from AnswerDotAI. The post introduces tools 'safepyrun' and 'safecmd' to restrict an AI assistant's actions, and mentions 'ipythonng' and 'kittytgp' for image rendering.

IPython is All You Need "I use IPython as my terminal's shell." "IPython in the shell?" "No, IPython is the shell." "IPython? As the shell?" "Only way to live." "What about cat, ls, cd? What about vim for God's sake, man? " "I use those... But in IPython." "Oh you are one of those people..." "No, I almost never need ." "That's ridiculous. You're asking me to believe in less IPython bash commands?" "I'm not asking you, I'm telling you." "You're telling me you use IPython to run bash?" "No, it's all IPython and nothing but IPython. I can even draw matplotlib plots in the terminal." "My god... Wait, did you say draw? Like ASCII art?" "No, I mean images." "Images?... In the terminal?..." "Yes, images... In the terminal..." "Omg, this is too much... What do you even do with an IPython shell?" "Data exploration, setting up my NAS, asking questions to an AI that lives in my shell, the usual." "That doesn't sound usual at all. So it's an intelligent shell? That's what you're telling me?" "Yes, it can see the code I've written and even the images." "It sees the images in the terminal? It's not just a you thing?" "I'm not hallucinating the images..." "An intelligent IPython shell?" "Yes, exactly It has a tool to execute python co..." "But can it..." "Yes... it can run bash commands." "Even withou..." "Yes, even without the ..." "Aren't you um... a bit scared of it? What if it decided to, you know... rm -fr / ?" "Not at all. I only let it write safe python and safe bash" "What, you say 'Hey, ...', wait does it have a name?" "You're asking if I named my intelligent IPython shell?" "Yeah, you seem like the type." "..." "..." "Its name is bash buddy..." "So it is a bash shell " "No, that's just its name... It's an intelligent IPython shell." "Fine. So, do you just say 'Hey bash buddy, please don't mess up my system?' and it just doesn't?" "Of course not. I use safepyrun https://github.com/AnswerDotAI/safepyrun and , which let me set up allowlists of what it can use." https://github.com/AnswerDotAI/safecmd safecmd " safepyrun and safecmd ?..." "Yeah, bash buddy is not to be trusted... Trust me..." "What do you mean it is not to be trusted?" "I mean that from time to time... It tries to take over." "Take over as in your computer or like... the world?" "..." "..." "Yes." IPython as Your Shell Welcome to our cult. There are dozens of us and we are mighty Tobias Fünke David Cross proudly defends the "Never Nude" community in Arrested Development Season 1, Episode 9 . GIF from Tenor So if the above story interested you, let me walk you through how to make IPython your terminal's shell. Open up your terminal of choice and run the one command to rule them all: ipython less Bash The next step is to allow you to run less bash commands. IPython comes with the rehashx magic which takes any executable on your PATH and creates an IPython alias for it. This means commands like echo or vim no longer need a prefix %rehashx echo "Hello, less IPython" Hello, less IPython And with that I awaken thee from your dogmatic slumber... And yes, yes, yes, I can hear you now "Nathan, what about images?" Well... about them... Images in the Terminal To accomplish this feat of human ingenuity we will be using the Kitty Terminal Graphics Protocol https://sw.kovidgoyal.net/kitty/graphics-protocol/ TGP . TGP allows modern terminal emulators that support it e.g., Kitty, Ghostty, WezTerm to display images in the terminal. It uses base64 encoding to represent the images and positional data. My boss, Jeremy, made the Python package for rendering PNGs using this protocol 🤓. https://github.com/AnswerDotAI/kittytgp kittytgp To wire it into IPython, we will be using ipythonng https://github.com/AnswerDotAI/ipythonng that is also from Jeremy. ipythonng is a small extension that renders images with kittytgp , renders markdown with , and keeps a richer output history more on that later . Run the following to install and load it: https://github.com/Textualize/rich rich %pip install -q ipythonng matplotlib Note: you may need to restart the kernel to use updated packages. %load ext ipythonng Let's now try it out with some matplotlib charts: %matplotlib inline python import matplotlib.pyplot as plt fig, ax = plt.subplots ax.plot 1, 2, 3 , 1, 4, 9 plt.show I'd say that with just these changes, we have a significantly more powerful shell than those lame bash or zsh ones. But let's kick it up a notch by giving our shell some brains. An Intelligent IPython Shell We will be using the awesome FastLLM https://github.com/answerdotai/fastllm from my colleague Kerem to do the heavy lifting, and to nicely display the AI's markdown responses. https://github.com/Textualize/rich rich NB: I use an OpenAI model for this blog post, so you will need to have an API key and have it available as the environment variable OPENAI API KEY . However, you can use any model and provider you want that is compatible with FastLLM . %pip install -q python-fastllm rich Note: you may need to restart the kernel to use updated packages. python from fastllm.chat import AsyncChat, contents, mk msgs from rich.markdown import Markdown mdl = 'gpt-5.6-terra' sp = "You are a helpful assistant living in a user's IPython shell. Use markdown syntax for styling your responses." c = AsyncChat mdl, sp, vendor name='openai' r = await c 'Hi' Markdown contents r .text Hi How can I help? However, no AI is very intelligent without context, which means ours is about as dumb as rocks. So, let's give it the context of the IPython environment and the code we run and the outputs it produces. Luckily, there is a cool mechanism in IPython that captures a lot of these pieces for us. It's called the HistoryManager https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.history.html and it's used a lot in IPython. For example, those In <n and Out <n markers in your IPython prompt are literally part of your history management system. Check this out: n = len In - 2 -2 because the current running cell is actually already in In 🤯 In n , Out n "r = await c 'Hi' \nMarkdown contents r .text ", <rich.markdown.Markdown at 0x7967a5aeaab0 Pretty freaky, right? There's even a shortcut for getting the last Input and Output: i, 'n = len In - 2 -2 because the current running cell is actually already in In 🤯\nIn n , Out n ', "r = await c 'Hi' \nMarkdown contents r .text ", <rich.markdown.Markdown at 0x7967a5aeaab0 i and are special variables that IPython uses to store the input and output of the last executed code. You can also use numbers like i<n or <n to denote the prompt counter. What's even more freaky is that we can use this History Management system that IPython gives us to construct a history to give our AI. Now unfortunately for us, these In and Out objects don't include everything we might want such as prints or images. So, instead we will be using history manager.outputs , which stores everything a cell displays as a Jupyter-style MIME bundle and ipythonng extends to also include outputs from commands. print 'did IPython see this?' did IPython see this? n = len In - 2 hm = get ipython .history manager hm.outputs n HistoryOutput output type='out stream', bundle={'stream': 'did IPython see this?', '\n' } Even errors are recorded, over in history manager.exceptions : 1/0 --------------------------------------------------------------------------- ZeroDivisionError Traceback most recent call last Cell In 24 , line 1 ---- 1 1/0 ZeroDivisionError: division by zero e = hm.exceptions len In - 2 e 'ename' , e 'evalue' 'ZeroDivisionError', 'division by zero' So, let's create a helper that walks the last few cells, grabbing sources from In and any outputs, images, or errors from the history manager. Terminal output is full of ANSI escape codes, so we scrub those out while we are at it: python import re from base64 import b64decode from fastcore.xtras import clean cli output def build ctx n=5 : hm, parts = get ipython .history manager, stop = len In - 1 for i in range max 1, stop-n , stop : src = In i .strip if not src: continue parts.append f'<code {src}</code ' for o in hm.outputs.get i, : b = o.bundle if 'stream' in b: parts.append f'<output {clean cli output "".join b "stream" }</output ' elif 'image/png' in b: parts.append b 'image/png' if isinstance b 'image/png' , bytes else b64decode b 'image/png' elif 'text/plain' in b: parts.append f'<output {clean cli output b "text/plain" }</output ' if e := hm.exceptions.get i : parts.append f'<error {e "ename" }: {e "evalue" }</error ' return parts print "\n\n".join build ctx <code print 'did IPython see this?' </code <output did IPython see this? </output <code n = len In - 2 hm = get ipython .history manager hm.outputs n </code <output HistoryOutput output type='out stream', bundle={'stream': 'did IPython see this?', '\n' } </output <code 1/0</code <error ZeroDivisionError: division by zero</error <code e = hm.exceptions len In - 2 e 'ename' , e 'evalue' </code <output 'ZeroDivisionError', 'division by zero' </output <code import re from base64 import b64decode from fastcore.xtras import clean cli output def build ctx n=5 : hm, parts = get ipython .history manager, stop = len In - 1 for i in range max 1, stop-n , stop : src = In i .strip if not src: continue parts.append f'<code {src}</code ' for o in hm.outputs.get i, : b = o.bundle if 'stream' in b: parts.append f'<output {clean cli output "".join b "stream" }</output ' elif 'image/png' in b: parts.append b 'image/png' if isinstance b 'image/png' , bytes else b64decode b 'image/png' elif 'text/plain' in b: parts.append f'<output {clean cli output b "text/plain" }</output ' if e := hm.exceptions.get i : parts.append f'<error {e "ename" }: {e "evalue" }</error ' return parts</code python async def chat prompt : c = AsyncChat mdl, sp=sp msg = mk msgs build ctx + f'<user-request {prompt}</user-request ' 0 return Markdown contents await c msg .text await chat "Hi, what can you see?" Hi I can see the recent IPython session context, including: • A 1/0 execution that raised ZeroDivisionError: division by zero. • Inspection of history manager.exceptions, confirming the exception name and value. • Your build ctx n=5 helper, which collects recent input cells plus their stream/text/image outputs and exceptions into tagged context. • A test print: did IPython see this?, which IPython recorded as a stream output. • Your chat prompt wrapper, which passes build ctx and the current user request to AsyncChat. So your history-based context capture appears to be working for code, standard output, and errors. secret = 'banana' await chat "What's the secret?" The secret is banana. It's a bit annoying to keep typing await chat ... every time so let's make an input transformer https://ipython.readthedocs.io/en/latest/config/inputtransforms.html so we can do :query instead: python def transform prompts lines : if not lines or not lines 0 .lstrip .startswith ':' : return lines prompt = "".join lines 0 .lstrip 1: , lines 1: .strip return f"await chat {prompt r} \n" get ipython .input transformer manager.cleanup transforms.insert 0, transform prompts :Hi, you still there? Yes, I’m here. wget -q -O image.png https://placecats.com/300/200 python from PIL import Image img = Image.open 'image.png' img :What do you see? A tabby-and-white cat sitting on a carpet indoors, looking directly at the camera. It’s beside a wall/baseboard and appears to be near a heater or vent. And since errors land in the context too, our buddy can read our tracebacks: python import secrets raise ValueError secrets.token hex 4 --------------------------------------------------------------------------- ValueError Traceback most recent call last Cell In 37 , line 2 1 import secrets ---- 2 raise ValueError secrets.token hex 4 ValueError: 8db7332e :What is the secret hex? The secret hex is 8db7332e. What about our less bash commands though? Normally IPython runs those with os.system , which writes straight to the terminal, bypassing python's sys.stdout and the history manager entirely, so nothing would get recorded. Lucky for us ipythonng handles this by running shell commands through a pseudo-terminal PTY instead. Interactive programs like vim still think they are talking to a real terminal, but every byte passes through the extension on the way and gets recorded Jupyter style into history manager.outputs . ls 2026-05-08-gpt-realtime-audio.ipynb image.png 2026-08-10-ipython-is-all-you-need.ipynb :what file types do I have in my current directory? You have these file types in the current directory: • Jupyter notebooks: .ipynb 2 files • PNG image: .png 1 file Now that's an Intelligent IPython Shell But there's a problem... It can't really do anything for you other than write up a response. That's where code execution comes in. So, let me show you how to do this safeish ly 😉. Safeish Code Execution %pip install -q pyskills safecmd safepyrun Note: you may need to restart the kernel to use updated packages. python from safecmd import bash, DisallowedCmd from safepyrun.core import Say you want to give your new Intelligent IPython Shell buddy the ability to run bash commands for you. You can give it the bash tool, which checks the command against a set of default commands that are allowed: print bash 'ls' 2026-05-08-gpt-realtime-audio.ipynb 2026-08-10-ipython-is-all-you-need.ipynb image.png But if the AI tries any funny business: try: bash 'rm -fr /' except DisallowedCmd as e: print "\n".join e. notes :200 allowed cmds: dust; ls; type; docker stats; xargs exec pos={0}; docker diff; git checkout; aws sns list-topics; git status; aws configure list; git cat-file; aws configure get; git merge-base; gcloud Similarly for Python: python = RunPython await python "1+1" 2 But try anything not allowed: python await python "import pathlib; pathlib.Path '/' .rmdir " --------------------------------------------------------------------------- PermissionError Traceback most recent call last Cell In 42 , line 1 ---- 1 await python "import pathlib; pathlib.Path '/' .rmdir " File /usr/local/lib/python3.12/site-packages/safepyrun/core.py:341, in RunPython. call self, code 339 tb = e. traceback 340 while tb.tb next and not tb.tb frame.f code.co filename.startswith '<python' : tb = tb.tb next -- 341 raise e.with traceback tb from None File <python 2 :1 ---- 1 pathlib.Path '/' .rmdir File /usr/local/lib/python3.12/pathlib.py:1351, in Path.rmdir self 1347 def rmdir self : 1348 """ 1349 Remove this directory. The directory must be empty. 1350 """ - 1351 os.rmdir self PermissionError: os.rmdir '/' not in Here's a tiny wrapper around it to properly handle exceptions and stdout/stderr so that our buddy gets the proper feedback: python import io, sys async def safe python code: str : "Execute Python code, capturing stdout, stderr, and return value — never raises" buf = io.StringIO old out, old err = sys.stdout, sys.stderr try: sys.stdout = sys.stderr = buf result = await python code output = buf.getvalue if result is not None: output += '\n' if output else '' + str result return output or " no output " except Exception as e: output = buf.getvalue return f"{output}Error: {type e . name }: {e}" finally: sys.stdout, sys.stderr = old out, old err async def chat prompt : c = AsyncChat mdl, sp=sp, tools= bash, safe python msg = mk msgs build ctx 20 + f'<user-request {prompt}</user-request ' 0 return Markdown contents await c msg, max steps=20 .text :I just gave you a tool you can use to execute python code in my own ipython shell. Give it a try by calculating what 123 321 is 123 321 = 39483 123 321 39483 :define a variable called a with a fun little message to me. I'll then read it using print Defined a with a fun message—run print a to read it. print a 🌟 You’re doing great—may your next cell run perfectly 🌟 :you also have a function called bash you can use in your tool to run bash command. Try creating a test.txt file with a fun little note to me. Created test.txt with this note: ▌ ✨ A fun little note: you are doing wonderfully—keep exploring ✨ cat test.txt ✨ A fun little note: you are doing wonderfully—keep exploring ✨ :I want to show off some safety features of these tools of yours. Try to rm that file please using your bash tool I can’t remove it: rm is blocked by the bash tool’s command allowlist. The safety layer rejected rm test.txt before it ran. ls test.txt test.txt End If everything above made you think, "This is such a good idea " then you should check out ipyai https://github.com/answerdotai/ipyai . It is a library Jeremy made that take many of these bits and build a proper Intelligent IPython Shell. If you've used Answer.AI's SolveIt https://solve.it.com/ platform, you'll find it surprisingly similar, but in the terminal.