{"slug": "loratop", "title": "loratop", "summary": "A developer released loratop, a small terminal UI monitor for tracking LoRA training runs in Ostris AI Toolkit. The tool reads a run's loss_log.db or parses loss, step and learning-rate values from tqdm training output, rendering loss and learning-rate graphs that adapt to narrow terminals and fall back to a sparkline when the optional plotext dependency is unavailable. It is Linux-only, locating the running trainer via /proc, and can be pointed at a run folder, a training log, or a trainer PID.", "body_md": "| | #!/usr/bin/env -S uv run --script | \n| | # SPDX-License-Identifier: MIT | \n| | # /// script | \n| | # dependencies = [\"rich\", \"plotext<6\"] # plotext 6 dropped clf/plotsize/build | \n| | # /// | \n| | \"\"\" | \n| | loratop - a tiny TUI monitor for Ostris AI Toolkit LoRA training. | \n| | It reads a run's loss_log.db (ai-toolkit writes it when logging.use_ui_logger | \n| | is true) or, failing that, parses loss/step/lr from the tqdm output in a | \n| | training log. It renders loss and lr graphs under two lines of progress. | \n| | Designed to be readable on tiny screens, so it adapts to narrow terminals | \n| | and degrades gracefully when optional deps are missing. | \n| | Linux only: finding the trainer reads /proc. | \n| | Usage | \n| | ----- | \n| | loratop # the run in this folder, else the one running trainer | \n| | loratop output/my_lora # a run folder: reads its loss_log.db, live or finished | \n| | loratop train.log # a training log (python run.py cfg > train.log 2>&1) | \n| | loratop 12345 # a running trainer's PID: its loss_log.db, else its log | \n| | Run it with uv (uv run loratop.py), or chmod +x it and put it on your PATH | \n| | as loratop. | \n| | Deps (plotext optional, falls back to a sparkline): | \n| | rich, plotext. Declared inline above, so uv installs them on first run. | \n| | \"\"\" | \n| | import argparse | \n| | import os | \n| | import re | \n| | import select | \n| | import shutil | \n| | import signal | \n| | import sqlite3 | \n| | import sys | \n| | import termios | \n| | import time | \n| | import tty | \n| | from collections import deque | \n| | # ---- optional dependencies ------------------------------------------------- | \n| | try: | \n| | import plotext as plt | \n| | if not hasattr(plt, \"clf\"): # plotext 6+: fall back to the sparkline | \n| | plt = None | \n| | except ImportError: | \n| | plt = None | \n| | try: | \n| | from rich.console import Console, Group | \n| | from rich.live import Live | \n| | from rich.panel import Panel | \n| | from rich.table import Table | \n| | from rich.text import Text | \n| | except ImportError: | \n| | sys.exit(\"loratop needs 'rich': run it with uv, or pip install rich\") | \n| | # ---- parsing patterns (tweak these if your log format differs) ------------- | \n| | # matches \"loss: 0.0823\", \"loss=1.2e-01\", \"avg_loss 0.05\", etc. | \n| | RE_LOSS = re.compile(r\"loss[=:\\s]+([0-9]*\\.?[0-9]+(?:[eE][-+]?[0-9]+)?)\", re.I) | \n| | # matches the tqdm \" 500/10000\" current/total fragment | \n| | RE_STEP = re.compile(r\"(\\d+)\\s*/\\s*(\\d+)\") | \n| | # matches \"lr: 1.0e-04\" | \n| | RE_LR = re.compile(r\"lr[=:\\s]+([0-9]*\\.?[0-9]+(?:[eE][-+]?[0-9]+)?)\", re.I) | \n| | # tqdm's own \"[elapsed log 2>&1): the trainer holds the file | \n| | for pid in writers: | \n| | if _comm(pid) != \"tee\": | \n| | return pid | \n| | # piped through tee: follow tee's stdin pipe back to its feeder | \n| | for tee_pid in writers: | \n| | try: | \n| | pipe = os.readlink(f\"/proc/{tee_pid}/fd/0\") | \n| | except OSError: | \n| | continue | \n| | if not pipe.startswith(\"pipe:\"): | \n| | continue | \n| | for pid in _pids(): | \n| | if pid == tee_pid: | \n| | continue | \n| | for fd in (1, 2): | \n| | try: | \n| | if os.readlink(f\"/proc/{pid}/fd/{fd}\") == pipe: | \n| | return pid | \n| | except OSError: | \n| | continue | \n| | return None | \n| | def db_of_pid(pid): | \n| | \"\"\"The loss_log.db `pid` has open, if any.\"\"\" | \n| | for _, link in _writes_to(pid, None): | \n| | if os.path.basename(link) == DB_NAME and os.path.isfile(link): | \n| | return link | \n| | return None | \n| | def _argv(pid): | \n| | try: | \n| | with open(f\"/proc/{pid}/cmdline\", \"rb\") as fh: | \n| | return fh.read().decode(errors=\"replace\").split(\"\\0\") | \n| | except OSError: | \n| | return [] | \n| | def find_trainers(): | \n| | \"\"\"{path: pid} for running trainers: processes holding a loss_log.db open, | \n| | or ai-toolkit's run.py with its output going to a file.\"\"\" | \n| | found = {} | \n| | for pid in _pids(): | \n| | db = db_of_pid(pid) | \n| | if db: | \n| | found.setdefault(db, pid) | \n| | elif any(os.path.basename(a) == \"run.py\" for a in _argv(pid)): | \n| | log = discover_log_from_pid(pid) | \n| | if log: | \n| | found.setdefault(log, pid) | \n| | # a trainer with a database also writes a log; show the database only | \n| | pids_with_db = {pid for path, pid in found.items() if os.path.basename(path) == DB_NAME} | \n| | return {path: pid for path, pid in found.items() | \n| | if os.path.basename(path) == DB_NAME or pid not in pids_with_db} | \n| | def resolve_source(target): | \n| | \"\"\"Turn the command-line target (or nothing) into (path, pid or None). | \n| | A folder means its loss_log.db, a file a database or log, a number a PID.\"\"\" | \n| | if target is None: | \n| | db = os.path.join(os.getcwd(), DB_NAME) | \n| | if os.path.isfile(db): | \n| | return db, None | \n| | found = find_trainers() | \n| | if len(found) == 1: | \n| | (path, pid), = found.items() | \n| | print(f\"found a running trainer: pid {pid}, {path}\", file=sys.stderr) | \n| | return path, pid | \n| | if found: | \n| | sys.exit(\"more than one trainer is running. pick one:\\n\" + \"\\n\".join( | \n| | f\" loratop {pid} # {path}\" for path, pid in sorted(found.items()))) | \n| | sys.exit(\"nothing to watch: no loss_log.db here and no running trainer.\\n\" | \n| | \"pass a run folder, a training log, or a PID\") | \n| | if os.path.isdir(target): | \n| | db = os.path.join(target, DB_NAME) | \n| | if os.path.isfile(db): | \n| | return db, None | \n| | sys.exit(f\"{target} has no {DB_NAME}. ai-toolkit writes one when \" | \n| | \"logging.use_ui_logger is true; otherwise pass the training log\") | \n| | if os.path.isfile(target): | \n| | return target, None | \n| | if target.isdigit(): | \n| | pid = int(target) | \n| | path = db_of_pid(pid) or discover_log_from_pid(pid) | \n| | if path: | \n| | print(f\"pid {pid}: {path}\", file=sys.stderr) | \n| | return path, pid | \n| | sys.exit(f\"pid {pid} has no {DB_NAME} open and its output isn't going \" | \n| | \"to a file. pass the run folder or log instead\") | \n| | sys.exit(f\"not a folder, file or PID: {target}\") | \n| | def fmt_secs(s): | \n| | if s is None or s != s or s < 0: | \n| | return \"--:--\" | \n| | s = int(s) | \n| | h, rem = divmod(s, 3600) | \n| | m, sec = divmod(rem, 60) | \n| | return f\"{h}:{m:02d}:{sec:02d}\" if h else f\"{m:02d}:{sec:02d}\" | \n| | def term_size(): | \n| | \"\"\"Current terminal size, straight from the tty ioctl. Ignores the | \n| | COLUMNS/LINES env vars, which multiplexers can leave stale.\"\"\" | \n| | for fh in (sys.__stdout__, sys.__stderr__): | \n| | try: | \n| | return os.get_terminal_size(fh.fileno()) | \n| | except (OSError, ValueError, AttributeError): | \n| | continue | \n| | return shutil.get_terminal_size((80, 24)) | \n| | def sparkline(vals, width): | \n| | if not vals: | \n| | return \"\" | \n| | vals = list(vals)[-width:] | \n| | lo, hi = min(vals), max(vals) | \n| | rng = (hi - lo) or 1.0 | \n| | return \"\".join(SPARK[min(7, int((v - lo) / rng * 7))] for v in vals) | \n| | # --------------------------------------------------------------------------- | \n| | class Tailer: | \n| | \"\"\"Incrementally reads a growing file, splitting on \\\\r and \\\\n so that | \n| | tqdm's in-place progress bar updates become a stream of parseable lines.\"\"\" | \n| | def __init__(self, path, backfill=8 * 1024 * 1024): | \n| | self.path = path | \n| | self.fh = open(path, \"r\", errors=\"replace\") | \n| | self.inode = os.fstat(self.fh.fileno()).st_ino | \n| | # seed history from the tail of the existing file | \n| | try: | \n| | size = os.path.getsize(path) | \n| | self.fh.seek(max(0, size - backfill)) | \n| | if size > backfill: | \n| | self.fh.readline() # discard partial line | \n| | except OSError: | \n| | pass | \n| | self.buf = \"\" | \n| | def lines(self): | \n| | # handle log rotation / truncation | \n| | try: | \n| | if os.stat(self.path).st_ino != self.inode or \\ | \n| | os.path.getsize(self.path) < self.fh.tell(): | \n| | self.fh.close() | \n| | self.fh = open(self.path, \"r\", errors=\"replace\") | \n| | self.inode = os.fstat(self.fh.fileno()).st_ino | \n| | self.buf = \"\" | \n| | except OSError: | \n| | pass | \n| | chunk = self.fh.read() | \n| | if not chunk: | \n| | return | \n| | self.buf += chunk | \n| | parts = re.split(r\"[\\r\\n]\", self.buf) | \n| | self.buf = parts.pop() # keep trailing partial fragment | \n| | for line in parts: | \n| | if line.strip(): | \n| | yield line | \n| | # --------------------------------------------------------------------------- | \n| | class AggSeries: | \n| | \"\"\"Fixed-capacity series covering the whole run: (step, value) points are | \n| | averaged into buckets of `stride` samples, and when the buffer fills, | \n| | adjacent buckets merge and the stride doubles. Memory stays bounded while | \n| | the x-axis always spans everything seen so far, in training steps.\"\"\" | \n| | def __init__(self, cap): | \n| | self.cap = max(4, cap - cap % 2) # even, so compaction pairs cleanly | \n| | self.stride = 1 | \n| | self.pts = [] | \n| | self._sx = self._sv = 0.0 | \n| | self._n = 0 | \n| | def append(self, x, v): | \n| | self._sx += x | \n| | self._sv += v | \n| | self._n += 1 | \n| | if self._n >= self.stride: | \n| | self.pts.append((self._sx / self._n, self._sv / self._n)) | \n| | self._sx = self._sv = 0.0 | \n| | self._n = 0 | \n| | if len(self.pts) >= self.cap: | \n| | self.pts = [((a[0] + b[0]) / 2, (a[1] + b[1]) / 2) | \n| | for a, b in zip(self.pts[::2], self.pts[1::2])] | \n| | self.stride *= 2 | \n| | def view(self): | \n| | \"\"\"(xs, values), including the in-progress bucket so the tail stays live.\"\"\" | \n| | out = list(self.pts) | \n| | if self._n: | \n| | out.append((self._sx / self._n, self._sv / self._n)) | \n| | return [p[0] for p in out], [p[1] for p in out] | \n| | class State: | \n| | def __init__(self, history, smooth, raw_smooth=0.8): | \n| | self._args = (history, smooth, raw_smooth) | \n| | self.raw = AggSeries(history) | \n| | self.ema = AggSeries(history) | \n| | self.lr_series = AggSeries(history) | \n| | self.smooth = smooth | \n| | # Ostris-style smoothing on the \"raw\" trace: keep raw_smooth of the | \n| | # previous value, so spikes don't blow out the y-axis | \n| | self.raw_alpha = 1.0 - raw_smooth | \n| | self._raw_ema = None | \n| | self._ema = None | \n| | self.step = self.total = None | \n| | self.lr = None | \n| | self.loss = None | \n| | self.min_loss = None | \n| | self.rate_pts = deque(maxlen=60) # (t, step), distinct steps only | \n| | self.eta_str = None # tqdm's own remaining-time, straight from the log | \n| | self.elapsed_str = None # tqdm's own elapsed, so it covers the whole run | \n| | self._loss_text = None | \n| | self.rate_str = None # tqdm's own it/s | \n| | self._t0 = None # first wall time, for database sources | \n| | self.finished = False # database source with no trainer writing it | \n| | def reset(self): | \n| | self.__init__(*self._args) | \n| | def _add_point(self, x, v, lr): | \n| | self.loss = v | \n| | self._raw_ema = v if self._raw_ema is None else \\ | \n| | self.raw_alpha * v + (1 - self.raw_alpha) * self._raw_ema | \n| | self.raw.append(x, self._raw_ema) | \n| | self._ema = v if self._ema is None else \\ | \n| | self.smooth * v + (1 - self.smooth) * self._ema | \n| | self.ema.append(x, self._ema) | \n| | self.min_loss = v if self.min_loss is None else min(self.min_loss, v) | \n| | if lr and lr > 0: | \n| | self.lr_series.append(x, lr) | \n| | def record(self, step, wall_time, loss, lr): | \n| | \"\"\"One step from loss_log.db.\"\"\" | \n| | self.step = step | \n| | if lr is not None: | \n| | self.lr = f\"{lr:.1e}\" | \n| | if loss is not None and loss == loss: # not NaN | \n| | self._add_point(step, loss, lr) | \n| | if wall_time is not None: | \n| | if self._t0 is None: | \n| | self._t0 = wall_time | \n| | self.rate_pts.append((wall_time, step)) | \n| | self.elapsed_str = fmt_secs(wall_time - self._t0) | \n| | def feed(self, line): | \n| | ms = RE_STEP.search(line) | \n| | if ms: | \n| | cur, tot = int(ms.group(1)), int(ms.group(2)) | \n| | # the training bar is the one carrying a loss; until one shows up, | \n| | # take any long bar. Sampling and caching bars are ignored after. | \n| | if tot >= 1 and (tot == self.total or RE_LOSS.search(line) | \n| | or (self.total is None and tot > 50)): | \n| | if self.step is not None and cur < self.step: | \n| | self.rate_pts.clear() # step went backwards: new run | \n| | # tqdm redraws the same step several times (set_postfix etc); | \n| | # only distinct steps carry rate information | \n| | if cur != self.step: | \n| | self.rate_pts.append((time.time(), cur)) | \n| | self.step, self.total = cur, tot | \n| | else: | \n| | ms = None # another bar (sampling, caching): ignore its timing | \n| | if ms: | \n| | # tqdm computes its own smoothed remaining/rate; prefer that | \n| | mel = RE_TQDM_ELAPSED.search(line) | \n| | if mel: | \n| | self.elapsed_str = mel.group(1) | \n| | me = RE_TQDM_ETA.search(line) | \n| | if me: | \n| | self.eta_str = me.group(1) | \n| | mr = RE_TQDM_RATE.search(line) | \n| | if mr: | \n| | try: | \n| | v, unit = float(mr.group(1)), mr.group(2) | \n| | self.rate_str = f\"{1 / v:.2f}\" if unit == \"s/it\" and v > 0 \\ | \n| | else f\"{v:.2f}\" | \n| | except ValueError: | \n| | pass | \n| | ml = RE_LR.search(line) | \n| | if ml: | \n| | self.lr = ml.group(1) | \n| | m = RE_LOSS.search(line) | \n| | # tqdm prints each loss twice: once when it's posted, and again on the | \n| | # next redraw when the step counter ticks. Count it once. | \n| | if m and m.group(1) != self._loss_text: | \n| | self._loss_text = m.group(1) | \n| | try: | \n| | v = float(m.group(1)) | \n| | except ValueError: | \n| | v = None | \n| | if v is not None and v == v: # not NaN | \n| | try: | \n| | lr = float(self.lr) if self.lr else None | \n| | except ValueError: | \n| | lr = None | \n| | self._add_point(self.step or 0, v, lr) | \n| | def rate(self): | \n| | if len(self.rate_pts) < 2: | \n| | return None | \n| | (t0, s0), (t1, s1) = self.rate_pts[0], self.rate_pts[-1] | \n| | dt = t1 - t0 | \n| | return (s1 - s0) / dt if dt >= 1.0 else None | \n| | def eta(self): | \n| | r = self.rate() | \n| | if r and self.total and self.step is not None and r > 0: | \n| | return (self.total - self.step) / r | \n| | return None | \n| | class LogSource: | \n| | def __init__(self, path): | \n| | self.tail = Tailer(path) | \n| | def poll(self, state): | \n| | for line in self.tail.lines(): | \n| | state.feed(line) | \n| | class DbSource: | \n| | \"\"\"Reads new steps from ai-toolkit's loss_log.db, read-only.\"\"\" | \n| | def __init__(self, path): | \n| | self.path = path | \n| | self.con = None | \n| | self.last = 0 | \n| | self.total = None | \n| | try: | \n| | with open(os.path.join(os.path.dirname(path), \"config.yaml\")) as fh: | \n| | m = RE_CONFIG_STEPS.search(fh.read()) | \n| | self.total = int(m.group(1)) if m else None | \n| | except OSError: | \n| | pass | \n| | def poll(self, state): | \n| | try: | \n| | if self.con is None: | \n| | self.con = sqlite3.connect(f\"file:{self.path}?mode=ro\", uri=True, timeout=1) | \n| | newest = self.con.execute(\"SELECT MAX(step) FROM steps\").fetchone()[0] or 0 | \n| | if newest < self.last: | \n| | # a resumed run deletes the steps after its checkpoint | \n| | state.reset() | \n| | self.last = 0 | \n| | rows = self.con.execute(\"\"\" | \n| | SELECT s.step, s.wall_time, | \n| | MAX(CASE WHEN m.key = 'loss/loss' THEN m.value_real END), | \n| | MAX(CASE WHEN m.key = 'learning_rate' THEN m.value_real END) | \n| | FROM steps s LEFT JOIN metrics m ON m.step = s.step | \n| | WHERE s.step > ? GROUP BY s.step ORDER BY s.step\"\"\", (self.last,)).fetchall() | \n| | except sqlite3.Error: | \n| | self.con = None # busy or mid-write; try again next tick | \n| | return | \n| | state.total = self.total | \n| | for step, wall_time, loss, lr in rows: | \n| | state.record(step, wall_time, loss, lr) | \n| | self.last = step | \n| | # --------------------------------------------------------------------------- | \n| | def _plot_start(w, h): | \n| | plt.clf() | \n| | # plotext clamps plotsize to its own terminal detection, which can | \n| | # lag reality (and reads 80x24 off-tty); we size it ourselves | \n| | for fn in (\"limit_size\", \"limitsize\"): | \n| | if hasattr(plt, fn): | \n| | getattr(plt, fn)(False, False) | \n| | break | \n| | plt.plotsize(max(20, w), max(7, h)) | \n| | plt.theme(\"dark\") | \n| | def build_loss_graph(state, w, h): | \n| | raw_x, raw = state.raw.view() | \n| | ema_x, ema = state.ema.view() | \n| | if plt and len(ema) >= 2: | \n| | _plot_start(w, h) | \n| | if len(raw) >= 2: | \n| | plt.plot(raw_x, raw, color=\"red\", label=\"raw\", marker=\"braille\") | \n| | plt.plot(ema_x, ema, color=\"cyan\", label=\"ema\", marker=\"braille\") | \n| | plt.xlabel(\"step\") | \n| | stride = state.raw.stride | \n| | plt.title(f\"loss (avg of {stride} steps)\" if stride > 1 else \"loss\") | \n| | return Text.from_ansi(plt.build().rstrip(\"\\n\")) | \n| | # fallback: unicode sparkline | \n| | if ema: | \n| | return Text(sparkline(ema, w - 4), style=\"cyan\") | \n| | return Text(\"waiting for loss data…\", style=\"dim\") | \n| | def build_lr_graph(state, w, h): | \n| | xs, lrs = state.lr_series.view() | \n| | if plt and len(lrs) >= 2: | \n| | _plot_start(w, h) | \n| | # log scale: automagic-style optimizers climb orders of magnitude | \n| | plt.yscale(\"log\") | \n| | plt.plot(xs, lrs, color=\"orange\", marker=\"braille\") | \n| | # plotext prints these as 0.00000600; label them in e-notation instead | \n| | lo, hi = min(lrs), max(lrs) | \n| | n = max(2, min(5, h // 3)) | \n| | ticks = [lo * (hi / lo) ** (i / (n - 1)) for i in range(n)] if hi > lo else [lo] | \n| | plt.yticks(ticks, [f\"{t:.1e}\" for t in ticks]) | \n| | plt.xlabel(\"step\") | \n| | plt.title(\"lr (log)\") | \n| | return Text.from_ansi(plt.build().rstrip(\"\\n\")) | \n| | if lrs: | \n| | return Text(sparkline(lrs, w - 4), style=\"yellow\") | \n| | return Text(\"waiting for lr data…\", style=\"dim\") | \n| | def header(state, pid_status): | \n| | \"\"\"Two lines: progress on top, loss and lr below.\"\"\" | \n| | pct = (state.step / state.total * 100) if state.step and state.total else None | \n| | r = state.rate() | \n| | def line(*pairs): | \n| | t = Text() | \n| | for i, (k, v) in enumerate(pairs): | \n| | if i: | \n| | t.append(\" \") | \n| | t.append(f\"{k} \", style=\"dim\") | \n| | t.append(v, style=\"bold\") | \n| | return t | \n| | top = line( | \n| | (\"step\", f\"{'--' if state.step is None else state.step}/{state.total}\" if state.total | \n| | else f\"{'--' if state.step is None else state.step}\"), | \n| | (\"progress\", f\"{pct:.1f}%\" if pct is not None else \"--\"), | \n| | (\"it/s\", state.rate_str or (f\"{r:.2f}\" if r else \"--\")), | \n| | (\"eta\", \"--\" if state.finished else state.eta_str or fmt_secs(state.eta())), | \n| | (\"elapsed\", state.elapsed_str or \"--:--\"), | \n| | ) | \n| | bottom = line( | \n| | (\"loss\", f\"{state.loss:.4f}\" if state.loss is not None else \"--\"), | \n| | (\"lr\", state.lr or \"--\"), | \n| | ) | \n| | if pid_status: | \n| | bottom.append(f\" {pid_status}\", style=\"dim\") | \n| | return [top, bottom] | \n| | def pid_status(pid): | \n| | if not pid: | \n| | return \"\" | \n| | try: | \n| | os.kill(pid, 0) | \n| | return f\"pid {pid}\" | \n| | except OSError: | \n| | return f\"pid {pid}: not running\" | \n| | def render(state, pid, console): | \n| | size = term_size() | \n| | # cols inside the outer panel: border (2) + padding (2) | \n| | cols = size.columns - 4 | \n| | head = header(state, pid_status(pid)) | \n| | # rows for graph panels: total minus outer borders (2) and the header, | \n| | # which wraps on narrow terminals | \n| | rows = size.lines - 2 - sum(len(t.wrap(console, cols)) for t in head) | \n| | # each graph panel spends 2 rows and 4 cols on its border and padding | \n| | if (rows - 4) // 2 >= STACK_MIN_ROWS: | \n| | h = (rows - 4) // 2 | \n| | graphs = Group( | \n| | Panel(build_loss_graph(state, cols - 4, h), border_style=\"white\", padding=(0, 1)), | \n| | Panel(build_lr_graph(state, cols - 4, rows - 4 - h), border_style=\"white\", padding=(0, 1)), | \n| | ) | \n| | elif cols // 2 - 4 >= MIN_GRAPH_COLS: | \n| | half = cols // 2 | \n| | graphs = Table.grid(expand=True) | \n| | graphs.add_column(ratio=1) | \n| | graphs.add_column(ratio=1) | \n| | graphs.add_row( | \n| | Panel(build_loss_graph(state, half - 4, rows - 2), border_style=\"white\", padding=(0, 1)), | \n| | Panel(build_lr_graph(state, cols - half - 4, rows - 2), border_style=\"white\", padding=(0, 1)), | \n| | ) | \n| | else: | \n| | graphs = Panel(build_loss_graph(state, cols - 4, rows - 2), | \n| | border_style=\"white\", padding=(0, 1)) | \n| | return Panel(Group(*head, graphs), | \n| | title=\"loratop\", subtitle=\"q to quit\", border_style=\"white\") | \n| | # --------------------------------------------------------------------------- | \n| | def main(): | \n| | ap = argparse.ArgumentParser(description=\"TUI monitor for Ostris AI Toolkit training\") | \n| | ap.add_argument(\"target\", nargs=\"?\", | \n| | help=\"run folder, training log, or PID (default: the run in \" | \n| | \"this folder, else the one running trainer)\") | \n| | ap.add_argument(\"--pid\", type=int, help=argparse.SUPPRESS) # old spelling of a PID target | \n| | ap.add_argument(\"--interval\", type=float, default=1.0, help=\"refresh seconds\") | \n| | ap.add_argument(\"--history\", type=int, default=600, | \n| | help=\"graph resolution: points kept before the plot \" | \n| | \"stride doubles to cover the whole run\") | \n| | ap.add_argument(\"--smooth\", type=float, default=0.1, help=\"EMA factor 0-1\") | \n| | ap.add_argument(\"--raw-smooth\", type=float, default=0.8, | \n| | help=\"Ostris-style smoothing 0-1 on the raw trace \" | \n| | \"(0 = truly raw)\") | \n| | ap.add_argument(\"--once\", action=\"store_true\", help=\"render one frame and exit (debug)\") | \n| | args = ap.parse_args() | \n| | # stale COLUMNS/LINES from a parent shell would freeze rich at the old | \n| | # size; drop them so every layer measures the tty directly | \n| | os.environ.pop(\"COLUMNS\", None) | \n| | os.environ.pop(\"LINES\", None) | \n| | target = args.target if args.target is not None else \\ | \n| | (str(args.pid) if args.pid else None) | \n| | path, pid = resolve_source(target) | \n| | pinned = pid is not None | \n| | pid = pid or discover_pid_from_log(path) | \n| | state = State(args.history, args.smooth, args.raw_smooth) | \n| | is_db = os.path.basename(path) == DB_NAME | \n| | source = DbSource(path) if is_db else LogSource(path) | \n| | source.poll(state) # backfill history | \n| | if not is_db: | \n| | # backfilled log lines all carry \"now\" as their timestamp, useless | \n| | # (and wildly wrong) for rate; start measuring fresh from here | \n| | state.rate_pts.clear() | \n| | state.finished = is_db and not pid | \n| | console = Console() | \n| | if args.once: | \n| | console.print(render(state, pid, console)) | \n| | return | \n| | def pid_alive(p): | \n| | try: | \n| | os.kill(p, 0) | \n| | return True | \n| | except OSError: | \n| | return False | \n| | resized = [False] | \n| | try: | \n| | signal.signal(signal.SIGWINCH, lambda *_: resized.__setitem__(0, True)) | \n| | except (ValueError, AttributeError): | \n| | pass # non-main thread or platform without SIGWINCH | \n| | # cbreak mode so a bare 'q' quits cleanly (exit 0, no SIGINT to the | \n| | # process group, plays nice with TUI wrappers that suspend around us) | \n| | kb_fd = old_attrs = None | \n| | if sys.stdin.isatty(): | \n| | kb_fd = sys.stdin.fileno() | \n| | old_attrs = termios.tcgetattr(kb_fd) | \n| | tty.setcbreak(kb_fd) | \n| | def key_pressed(timeout): | \n| | if kb_fd is None: | \n| | time.sleep(timeout) | \n| | return None | \n| | r, _, _ = select.select([kb_fd], [], [], timeout) | \n| | return os.read(kb_fd, 1) if r else None | \n| | last_disc = 0.0 | \n| | try: | \n| | with Live(render(state, pid, console), console=console, | \n| | refresh_per_second=4, screen=True) as live: | \n| | while True: | \n| | # re-discover if the trainer died/restarted (unless pinned) | \n| | if not pinned and (pid is None or not pid_alive(pid)) \\ | \n| | and time.monotonic() - last_disc > 5: | \n| | last_disc = time.monotonic() | \n| | pid = discover_pid_from_log(path) | \n| | source.poll(state) | \n| | state.finished = is_db and not (pid and pid_alive(pid)) | \n| | live.update(render(state, pid, console), refresh=True) | \n| | deadline = time.monotonic() + args.interval | \n| | while time.monotonic() < deadline: | \n| | if resized[0]: | \n| | resized[0] = False | \n| | live.update(render(state, pid, console), | \n| | refresh=True) | \n| | if key_pressed(0.05) in (b\"q\", b\"Q\"): | \n| | return | \n| | except KeyboardInterrupt: | \n| | pass | \n| | finally: | \n| | if old_attrs is not None: | \n| | termios.tcsetattr(kb_fd, termios.TCSADRAIN, old_attrs) | \n| | if __name__ == \"__main__\": | \n| | main() |", "url": "https://wpnews.pro/news/loratop", "canonical_source": "https://gist.github.com/syncreticgames/cc8a4523263e96289b2d9cc2b2a23e8b", "published_at": "2026-10-10 12:52:06+00:00", "updated_at": "2026-10-12 02:07:52.207237+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "machine-learning"], "entities": ["Ostris AI Toolkit", "loratop", "plotext", "rich", "uv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/loratop", "markdown": "https://wpnews.pro/news/loratop.md", "text": "https://wpnews.pro/news/loratop.txt", "jsonld": "https://wpnews.pro/news/loratop.jsonld"}}