Welcome to the v0.0.2 release of Knowledge-and-Memory-Management, a tool designed for ingesting and managing knowledge from diverse sources. This release marks a clean release, stripping all hardcoded personal paths and replacing them with the portable $AGENT_HOME
environment variable. For experienced developers, this version brings consistency and ease of deployment across environments without sacrificing the core functionality of knowledge collection and memory management.
The project focuses on three primary collection pipelines: web, video, and articles. Each pipeline is modular, allowing you to configure, extend, or replace components based on your stack. The memory management layer ensures that collected data is indexed, stored, and retrievable via semantic search, making it practical for building personal knowledge bases or feeding into larger systems like agents or RAG pipelines.
The collection module is source-agnostic at its core but ships with specialized handlers for common content types.
Web collection uses a configurable web scraper that supports depth limits, domain filtering, and content extraction via readability algorithms. You can target specific sections, strip ads, and normalize HTML into markdown. The scraper respects robots.txt
and supports session management for authenticated sites.
Video collection transcribes audio using a local or remote ASR model. The pipeline extracts audio tracks, splits them into chunks, and generates timestamped transcripts. This is particularly useful for processing lectures, talks, or screencasts. The transcript is treated as a text document for further processing.
Article collection handles RSS/Atom feeds and direct URLs. It parses feeds, fetches full content using readability engines, and deduplicates entries. Articles are converted into a consistent schema: title, author, published date, body text, and metadata.
All collected data passes through a normalizer that converts content into a standard chunk format—suitable for embedding and memory storage. The normalizer handles language detection, tokenization, and optional summarization (the “S” in the original feature set, referring to semantic segmentation or summarization depending on configuration).
The memory layer stores knowledge chunks in a vector database with associated metadata. Each chunk is assigned an embedding generated from a configurable model (e.g., sentence-transformers). The store supports:
In v0.0.2, the memory management has been refactored to use $AGENT_HOME
as the base directory for all persistent files, including the vector store index and logs. This removes any previous machine-specific paths, making replication across development, staging, and production environments trivial.
$AGENT_HOME
The primary change in this release is the elimination of hardcoded user paths. Previously, configuration files and data directories referenced absolute paths like /home/user/.agent/
or C:\Users\user\.agent\
. Now, everything is relative to $AGENT_HOME
. Set this variable to the project root, and the system resolves all paths automatically.
export AGENT_HOME=/path/to/project
This change applies to:
If $AGENT_HOME
is unset, the system falls back to the current working directory. This ensures backward compatibility for local runs while enforcing portability for deployment.
Below is a minimal example showing how to collect a web article and store it in memory:
from agent_knowledge import Collector, MemoryStore
store = MemoryStore(base_path="$AGENT_HOME/data")
collector = Collector(store=store)
result = collector.collect_web("https://example.com/technical-blog")
print(f"Ingested {len(result.chunks)} chunks")
The base_path
parameter accepts $AGENT_HOME
directly; the system expands the variable at runtime. After ingestion, you can query the memory store with natural language:
matches = store.search("background on memory management")
for chunk in matches:
print(chunk.text[:200])
If you are upgrading from a previous version, move your existing data directory to the new location under $AGENT_HOME
or update your environment variable. The v0.0.2 detects the old format and migrates it automatically, but the path resolution is now strictly based on $AGENT_HOME
.
v0.0.2 delivers a cleaner, more portable foundation for knowledge collection and memory management. The shift to $AGENT_HOME
eliminates a common friction point when moving between systems. The collection pipelines—web, video, and articles—remain stable and performant, while the memory layer provides a solid base for building applications that require persistent, searchable knowledge.
For developers integrating this into larger workflows, the API surface is unchanged, so existing integrations continue to work. Future releases will focus on incremental improvements to the embedding pipeline and support for additional source types. Until then, this release stands as a reliable, portable version of the system.