# Seeking AI/ML Collaborators: Building Marven — A Local AI with Persistent Memory, Continual Learning, and Human-State Awareness > Source: > Published: 2026-08-28 21:24:36+00:00 Hi everyone, I’m the founder of [Heshware LLC](https://heshware.it.com), and I’m currently developing an AI project called **Marven**. Marven started as an experiment in building a more personal AI assistant, but over time the project has grown into a larger research and engineering question: **What would it take to build a local-first AI system that doesn’t simply respond to prompts, but develops meaningful continuity with a user over long periods of time?** The goal is not to claim that Marven is conscious or to build another wrapper around an LLM. I’m interested in developing an architecture where an AI can maintain persistent memory, understand changing human context, learn from long-term interactions, preserve continuity across sessions, and operate primarily on infrastructure controlled by the user. **What I’m Trying to Build** The long-term vision for Marven is a **Personal Sovereign AI** — an AI system whose memory, models, identity data, and personal context can remain under the user’s control rather than existing entirely inside a centralized cloud platform. Some of the capabilities I’m exploring include: One of the core questions behind Marven is: **Can an AI maintain continuity with someone for years while still giving that person ownership and control over what the AI remembers?** Marven is still experimental and the architecture continues to evolve. The current/local stack has included: **LLM inference** **Memory** I’ve also been experimenting with what I call the **MetaMirror** system. MetaMirror is intended to represent more than conversation history. The idea is to maintain structured observations about interaction context and how that context changes over time. For example: ``` Interaction ↓ Conversation / Voice Input ↓ Context Analysis ↓ Working Memory ↓ Memory Importance Evaluation ↓ Episodic / Semantic Memory ↓ Long-Term Memory Archive ↓ Retrieval + Context Reconstruction ↓ Future Marven Interaction ``` The architecture is moving toward separating memory into different layers instead of continuously dumping conversations into a vector database. That distinction is becoming increasingly important to me. Remembering everything is not necessarily intelligence. Knowing **what should be remembered, what should be forgotten, what should be summarized, and when something should be retrieved** is a much harder problem. Another part of the project is what I’m currently thinking of as a **Human-State Engine**. The idea is to give Marven additional context regarding *how* someone is communicating rather than only processing the literal words they say. Potential signals include: Future experiments could potentially include visual signals, but I want this architecture to be privacy-conscious and based around explicit user consent. For example, instead of simply receiving: ``` "I'm fine." ``` the system might internally receive contextual information closer to: ``` Text: "I'm fine." Voice indicators: - lower energy than personal baseline - longer pause before response - reduced speaking speed Conversation context: - previously discussing stressful event Confidence: - uncertain emotional state ``` The LLM still determines how to respond, but it receives richer contextual information. Importantly, I do **not** want Marven confidently diagnosing someone’s emotions. Human-state detection should be probabilistic and uncertain by design. I’ve also been working on a persistent visual representation of Marven. The current direction uses a 3D digital-human interface with states such as: The goal isn’t simply to create an animated chatbot. I’m interested in whether maintaining a consistent visual and behavioral presence contributes to the feeling of continuity in long-term human-AI interaction. The voice pipeline has experimented with technologies including: Eventually I want voice interaction to operate with sufficiently low latency that Marven feels less like: **Speak → wait → AI responds** and more like a natural conversational system. That requires work across: ``` Wake word ↓ Voice activity detection ↓ Streaming STT ↓ Context + memory retrieval ↓ Model inference ↓ Streaming TTS ↓ Digital-human animation ``` Latency across that entire pipeline is one of the engineering challenges I’m currently investigating. The part of Marven I’m currently most interested in improving is the memory architecture. Most current AI assistants effectively have some combination of: ``` Conversation History + Vector Database + Retrieval ``` That works, but I’m interested in something more structured. I’m experimenting conceptually with a system closer to: ``` Sensory / Interaction Memory ↓ Working Memory ↓ Importance Scoring ↓ ┌────────┴─────────┐ ↓ ↓ Episodic Semantic Memory Memory ↓ ↓ └────────┬─────────┘ ↓ Memory Consolidation ↓ Long-Term Archive ↓ Contextual Retrieval ``` Some questions I’m investigating: **How should memory importance be calculated?** Something mentioned once may be extremely important, while something repeated every day may not be. **How should memory decay work?** Human memory isn’t an infinitely growing database. **How should contradictions be handled?** If someone’s preferences or beliefs change, the AI shouldn’t continue retrieving outdated information as if it were current. **How do you prevent memory poisoning?** If an attacker can manipulate persistent memory, prompt injection becomes much more dangerous. **How should memories be consolidated?** Hundreds of interactions about the same subject should potentially become a higher-level semantic representation rather than hundreds of individual memories. **How can the model know why a memory matters?** Similarity alone may not be enough for good retrieval. Another challenge is determining how much of Marven’s learning should happen through: I don’t think continuously fine-tuning the base model on every user interaction is necessarily the correct solution. My current thinking is that most personalization should initially happen through memory and contextual systems, while actual model-weight updates happen much more deliberately. I’d be especially interested in hearing from anyone researching continual learning or catastrophic forgetting. Persistent memory also creates an entirely new security problem. I’m actively thinking about threats including: A personal AI could eventually know more sensitive contextual information about someone than almost any other piece of software they use. That means security cannot be something added after the architecture is finished. I’m particularly interested in architectures where memories have permissions, provenance, confidence scores, timestamps, and potentially cryptographic integrity verification. One of Marven’s fundamental design principles is that users should eventually have the option to run significant portions of the system locally. That could mean: ``` Personal Device ↓ Local Marven Node ↓ Local Memory Database ↓ Local / Home LLM Server ↓ Optional Cloud Models ``` The cloud should ideally become an optional compute resource rather than the permanent owner of someone’s AI memory. I’m also exploring home-server/NAS-style architectures where an individual’s long-term AI memory could exist on storage they physically control. Marven isn’t only a concept. I’ve already experimented with or built portions of: A lot of these pieces are currently separate experiments. One of my current priorities is turning them into a cleaner unified architecture. I’m posting here because I would really like feedback from people who work in areas such as: I’m especially interested in contributors who enjoy experimenting with research ideas and turning them into working prototypes. You don’t need to be an expert in every area. Even feedback like: “Your memory architecture will eventually break because of X.” or “There’s a paper/project you should study that already tackles part of this.” would be extremely useful. I’d particularly like opinions on a few questions: **1. If you were designing long-term memory for an AI intended to interact with someone for 5–10+ years, what architecture would you use?** **2. Would you combine vector retrieval with a knowledge graph, or use another architecture entirely?** **3. What is the best way to separate episodic memory from semantic/user-profile memory?** **4. How would you design memory consolidation and forgetting?** **5. How would you defend persistent AI memory against poisoning and prompt injection?** **6. What parts of personalization actually deserve model fine-tuning versus external memory?** **7. Are there open-source projects or papers exploring similar long-duration human-AI relationships that you think I should study?** I’m also open to collaborating with developers or researchers who find these problems interesting. Right now I’m particularly interested in people who could help experiment with: **Memory / RAG** **Local AI** **Continual Learning** **Voice / Multimodal** **AI Security** This is still an early-stage research and development project, so I’m interested in people who enjoy exploring difficult questions rather than only implementing a predetermined product specification. Marven is being developed through **Heshware LLC**, but I’m also interested in keeping portions of the research and experimentation open enough that other developers can learn from it and contribute ideas. Ultimately, the question I’m trying to explore with Marven is pretty simple: **What happens when an AI stops being something you repeatedly start a new conversation with and instead becomes a system capable of maintaining years of meaningful, user-controlled continuity?** There are a lot of unanswered engineering, security, privacy, and research questions between here and there. That’s exactly why I’m interested in building it. I’d love to hear what the Hugging Face community thinks.