Today I'm releasing Pythonaibrain-NLP 0.2.0, the latest public release of my Python NLP framework.
The package is now available on PyPI, and the complete source code, documentation, architecture notes, examples, and tests are available on GitHub.
Install it with:
pip install pythonaibrain-nlp
Pythonaibrain-NLP was built around a different idea.
Instead of making a transformer the center of everything, I wanted to build a more structured NLP system where understanding, dialogue state, retrieval, and generation are explicit components of the architecture.
The current system combines:
The goal isn't to replace every modern NLP architecture.
The goal is to provide a structured, understandable, trainable NLP pipeline that can be integrated into Python applications.
The core pipeline is:
User Input
│
▼
┌─────────────┐
│ NLU │
│ │
│ Intent │
│ + Slots │
└──────┬──────┘
│
▼
┌─────────────────┐
│ Dialogue State │
│ + Context │
└────────┬────────┘
│
┌───────┴────────┐
▼ ▼
Function/API RAG
Dispatch Retrieval
│ │
└───────┬────────┘
▼
┌─────────────┐
│ NLG │
│ SC-LSTM │
└──────┬──────┘
│
▼
Response
This separation makes each stage independently accessible and easier to experiment with.
The NLU subsystem uses a joint neural architecture for:
Intent classification + slot tagging
The model is designed to understand both what the user wants and which pieces of information are present in the input.
For example, a request such as:
"Book a flight to Delhi tomorrow"
can be represented through an intent together with structured slot information rather than treating the entire sentence as an opaque classification problem.
This structured representation can then be passed to the dialogue and application layers.
One of the central parts of Pythonaibrain-NLP is its Semantically-Controlled LSTM (SC-LSTM) based NLG system.
The idea is to condition generation on semantic information rather than simply generating text from an unconstrained language representation.
That gives the application a much more explicit relationship between:
Semantic representation
↓
NLG controller
↓
Generated text
This is particularly useful for task-oriented dialogue systems where the generated response should correspond to a known semantic intent.
Pythonaibrain-NLP also maintains conversational information through:
This allows the system to work with multi-turn interactions rather than processing every sentence completely independently.
A conversation can therefore evolve as structured state instead of being treated as a sequence of unrelated strings.
The framework includes a lightweight retrieval component based on TF-IDF and cosine similarity.
This provides a simple way to retrieve relevant information when the system needs knowledge from a local corpus.
The important part here is that retrieval doesn't have to replace the rest of the NLP pipeline.
It can participate in the overall decision process:
NLU
↓
Intent + Slots
↓
Application logic / Retrieval
↓
NLG
↓
Response
Pythonaibrain-NLP does not require a transformer architecture for its core NLP pipeline.
Instead, it uses recurrent neural architectures and structured components.
That makes the project interesting for situations where developers want:
This isn't an argument that transformers are bad.
It's simply a different engineering trade-off.
The interesting part of the project isn't one model by itself.
It's the combination.
Pythonaibrain-NLP brings together:
┌──────────────┐
│ NLU │
└──────┬───────┘
│
Intent + Slots
│
▼
┌──────────────────┐
│ Dialogue Context │
└────────┬─────────┘
│
┌─────┴─────┐
│ │
▼ ▼
Retrieval Application
│ │
└─────┬─────┘
│
▼
┌───────────┐
│ NLG │
└─────┬─────┘
│
▼
Response
Each layer has a defined responsibility.
That structure is one of the main things I wanted from the project.
The repository includes:
The project also includes measured experiments and evaluation information in the repository rather than presenting the architecture only as a theoretical design.
pip install pythonaibrain-nlp
Then import the package from Python and use the APIs documented in the project.
The repository contains examples for getting started.
The complete project is available on GitHub:
https://github.com/DivyanshuSinha136/Pythonaibrain-NLP
And the package is available on PyPI:
https://pypi.org/project/Pythonaibrain-NLP/
Version 0.2.0 is now the public baseline.
Future development can build on top of this foundation rather than changing the fundamental direction of the project.
I'm particularly interested in seeing what developers can build with the architecture and where the limitations become apparent when it is used outside my own experiments.
That's one of the most useful parts of releasing a project publicly: the architecture stops being something that exists only on my machine.
It becomes something other people can install, inspect, test, extend, and challenge.
Pythonaibrain-NLP 0.2.0 is now public. 🚀
If you're interested in structured NLU/NLG systems, dialogue architectures, recurrent neural NLP, or building NLP systems directly in Python, I'd love to see what you do with it.