Hi, I am building a custom logits processor that forces the generation of models to exactly follow the input text, plus classify parts of the text using special inline tags for precise text classification using generative models.
I successfully implemented the processor for models that generate the output directly without any chain of thought, deep reasoning, or thinking.
Now I would like to follow up on this by doing a two-phase generation, where I let the model reason however it wants, and then start the verbatim generation of the input text plus the classified spans. However, I am now thinking about how to divide this generation, first I thought about checking for the end-of-thinking token during generation, but it seems naive, as Ollama api or vLLM divide these, and you can simply pick the info you need from the data structure.
I need the possibility to access the logits during generation, which Ollama does not allow, and vLLM added this after I created my processor to fit HF, so I was wondering if there is a way to divide the two phases using only the HF library.