Your bottleneck is likely retrieval, not reranking: a cross-encoder can only reorder the top 20 it is given, so first measure recall@20 of the first stage alone on 50–100 real queries rewritten as paraphrases — if the right thread is not in those 20, no reranker or larger embedder will recover it. For the index, embed each answer post with its thread’s question prepended, so every chunk keeps the question–answer pairing without diluting it across a whole thread, and pair a small bi-encoder (the bge-small or e5-small class is plenty for 40k posts on CPU, run hybrid with BM25 for the specialized terms embeddings smooth over) with your cross-encoder on top. Skip hard-negative fine-tuning until that recall number exists: if recall@20 is already high and the final ranking is wrong, fine-tune the reranker; if it is low, BM25-mined negatives for the bi-encoder are the one training step worth doing, and the same measurement tells you whether it worked.
What a Small RAG Fine-Tuning Experiment Can Actually Prove