I encountered a closely related problem while developing SEMA, although in an explicit relational-memory setting rather than an LLM prompt.
During retrieval, query fragments climb through the graph toward the learned contexts that contain them. Initially, allowing more reachable evidence seemed beneficial, but common fragments behaved like hubs: they reached too many contexts, contributed mostly noise, and diluted the genuinely discriminative parts of the query.
I addressed this with an adaptive saturation mechanism rather than a fixed global k. Each region expands only until the system can decide whether it yields a concrete, discriminative vote or has become saturated. Fan-out is capped at √N, votes are weighted by inverse context frequency, ambiguous matches must clear a contrastive noise margin, and saturated regions abstain instead of adding weak evidence. A context is accepted only after pooled evidence exceeds what any single region could contribute alone, requiring actual cross-region corroboration.
A useful consequence is that common boilerplate is not merely discarded: it is marked as scaffolding, allowing later evidence to be evaluated without the preamble diluting the payload.
So I strongly agree that the useful evidence count is task- and objective-dependent. My experience suggests that k may be better treated not only as a calibrated retrieval parameter, but as an endogenous stopping decision: continue adding evidence only while it increases discrimination or independent corroboration, and stop when additional evidence primarily increases reach rather than information.