{"slug": "compact-models-for-patient-level-histopathology-mil-baselines-and-controlled", "title": "Compact models for patient-level histopathology MIL: baselines and controlled evaluation", "summary": "A Colab Tesla T4 feature-extraction test found that all three public pathology-pretrained ViT-S/16 checkpoints — Kaiko ViT-S, Lunit DINO ViT-S and DistillPath-KS16-HOpt0 — loaded through the timm library and produced 384-dimensional pooled crop embeddings, with model(x) and num_classes=0 returning [B, 384] and forward_features(x) returning [B, 197, 384]. The author cautions the check establishes only compatibility and a limited cost reference, not patient-level macro-F1, clinical utility or a distillation gain. The write-up recommends frozen crop embeddings with a small patient-level multiple-instance-learning head, mean pooling and attention-based MIL under an explicit crop budget, and holding encoder and crop IDs fixed when comparing aggregation methods.", "body_md": "For now, I found a few compact models and implementation examples that might be relevant:\n\n**1. Compact encoders and frozen features versus fine-tuning.** I would keep the project’s existing compact CNN or ViT as a reference and compare it with at least one released, pathology-pretrained small encoder. Three concrete ViT-S/16 candidates are [Kaiko ViT-S](https://huggingface.co/1aurent/vit_small_patch16_224.kaiko_ai_towards_large_pathology_fms), [Lunit DINO ViT-S](https://huggingface.co/1aurent/vit_small_patch16_224.lunit_dino), and [DistillPath-KS16-HOpt0](https://huggingface.co/RamonK/DistillPath-KS16-HOpt0). The first two offer different pathology-pretraining references; the third is a small student initialized from Kaiko and subsequently distilled from a larger pathology teacher. None is necessarily best for an unseen patient-level task. **Frozen crop embeddings followed by a small patient-level MIL head** are a particularly interpretable and inexpensive first comparison. End-to-end or partial fine-tuning can be a separate branch if the frozen representation appears limiting.\n\n**2. Variable crop counts and heterogeneous tissue.** Start with **mean pooling and attention-based MIL** using identical cached embeddings and an explicit crop budget. The classic [attention-based deep MIL paper](https://proceedings.mlr.press/v80/ilse18a.html) provides a simple learnable set aggregator. Vary the number of crops per patient and repeat crop sampling with different seeds; this often tells you more than immediately adding a complex head. Max/top-k or class-aware selection are conditional alternatives when evidence is sparse. If a patient has several identifiable slides, compare flat crop-to-patient pooling with slide-first aggregation rather than assuming either hierarchy is universally better.\n\n**3. Controls that separate methodological gains.** Hold **encoder and crop IDs fixed** when comparing aggregation methods; hold **MIL head, splits, crop budget, and evaluation protocol fixed** when comparing encoders. A released distilled checkpoint versus its released initialization is a practical comparison of *weights as distributed*, not a clean causal estimate of a distillation loss: the distilled student also had additional training data and compute. To study the distillation objective itself, compare the same student initialized the same way under suitably matched training conditions, with and without the teacher signal. Report patient-level macro-F1 and class recall beside crop counts and costs, rather than treating parameter count alone as a compute control.\n\nThe earlier reply already suggests frozen pathology features and one-factor-at-a-time comparisons. To complement it, I have included specific weights, implementation references, conditional choices, and a bounded T4 engineering check. None requires access to unpublished methods or private results.\n\nIn a short **Colab Tesla T4 feature-extraction test**, all three public ViT-S checkpoints loaded through `timm` and yielded **384-dimensional pooled crop embeddings**. This checks compatibility and a limited cost reference, **not** patient macro-F1, clinical utility, or a distillation gain. The tested scope appears below.\n\n`model(x)` with `num_classes=0` gives `[B, 384]`; `forward_features(x)` gives `[B, 197, 384]`. Feeding tokens instead of one pooled vector per crop changes both the model and its compute. See the `timm` feature-extraction guide\nThe expanded sections provide details and references for the branches that turn out to matter.\n\nCompact pathology encoders, feature extraction, and a small T4 check", "url": "https://wpnews.pro/news/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled", "canonical_source": "https://discuss.huggingface.co/t/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled-evaluation/190814#post_3", "published_at": "2026-10-21 08:44:37+00:00", "updated_at": "2026-10-21 09:14:32.321690+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "computer-vision", "ai-tools"], "entities": ["Kaiko ViT-S", "Lunit DINO ViT-S", "DistillPath-KS16-HOpt0", "timm", "Colab Tesla T4", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled", "markdown": "https://wpnews.pro/news/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled.md", "text": "https://wpnews.pro/news/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled.txt", "jsonld": "https://wpnews.pro/news/compact-models-for-patient-level-histopathology-mil-baselines-and-controlled.jsonld"}}