Task type first, that narrows it from thousands to a handful immediately (classification vs generation vs OCR vs embeddings, etc.), pick a model family built for that task rather than a general-purpose one. After that, for a small project specifically: model size matters more than benchmarks upfront, since a 7B+ model you can’t actually run locally is useless no matter how good its score is, so check what fits your hardware (or whether you’re using the free Inference API) before comparing quality.
Downloads/likes are a decent tie-breaker for “is this actively maintained and not abandoned,” but they’re a weak signal on their own, a model with 2M downloads can still be the wrong architecture for your task. Better signal: sort by task on the Hub’s model list, skim the model card for the dataset it was trained on (does it resemble your data?), and just try 1-2 candidates on a handful of your own real examples rather than trusting a benchmark number, benchmarks rarely match your specific data distribution anyway.