AstraZeneca researchers developed an LLM-as-a-Judge evaluation framework for ChatInvent, an agentic drug discovery assistant, achieving human-aligned scoring with a majority-vote agreement of 0.86 aft…
PromptCube, a platform for AI developers, emphasizes the value of specialized AI communities over general social media for solving technical problems, citing a case where a RAG pipeline latency droppe…
Meta's Llama 3.1 70B model can now run on a single 24GB consumer GPU using GGUF or EXL2 quantization, achieving 5-10 tokens per second on an RTX 3090, according to a deployment guide. The guide recomm…
Cast AI benchmark testing shows that continuous batching at batch size 8 reduces Llama 3.1 70B inference cost on a single H100 from approximately $0.60-$0.80 per million tokens to $0.15-$0.25 per mill…
Ray AI libraries (Serve, Data, Train) now support Google TPU slices through a topology field that reserves a whole ICI-connected slice, preventing multi-host deployment hangs. Ray Serve serves LLMs vi…
Distilling from Google's Gemma 3 27B IT model into a smaller student model transfers depressive traits, with the student scoring a mean depression of 0.68 on the Gemma Needs Help eval even after aggre…
An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
Hugging Face users are experiencing confusion over Llama 3.1 70B API access via Inference Providers like Featherless. The issue is likely a provider-specific model availability mismatch rather than a …
A new study evaluating large language models in the social deduction game Secret Hitler found that current architectures remain ineffective at complex, multi-turn manipulation and deception. Models li…