Verihubs, an Indonesian AI company, rebuilt its data warehouse on ClickHouse, achieving up to 98% faster queries and 50% lower cloud costs. The new architecture replaced daily Postgres batch pulls wit…
A developer introduces agentic data engineering, a practice where autonomous AI agents design, build, and maintain data pipelines from natural-language intent, shifting from manual coding to specifyin…
Loop Engineering introduces a governance architecture for AI agents that prevents costly errors by adding structured control layers, including task definitions, state separation, tool guards, observat…
A data engineer created Ghost Skills, a collection of methodology instructions for AI coding agents, to address the gap between model capability and the contextual knowledge needed for real-world data…
Two blog posts from DBOS and Obelisk sparked a debate on using Postgres versus SQLite for durable workflow execution, with both arguing their preferred database can replace external orchestrators like…
A 2026 career guide breaks down the distinct roles of data engineers, who build and maintain data infrastructure and pipelines, and data scientists, who analyze data and build machine learning models.…
Data Workers, an open-source autonomous agent swarm for data engineering, bet on the Model Context Protocol (MCP) to connect its AI agents to dozens of tools in the modern data stack, rather than buil…
Nearly 40% of active Data Engineer job postings now mention AI, with 17.4% explicitly requiring generative AI skills like LLMs, RAG, and AI agents, according to an analysis of 6,736 listings on Interv…
A developer demonstrated that MCP (Model Context Protocol) servers are essentially REST APIs with a polite wrapper, showing how to access the Helium MCP server's underlying HTTP endpoints directly usi…
Building a custom task framework for running LLM tasks on AWS SQS, choosing to create their own solution rather than using existing frameworks like Celery or Airflow. They explain that SQS is better s…