{"slug": "making-type-coverage-visible-in-dify-s-ci", "title": "Making Type Coverage Visible in Dify's CI", "summary": "Dify, an open-source platform for building LLM applications, integrated Pyrefly static analysis into its CI pipeline to improve type coverage without blocking daily work. The rollout uses a blocking check for enforceable files and a non-blocking full-project coverage report that compares PRs against the base branch, displaying metrics like type coverage and typed symbols in PR comments. This approach allows contributors to see incremental typing improvements without requiring a complete migration.", "body_md": "# Making Type Coverage Visible in Dify's CI\n\nDify is a large open-source platform for building LLM applications. Its backend is a Python Flask application with workflows, RAG pipelines, model providers, agents, Celery tasks, database migrations, and a large test suite. That makes it a useful case study for Pyrefly adoption: a real codebase where static analysis needs to fit into existing CI without blocking daily work.\n\nThe goal was not to make every Pyrefly diagnostic fail CI on day one. That would have been noisy and counterproductive. The better approach was to split the rollout into two CI surfaces: a blocking check for the files we were ready to enforce, and full-project reporting that stayed non-blocking and showed up in PR comments.\n\n## Start with Measurement\n\nFor Dify, Pyrefly was added as a backend development dependency and configured in `api/pyproject.toml`\n\n:\n\n```\n[tool.pyrefly]project-includes = [\".\"]project-excludes = [\".venv\", \"migrations/\"]python-platform = \"linux\"python-version = \"3.12.0\"infer-with-first-use = truemin-severity = \"warn\"\n```\n\nThat project configuration gives Pyrefly enough information to analyze the backend in the same Python version and platform CI uses. But Dify also needed a second layer for adoption: a local exclude list at `api/pyrefly-local-excludes.txt`\n\n.\n\nThe blocking path uses that file to exclude known-problem areas while still failing on Pyrefly errors in the enforceable subset. The script builds a stricter command line, disables ignore-file heuristics, excludes broad legacy areas such as migrations and tests, then adds each path from `pyrefly-local-excludes.txt`\n\nas a `--project-excludes`\n\nentry.\n\nThat gives CI a practical gate: Pyrefly can block regressions where the project is ready, without pretending the whole repository is already clean.\n\n## Keep a Full Non-Blocking Signal\n\nThe second path runs over the full backend and reports what changed.\n\n```\nuv run --directory api --dev pyrefly coverage report\n```\n\nThe report produces structured JSON. Dify then renders the summary into a compact Markdown table for CI. A local run currently reports:\n\n| Metric | Value |\n|---|---|\n| Modules | 2790 |\n| Typable symbols | 54,901 |\n| Typed symbols | 24,989 |\n| Untyped symbols | 29,635 |\n| Any symbols | 277 |\n| Type coverage | 46.02% |\n| Strict coverage | 45.52% |\n\nThat number is a full-project baseline.\n\n## Compare PRs\n\nThe non-blocking coverage workflow runs `pyrefly report`\n\ntwice: once on the pull request branch and once on the base branch. A helper script extracts the summary and renders a comparison table with deltas for type coverage, strict coverage, typed symbols, untyped symbols, and module count.\n\nThat design matters. Large projects rarely improve through one giant typing push. They improve when contributors can see that a PR added 200 typed symbols, removed 20 untyped symbols, or accidentally moved coverage in the wrong direction.\n\nThe PR comment becomes a lightweight code review signal:\n\n```\n### Pyrefly Type Coverage| Metric | Base | PR | Delta || --- | ---: | ---: | ---: || Type coverage | 45.90% | 46.02% | +0.12% || Strict coverage | 45.40% | 45.52% | +0.12% || Typed symbols | 24,700 | 24,989 | +289 || Untyped symbols | 29,800 | 29,635 | -165 |\n```\n\nThis encourages better typing without turning the first rollout into a migration project.\n\nThe workflow also avoids comment spam. Each comment starts with a stable marker like `### Pyrefly Type Coverage`\n\n; the GitHub Action looks for an existing comment with that marker and updates it in place. If no matching comment exists, it creates one. That keeps repeated pushes from filling the PR timeline with near-identical coverage tables.\n\n## Keep Diagnostics Reviewable\n\nDify also has a non-blocking Pyrefly diff workflow. It runs `pyrefly check`\n\non the PR branch and the base branch, normalizes the output, and posts a diff only when the diagnostic line count changes.\n\nThe normalization step is small but important. Full checker output includes source excerpts and caret lines, which create noisy diffs. Dify's helper keeps the diagnostic headline and location line:\n\n``` php\n_DIAGNOSTIC_PREFIXES = (\"ERROR \", \"WARNING \")_LOCATION_PREFIX = \"-->\"\n```\n\nThat gives reviewers the part they need: what changed, and where. It avoids burying the signal inside pages of repeated context.\n\nThe diff comment uses the same pattern: one stable `### Pyrefly Diff`\n\ncomment that gets updated as the PR changes.\n\n## Handle Fork PRs Safely\n\nOne subtle CI detail is forked pull requests. Same-repository PRs can post comments directly from the pull request workflow. Forked PRs need a safer path.\n\nDify solves this by uploading structured artifacts from the untrusted workflow, then using a separate `workflow_run`\n\njob on trusted default-branch code to download the artifacts and render the comment. The trusted workflow posts the final PR comment.\n\nThat separation keeps the useful contributor experience while avoiding the common mistake of giving untrusted PR code write permissions.\n\n## Fit Type Coverage Alongside Test Coverage\n\nPyrefly coverage is not a replacement for runtime coverage. Dify's backend test workflow still runs unit tests and integration tests separately, uploads coverage artifacts, combines them, prints a coverage summary, and uploads XML coverage to Codecov.\n\nThe two signals answer different questions:\n\n- Runtime coverage asks: \"Did tests execute this code?\"\n- Type coverage asks: \"Can static analysis understand this interface?\"\n\nFor a codebase like Dify, both matter. Runtime coverage catches behavior regressions. Type coverage catches API drift, missing annotations, accidental `Any`\n\n, and unclear boundaries before they become harder to reason about.\n\n## What Worked\n\nThe most effective choices were simple:\n\n- Keep two Pyrefly configurations: one blocking path with local excludes, and one full-project path that reports in comments.\n- Compare PRs against base instead of enforcing a global threshold immediately.\n- Render comments in Markdown so reviewers do not need to open CI logs.\n- Update one stable PR comment per signal instead of creating a new comment on every push.\n- Normalize diagnostics before diffing them.\n- Keep fork-PR commenting on trusted code.\n- Use the same dependency manager and working directory shape locally and in CI.\n\nThis setup makes typing visible, reviewable, and incremental. That is the part that matters most at the beginning.\n\n## Looking Ahead\n\nOnce a project has stable reporting, it can decide where to tighten. Some teams may add thresholds. Others may focus on high-value modules first: request builders, service boundaries, decorators, provider contracts, or database-facing code.\n\nFor Dify, Pyrefly's value is already visible before strict enforcement. It gives contributors a concrete way to see whether a change made the Python backend easier or harder to understand. In a fast-moving LLM application platform, that feedback loop is worth a lot.", "url": "https://wpnews.pro/news/making-type-coverage-visible-in-dify-s-ci", "canonical_source": "https://pyrefly.org/blog/dify-pyrefly-coverage-ci/", "published_at": "2026-06-18 14:50:53+00:00", "updated_at": "2026-06-18 14:53:13.237168+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["Dify", "Pyrefly", "GitHub", "Python", "Flask", "Celery"], "alternates": {"html": "https://wpnews.pro/news/making-type-coverage-visible-in-dify-s-ci", "markdown": "https://wpnews.pro/news/making-type-coverage-visible-in-dify-s-ci.md", "text": "https://wpnews.pro/news/making-type-coverage-visible-in-dify-s-ci.txt", "jsonld": "https://wpnews.pro/news/making-type-coverage-visible-in-dify-s-ci.jsonld"}}