Building ShouldWeAutomate: A Decision Intelligence Platform for Workflow Automation

How we built an open-source platform that tells you whether your business process is ready for AI automation — with deterministic scoring, gamified UX, and optional LLM inference.

Every week, someone asks: "Can we automate this workflow?" The answer is never simple. It depends on data quality, process stability, regulatory exposure, exception rates, integration readiness, decision complexity, and ROI potential — seven dimensions that interact in non-obvious ways.

Most automation decisions are made on gut feel. Teams spend months building automation only to discover the process changes too frequently, the data is too messy, or the compliance team blocks it.

We wanted to build a tool that makes this evaluation systematic, data-driven, and interactive — something a team can open in a browser, describe their workflow, and get a defensible answer in seconds.

Single-page Flask application rendered server-side with Jinja2 templates. The frontend is vanilla JavaScript with Chart.js for the radar visualization and a custom SVG gauge for the overall score.

Key design decisions:

// Core rendering — dimension cards with aggregate + fine-tune
function createDimSection(key, dim, prefix) {
  const aggDefault = Math.round(
    dim.questions.reduce((s, q) => s + q.default, 0) / dim.questions.length
  );
  const tier = getTier(aggDefault);
  // ... builds the HTML with aggregate slider + expandable sub-sliders
}

// Live preview — recompute overall on every slider change
function updateLivePreview() {
  const weights = [0.20, 0.20, 0.15, 0.15, 0.10, 0.10, 0.10];
  dimKeys.forEach((key, i) => total += getAggregateValue(key) * weights[i]);
  // Update gauge SVG dashoffset, tier badge, recommendation text
}

Flask acts as both the web server and the decision engine. The architecture follows a modular design:

engine/
├── scorer.py          # Dimension scoring logic, defaults, recommendations
├── analyzer.py        # Orchestrator — ties all modules together
├── explainer.py       # Score breakdown with pull-up/pull-down analysis
├── what_if.py         # What-if simulation and sensitivity analysis
├── roi_calculator.py  # Quantitative ROI with NPV, payback, FTE impact
├── remediation.py     # Remediation playbooks per dimension
├── regulations.py     # Regulatory framework mapping (HIPAA, GDPR, SOX, etc.)
├── similarity.py      # Benchmark similarity search
├── sub_process.py     # Multi-process decomposition and aggregation
└── llm.py             # OpenAI-compatible LLM gateway

The core scoring logic in scorer.py

defines 7 dimensions, each with 5 weighted sub-questions:

SCORING_DEFAULTS = {
    "data_quality": {
        "label": "Data Quality",
        "weight": 0.20,
        "questions": [
            {"id": "data_completeness", "text": "How complete is your data?", ...},
            {"id": "data_consistency", "text": "How consistent is data format?", ...},
        ],
    },
}

The overall score is a weighted average. The recommendation tier is determined by thresholds inspired by Capability Maturity Model (CMM) levels:

def get_recommendation(overall_score):
    if overall_score < 30:
        return {"level": "DO NOT AUTOMATE", ...}
    elif overall_score < 50:
        return {"level": "IMPROVE PROCESS FIRST", ...}
    elif overall_score < 70:
        return {"level": "HUMAN-IN-THE-LOOP AI", ...}
    elif overall_score < 85:
        return {"level": "AI ASSISTED AUTOMATION", ...}
    else:
        return {"level": "AGENT AUTOMATION READY", ...}

The LLM integration in engine/llm.py

is optional and modular. It follows the OpenAI chat completions format, making it compatible with LM Studio, Ollama, OpenAI, Anthropic, or any other provider.

When enabled, the AI performs three tasks:

def infer_workflow(description, industry):
    user_prompt = f"Industry: {industry}\n\nWorkflow Description:\n{description}"
    result = _call_llm(SYSTEM_WORKFLOW_ANALYSIS, user_prompt)
    if result and "dimension_scores" in result:
        scores = {k: max(0, min(100, int(v))) for k, v in result["dimension_scores"].items()}
        return result
    return None

The system prompt instructs the LLM to be skeptical and default to moderate scores unless the description strongly suggests otherwise — preventing over-optimistic AI outputs.

The data/benchmark_generator.py

creates 600+ synthetic workflows across 10 industries with deliberately injected failure modes:

FAILURE_PROFILES = {
    "contradictory_rules": "Business rules are contradictory across departments",
    "broken_apis": "Legacy systems have no stable API endpoints",
    "regulatory_churn": "Regulations change quarterly, invalidating logic",
    "data_rot": "Historical data uses outdated schemas",
    "seasonal_spikes": "Volume varies 10x between peak and off-peak",
    "fraud_scenarios": "Fraud patterns evolve faster than detection rules",
}

Each workflow gets randomized dimension scores, a metadata profile, and injected failure modes. The result is a realistic benchmark for similarity matching — when a user analyzes their workflow, we find the 5 most similar synthetic workflows.

The original UI had 35 range sliders visible at once. Users found it overwhelming. We redesigned it with three principles:

function getTier(score) {
  if (score >= 85) return { text: "Mythic", icon: "🏆", cls: "tier-excellent" };
  if (score >= 70) return { text: "Gold",   icon: "🥇", cls: "tier-good" };
  if (score >= 50) return { text: "Silver", icon: "🥈", cls: "tier-moderate" };
  if (score >= 30) return { text: "Bronze", icon: "🥉", cls: "tier-poor" };
  return { text: "Critical", icon: "⛔", cls: "tier-critical" };
}

AI auto-fill is now the default path. Users describe their workflow in a textarea, click "Auto-fill Scores," and the AI pre-fills all 35 sub-scores. Users can then fine-tune before analyzing.

After analysis, users get a comprehensive dashboard with seven tabs:

Deterministic engines are underrated. The LLM is a nice-to-have, but the deterministic scoring engine handles 90% of use cases. It's fast (~1 second), predictable, and doesn't require users to set up external services.

Gamification reduces friction. Users engaged more with tier badges and live preview than with a static form. The instant feedback loop makes the evaluation feel like a game rather than a survey.

AI prefill is a trust cliff. When AI prefills scores, users trust it more if they can see and tweak every value. The fine-tune section is critical for building confidence.

Synthetic benchmarks are surprisingly useful. Even though they're generated, they provide a reference frame. Users want to know how their scores compare to "similar" workflows.

git clone https://github.com/harishkotra/ShouldWeAutomate.git
cd ShouldWeAutomate
pip install -r requirements.txt
python app.py

Code & more: https://www.dailybuild.xyz/project/148-should-we-automate