{"slug": "four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself", "title": "Four reviewers told me the one thing I couldn't fix by myself", "summary": "A developer's open-source protocol for AI-device interaction was rejected by IEEE WF-IoT 2026 despite positive reviews, highlighting the challenge of self-evaluation. The reviewers praised the architecture but noted the lack of independent annotators, a gap the developer cannot close alone. The developer has since expanded the evaluation corpus and fixed bugs found through self-testing.", "body_md": "*What happens when you submit a solo open-source project to academic peer review:\nthe reviews were good, the paper was rejected, and the gap they found was the one\nI had no way to close on my own.*\n\nI've been building [DoSync](https://github.com/giulianireg-spec/dosync-protocol),\n\nan open-source protocol that sits between AI agents and physical devices — I\n\nwrote about\n\n[why it exists here](https://dev.to/giulianiregspec/why-your-smart-home-ai-cant-open-the-door-in-an-emergency-and-how-to-fix-it-5d5i)\n\nif you want the background.\n\nI tested it the only way I could: on my own hardware, against my own\n\nexpectations. Then I sent it to IEEE WF-IoT 2026, because I wanted it examined by\n\npeople who weren't me.\n\nFive reviews came back. Four were substantive. It was rejected.\n\nAnd here's the part that took me longest to understand: **the reviews were\ngood.** Two scored relevance 5 out of 5. One recommended minor revisions. The\n\nRejected anyway. Not because the idea was wrong. Because of this, from the most\n\ndetailed review:\n\nThe evaluation is small and self-judged: 15 scenarios with ground truth\n\ndefined by the author alone.\n\nThat sentence is correct, and no amount of rewriting section 4 was going to\n\nchange it.\n\nIt took me a few days to notice that the rejection was the thing I'd submitted\n\nfor.\n\nI didn't send the paper hoping for a stamp. I sent it because after months of\n\nbeing the only person who had ever evaluated this, I had no way to tell the\n\ndifference between \"this works\" and \"this works the way I expect it to.\" Four\n\nstrangers read it carefully and told me. That is not a consolation prize — it is\n\nthe entire reason to submit.\n\nWhat they said, in summary: the architecture holds up, and the evidence doesn't\n\nreach as far as the claims. One reviewer went further and put the knife somewhere\n\nspecific. My central claim is that the project removes the manual burden of\n\nwriting device-by-device automation rules. Their objection:\n\nif resolution quality depends on well-curated tags, the manual configuration\n\nburden has not been eliminated, it has moved from rules to manifest curation\n\nI've had months to find a counter-argument. I don't have one yet.\n\nThe request that came up in almost every review was for independent annotators.\n\nNot more scenarios — different people. Someone other than me who had read a\n\nscenario, looked at the devices available, and formed their own opinion about\n\nwhich ones should respond.\n\nI couldn't produce one. Every judgement about what the resolver *should* pick —\n\nin all fifteen scenarios, in the paper and in the benchmark files — was mine.\n\nWhen I disagreed with the resolver, I was the one who had decided what the right\n\nanswer was in the first place.\n\nEverything else on the reviewers' list, I could work on alone, and did. The\n\nevaluation used to run on a single home registry, which is a poor way to support\n\na claim of being domain-agnostic; there's now a\n\n[multi-domain corpus](https://github.com/giulianireg-spec/dosync-protocol/tree/main/benchmarks/corpus)\n\ncovering industrial and clinical registries, published so the numbers are\n\nreproducible by someone who isn't me. The precision/recall conventions that made\n\ntwo rows of a results table mathematically impossible are defined and the bug is\n\ntraced. There's a sensitivity analysis on the scoring weights.\n\nNone of that required anyone else. The annotators do. That asymmetry turned out\n\nto be the most useful thing in the whole review packet.\n\nSince the reviews, I've been doing the closest thing to third-party testing that\n\none person can do: installing my own project on machines that had never run it,\n\nfollowing my own published instructions, on hardware that owed me nothing.\n\nIt found more than a dozen defects in a week, on a codebase with about a thousand\n\npassing tests. My favourite: device discovery had been silently dead in\n\nproduction for days, because a dependency I'd added shipped an event loop that\n\n*declares* a method and raises when you call it. My check was `hasattr`\n\n. It\n\nreturned `True`\n\n. Every test passed and nothing was ever discovered.\n\nNot one of those was findable from the inside. My test suite ran on one operating\n\nsystem, one event loop, one machine that already had everything configured — it\n\nwas measuring a world only I lived in. Which is, word for word, what the reviewer\n\nsaid about my evaluation.\n\nThat's the part I'd pass on to anyone building something alone: **the hardest\nwork isn't the next feature. It's getting the thing into the hands of someone who\ndoesn't already know how it's supposed to behave** — including your own hands, on\n\nThe protocol is open, the corpus is public, and the one thing it still needs is\n\nan opinion that isn't mine.\n\n**GitHub:** [github.com/giulianireg-spec/dosync-protocol](https://github.com/giulianireg-spec/dosync-protocol)\n\n**Web:** [https://dosync.dev/](https://dosync.dev/)\n\n**License:** Apache 2.0\n\n*DoSync — the semantic layer between AI agents and physical systems.*\n\n*(The reviews are summarised faithfully and the reviewers stay anonymous, as they\nshould.)*", "url": "https://wpnews.pro/news/four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself", "canonical_source": "https://dev.to/giulianiregspec/four-reviewers-told-me-the-one-thing-i-couldnt-fix-by-myself-4p08", "published_at": "2026-08-30 01:51:03+00:00", "updated_at": "2026-08-30 02:22:14.144182+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "developer-tools"], "entities": ["DoSync", "IEEE WF-IoT 2026"], "alternates": {"html": "https://wpnews.pro/news/four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself", "markdown": "https://wpnews.pro/news/four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself.md", "text": "https://wpnews.pro/news/four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself.txt", "jsonld": "https://wpnews.pro/news/four-reviewers-told-me-the-one-thing-i-couldn-t-fix-by-myself.jsonld"}}