Add coder-eval plugin (Code Quality Testing) - #415
Open
uipreliga wants to merge 1 commit into
Open
Conversation
Signed-off-by: uipreliga <tomasz.religa@uipath.com> Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds coder-eval to
Code Quality Testing.Disclosure: I maintain this project. Happy to reword, shorten, or move the entry — just say which.
What it is
Most quality tooling checks the files a plugin ships. coder-eval checks the behavior: given a labelled set of prompts, does the skill actually fire when it should and stay quiet when it shouldn't? It reports suite-level precision / recall / F1 and can fail a GitHub Actions run when the numbers drop, so a
description:edit that quietly breaks activation gets caught in review.It runs the same suite against Claude Code, Codex, OpenCode and Antigravity, which is how you tell "the skill is wrong" apart from "this harness didn't route to it."
/coder-eval:init,:check-skill,:task,:lint-tasks,:analyze,:ci)Two things I left out on purpose, tell me if you want them
External Marketplaces— coder_eval is also a marketplace (claude plugin marketplace add UiPath/coder_eval), so it would qualify for that table too. I did not add a second row, because putting myself in two places in one PR seemed presumptuous. Say the word and I'll add it.README-zh.md— I did not touch it. It currently carries 1 of the 6 external entries thatREADME.mdhas, so it looks like you sync it yourself rather than expecting contributors to. If you'd prefer the row there too, here it is:Diff is one line. Thanks for maintaining the list.