Hi — I maintain EvalPort (OpenEval), an open, portable schema for LLM/agent evaluation results, and I'd like to propose a small conversion adapter for SkillEvaluator, per the "discuss substantial changes in an issue first" note in CONTRIBUTING.md.
What I looked at in the actual source: src/skillevaluator/models/result.py has a real, structured result object across the Tier 1 validation pipeline — ValidationResult (validator_name, passed, findings: list[Finding], success_details: list[SuccessDetail], summary: ValidationSummary, plus a derived status property that's "passed"/"failed"/"incomplete"). Finding carries category, severity: Severity (CRITICAL/HIGH/MEDIUM/LOW/INFO), check_name, message, file_path/line_number, and suggestion. ValidationSummary aggregates counts (files_scanned, checks_performed, per-severity counters). This is exactly the kind of structured, gradeable result object OpenEval's ResultSet is meant to carry — each Finding maps naturally onto a graded test-case/criterion result, and ValidationSummary onto suite-level rollup stats.
What I'm offering: a small standalone adapter package (following the shape of the existing adapters in adapters/) with to_openeval()/from_openeval() converting ValidationResult to/from a payload that validates against openeval.validate.validate_suite()/validate_result_set(). Given SkillEvaluator's multi-tier design, this would likely start with Tier 1 (validate/quality-check/security-scan/etc., all producing ValidationResult) since that's the clearest 1:1 fit; Tier 3's live-evaluation reports could follow separately if there's interest.
Happy to build a first draft and open a PR, or just hand off pointers if someone here wants to take it — this is meant as a low-pressure invitation, not a demand. Spec: https://github.com/adhabnr-ux/evalport.
Hi — I maintain EvalPort (OpenEval), an open, portable schema for LLM/agent evaluation results, and I'd like to propose a small conversion adapter for SkillEvaluator, per the "discuss substantial changes in an issue first" note in CONTRIBUTING.md.
What I looked at in the actual source:
src/skillevaluator/models/result.pyhas a real, structured result object across the Tier 1 validation pipeline —ValidationResult(validator_name,passed,findings: list[Finding],success_details: list[SuccessDetail],summary: ValidationSummary, plus a derivedstatusproperty that's"passed"/"failed"/"incomplete").Findingcarriescategory,severity: Severity(CRITICAL/HIGH/MEDIUM/LOW/INFO),check_name,message,file_path/line_number, andsuggestion.ValidationSummaryaggregates counts (files_scanned,checks_performed, per-severity counters). This is exactly the kind of structured, gradeable result object OpenEval'sResultSetis meant to carry — eachFindingmaps naturally onto a graded test-case/criterion result, andValidationSummaryonto suite-level rollup stats.What I'm offering: a small standalone adapter package (following the shape of the existing adapters in
adapters/) withto_openeval()/from_openeval()convertingValidationResultto/from a payload that validates againstopeneval.validate.validate_suite()/validate_result_set(). Given SkillEvaluator's multi-tier design, this would likely start with Tier 1 (validate/quality-check/security-scan/etc., all producingValidationResult) since that's the clearest 1:1 fit; Tier 3's live-evaluation reports could follow separately if there's interest.Happy to build a first draft and open a PR, or just hand off pointers if someone here wants to take it — this is meant as a low-pressure invitation, not a demand. Spec: https://github.com/adhabnr-ux/evalport.