TIKA-4745 - add cohort-specific caps

[tballison]

Small tweak to improve sbcs -> cjk confusion

[Copilot]

Pull request overview

This PR updates Mojibuster’s Naive Bayes bigram encoding detector to reduce SBCS→CJK misclassification by changing the per-distinct-bigram clipping rule to be cohort-aware (script-family aware), and adds an internal .skills/ write-up describing a workflow for running encoding regression comparisons with tika-eval.

Changes:

• Replace “top-vs-runner-up” per-bigram clipping with a “top-vs-best cross-cohort” cap and introduce a Cohort mapping for model labels.
• Tighten the cap magnitude (CAP_PER_BIGRAM_NATS) from 20.0 to 10.0 and enforce model-label→cohort coverage at model load time.
• Add a .skills/ document with a condensed procedure (commands + SQL) for encoding regression hunts using a single app build and multiple configs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/NaiveBayesBigramEncodingDetector.java	Adds cohort taxonomy/mapping and changes per-bigram capping to be cross-cohort aware to reduce SBCS→CJK confusion.
.skills/tika-eval-encoding-regression.md	Adds a workflow doc for running encoding regression comparisons and investigating flips via tika-eval + SQL + trace metadata.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.