What should the file have been detected as? What has the file been misdetected as?
Shell script (sh) misdetected as Tcl. The file xdg-mime (/usr/bin/xdg-mime)
is a 1560-line shell script that Magika 1.1.0 classifies as tcl instead of sh."
Misdetected file
xdg-mime.zip
Additional context
A 1560-line shell script (xdg-mime from /usr/bin) is confidently misclassified as Tcl (75% score) by both the CLI and the Python API.
Steps to reproduce
magika -l -s xdg-mime
# output: xdg-mime: tcl 75%
magika version: 1.1.0 standard_v3_3
Root cause analysis
The script contains a large awk program embedded in a heredoc near the end of the file (~lines 1400–1550):
/^mime-info/, /^\/mime-info/ {
if (match($0,/type="[^"]*/)) {
print "[Desktop Entry]"
...
}
}Magika uses the last ~512 bytes as features. Those bytes are dominated by the awk block, which uses {}, match(), print, /regex/ patterns, and curly-brace blocks — syntax that resembles Tcl. The rest of the file (~1400 lines) is standard POSIX shell, including extensive [ -z "..." ] and [ -r "$file" ] conditionals, which also use brackets characteristic of Tcl.
Why this matters
This is a representative pattern: shell scripts that embed other languages (awk, sed, Python) in heredocs are common in system utilities (xdg-open, xdg-icon-resource, etc.). The tail-bytes feature extraction makes these systematically hard to classify correctly.
Suggested fix
Adding shell scripts with embedded awk/other-language heredocs to the training corpus would address this class of misclassification.
What should the file have been detected as? What has the file been misdetected as?
Shell script (
sh) misdetected as Tcl. The filexdg-mime(/usr/bin/xdg-mime)is a 1560-line shell script that Magika 1.1.0 classifies as
tclinstead ofsh."Misdetected file
xdg-mime.zip
Additional context
A 1560-line shell script (
xdg-mimefrom/usr/bin) is confidently misclassified as Tcl (75% score) by both the CLI and the Python API.Steps to reproduce
magika -l -s xdg-mime # output: xdg-mime: tcl 75%magika version: 1.1.0 standard_v3_3
Root cause analysis
The script contains a large
awkprogram embedded in a heredoc near the end of the file (~lines 1400–1550):Magika uses the last ~512 bytes as features. Those bytes are dominated by the awk block, which uses
{},match(),print,/regex/patterns, and curly-brace blocks — syntax that resembles Tcl. The rest of the file (~1400 lines) is standard POSIX shell, including extensive[ -z "..." ]and[ -r "$file" ]conditionals, which also use brackets characteristic of Tcl.Why this matters
This is a representative pattern: shell scripts that embed other languages (awk, sed, Python) in heredocs are common in system utilities (
xdg-open,xdg-icon-resource, etc.). The tail-bytes feature extraction makes these systematically hard to classify correctly.Suggested fix
Adding shell scripts with embedded awk/other-language heredocs to the training corpus would address this class of misclassification.