[]byte-kind shortcut overrides element CustomEncoder/CustomDecoder for named uint8 slices

[gdamore]

When a slice's element type has kind uint8 but implements CustomEncoder/CustomDecoder (or Marshaler/Unmarshaler), the library still treats the slice as a binary blob (bin) instead of honoring the element's custom codec.

The decision is made from the element kind alone, before checking whether the element type provides a custom codec. A scalar value of the same type encodes correctly (custom codec is consulted), so the inconsistency is easy to miss until a slice field silently serializes wrong.

I'd argue an explicit CustomEncoder/CustomDecoder on the element type should take precedence over the kind-based []byte shortcut.

Minimal repro:

package main

import (
	"fmt"
	"github.com/vmihailenco/msgpack/v5"
)

// A named uint8 type that wants to encode as a string.
type Role uint8

func (r Role) EncodeMsgpack(enc *msgpack.Encoder) error  { return enc.EncodeString("BACKEND") }
func (r *Role) DecodeMsgpack(dec *msgpack.Decoder) error { _, err := dec.DecodeString(); return err }

func main() {
	// Scalar: custom codec IS used -> msgpack string.
	b, _ := msgpack.Marshal(Role(1))
	fmt.Printf("scalar Role   -> % x\n", b) // a7 42 41 43 4b 45 4e 44  ("BACKEND")

	// Slice: custom codec is BYPASSED -> packed as bin blob.
	b, _ = msgpack.Marshal([]Role{1, 2})
	fmt.Printf("[]Role        -> % x\n", b) // c4 02 01 02   (bin, raw bytes — wrong)

	// And decode of an array-of-strings into []Role fails:
	src, _ := msgpack.Marshal([]string{"FOO", "BAR"})
	var out []Role
	err := msgpack.Unmarshal(src, &out)
	fmt.Printf("decode []Role <- [\"FOO\",\"BAR\"]: err=%v\n", err)
}

Output:

scalar Role   -> a7 42 41 43 4b 45 4e 44
[]Role        -> c4 02 01 02
decode []Role <- ["FOO","BAR"]: err=msgpack: invalid code=91 decoding string/bytes length

Expected: []Role encodes as a 2-element array of strings (matching the scalar codec), and decoding an array-of-strings into []Role succeeds.

Version: v5.4.1, Go 1.26.

Workaround: widen the element's underlying type off uint8 (e.g. uint16) so the []byte shortcut no longer matches; the element codec is then used.