Feature request: Support Confluent Schema Registry in avro

[iqorex]

Use case

Currently powertools only supports raw avro record, we have a case were we have messages in confluent format with magic header in that case deserialization will throw KafkaConsumerDeserializationError.

Solution/User Experience

I think good universal solution would be to have optional option to set how many bytes should be skipped in schemaConfig.

const deserialize = (data: string, schema: string, bytesToSkip?: number) => {
  try {
    const type = avro.parse(schema);
    const buffer = Buffer.from(data, 'base64');

    if (bytesToSkip) return type.fromBuffer(buffer.subarray(bytesToSkip))
    return type.fromBuffer(buffer);
  } catch (error) {
    throw new KafkaConsumerDeserializationError(
      `Failed to deserialize Avro message: ${error}, message: ${data}, schema: ${schema}`
    );
  }

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @iqorex - thanks for opening this issue.

I'll let our expert @leandrodamascena reply on this as I am not the most knowledgeable on Kafka-related topics. Leandro, do you think this makes sense to add?

[Zelys-DFKH]

deserialize() in consumer.ts already computes schemaMetadata and passes it to the protobuf deserializer, which uses schemaMetadata.schemaId to strip a registry header before decoding. The avro branch drops schemaMetadata entirely, so there's no way to skip a header today - that's what you're hitting.

Two framings to handle, not one: Confluent uses a 5-byte header (1 magic byte + 4-byte big-endian schema ID); Glue uses a different, fixed 18-byte header (version byte + compression byte + 16-byte UUID), the same shape across Avro, JSON, and Protobuf. A flat 5-byte strip would corrupt Glue-framed messages instead of just failing on them.

Cleanest fix: pull the registry-framing detection (Confluent numeric ID vs. Glue UUID) into a shared helper that both the avro and protobuf deserializers call, rather than hardcoding a byte count in avro alone.

If this scope works, I'll send a PR: the shared helper, tests for raw, Confluent-framed, and Glue-framed Avro, and a doc note in the Kafka consumer utility page.