Skip to content

Python producers ignore inherited Avro content type; Kafka binary marshaller always emits JSON #670

Closed
clemensv/real-time-sources
#1743

Description

@clemensv

Reproducer

Toolchain: xrcg 0.11.0, avrotize 3.9.1, Python.

Generate the public Bluesky contract with an inherited Avro message variant:

{
  "messageid": "Bluesky.Feed.Post",
  "basemessage": "/messagegroups/BlueskyFirehose/messages/Bluesky.Feed.Post",
  "dataschemaformat": "Avro",
  "dataschemauri": "#/schemagroups/BlueskyFirehose.avro/schemas/Bluesky.Feed.Post",
  "envelopemetadata": {
    "datacontenttype": {
      "type": "string",
      "value": "application/vnd.apache.org.avro+avro"
    }
  }
}

The base message uses CloudEvents/1.0, with type, source and subject metadata. The referenced schema is an Avro record. Associate the variant group with a Kafka endpoint and generate:

xrcg generate --style kafkaproducer --language py --projectname bluesky_producer --definitions bluesky.xreg.json --output generated

The generated variant class contains the declared content type in its attributes, but its send method has:

content_type: str = "application/json"
# ...
attributes["datacontenttype"] = content_type

Its binary marshaller ignores content_type:

payload = data.to_byte_array("application/json")

Observed behavior

Executed generated BlueskyFirehoseAvroBinaryEventProducer.send_bluesky_feed_post in CloudEvents binary mode using a generated Post and a recording Kafka producer:

Invocation Kafka content-type Body
Default content_type application/json UTF-8 JSON object
Explicit application/vnd.apache.org.avro+avro application/vnd.apache.org.avro+avro Same UTF-8 JSON object

This affects serialization, not only metadata. An explicit argument does not fix Kafka output. The structured branch also always uses json.loads(x.to_json()), rather than serializing binary Avro and representing it as CloudEvents JSON data_base64.

The inherited classes and their six message methods are generated, and all generated Python compiles. Inheritance itself resolves the declared metadata; the producer template then overrides it.

AMQP generation from the same contract also defaults the binary-Avro variant's content_type to application/json. No AMQP wire claim is made here.

Expected

  • Default the payload content type from effective inherited CloudEvents metadata.
  • Select the payload serializer using that content type.
  • Binary Avro must produce Avro bytes, including when nested in a structured CloudEvent.
  • Preserve the JSON variant's application/json encoding even when its schema is Avro.

No generated-code edits or test workarounds were applied.

Activity

  1. clemensv commented on Nov 13, 2026

    @clemensv
    CollaboratorAuthor

    Implemented in #671. The shared Python default now respects declared/inherited Avro media types, including JSON variants over an Avro schema. The reported application/vnd.apache.org.avro+avro spelling is normalized only for Avrotize serialization; wire metadata retains it. Structured Kafka envelopes now retain datacontenttype as well as data_base64. The regression executes full generated send methods and independently decodes Avro wire bytes; AMQP defaults are also covered. Standard PR CI is running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions