A lot of components in the Kafka and hadoop ecosystem assume you are workingwith avro messages. In this talk I will highlight some of the advantages andchallenges of stream processing with protobuf encoded messages, and why itmight be worth using them.
A powerful feature of protobuf messages is its handling of unknown fields, andthe ability to partially parse messages with different schema versions or noschema at all. This helps in stream processing when you don't actually need toknow all message fields in order to process and forward them. |