This message was deleted.
# generals
Slackbot
11/03/2023, 4:41 PMThis message was deleted.
l
Luiz Augusto
11/03/2023, 4:44 PMIt doesn’t need an
avroBytesDecoderprotoBytesDecoderb
Bibek Sahoo
11/07/2023, 6:57 AMI believe below article explains the procedure :
https://blog.hellmar-becker.de/2021/10/19/reading-avro-streams-from-confluent-cloud-into-druid/
l
Luiz Augusto
11/07/2023, 10:33 AMHey @Bibek Sahoo, sadly it doesn’t.
(You can also use JSON or Protobuf with Schema Registry, but that’s a story for another time.)@Hellmar Becker mentions it, but I can’t find any example on his blog (or anywhere else) how to ingest JSON with Schema Registry.
h
Hellmar Becker
11/07/2023, 4:54 PM@Luiz Augusto my bad, I actually never tested JSON-SR. I’ll find out
l
Luiz Augusto
11/07/2023, 5:07 PMThanks @Hellmar Becker!
h
Hellmar Becker
11/08/2023, 7:28 AMit’s actually not supported now, I corrected the blogpost
😐 1
c
Clint Wylie
11/08/2023, 7:49 AMno one has been super motivated to implement it because we can’t really ship the jar to make it work out of the box (like protobuf + schema-registry actually, though someone was motivated to implement that one)
Clint Wylie
11/08/2023, 7:50 AMit probably wouldn’t be terribly hard, just make an extension that uses the json-schema-provider library and probably looks pretty similar to avro and protobuf implementations
Clint Wylie
11/08/2023, 7:51 AMi assume could re-use a bunch of stuff from standard json input format
Clint Wylie
11/08/2023, 7:52 AMthe avro implementation is apache licensed and part of regular schema-registry client, but protobuf and json are confluent license
Clint Wylie
11/08/2023, 7:55 AMif anyone is interested in implementing it i’d be happy to guide and review (assuming implement it for real and deserialize json per the schema rather than just ignoring the schema part)
Clint Wylie
11/08/2023, 7:56 AMClint Wylie
11/08/2023, 7:57 AMand https://github.com/apache/druid/pull/10839 has discussion about license stuff, though it implements older parseSpec stuff that isn’t necessary for new stuff, but https://github.com/apache/druid/pull/11018 would be the main thing to model after i think in order to actually implement
l
Luiz Augusto
11/08/2023, 12:15 PMWhat would be the real advantage of using the JSON schema here?
For Avro and Proto it makes sense as it requires deserialization, and It’s not like they are actually building up the full ingest schema from the SR output. I still need to manually (or programmatically) fill the
flattenSpecdimensionsSpec