Good afternoon everyone, so I am facing a problem that is quite strange. The past week I did a test ingestion to reproduce how a "normal" ingestion in an production environment would be (I tried ingesting 40 tables, with one of them over 250 000 rows). The thing is that after that I have been getting some errors and the problem seemed to be that my kafka was storing a lot of info. I looked up and saw the following:

512     /var/volumes/kafka-broker/var/lib/kafka/data3/cleaner-offset-checkpoint
21K     /var/volumes/kafka-broker/var/lib/kafka/data3/connect_config-0
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-11
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-14
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-17
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-2
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-20
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-23
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-5
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_offset-8
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/connect_status-2
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-0
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-1
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-10
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-11
1.5M    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-12
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-13
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-14
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-15
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-16
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-17
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-18
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-19
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-2
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-20
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-21
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-22
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-23
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-24
4.5M    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-25
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-26
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-27
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-28
12K     /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-29
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-3
8.0M    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-30
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-31
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-32
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-33
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-34
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-35
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-36
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-37
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-38
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-39
1.6M    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-4
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-40
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-41
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-42
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-43
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-44
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-45
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-46
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-47
1.6M    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-48
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-49
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-5
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-6
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-7
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-8
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/__consumer_offsets-9
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/DataHubUsageEvent_v1-0
512     /var/volumes/kafka-broker/var/lib/kafka/data3/log-start-offset-checkpoint
1.4M    /var/volumes/kafka-broker/var/lib/kafka/data3/MetadataAuditEvent_v4-0
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/MetadataChangeEvent_v4-0
583K    /var/volumes/kafka-broker/var/lib/kafka/data3/MetadataChangeLog_Timeseries_v1-0
14G     /var/volumes/kafka-broker/var/lib/kafka/data3/MetadataChangeLog_Versioned_v1-0
5.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/MetadataChangeProposal_v1-0
512     /var/volumes/kafka-broker/var/lib/kafka/data3/meta.properties
2.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/recovery-point-offset-checkpoint
2.0K    /var/volumes/kafka-broker/var/lib/kafka/data3/replication-offset-checkpoint
493K    /var/volumes/kafka-broker/var/lib/kafka/data3/_schemas-0

Is it normal to have so much info under the topic

MetadataChangeLog_Versioned_v1

? Can something be done to control the size of the topics or erase part of the information stored in them? Note: Datahub is only used by me and no one else. This is the first time this has happened to me and I have been using Datahub for quite a few months (with small ingestions, with max of hundreds of rows) The version that I am using is the 0.8.45

Hi Pablo, is this problem still persisting for you?

Hi Paul, no this problem hasn't appear again in the same executions done using v0.9.0. Thanks!