Hi I did some bigquery ingestion in bulk and then ingested l DataHub #troubleshoot

Hi , I did some bigquery ingestion in bulk and the...

important-wire-73

04/25/2022, 7:22 AM

Hi , I did some bigquery ingestion in bulk and then ingested looker data. jobs completed. gms logs keep showing bulk requests (ES) processing. I checked elasticsearch there’s no data for dashboards, but when i put url of looker dataset in datahub UI it shows all the data.

important-wire-73

04/25/2022, 7:23 AM

Any suggestion? Where can I check pending process/events?

important-wire-73

04/25/2022, 11:44 AM

jobs are stuck at this offset

Copy code

I have no name!@prerequisites-kafka-0:/opt/bitnami/kafka$ ./bin/kafka-consumer-groups.sh --describe --group generic-mae-consumer-job-client --bootstrap-server localhost:9092

GROUP                           TOPIC                           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                     HOST            CLIENT-ID
generic-mae-consumer-job-client MetadataChangeLog_Timeseries_v1 0          0               1325            1325            consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8
generic-mae-consumer-job-client MetadataChangeLog_Versioned_v1  0          1429276         1526817         97541           consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8

green-football-43791

04/25/2022, 7:25 PM

how are you running datahub?

important-wire-73

04/25/2022, 9:07 PM

on GKE

important-wire-73

04/25/2022, 9:12 PM

everything was running fine earlier and then after some days I did some ingestion of bq-usage and looker then it’s stuck

important-wire-73

04/25/2022, 9:12 PM

gms is repeating these logs

Copy code

21:10:47.843 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:28 - Successfully fed bulk request. Number of events: 2 Took time ms: -1
21:10:48.534 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:48.534 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:49.572 [generic-mae-consumer-job-client-0-C-1] INFO  c.l.m.kafka.hook.UpdateIndicesHook:177 - Here's the relationship types found [OwnedBy]
21:10:49.843 [I/O dispatcher 1] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://elasticsearch-master:9200/_bulk?timeout=1m>] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security."]
21:10:49.844 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:28 - Successfully fed bulk request. Number of events: 2 Took time ms: -1
21:10:51.543 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:51.543 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing

green-football-43791

04/25/2022, 9:12 PM

how is your kafka doing?

green-football-43791

04/25/2022, 9:12 PM

it sounds like kafka may have gone down from that message

green-football-43791

04/25/2022, 9:13 PM

hmm or what about logs from your mae consuemr?

green-football-43791

04/25/2022, 9:13 PM

any logs there?

important-wire-73

04/25/2022, 9:14 PM

kafka pod is running and pod logs are

Copy code

[2022-04-25 20:56:06,864] INFO [GroupCoordinator 0]: Preparing to rebalance group generic-mae-consumer-job-client in state PreparingRebalance with old generation 1233 (__consumer_offsets-30) (reason: Adding new member consumer-generic-mae-consumer-job-client-8-2239ed72-3059-4f64-95a6-b7fa6648b0f6 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 20:57:38,862] INFO [GroupCoordinator 0]: Member[group.instance.id None, member.id consumer-generic-mae-consumer-job-client-8-3f02fe1a-aa1e-4f7f-b245-68827ffe6d5f] in group generic-mae-consumer-job-client has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 20:57:38,863] INFO [GroupCoordinator 0]: Stabilized group generic-mae-consumer-job-client generation 1234 (__consumer_offsets-30) with 2 members (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 20:57:38,865] INFO [GroupCoordinator 0]: Assignment received from leader for group generic-mae-consumer-job-client for generation 1234. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:00:35,975] INFO [GroupCoordinator 0]: Preparing to rebalance group generic-mae-consumer-job-client in state PreparingRebalance with old generation 1234 (__consumer_offsets-30) (reason: Adding new member consumer-generic-mae-consumer-job-client-8-0104f086-ad99-4657-90ad-f25f44fc06a5 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:02:38,986] INFO [GroupCoordinator 0]: Member[group.instance.id None, member.id consumer-generic-mae-consumer-job-client-8-2239ed72-3059-4f64-95a6-b7fa6648b0f6] in group generic-mae-consumer-job-client has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:02:38,988] INFO [GroupCoordinator 0]: Stabilized group generic-mae-consumer-job-client generation 1235 (__consumer_offsets-30) with 2 members (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:02:38,990] INFO [GroupCoordinator 0]: Assignment received from leader for group generic-mae-consumer-job-client for generation 1235. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:06:05,975] INFO [GroupCoordinator 0]: Preparing to rebalance group generic-mae-consumer-job-client in state PreparingRebalance with old generation 1235 (__consumer_offsets-30) (reason: Adding new member consumer-generic-mae-consumer-job-client-8-7b12659a-d954-47ce-838d-dffeafe49053 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:07:39,101] INFO [GroupCoordinator 0]: Member[group.instance.id None, member.id consumer-generic-mae-consumer-job-client-8-0104f086-ad99-4657-90ad-f25f44fc06a5] in group generic-mae-consumer-job-client has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:07:39,102] INFO [GroupCoordinator 0]: Stabilized group generic-mae-consumer-job-client generation 1236 (__consumer_offsets-30) with 2 members (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:07:39,103] INFO [GroupCoordinator 0]: Assignment received from leader for group generic-mae-consumer-job-client for generation 1236. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:10:37,028] INFO [GroupCoordinator 0]: Preparing to rebalance group generic-mae-consumer-job-client in state PreparingRebalance with old generation 1236 (__consumer_offsets-30) (reason: Adding new member consumer-generic-mae-consumer-job-client-8-fac82336-6169-4456-8e65-af3198180d75 with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:12:39,145] INFO [GroupCoordinator 0]: Member[group.instance.id None, member.id consumer-generic-mae-consumer-job-client-8-7b12659a-d954-47ce-838d-dffeafe49053] in group generic-mae-consumer-job-client has left, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:12:39,146] INFO [GroupCoordinator 0]: Stabilized group generic-mae-consumer-job-client generation 1237 (__consumer_offsets-30) with 2 members (kafka.coordinator.group.GroupCoordinator)
[2022-04-25 21:12:39,148] INFO [GroupCoordinator 0]: Assignment received from leader for group generic-mae-consumer-job-client for generation 1237. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)

important-wire-73

04/25/2022, 9:15 PM

hmm or what about logs from your mae consuemr?

I was trying to find it, there’s no pod for it, where can i find this

green-football-43791

04/25/2022, 9:15 PM

oh interesting

green-football-43791

04/25/2022, 9:15 PM

there are 2 options

green-football-43791

04/25/2022, 9:16 PM

you can have the mae consumer independently or it can run inside of gms

green-football-43791

04/25/2022, 9:16 PM

do you know which you opted for?

important-wire-73

04/25/2022, 9:16 PM

i chose the default one (datahub-helm) so i think it’s inside gms

green-football-43791

04/25/2022, 9:16 PM

k then i would be curious to see if you can find anything inside of gms

important-wire-73

04/25/2022, 9:18 PM

any suggestion where should i find anything unusual, logs I have already shared.

green-football-43791

04/25/2022, 9:21 PM

you shared the gms logs?

important-wire-73

04/25/2022, 9:22 PM

yes this one

Copy code

21:10:47.843 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:28 - Successfully fed bulk request. Number of events: 2 Took time ms: -1
21:10:48.534 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:48.534 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:49.572 [generic-mae-consumer-job-client-0-C-1] INFO  c.l.m.kafka.hook.UpdateIndicesHook:177 - Here's the relationship types found [OwnedBy]
21:10:49.843 [I/O dispatcher 1] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://elasticsearch-master:9200/_bulk?timeout=1m>] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security."]
21:10:49.844 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:28 - Successfully fed bulk request. Number of events: 2 Took time ms: -1
21:10:51.543 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing
21:10:51.543 [kafka-coordinator-heartbeat-thread | generic-mae-consumer-job-client] INFO  o.a.k.c.c.i.AbstractCoordinator:1054 - [Consumer clientId=consumer-generic-mae-consumer-job-client-8, groupId=generic-mae-consumer-job-client] Attempt to heartbeat failed since group is rebalancing

important-wire-73

04/25/2022, 9:23 PM

i just checked gms.debug.log in that offset is increasing, How can I check pending events?

important-wire-73

04/25/2022, 9:45 PM

as per gms.debug.log it gets reset to 1429276. which is same as

Copy code

I have no name!@prerequisites-kafka-0:/opt/bitnami/kafka$ ./bin/kafka-consumer-groups.sh --describe --group generic-mae-consumer-job-client --bootstrap-server localhost:9092

GROUP                           TOPIC                           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                     HOST            CLIENT-ID
generic-mae-consumer-job-client MetadataChangeLog_Timeseries_v1 0          0               1325            1325            consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8
generic-mae-consumer-job-client MetadataChangeLog_Versioned_v1  0          1429276         1526817         97541           consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8

Copy code

bash-5.1$ cat  /tmp/datahub/logs/gms/gms.debug.log | grep 1429276
14:32:32 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
14:42:32 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
14:52:33 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:02:33 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:12:33 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:22:33 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:32:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:42:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
15:52:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:02:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:12:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:22:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:32:34 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:42:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
16:52:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:02:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:12:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:22:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:32:35 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:42:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
17:52:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:02:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:12:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:22:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:32:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:42:36 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
18:52:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:02:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:12:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:22:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:32:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:42:37 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
19:52:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:02:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:12:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:22:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:32:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:42:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
20:52:38 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
21:02:39 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
21:12:39 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
21:22:39 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
21:32:39 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
21:42:39 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276

important-wire-73

04/25/2022, 9:57 PM

for e.g. after 1429775 it got reset to 1429276 again and this is happening repeatedly

Copy code

11:50:26 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429775
11:50:26 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Successfully converted Avro MCL to Pegasus MCL. urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.b.c,DEV), key: null
11:50:26 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Invoking MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.b.c,DEV), key: null
11:50:26 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.s.e.ElasticSearchService - Upserting Search document entityName: dataset, document: {"urn":"urn:li:dataset:(urn:li:dataPlatform:bigquery,a.b.c,DEV)","browsePaths":["/dev/bigquery/a/b/c"]}, docId: urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Abigquery%2Ca.b.c%2CDEV%29
11:50:26 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Successfully completed MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.b.c,DEV), key: null
11:51:57 [pool-6-thread-1] DEBUG c.l.m.e.ebean.EbeanEntityService - Invoked listUrns with entityName: dataHubPolicy, start: 0, count: 30
11:52:30 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Got Generic MCL on topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 1429276
11:52:30 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Successfully converted Avro MCL to Pegasus MCL. urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.e.d,DEV), key: null
11:52:30 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Invoking MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.e.d,DEV), key: null
11:52:30 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.s.e.ElasticSearchService - Upserting Search document entityName: dataset, document: {"urn":"urn:li:dataset:(urn:li:dataPlatform:bigquery,a.e.d,DEV)","platform":"urn:li:dataPlatform:bigquery"}, docId: urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Abigquery%2Ca.e.d%2CDEV%29
11:52:30 [generic-mae-consumer-job-client-0-C-1] DEBUG c.l.m.k.MetadataChangeLogProcessor - Successfully completed MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:bigquery,a.e.d,DEV), key: null

important-wire-73

04/26/2022, 3:43 AM

@little-megabyte-1074 could you connect me with someone who can help on this

orange-night-91387

04/26/2022, 3:45 PM

Your Kafka consumer group is stuck in a rebalancing state, the GMS side consumer is unable to heartbeat to the group and so it is unable to update the offset is what it looks like is happening. Have you tried restarting the broker or the client to see if they are able to reconnect properly?

important-wire-73

04/26/2022, 4:00 PM

yes restarted multiple time but same result. after 500 msg it rebalances . (Moved mae consumer to separate pod too)

orange-night-91387

04/26/2022, 4:31 PM

MAE has a 500 or Kafka does?

important-wire-73

04/26/2022, 7:09 PM

Copy code

I have no name!@prerequisites-kafka-0:/opt/bitnami/kafka$ ./bin/kafka-consumer-groups.sh --describe --group generic-mae-consumer-job-client --bootstrap-server localhost:9092

GROUP                           TOPIC                           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                     HOST            CLIENT-ID
generic-mae-consumer-job-client MetadataChangeLog_Timeseries_v1 0          0               1325            1325            consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8
generic-mae-consumer-job-client MetadataChangeLog_Versioned_v1  0          1429276         1526817         97541           consumer-generic-mae-consumer-job-client-8-634ce173-ffbf-47e7-917d-e6203a48b28f /10.84.0.88     consumer-generic-mae-consumer-job-client-8

important-wire-73

04/26/2022, 7:10 PM

MAE reads data in chunks of 500 as per default settings . https://datahubspace.slack.com/archives/C029A3M079U/p1650986262996129?thread_ts=1630349465.042800&cid=C029A3M079U

lemon-cat-72045

11/07/2022, 9:40 AM

Hi @important-wire-73, I am seeing the same issue that the

generic-mae-consumer-job-client

gets stuck in the rebalancing state. Did you find any solution to this problem? Thanks!

Open in Slack

Previous Next