https://datahubproject.io logo
Join Slack
Powered by
After running ingestion I have the following messa...
# ingestion
s
After running ingestion I have the following message. I wrote a single table in my recipe so this is as expected
Copy code
Sink (datahub-rest) report:
{'failures': [], 'records_written': 1, 'warnings': []}
But when I go to the UI there is no dataset. Is there supposed to be some delay? Should I check the errors of some service. If yes, which one?
e
Can you see gms logs to see if it did wrote to the search index?
s
It does say this
Copy code
05:26:07.692 [qtp544724190-12] INFO c.l.m.r.entity.EntityResource - INGEST urn urn:li:dataset:(urn:li:dataPlatform:postgres,superset.public.logs,PROD)
05:26:07.739 [pool-9-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=ingest - ingest - 200 - 50ms
which means the request did come to gms
Is there supposed to be something else here too?
I tried it again after removing gms and adding it again and got these logs
Copy code
2021-07-16 08:08:24.076:INFO:oejs.AbstractConnector:main: Started ServerConnector@5fdef03a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2021-07-16 08:08:24.077:INFO:oejs.Server:main: Started @32328ms
08:12:13.533 [qtp544724190-15] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
{}
08:12:13.575 [qtp544724190-15] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
{}
08:12:13.586 [qtp544724190-15] INFO c.l.r.s.c.ResourceMethodConfigProviderImpl - RestLi MethodLevel Configuration for property timeoutMs sorted by priority - first match gets applied:
*.* = 0
{}
08:12:13.720 [qtp544724190-15] INFO c.l.m.r.entity.EntityResource - INGEST urn urn:li:dataset:(urn:li:dataPlatform:postgres,superset.public.logs,PROD)
08:12:14.277 [qtp544724190-15] INFO c.l.parseq.TaskDescriptorFactory - No provider found for TaskDescriptor, falling back to DefaultTaskDescriptor
08:12:14.451 [pool-9-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=ingest - ingest - 200 - 738ms
After that there were no logs On the console just like before it said
Copy code
Sink (datahub-rest) report:
{'failures': [], 'records_written': 1, 'warnings': []}
There does not seem to be any errors
Copy code
health status index                                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   datahub_dataflowindex_v2                          UOgGqhKLQK6J2jbDlw9x7Q   1   1          0            0       416b           208b
green  open   datahub_chartindex_v2                             JMwIB7ZDROaxn0T3-YtOZg   1   1          0            0       416b           208b
green  open   datahub_tagdocument                               sh_ZSW0sQTm95YTgBqQsuw   1   1          0            0       416b           208b
green  open   datahub_mlmodeldocument                           nnDqceQxRvWpGI2M5dwhpA   1   1          0            0       416b           208b
green  open   datahub_dashboarddocument                         TeVcALh-QDCerAy6f4wc1g   1   1          0            0       416b           208b
green  open   datahub_usagestats_v1                             6KsJkG5QRv2Dwd7ntEYwdg   1   1          0            0       416b           208b
green  open   datahub_glossaryterminfodocument                  e6RdV1wzQ9CfE2SiQ_CwpA   1   1          0            0       416b           208b
green  open   datahub_glossarytermindex_v2                      HAAqJuIBRA6EBm8oswd6jg   1   1          0            0       416b           208b
green  open   datahub_mlmodelindex_v2                           1Bna5h8wSbKGx3_irK_d1A   1   1          0            0       416b           208b
green  open   datahub_corpuserinfodocument                      R7xJfGsXT4SiUAlb722ZtQ   1   1          0            0       416b           208b
green  open   datahub_corpuserindex_v2                          DvoHJNqGRJ6XseYnJLKzhw   1   1          0            0       416b           208b
green  open   datahub_dataflowdocument                          B7QHiT4hQqyWsubSgbPwNQ   1   1          0            0       416b           208b
green  open   datahub_corpgroupindex_v2                         6iPSH5LcQ8eziQopJiEHKQ   1   1          0            0       416b           208b
green  open   datahub_chartdocument                             I36rig3JSH6WwqcEM98aug   1   1          0            0       416b           208b
green  open   datahub_datasetindex_v2                           ZtnfBu4pQuCI51agk8mUNg   1   1          0            0       416b           208b
green  open   datahub_mlfeatureindex_v2                         YZZ3NBplT8iVHpoedeuIMg   1   1          0            0       416b           208b
green  open   datahub_datasetdocument                           Y2_SUto5QeWfAmoDr2bFLg   1   1          0            0       416b           208b
green  open   datahub_dataprocessdocument                       y_Z-zaDDQGi1sh-W7S4jPg   1   1          0            0       416b           208b
green  open   datahub_mlprimarykeyindex_v2                      pZHmEUlDS2yvoqnZUhEH9A   1   1          0            0       416b           208b
green  open   datahub_glossarynodeinfodocument                  8L7necASRVuKjyfuzOGiJQ   1   1          0            0       416b           208b
green  open   .ds-datahub_datahub_usage_event-2021.07.15-000001 oHDGZdGUTxOvN23tgTDogA   1   1        172            0    350.1kb          187kb
green  open   datahub_glossarynodeindex_v2                      408DOxnkQ0GXh7MGCBUK_Q   1   1          0            0       416b           208b
green  open   datahub_datajobindex_v2                           LFpSvY9gQIu97OOipjwTvw   1   1          0            0       416b           208b
green  open   datahub_mlfeaturetableindex_v2_1626152292856      ODENvYpiTpym3MXNz1Tz2w   1   1          0            0       416b           208b
green  open   datahub_tagindex_v2                               n2SZ64JWQZq2VLDWIFXiyA   1   1          0            0       416b           208b
green  open   datahub_dataplatformindex_v2                      DdWiMyovRMSriA-UKRDLcQ   1   1          0            0       416b           208b
green  open   datahub_datajobdocument                           ozi-r240SbS8jOiPFhNxkA   1   1          0            0       416b           208b
green  open   datahub_dataprocessindex_v2                       n0cxyrRKRvixfe55WHIgmQ   1   1          0            0       416b           208b
green  open   datahub_graph_service_v1                          5gwrVveiQfWkf9WOnDwbmw   1   1          0            0       416b           208b
green  open   datahub_dashboardindex_v2                         fRHwJT2PTmaya2zGXmBr0g   1   1          0            0       416b           208b
Elasticsearch shows
This means that it is able to connect to elasticsearch as usage statistics are being shown. But it seems like it did not write the ingestion in elasticsearch
Does it need the schema registry. It might be some issue with connection to that. My global block is like this.
Copy code
global:
  graph_service_impl: elasticsearch
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: true

  kafka:
    bootstrap:
      server: "{{ kafka_bootstrap_server_url }}"
    schemaregistry:
      url: "https://{{ kafka_schema_registry_key_id }}:{{ kafka_schema_registry_key_secret }}@{{ kafka_schema_registry_url }}"


  elasticsearch:
    host: "elasticsearch-master.apps.svc.cluster.local"
    port: "9200"
    indexPrefix: "datahub"

  sql:
    datasource:
      host: "{{ sql_host }}:3306"
      hostForMysqlClient: "{{ sql_host }}"
      port: "3306"
      url: "jdbc:mysql://{{ sql_host }}:3306/{{ sql_database }}?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2"
      driver: "com.mysql.jdbc.Driver"
      username: "{{sql_user}}"
      password:
        secretRef: datahub-sql-secret
        secretKey: token

  datahub:
    gms:
      port: "8080"
    mae_consumer:
      port: "9091"
    appVersion: "1.0"

  springKafkaConfigurationOverrides:
    security.protocol: "SASL_SSL"
    sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ kafka_key_id }}' password='{{ kafka_key_secret }}';"
    sasl.mechanism: "PLAIN"
I removed the api keys from bootstrap server but not schema registry because don't have a way to pass them So it is possible that I am unable to connect to schema registry and it is failing silently. Any suggestions?
@big-carpet-38439 So basically there is no error when running on K8s in 
gms
logs but nothing shows up in UI. I am not sure what to look out. I think I saw a schema registry error once related to avro serialization but could not reproduce it. So that might be the problem. Not sure
b
Yeah this could be the issue -- hopefully it would not be sailing silently but it could be happening within the Kafka Producer
Can you check Kafka Topic called "MetadataAuditEvent_v4" to see if any events have been published?
Confluent should show
s
There are no new logs in 
datahub-mce-consumer
or 
datahub-datahub-mae-consumer
Nothing in that topic either
b
So yes it appears that the write to Kafka is not going through then
Which does seem related to SchemaRegistry, I'd think
Because the UsageLogs are coming through right?
s
Yes DataHubUsageEvent_v1 have logs and analytics is showing
b
(in DataHubUsageEvent_v1)
yep
okay interesting
im guessing you've already tried removing "https" from schema registry config?
s
I did. There was some error related to missing protocol malformed URL
b
Yes it does appear to require it... Looking at the producer code
So located inside of your container
s
Are the schema versions being stored in schema registry? I am still not sure why that is here.
b
At /tmp/datahub/logs/gms there will be a debug log file. Do you mind downloading and sending that over?
(Inside your container)
If you are still running on docker compose
s
ok give me a minute
b
you can use
Copy code
docker exec --privileged <container-id> <shell-command>
For example
Copy code
docker exec --privileged 12345 cat /tmp/datahub/logs/gms/date.debug.log > debug.log
That will ensure that you are attempting to fire off an MAR
MAE
s
DMed you the logs. At the end it seems there is something for data ingestion. But there isn't anything for MAE event.
Is something else needs to be running?
b
Yes I’d recommend using an Avro consumer using Kafka cat or something similar! What are you seeing in terms of issues?
a
Is there any CLI or utility to check what was published to this topic ?
s
thank you 1
2 Views
Open in Slack
PreviousNext
Powered by