https://datahubproject.io logo
Join Slack
Powered by
Hey Team, on our deployed datahub we are unable to...
# all-things-deployment
p
Hey Team, on our deployed datahub we are unable to see most popular , recently viewed data, domains on homepage and also weekly active users are not updating on analytics page . What can be the reason (or have any extra configs to enable this) cc @gifted-kite-59905
teamwork 1
e
Hi! How do you deploy datahub? Is it docker or helm? In both cases, you need to set DATAHUB_ANALYTICS_ENABLED to true on both frontend and gms. Also make sure you run elasticsearch-setup-job with the above env set to true
g
Hi @early-lamp-41924 DATAHUB_ANALYTICS_ENABLED=true on both services... we are using AWS ES created template and opendistro policies as written in elastic search setup job. Also MAE and MCE are deployed separately.
e
and the MAE consumer has this set as well?
This is very weird
In your elasticsearch cluster, do you see indices that look like datahub_usage_event-00000 ?
g
Copy code
green open datahubexecutionrequestindex_v2_1648039307579 oBtAP2FlQH2up2t6lvK8YA 1 1   17   0  83.3kb  41.6kb
green open corpgroupindex_v2_1648039304166               njYN6FVAT5K0ihEP6AsVfQ 1 1    1   0  15.8kb   7.9kb
green open chartindex_v2_1648039324565                   HvMqBNt9S-6u_dyklvcZcA 1 1    0   0    416b    208b
green open dataset_datasetprofileaspect_v1               yOD-mk9oQoyg9RPK0AzPBw 1 1   74   0  97.5kb  52.6kb
green open dataprocessindex_v2_1648039305086             pj65ws7QRtqb73Ym7-dkEA 1 1    0   0    416b    208b
green open glossarytermindex_v2_1648039311719            lIdIvomdRUKUUsgW8Cb15Q 1 1    0   0    416b    208b
green open datasetindex_v2_1648039323716                 pDvWNGo7QVyefY7WpIjx8A 1 1  658 228   1.5mb 791.1kb
green open datahubingestionsourceindex_v2_1648039321043  kQbdKnsCRR-4ZkBN8I07cQ 1 1    1   0  10.1kb     5kb
green open domainindex_v2_1648039321912                  LYU23p6GQPG88Rdb6tNATw 1 1    2   0  19.5kb   9.7kb
green open mlmodelgroupindex_v2_1648039306762            _aYy3f98R6yYkGnnG0r1VA 1 1    0   0    416b    208b
green open assertionindex_v2_1648039309245               jhN3fYTjRSOfRWvVWoU9kQ 1 1    0   0    416b    208b
green open datajobindex_v2_1648039308406                 zJ1LIzIFT4WoogA-N30k0g 1 1    0   0    416b    208b
green open datahubsecretindex_v2_1648039322814           cus2kgLdR9yEW6viricRjw 1 1    2   0    16kb  10.3kb
green open datajob_datahubingestioncheckpointaspect_v1   PFDlWWtFSNaftrKKDD8KtA 1 1    0   0    416b    208b
green open assertion_assertionruneventaspect_v1          -R9oL5dXSfWxs9xtYY78hQ 1 1    0   0    416b    208b
green open containerindex_v2_1648039297303               rwKrNx4OQiSM9Frm1Mbr_g 1 1   30  12  60.4kb  38.6kb
green open dataflowindex_v2_1648039320220                T8k23o3KS9KbQGhrzqp7oA 1 1    0   0    416b    208b
green open mlfeatureindex_v2_1648039319358               cr07xyXmRC6doyZqwWKZPw 1 1    0   0    416b    208b
green open glossarynodeindex_v2_1648039318523            znZ3F5BURwOM8UqyRrGY8g 1 1    0   0    416b    208b
green open mlprimarykeyindex_v2_1648039312523            6W-9zXKTRiKWmRdDPl3q-w 1 1    0   0    416b    208b
green open datajob_datahubingestionrunsummaryaspect_v1   KZjnDpmvSd6zezaWdbv-SQ 1 1    0   0    416b    208b
green open schemafieldindex_v2_1648039310054             e11JfrpcRXaUhlV0hAPUqQ 1 1    0   0    416b    208b
green open mlmodeldeploymentindex_v2_1648039314357       I01VfUAMSLG0hYBAfKz8fw 1 1    0   0    416b    208b
green open graph_service_v1                              csiL9NtgQ5Wrq9RY2zZAdg 1 1  786 186 289.5kb 146.6kb
green open dataset_operationaspect_v1                    xC8B1cLrQveUFuVq3hOFJw 1 1    0   0    416b    208b
green open .kibana_1                                     3X3lsK63T8C5hOoz_g2cig 1 1    1   0  10.1kb     5kb
green open datahub_usage_event-000002                    brIV-fuXQliEZxCXdEPawA 5 1 5329   0   4.7mb   2.3mb
green open mlfeaturetableindex_v2_1648039305931          77yFKDPoRa-8LhVQTsk--A 1 1    0   0    416b    208b
green open datahub_usage_event-000001                    eaLqL3iSRZeOC9gPEb9pig 5 1    0   0     2kb     1kb
green open .tasks                                        dnDyxvdxSYWSZ7ifrMQapA 1 1   28   0 107.7kb  53.8kb
green open mlmodelindex_v2_1648039317680                 Jj6V5q-ZT4GkXoTWQbj_QA 1 1    0   0    416b    208b
green open datahubretentionindex_v2_1648039315178        4KtZizcKSM2nuvKC-uNybg 1 1    0   0    416b    208b
green open corpuserindex_v2_1648039316863                i-axriNkR5Wy8eLw11elqQ 1 1    5   2    60kb    30kb
green open notebookindex_v2                              Ohe-EhdLRDeu6D1xxJ4mnQ 1 1    0   0    416b    208b
green open system_metadata_service_v1_1648039325401      SYvurdaoRaiie3YGxy04FQ 1 1 4709   3     1mb 525.8kb
green open tagindex_v2_1648039310909                     uL5IHZrsRDCy89IVl2G2YQ 1 1    1   0  19.8kb   6.3kb
green open dataplatformindex_v2_1648039316028            MJojQBYDR8-ok7T1RRlv0w 1 1    0   0    416b    208b
green open dashboardindex_v2_1648039313336               5CvL80LRRqCL6Xx5Ue5FCQ 1 1    0   0    416b    208b
green open .opendistro-job-scheduler-lock                ybx1UC1xR9SMjcby1n9Qkg 5 1    2   1  29.1kb  18.7kb
green open datahubpolicyindex_v2_1648039303333           stIJ_kSAT1C2frsOZF6vOA 1 1    0   0    416b    208b
green open dataset_datasetusagestatisticsaspect_v1       bpyYF5YoTU--_Jv_R7tDvA 1 1    0   0    416b    208b
yes
e
could you check the documents inside datahub_usage_event-000002 ? Want to see if its an issue with frontend pushing usage events or with gms reading from this index
g
@early-lamp-41924
Copy code
{
"_index": "datahub_usage_event-000002",
"_type": "_doc",
"_id": "LogInEvent_urn%3Ali%3Acorpuser%3Adatahub_1648025919416",
"_score": 1,
"_source": {
"type": "LogInEvent",
"actorUrn": "urn:li:corpuser:datahub",
"timestamp": 1648025919416,
"date": "Wed Mar 23 2022 14:28:39 GMT+0530 (India Standard Time)",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.74 Safari/537.36",
"browserId": "c074907a-f7ce-4011-b67d-c3ba92b75633",
"@timestamp": 1648025919416
}
}
{
"_index": "datahub_usage_event-000002",
"_type": "_doc",
"_id": "PageViewEvent_urn%3Ali%3Acorpuser%3Adatahub_1648025931893",
"_score": 1,
"_source": {
"title": "DataHub",
"url": "https://<frontend endpoint>/",
"path": "/",
"hash": "",
"search": "",
"width": 1920,
"height": 944,
"referrer": "https://<frontend endpoint>/",
"prevPathname": "https://<frontend endpoint>/",
"type": "PageViewEvent",
"actorUrn": "urn:li:corpuser:datahub",
"timestamp": 1648025931893,
"date": "Wed Mar 23 2022 14:28:51 GMT+0530 (India Standard Time)",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.74 Safari/537.36",
"browserId": "c074907a-f7ce-4011-b67d-c3ba92b75633",
"@timestamp": 1648025931893,
"corp_user_name": "DataHub",
"corp_user_username": "datahub"
}
}
e
hmn this is very odd. everything looks good to me
so even if you visit a few entity pages
you don’t see recently viewed and popular modules right?
p
yes
g
@early-lamp-41924 just saw in documentation indexing is like
Copy code
.ds-datahub_usage_event-000002
Is it creating problem ? 2nd do we need to specify USE_AWS_ELASTICSEARCH=true in our gms and frontend too?
e
No. so that is the syntax when you are on latest version of elasticsearch (that uses data streams) instead of opensearch
the above env is only needed for elasticsearch-setup-job
one last thing
could you check the information on the datahub_usage_event-000002 index and see if it has an alias for datahub_usage_event ?
p
Copy code
{
  "datahub_usage_event-000002": {
    "aliases": {
      "datahub_usage_event": {
        "is_write_index": true
      }
    }
  }
}
e
hmn this all looks good
g
that was datahub_usage_event-000002 , datahub_usage_event-000001 is empty
Copy code
"datahub_usage_event-000001": {
"aliases": {
"datahub_usage_event": {
"is_write_index": false
}
}
}
datahub_usage_event-000001 has no documents
e
That should be fine
Hmn running out of ideas here. Do you mind posting your helm values.yaml file? Feel free to obfuscate all endpoints
g
Copy code
"env": [
          {
            "name": "DATASET_ENABLE_SCSI",
            "value": "false"
          },
          {
            "name": "KAFKA_BOOTSTRAP_SERVER",
            "value": "<host>:<port>"
          },
          {
            "name": "KAFKA_SCHEMAREGISTRY_URL",
            "value": "<host>"
          },
          {
            "name": "ELASTICSEARCH_HOST",
            "value": "<http://vpc-xxxxxxxxxxxxx.es.amazonaws.com|vpc-xxxxxxxxxxxxx.es.amazonaws.com>"
          },
          {
            "name": "ELASTICSEARCH_PORT",
            "value": "443"
          },
          {
            "name": "ELASTICSEARCH_USE_SSL",
            "value": "true"
          },
          {
            "name": "GRAPH_SERVICE_IMPL",
            "value": "elasticsearch"
          },
          {
            "name": "JAVA_OPTS",
            "value": "-Xms1g -Xmx1g"
          },
          {
            "name": "MAE_CONSUMER_ENABLED",
            "value": "false"
          },
          {
            "name": "MCE_CONSUMER_ENABLED",
            "value": "false"
          },
          {
            "name": "UI_INGESTION_ENABLED",
            "value": "true"
          },
          {
            "name": "UI_INGESTION_DEFAULT_CLI_VERSION",
            "value": "0.8.26.6"
          },
          {
            "name": "METADATA_AUDIT_EVENT_NAME",
            "value": "<xxx>"
          },
          {
            "name": "METADATA_CHANGE_EVENT_NAME",
            "value": "<xxx>"
          },
          {
            "name": "FAILED_METADATA_CHANGE_EVENT_NAME",
            "value": "<xxx>"
          },
          {
            "name": "EBEAN_DATASOURCE_USERNAME",
            "value": "<xxx>"
          },
          {
            "name": "EBEAN_DATASOURCE_PASSWORD",
            "value": "<xxx>"
          },
          {
            "name": "EBEAN_DATASOURCE_HOST",
            "value": "<host>:<port>"
          },
          {
            "name": "EBEAN_DATASOURCE_URL",
            "value": "jdbc:<host>:<port>/<dbname>?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2"
          },
          {
            "name": "EBEAN_DATASOURCE_DRIVER",
            "value": "com.mysql.jdbc.Driver"
          },
          {
            "name": "ENTITY_REGISTRY_CONFIG_PATH",
            "value": "/datahub/datahub-gms/resources/entity-registry.yml"
          },
          {
            "name": "DATAHUB_ANALYTICS_ENABLED",
            "value": "true"
          }
        ],
@early-lamp-41924
e
Hey. Are you using helm or docker to deploy to aws?
If it’s helm, the whole values.yaml file will be helpful
g
We have deployed to kubernetes using docker
e
Could you post the values.yaml you used?
g
@early-lamp-41924 we didn't use helm chart to deploy . have our own deployment setup.
7 Views
Open in Slack
PreviousNext
Powered by