https://datahubproject.io logo
Join Slack
Powered by
How stable is the new release, 0.13.0, and is the ...
# all-things-deployment
b
How stable is the new release, 0.13.0, and is the chart updated to support it?
b
its released with helm chart 
0.4.0
plus1 1
l
I tested 
0.4.0
chart today and the deployment seems stable.
thank you 1
f
Ingested data doesn’t seem to show in UI.
b
what type of data did you ingest? and which cli version?
f
Copy code
helm install datahub datahub/datahub --version 0.4.0
The UI ingestion and the cli (v0.13.0) doesn’t bring in the assets.
l
I'll test a ingestion now.
@flat-bear-65100, are you using ElasticSearch or Opensearch?
f
From this:
Copy code
helm install datahub datahub/datahub --version 0.4.0
Also
Copy code
helm install prerequisites datahub/datahub-prerequisites
l
okok.
Facing a smilar issue here
Copy code
warnings': {},
 'failures': {},
 'total_num_files': 1,
 'num_files_completed': 1,
 'files_completed': ['outputs/output.json'],
 'percentage_completion': '0%',
 'estimated_time_to_completion_in_minutes': -1,
 'total_bytes_read_completed_files': 387608,
 'current_file_size': 387608,
 'total_parse_time_in_seconds': 0.02,
 'total_count_time_in_seconds': 0.0,
 'total_deserialize_time_in_seconds': 0.15,
 'start_time': '2024-03-08 15:17:55.380426 (2.11 seconds ago)',
 'running_time': '2.11 seconds'}
Sink (datahub-rest) report:
{'total_records_written': 148,
 'records_written_per_second': 68,
 'warnings': [],
 'failures': [],
 'start_time': '2024-03-08 15:17:55.339672 (2.16 seconds ago)',
 'current_time': '2024-03-08 15:17:57.494660 (now)',
 'total_duration_in_seconds': 2.15,
 'max_threads': 15,
 'gms_version': 'v0.13.0',
 'pending_requests': 0}

 Pipeline finished successfully; produced 148 events in 2.11 seconds.
and the ingested data doesn’t seem to show in UI.
💯 1
b
We had the same as well and I did a rollback so cannot test again 😅 need to setup a test environment before
l
I'm facing the same issue with 
v0.12.1
, but I'm testing a ingestion pipeline that is working in Production for 
0.8.31
😬
Copy code
warnings': {},
 'failures': {},
 'total_num_files': 1,
 'num_files_completed': 1,
 'files_completed': ['outputs/output.json'],
 'percentage_completion': '0%',
 'estimated_time_to_completion_in_minutes': -1,
 'total_bytes_read_completed_files': 387608,
 'current_file_size': 387608,
 'total_parse_time_in_seconds': 0.0,
 'total_count_time_in_seconds': 0.0,
 'total_deserialize_time_in_seconds': 0.06999999999999999,
 'start_time': '2024-03-08 16:08:46.741776 (2.17 seconds ago)',
 'running_time': '2.17 seconds'}
Sink (datahub-rest) report:
{'total_records_written': 148,
 'records_written_per_second': 66,
 'warnings': [],
 'failures': [],
 'start_time': '2024-03-08 16:08:46.685271 (2.23 seconds ago)',
 'current_time': '2024-03-08 16:08:48.915821 (now)',
 'total_duration_in_seconds': 2.23,
 'gms_version': 'v0.12.1',
 'pending_requests': 0}

 Pipeline finished successfully; produced 148 events in 2.17 seconds
I just tested the ingestion pipeline using DataHub's Docker quickstart (version 
0.13.0
) instead of using Minikube and the chart. And it worked. 😕 Regarding the test conducted with the DataHub Helm chart version 
0.4.0
, I changed Elasticsearch to version 
8.X.X
. It could be something related to this change, or perhaps another configuration issue.
f
@lemon-receptionist-90470, I’ll try again today. Thank you!
Which version did you change elasticsearch to (just so we’re in sync)?
l
Worked when I tested with DataHub's Docker Quickstart (which used this docker-compose), so the Elasticsearch version used was 
7.10.1
Not worked when tested with the following Charts: Prerequisites:
Copy code
dependencies:
  - name: elasticsearch
    version: 8.5.1
    repository: <https://helm.elastic.co>
    condition: elasticsearch.enabled
  - name: postgresql
    version: 11.2.6
    repository: <https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami>
    condition: postgresql.enabled
  # This chart deploys a community version of kafka
  - name: kafka
    version: 26.11.2
    repository: <https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami>
    condition: kafka.enabled
Datahub:
Copy code
dependencies:
  - name: datahub
    version: 0.4.0
    repository: "<https://helm.datahubproject.io>"
    condition: datahub.enabled
Elasticsearch with the following configurations:
Copy code
elasticsearch:
    useSSL: "true"
    skipcheck: "true"
    insecure: "true" # Change before go to Production
    auth:
      username: "elastic"
      password:
        secretRef: elasticsearch-master-credentials
        secretKey: password
f
@lemon-receptionist-90470, I think I read that wrong when you posted it. So those configs don’t work?
l
Hey, @flat-bear-65100, what are doubts?
f
You mentioned “Not worked when tested with the following Charts:” so my question is what did work?
r
Elasticsearch 8.x is not currently supported. Please use ES 7.x or an OpenSearch 1.x or 2.x chart for the search datastore.
l
Yeah. It’s working with ES 7.x.
f
ok, I’ll try it, thank you
l
@flat-bear-65100, to clarify: So, first I tried deploying DataHub on Kubernetes using Minikube, along with the DataHub Helm chart (version 
0.4.0
). For this setup, I decided to upgrade Elasticsearch within the 
datahub-prerequisites
to version 
8.x.x
as part of the Helm chart configuration. Unfortunately, this led to some unexpected issues. To validate that the problem is related to ES 
8.x.x
, I tested the my ingestion pipeline with DataHub locally using DataHub's Docker Quickstart (version 
0.13.0
), which avoids the complexity of Kubernetes by running DataHub directly with Docker. This uses Elasticsearch 7.x by default. And it worked, confirming that the problem is related to the Elasticsearch 8.x incompatibility (as @gentle-night-56466 reinforced) . If you want to test Datahub in a simple way, check the following page: https://datahubproject.io/docs/quickstart/ Best.
f
@lemon-receptionist-90470, will that work for the latest version, ie like helm chart: --version 0.4.0?
a
fwiw, I'm brand new to datahub and faced a similar issue on a fresh install of the most recent helm chart (version 0.4.0). reinstalling fresh from chart version 0.3.30 fixed the issue. no other changes were made. if there is anything I can provide anyone to help debug, happy to do so. otherwise just here to thank folks on this thread for helping me feel less crazy.
f
Would be nice to get 0.4.x to work
2 Views
Open in Slack
PreviousNext
Powered by