Hi,
Need some information for deployment with kub...
# all-things-deploymentm
mammoth-fountain-32989
06/01/2022, 1:48 PMHi,
Need some information for deployment with kubernetes.
Any guidance on how to estimate the resources required in terms of CPU, memory and disk space with such deployment.
Planning to use repository as Postgresql outside the containers to persist the data.
Remaining services will be with the containers.
Please help with some pointers on how to estimate the resource requirement. Will have few hundreds of datasets to start with and will scale up if needed.
Thanks.
e
early-lamp-41924
06/01/2022, 4:43 PMHi. If you are planning on running elasticsearch within the kubernetes cluster, you will need at least 3 nodes with 2 CPUs and 8 GB memory each. Note elasticsearch uses a ton of resources unfortunately
b
bright-receptionist-94235
06/01/2022, 8:37 PMHi
how do I define Datahub to work with exists external mysql instance?
e
early-lamp-41924
06/01/2022, 8:44 PMAre you using docker for deploy or kubernetes?
m
mammoth-fountain-32989
06/02/2022, 3:35 AMI already have an instance of elastic search running outside kubernetes, how to configure the deployment to use it. and how to configure the external PostgresDB as well. And with this setup, any pointer on the capacity requirements given that DB and elasticsearch are outside. Thanks
mammoth-fountain-32989
06/02/2022, 8:31 AM@early-lamp-41924
e
early-lamp-41924
06/02/2022, 5:11 PMPlease refer to this doc on the fields to update! https://datahubproject.io/docs/deploy/aws#use-aws-managed-services-for-the-storage-layer Although it is for AWS, most of the fields that needs to get updated is the same
early-lamp-41924
06/02/2022, 5:16 PMFor postgres, you need to update host and port to the correct link (you can omit hostForMysqlClient). url should be “jdbcpostgresql//<<host>>:<<port>>/datahub . driver should be “org.postgresql.Driver”
early-lamp-41924
06/02/2022, 5:16 PMYou will want to disable mysqlSetupJob and enable postgresSetupJob as well
early-lamp-41924
06/02/2022, 6:42 PMYou also need to set hostForpostgresqlClient (note the capitalization is important, need to fix this) analogous to hostForMysqlClient (so host:port)
b
bright-receptionist-94235
06/03/2022, 12:41 PMYes I want to run on k8s, I will check the documentation, thx