https://datahubproject.io logo
Join Slack
Powered by
Hi guys , i try to resintall datahub using helm ``...
# troubleshoot
f
Hi guys , i try to resintall datahub using helm
Copy code
ehlm install datahub  datahub/datahub  -f helm_custom_settings_custom_helm.yaml
but i have this error
Copy code
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
i think its a problem with datahub-auth-secrets.yml
e
The above is when setup jobs are failing. Can you post the result of
Copy code
kubectl get pods
and if any of the setup jobs are hanging or failing
get the logs of that pod
Copy code
kubectl logs <<pod-name>>
f
the gms has an error, and i try to reinstall datahub
but i used uninstall so, i lost that pod
also, the server failed yesterday, but i didint do any deploy, so i think its becasuse of datahub-datahub-upgrade-job
I also see this warning on datahub-upgrade-job, but i dont know if it is related or not
Copy code
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.8
e
you see the mysql setup job
is still running
any logs there?
f
kubectl describe pod datahub-mysql-setup-job
Copy code
vents:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  26m   default-scheduler  Successfully assigned default/datahub-mysql-setup-job-qbtrl to gke-datahub-prod-default-pool-3972b874-x5cd
  Normal  Pulled     26m   kubelet            Container image "acryldata/datahub-mysql-setup:v0.8.26.0" already present on machine
  Normal  Created    26m   kubelet            Created container mysql-setup-job
  Normal  Started    26m   kubelet            Started container mysql-setup-job
e
any logs?
f
also, logs from k8s
now its 10:31 for me
so, i think its ok
e
can you run
Copy code
kubectl logs <<pod-name>>
and post the whole thing?
hmn tho frmo above seems like
it’s stuck somewhere
f
kubectl logs datahub-mysql-setup-job-qbtrl
Copy code
2022/02/23 08:03:35 Waiting for: <tcp://prerequisites-mysql:3306>
2022/02/23 08:03:35 Connected to <tcp://prerequisites-mysql:3306>
-- create datahub database
CREATE DATABASE IF NOT EXISTS datahub CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
USE datahub;

-- create metadata aspect table
create table if not exists metadata_aspect_v2 (
  urn                           varchar(500) not null,
  aspect                        varchar(200) not null,
  version                       bigint(20) not null,
  metadata                      longtext not null,
  systemmetadata                longtext,
  createdon                     datetime(6) not null,
  createdby                     varchar(255) not null,
  createdfor                    varchar(255),
  constraint pk_metadata_aspect_v2 primary key (urn,aspect,version)
);

-- create default records for datahub user if not exists
CREATE TABLE temp_metadata_aspect_v2 LIKE metadata_aspect_v2;
INSERT INTO temp_metadata_aspect_v2 (urn, aspect, version, metadata, createdon, createdby) VALUES(
  'urn:li:corpuser:datahub',
  'corpUserInfo',
  0,
  '{"displayName":"Data Hub","active":true,"fullName":"Data Hub","email":"<mailto:datahub@linkedin.com|datahub@linkedin.com>"}',
  now(),
  'urn:li:corpuser:__datahub_system'
), (
  'urn:li:corpuser:datahub',
  'corpUserEditableInfo',
  0,
  '{"skills":[],"teams":[],"pictureLink":"<https://raw.githubusercontent.com/linkedin/datahub/master/datahub-web-react/src/images/default_avatar.png>"}',
  now(),
  'urn:li:corpuser:__datahub_system'
);
-- only add default records if metadata_aspect is empty
INSERT INTO metadata_aspect_v2
SELECT * FROM temp_metadata_aspect_v2
WHERE NOT EXISTS (SELECT * from metadata_aspect_v2);
DROP TABLE temp_metadata_aspect_v2;

-- create metadata index table
CREATE TABLE IF NOT EXISTS metadata_index (
 `id` BIGINT NOT NULL AUTO_INCREMENT,
 `urn` VARCHAR(200) NOT NULL,
 `aspect` VARCHAR(150) NOT NULL,
 `path` VARCHAR(150) NOT NULL,
 `longVal` BIGINT,
 `stringVal` VARCHAR(200),
 `doubleVal` DOUBLE,
 CONSTRAINT id_pk PRIMARY KEY (id),
 INDEX longIndex (`urn`,`aspect`,`path`,`longVal`),
 INDEX stringIndex (`urn`,`aspect`,`path`,`stringVal`),
 INDEX doubleIndex (`urn`,`aspect`,`path`,`doubleVal`)
);
Hmm i. found it prerequisites-mysql
Copy code
2022-02-23 10:33:29.252 EET2022-02-23T08:33:29.252163Z 679418 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000010' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
e
huh wonder why it got full
you can try deleting pvc for the mysql
and reinstalling
f
hmm, i will lose da ingestion data right?
it was the pvc mysql storage. i increase it at 30 gb
and it work, but i still have some error on
s
That is just a warning. You can ignore that.
Open in Slack
PreviousNext
Powered by