Hello, I have a question, once I ingest metadata, ...
# ingestion
m
Hello, I have a question, once I ingest metadata, but accidentally shut down the schema-Registry container, but I didn't notice, and then continued to start the DataHub client to ingest metadata, resulting in an exception. However, when I started the schema-Registry container again, I took the same YML file again. There was no exception in the client of DataHub, nor in GMS, but there was no data source information on Datahub. When I modify the instance name, the ingested data source information can appear. I want to know how can I ingest metadata normally without modifying the yml file
i
Hello Yuhui, I’m trying to understand your situation here. 1.) During ingestion you killed the schema registry container and ingestion failed with exceptions. 2.) After re-launching the schema registry no exceptions happened during ingestion but no data showed in the UI. Is my interpretation correct?
m
I just reproduced this problem. I correct it. This is how it happened: after I installed the container of the datahub, I haven't ingested metadata for the first time. I closed the schema registry container, resulting in metadata ingestion failure. Then the following exception was reported, and the client also showed failure. But when I started metadata ingestion for the second time, the client showed normal, the datahub frontend react shows no data information
I conducted another test. After installing the datahub container, I successfully performed metadata ingestion first, and then closed the schema registry container. Miraculously, when I performed metadata ingestion of other data sources again, it succeeded, and the datahub frontend react showed the data source information. I guess the schema information will be loaded into memory after the metadata is ingested, so the subsequent task is successful
i
Schema registry must never be closed, it is meant to be a long-running service along side other DataHub containers. Why are you closing it?
m
It's not that I want to close it, it's that I've had an unexpected shutdown for an unknown reason. I need to simulate what happens when the container closes unexpectedly, and how do I handle it. For example, whether it can be used normally after the restart
i
It should be usable normally after the restart
m
But the reality is currently abnormal. In the case of the correction I mentioned earlier
i
My apologies Yuhui but I’m not following.
If the schema registry shuts down DataHub will not function correctly if it needs to use the schema registry, that is what you saw.
m
That's all right. I just wanted to ask if anyone had encountered this anomaly before