https://datahubproject.io logo
Join Slack
Powered by
Hi, I am trying to ingest some data from an elasti...
# troubleshoot
w
Hi, I am trying to ingest some data from an elasticsearch 7.x cluster. I'm running datahub locally, and ingesting the following recipe:
Copy code
source:
  type: "elasticsearch"
  config:
    # Coordinates
    host: '<https://the_host:9200>'
    # Credentials
    username: the-user
    password: the-pass
    ca_certs: False
    verify_certs: False
    # Options
    # url_prefix: "" # optional url_prefix
    env: "DEV"
    # index_pattern:
      # allow: [".*some_index_name_pattern*"]
      # deny: [".*skip_index_name_pattern*"]
sink:
    type: "datahub-rest"
    config:
        server: "<http://localhost:8080>"
Running the ingestion I get:
Copy code
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py", line 359, in _extract_mcps
    340  def _extract_mcps(self, index: str) -> Iterable[MetadataChangeProposalWrapper]:
 (...)
    355      # 1.1 Generate the schema fields from ES mappings.
    356      index_mappings = raw_index_metadata["mappings"]
    357      index_mappings_json_str: str = json.dumps(index_mappings)
    358      md5_hash = md5(index_mappings_json_str.encode()).hexdigest()
--> 359      schema_fields = list(
    360          ElasticToSchemaFieldConverter.get_schema_fields(index_mappings)
    ..................................................
     self = ElasticsearchSource(ctx=<datahub.ingestion.api.common.PipelineContext object at 0x13026d280>)
     index = '.signals_watches_trigger_state'
     Iterable = typing.Iterable
     MetadataChangeProposalWrapper = <class 'datahub.emitter.mcp.MetadataChangeProposalWrapper'>
     index_mappings = {}
     raw_index_metadata = {'aliases': {},
                           'mappings': {},
                           'settings': {'index': {...}}}
     index_mappings_json_str = '{}'
     json.dumps = <function 'dumps' __init__.py:183>
     md5_hash = '99914b932bd37a50b983c5e7c90ae93b'
    ..................................................

File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py", line 158, in get_schema_fields
    152  def get_schema_fields(
    153      cls, elastic_mappings: Dict[str, Any]
    154  ) -> Generator[SchemaField, None, None]:
    155      converter = cls()
    156      properties = elastic_mappings.get("properties")
    157      if not properties:
--> 158          raise ValueError(
    159              f"Missing 'properties' in elastic search mappings={json.dumps(elastic_mappings)}!"
    ..................................................
     cls = <class 'datahub.ingestion.source.elastic_search.ElasticToSchemaFieldConverter'>
     elastic_mappings = {}
     Dict = typing.Dict
     Any = typing.Any
     Generator = typing.Generator
     SchemaField = <class 'datahub.metadata.schema_classes.SchemaFieldClass'>
     converter = <datahub.ingestion.source.elastic_search.ElasticToSchemaFieldConverter object at 0x134858940>
     properties = None
    ..................................................

---- (full traceback above) ----
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/entrypoints.py", line 149, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
File "/Users/x/.pyenv/versions/forter-3.8.12/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/telemetry/telemetry.py", line 317, in wrapper
    raise e
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/telemetry/telemetry.py", line 269, in wrapper
    res = func(*args, **kwargs)
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/utilities/memory_leak_detector.py", line 102, in wrapper
    res = func(*args, **kwargs)
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/cli/ingest_cli.py", line 128, in run
    raise e
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/cli/ingest_cli.py", line 114, in run
    pipeline.run()
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/run/pipeline.py", line 214, in run
    for wu in itertools.islice(
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py", line 308, in get_workunits
    for mcp in self._extract_mcps(index):
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py", line 359, in _extract_mcps
    schema_fields = list(
File "/Users/x/Development/data-hub/datahub/metadata-ingestion/src/datahub/ingestion/source/elastic_search.py", line 158, in get_schema_fields
    raise ValueError(

ValueError: Missing 'properties' in elastic search mappings={}!
[2022-05-09 17:25:49,344] INFO     {datahub.entrypoints:176} - DataHub CLI version: 0.0.0.dev0 at /Users/cristicalugaru/Development/data-hub/datahub/metadata-ingestion/src/datahub/__init__.py
[2022-05-09 17:25:49,344] INFO     {datahub.entrypoints:179} - Python version: 3.8.12 (default, Jan 31 2022, 11:27:11)
[Clang 13.0.0 (clang-1300.0.27.3)] at /Users/cristicalugaru/.pyenv/versions/my-env/bin/python on macOS-12.0.1-arm64-arm-64bit
[2022-05-09 17:25:49,344] INFO     {datahub.entrypoints:182} - GMS config {'models': {}, 'versions': {'linkedin/datahub': {'version': 'v0.8.34', 'commit': '9422578e419a30231bdb83bd5f4cd42607781942'}}, 'managedIngestion': {'defaultCliVersion': '0.8.34.1', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'noCode': 'true'}
I see some indexes ingested but not the main ones:
Copy code
[2022-05-09 17:25:49,375] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.searchguard_resource_owner
[2022-05-09 17:25:49,424] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.searchguard_resource_owner
[2022-05-09 17:25:49,808] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.kibana-event-log-7.13.2
[2022-05-09 17:25:49,820] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.kibana-event-log-7.13.2
[2022-05-09 17:25:49,835] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.kibana-event-log-7.13.2
[2022-05-09 17:25:49,888] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.ds-ilm-history-5-2022.03.10-000001
[2022-05-09 17:25:49,911] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.ds-ilm-history-5-2022.03.10-000001
[2022-05-09 17:25:49,926] INFO     {datahub.ingestion.run.pipeline:103} - sink wrote workunit index-.ds-ilm-history-5-2022.03.10-000001
Any idea if I'm doing something wrong here?
h
Hi @worried-motherboard-80036, do the indices you are seeing errors on have the 
properties
defined under the 
mappings
?
w
@helpful-optician-78938 they do. For example, on one of the indexes, I do: 
GET /important-index-2022-04/_mapping
Copy code
{
  "important-index-2022-04" : {
    "mappings" : {
      "properties" : {
        "Prop1" : {
          "type" : "float"
        },
        "Prop2" : {
          "type" : "boolean"
        },
.....
h
Could you verify it on the index from the log snippet above (
index = '.signals_watches_trigger_state'
)?
w
ok I take that back - for that index it does not:
GET /.signals_watches_trigger_state/_mapping
Copy code
{
  ".signals_watches_trigger_state": {
    "mappings": {}
  }
}
not even sure what type of index this is - but should it not process the others that have mappings? Is it failing fast?
since that index starts with a . (dot) presumably it's an internal index?
@helpful-optician-78938 any idea about this - I just want to understand what type of indexes does it support when ingesting? Some of my indexes have the properties defined, but I'm not seeing them ingested: e.g.
Copy code
{
  "analytics-sessions-2022-04" : {
    "mappings" : {
      "properties" : {
h
Hi @worried-motherboard-80036, just merged PR #4853 that addresses this issue. If you build from latest master, this would unblock you.
w
thanks a lot, will check it out and get back to you!
So I tried this again, ingestion succeeds this time: 
Sink (datahub-rest) report:
{'records_written': 92,
'warnings': [],
'failures': [],
'downstream_start_time': datetime.datetime(2022, 5, 11, 22, 42, 58, 874823),
'downstream_end_time': datetime.datetime(2022, 5, 11, 22, 43, 2, 779106),
'downstream_total_latency_in_seconds': 3.904283,
'gms_version': 'v0.8.34'}
But when I go to the UI to check on what was ingested, I select one of the indexes and get:
h
There was probably some bad data sitting from previous ingestions. Could you do 
datahub delete -p elasticsearch --hard
and rerun ingestion and see if this error goes away?
w
I did remove everything with the nuke option. Anyway, trying what you say:
Copy code
→ datahub delete -p elasticsearch --hard
No ~/.datahubenv file found, generating one for you...
This will permanently delete data from DataHub. Do you want to continue? [y/N]: y
[2022-05-12 10:01:03,205] INFO     {datahub.cli.delete_cli:209} - datahub configured with <http://localhost:8080>
[2022-05-12 10:01:03,360] INFO     {datahub.cli.delete_cli:222} - Filter matched 3 entities. Sample: ['urn:li:dataset:(urn:li:dataPlatform:elasticsearch,analytics-sessions-2022-03,DEV)', 'urn:li:dataset:(urn:li:dataPlatform:elasticsearch,analytics-sessions-2022-04,DEV)', 'urn:li:dataset:(urn:li:dataPlatform:elasticsearch,sessions,DEV)']
This will delete 3 entities. Are you sure? [y/N]: y
100% (3 of 3) |#############################################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Took 17.25 seconds to hard delete 20 rows for 3 entities
Re-ingesting:
Copy code
Sink (datahub-rest) report:
{'records_written': 11,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 5, 12, 10, 1, 26, 190526),
 'downstream_end_time': datetime.datetime(2022, 5, 12, 10, 1, 26, 948399),
 'downstream_total_latency_in_seconds': 0.757873,
 'gms_version': 'v0.8.34'}
Going to the UI, I see the indexes and when I click them I get the same 
Validation error of type MissingFieldArgument: Missing field argument resource @ 'dataset/usageStats'
4 Views
Open in Slack
PreviousNext
Powered by