The calling method's class, org.springframework.beans.factory.xml.XmlBeanDefinitionReader, is available from the following locations:

    jar:file:/Users/dseredenko/.m2/repository/org/springframework/spring-beans/5.3.21/spring-beans-5.3.21.jar!/org/springframework/beans/factory/xml/XmlBeanDefinitionReader.class
    jar:file:/Users/dseredenko/.m2/repository/io/acryl/datahub-client/0.8.44/datahub-client-0.8.44.jar!/org/springframework/beans/factory/xml/XmlBeanDefinitionReader.class

volumes:
    - ../mysql/init.sql:/docker-entrypoint-initdb.d/init.sql

Can't initialize batch_readline - may be the input source is a directory or a block device.

datahub get --urn "urn:li:dataset:(urn:li:dataPlatform:bigquery,project_id.dataset.table,DEV)" --aspect schemaMetadata  | jq >schema.json

datahub ingest -c ingest_schema.dhub.yaml

ingest_schema.dhub.yaml

# see <https://datahubproject.io/docs/generated/ingestion/sources/file> for complete documentation
source:
  type: "file"
  config:
    filename: schema.json

# see <https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub> for complete documentation
sink:
  type: "datahub-rest"
  config:
    server: "<https://gms.topic-dev.name.dev>"

Command failed with com.linkedin.pegasus2avro.usage.UsageAggregation is missing required field: bucket.

UsageAggregation

spark-lineage

'com.linkedin.common.BrowsePaths': {
					'paths': ['/spark/spark_spark-master_7077/pythonhdfsin2hivecreatetable']
				}

/pythonhdfsin2hivecreatetable

'com.linkedin.common.BrowsePaths': {
					'paths': ['/spark/spark_spark-master_7077']
				}

appName

searchAcrossLineage

graph_service_v1

transformers:
  - type: "simple_add_dataset_ownership"
    config:
      owner_urns:
        - "urn:li:corpGroup:edwh"
      ownership_type: "TECHNICAL_OWNER"

[2022-09-12, 07:08:52 UTC] {process_utils.py:173} INFO - [[34m2022-09-12, 07:08:52 UTC[0m] {[34mpipeline.py:[0m57} ERROR[0m -  failed to write record with workunit txform-urn:li:dataPlatform:postgres-edwh.raw_tn_marketing.t_customer-TEST-ownership with ('Unable to emit metadata to DataHub GMS', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:422]: Failed to validate record with class com.linkedin.common.Ownership: ERROR :: /lastModified/actor :: "Provided urn " is invalid\n\n\tat com.linkedin.metadata.resources.entity.AspectResource.lambda$ingestProposal$3(AspectResource.java:149)', 'message': 'Failed to validate record with class com.linkedin.common.Ownership: ERROR :: /lastModified/actor :: "Provided urn " is invalid', 'status': 422}) and info {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'stackTrace': 'com.linkedin.restli.server.RestLiServiceException [HTTP Status:422]: Failed to validate record with class com.linkedin.common.Ownership: ERROR :: /lastModified/actor :: "Provided urn " is invalid\n\n\tat com.linkedin.metadata.resources.entity.AspectResource.lambda$ingestProposal$3(AspectResource.java:149)', 'message': 'Failed to validate record with class com.linkedin.common.Ownership: ERROR :: /lastModified/actor :: "Provided urn " is invalid', 'status': 422}[0m

aws_connection

'1 validation error for DBTConfig\n'
           'aws_connection\n'
           '  extra fields not permitted (type=value_error.extra)\n',

~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': '8cc88fff-6200-465e-9d09-703160d417fc',
 'infos': ['2022-09-12 13:20:11.799891 [exec_id=8cc88fff-6200-465e-9d09-703160d417fc] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-09-12 13:20:11.801144 [exec_id=8cc88fff-6200-465e-9d09-703160d417fc] INFO: Caught exception EXECUTING '
           'task_id=8cc88fff-6200-465e-9d09-703160d417fc, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 121, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 71, in execute\n'
           '    validated_args = SubProcessIngestionTaskArgs.parse_obj(args)\n'
           '  File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj\n'
           '  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__\n'
           'pydantic.error_wrappers.ValidationError: 1 validation error for SubProcessIngestionTaskArgs\n'
           'debug_mode\n'
           '  extra fields not permitted (type=value_error.extra)\n']}
Execution finished with errors.

Result window is too large

log4j.logger.datahub.spark=DEBUG
log4j.logger.datahub.client.rest=DEBUG

Application end event received, but start event missing for appId spark-xxx

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"

PodDisruptionBudget

source:
  type: snowflake
  config:
    account_id: ${DH_SNOWFLAKE_ACCOUNT_ID}
    warehouse: ${DH_SNOWFLAKE_WAREHOUSE}
    username: ${DH_SNOWFLAKE_USER}
    password: ${DH_SNOWFLAKE_PASSWORD}
    role: ${DH_SNOWFLAKE_ROLE}
    include_tables: True
    include_views: True

    ignore_start_time_lineage: true

    stateful_ingestion:
      enabled: True 
      remove_stale_metadata: True 

    profiling:
      enabled: true
    profile_pattern:
      allow:
        - 'DATABASE_NAME.SCHEMA_NAME.*'
    database_pattern:
      allow:
        - "OIP"
    schema_pattern:
      allow:
        - "SCHEMA_NAME"
    table_pattern:
      deny:
       - '.*\._AIRBYTE_.*'

pipeline_name: "snowflake_ingestion"

sink:
  type: datahub-rest
  config:
    server: ${DATAHUB_SERVER}

mlfeatureindex_v2