A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Config for recipe :
```{'sink':
 {'type': 'datahub-kafka',
  'config': {
  	'connection': {
  		'bootstrap': '&lt;bootstrapserver&gt;:9092',
  		'schema_registry_url': 'https://&lt;schemaregistry url&gt;'
  		}
  },
  'source': {
  	'type': 'glue',
  	'config': {
  		'env': 'PROD',
  		'aws_region': 'us-east-1',
  		'extract_transforms': 'false',
  		'table_pattern':
  		{'allow': ['test.*'], 'ignoreCase': 'false'}
  		}
  	},
  'transformers': [{'type': 'simple_remove_dataset_ownership', 'config': {}}, {'type': 'simple_add_dataset_ownership', 'config': {'owner_urns': ['urn:li:corpuser:user1']}}, {'type': 'set_dataset_browse_path', 'config': {'path_templates': ['/Platform/PLATFORM/DATASET_PARTS']}}]}```

Output to curl "<https://schemaregistryurl/subjects>" -k -v
["MetadataChangeLog_Versioned_v1-value","MetadataAuditEvent_v4-value"]* Closing connection 0

Made some progress here, now  getting
```datahub.configuration.common.PipelineExecutionError: ('Sink reported errors', SinkReport(records_written=0, warnings=[], failures=[KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}], downstream_start_time=None, downstream_end_time=None, downstream_total_latency_in_seconds=None))
[2022-03-15 16:05:46,123] {local_task_job.py:146} INFO - Task exited with return code 1```

This typically happens if the broker is down. Can you check if the broker is up or if it ran into any issues(from the logs)?

<@U028L1V9BE1> - I dont see any errors in the logs. Trying to check if the broker is up.

<@U028L1V9BE1> - Getting this -
```botocore.exceptions.ClientError: An error occurred (ServiceUnavailableException) when calling the GetTables operation (reached max retries: 4): 
[2022-03-21 17:54:13,625] {taskinstance.py:1532} INFO - Marking task as UP_FOR_RETRY. dag_id=metadata_push_ingestion_dag, task_id=ingest_metadata, execution_date=20220321T175240, start_date=20220321T175242, end_date=20220321T175413
[2022-03-21 17:54:13,676] {local_task_job.py:146} INFO - Task exited with return code 1```