Hi! Please, Could someone supply me with an example of how to ingest feast data? I'm very confuse with this: Note: Feast ingestion requires Docker to be installed. Extracts: • List of feature tables (modeled as `MLFeatureTable`s), features (`MLFeature`s), and entities (`MLPrimaryKey`s) • Column types associated with each feature and entity Note: this uses a separate Docker container to extract Feast's metadata into a JSON file, which is then parsed to DataHub's native objects. This was done because of a dependency conflict in the 

feast

 module.

source:
  type: feast
  config:
    core_url: localhost:6565 # default
    env: "PROD" # Optional, default is "PROD"
    use_local_build: False # Whether to build Feast ingestion image locally, default is False

Thanks

Hi Henrry! did you already try to run this recipe directly^^^

@chilly-holiday-80781 can you please take a look?

Hi @freezing-teacher-87574! To elaborate – our requirement here is that you need to have Docker running because our Feast connector spawns a Docker container to handle the ingestion. If you have Docker installed and running, all you need to do is point the recipe to your Feast core URL – no need to set

use_local_build

or anything fancy.

Hi 👋 I'm trying to run a feast recipe. My feature store is in a bucket in S3 that I'm setting in "core_url":

# see <https://datahubproject.io/docs/metadata-ingestion/source_docs/feast> for complete documentation
source:
  type: feast
  config:
    core_url: <s3://featurestore/dev/feature-store-dev.db> 
    env: "DEV" # Optional, default is "PROD"
    use_local_build: False # Whether to build Feast ingestion image locally, default is False

# see <https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub> for complete documentation
sink:
  type: "datahub-rest"
  config:
    server: "<http://xxx.xxx.xxx.xx:8080>"

It's giving me an error in connection with the following trace:

[2021-11-12 12:37:18,158] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1356, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/repo/DataHub/feast/dev/dags/datahub-ingest-dag-feast_dev.py", line 39, in datahub_recipe
    pipeline.run()
  File "/home/airflow/.local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 125, in run
    for wu in self.source.get_workunits():
  File "/home/airflow/.local/lib/python3.8/site-packages/datahub/ingestion/source/feast.py", line 269, in get_workunits
    docker_client.containers.run(
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/models/containers.py", line 840, in run
    raise ContainerError(
docker.errors.ContainerError: Command 'python3 ingest.py --core_url=<s3://featurestore/dev/feature-store-dev.db> --output_path=/out.json' in image 'acryldata/datahub-ingestion-feast-wrapper' returned non-zero exit status 1: b'Traceback (most recent call last):\n  File "/usr/local/lib/python3.8/site-packages/feast/grpc/grpc.py", line 48, in create_grpc_channel\n    grpc.channel_ready_future(channel).result(timeout=timeout)\n  File "/usr/local/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result\n    self._block(timeout)\n  File "/usr/local/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block\n    raise grpc.FutureTimeoutError()\ngrpc.FutureTimeoutError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "ingest.py", line 138, in <module>\n    cli()\n  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__\n    return self.main(*args, **kwargs)\n  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main\n    rv = self.invoke(ctx)\n  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke\n    return callback(*args, **kwargs)\n  File "ingest.py", line 26, in cli\n    tables = client.list_feature_tables()\n  File "/usr/local/lib/python3.8/site-packages/feast/client.py", line 683, in list_feature_tables\n    feature_table_protos = self._core_service.ListFeatureTables(\n  File "/usr/local/lib/python3.8/site-packages/feast/client.py", line 134, in _core_service\n    channel = create_grpc_channel(\n  File "/usr/local/lib/python3.8/site-packages/feast/grpc/grpc.py", line 51, in create_grpc_channel\n    raise ConnectionError(\nConnectionError: Connection timed out while attempting to connect to <s3://featurestore/dev/feature-store-dev.db>\n'
[2021-11-12 12:37:18,161] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=datahub_ingest_using_recipe_feast, task_id=ingest_using_recipe, execution_date=20211112T122950, start_date=20211112T123705, end_date=20211112T123718
[2021-11-12 12:37:18,239] {local_task_job.py:149} INFO - Task exited with return code 1

Does anyone have any idea how to fix it? Thank you

What Feast version are you working with? We’re still working on support for the latest versions

Thanks for the reply @chilly-holiday-80781. The feast version is 0.14.1.