Traceback (most recent call last): File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 126, in _dispatch_execute outputs = task_def.dispatch_execute(ctx, idl_input_literals) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 800, in dispatch_execute literals_map, native_outputs_as_map = run_sync( File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 93, in run_sync return self._runner_map[name].run(coro) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 72, in run res = fut.result(None) File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 655, in _output_to_literal_map raise e File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1299, in async_to_literal lv = transformer.to_literal(ctx, python_val, python_type, expected) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/types/directory/types.py", line 502, in to_literal ctx.file_access.put_data(source_path, remote_directory, is_multipart=True, batch_size=batch_size) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 593, in put_data raise FlyteUploadDataException( Message: FlyteUploadDataException: SYSTEMUploadDataError error=Failed to convert outputs of task 'bert-fine-tune.download_dataset' at position 0. Failed to convert type <class 'pathlib.PosixPath'> to type <class 'flytekit.types.directory.types.FlyteDirectory'>. Error Message: Failed to put data from /tmp/flyte-ed1a6zqr/sandbox/local_flytekit/a4bbd6b3fd6241ad67f02f37ba794cbf/dataset_cache to union:///uq/lowrystcol/development/flytesnacks/aglkh9mczm9xrw4xhwvt-n0-3/c14f82ba574e70a47e3859292e37dba1 (recursive=True). Original exception: This event loop is already running., cause=This event loop is already running

@stocky-art-46888 it sounds like you may be having the same issue?

that was for download_dataset. This is for download_model Traceback (most recent call last): File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 126, in _dispatch_execute outputs = task_def.dispatch_execute(ctx, idl_input_literals) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 800, in dispatch_execute literals_map, native_outputs_as_map = run_sync( File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 93, in run_sync return self._runner_map[name].run(coro) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 72, in run res = fut.result(None) File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 655, in _output_to_literal_map raise e File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1299, in async_to_literal lv = transformer.to_literal(ctx, python_val, python_type, expected) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/types/directory/types.py", line 502, in to_literal ctx.file_access.put_data(source_path, remote_directory, is_multipart=True, batch_size=batch_size) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 593, in put_data raise FlyteUploadDataException( Message: FlyteUploadDataException: SYSTEMUploadDataError error=Failed to convert outputs of task 'bert-fine-tune.download_model' at position 0. Failed to convert type <class 'pathlib.PosixPath'> to type <class 'flytekit.types.directory.types.FlyteDirectory'>. Error Message: Failed to put data from /tmp/flyte-228ab61n/sandbox/local_flytekit/dac9b973e3f082851dd2cf1f15c45517/model_cache to union:///bg/lowrystcol/development/flytesnacks/afhgqdx24fjkhpfpbw27-n1-0/cd1f94633afd359437b90a55ff936cdf (recursive=True). Original exception: This event loop is already running., cause=This event loop is already running

Pinging the team! Thank you for providing the error messages!

image.png

In the first task it's able to download the dataset, while uploading to cache_dir may be data type of dataset path and cache dir path is not matching Failed to convert type <class 'pathlib.PosixPath'> to type <class 'flytekit.types.directory.types.FlyteDirectory'>.

image.png

Failed to put data from /tmp/flyte-w_orrry9/sandbox/local_flytekit/c1fdd953a87872a0398b3584bca94e76/dataset_cache to union:///no/kiran73code/development/flytesnacks/afpdxwp5bcpk7xf5jhxd-n0-0/b8689538d22c70a4ab7ff8a3beec7187 (recursive=True).

The FlyteDirectory class takes a path to try to upload to (s3, GCS, etc.) and returns that remote path. I guess there are some issues while uploading this data. Error Message: Failed to put data from /tmp/flyte-7r28p3sc/sandbox/local_flytekit/4caa1910fce7d3d52380f87cc9598b27/dataset_cache to union:///05/kiran73code/development/flytesnacks/axpt9vsfws5zndvt2b8f-n0-0/7a233f69f804ff16ed9da163620753fb (recursive=True).

It looks like it may be a image build issue getting a different version of Flytekit than expected. Since I had my image already built its probably why it wasn't affected on my end.

fix?

Engineering is looking at it! If you want, you could try upgrading your version of the

pip install union==0.1.83

not 100% sure if it will fix,

k, I'll try

let me try..

Traceback (most recent call last): File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 126, in _dispatch_execute outputs = task_def.dispatch_execute(ctx, idl_input_literals) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 800, in dispatch_execute literals_map, native_outputs_as_map = run_sync( File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 93, in run_sync return self._runner_map[name].run(coro) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 72, in run res = fut.result(None) File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/opt/micromamba/envs/runtime/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/base_task.py", line 655, in _output_to_literal_map raise e File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1299, in async_to_literal lv = transformer.to_literal(ctx, python_val, python_type, expected) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/types/directory/types.py", line 502, in to_literal ctx.file_access.put_data(source_path, remote_directory, is_multipart=True, batch_size=batch_size) File "/opt/micromamba/envs/runtime/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 593, in put_data raise FlyteUploadDataException( Message: FlyteUploadDataException: SYSTEMUploadDataError error=Failed to convert outputs of task 'bert-fine-tune.download_dataset' at position 0. Failed to convert type <class 'pathlib.PosixPath'> to type <class 'flytekit.types.directory.types.FlyteDirectory'>. Error Message: Failed to put data from /tmp/flyte-jrh86p9u/sandbox/local_flytekit/4cb72183fa34f61963756793a543249b/dataset_cache to union:///mc/lowrystcol/development/flytesnacks/aqxd5gd6mtsd6gp4bwnq-n0-0/fbd03fe5c73a8be19b643e1eeb61f640 (recursive=True). Original exception: This event loop is already running., cause=This event loop is already running

image.png

Ah, sorry i did say pip install, but I was wrong and meant in the ImageSpec try upgrading. This should trigger a new image build. Still not 100% sure if it will fix. If it doesn't I'll get back to you when engineering has an answer

image = ImageSpec(
    packages=[
        "union==0.1.83",
        "transformers==4.39.1",
        "datasets==2.18.0",
        "matplotlib==3.8.3",
        "torch==2.0.1",
        "accelerate==0.27.2",
        "scikit-learn==1.5.1"
    ],
)

testing on my end now too actually do I can let you know. Since i didn't have that built I should probably have the same issue if it persists.

The downloading dataset and model worked on my end after the new image build and overwriting cache so I know they did write. If you do try it on your end let me know if you still get the same issue, since we're still troubleshooting what caused it in the first place.

looks good so far

Thank you both for sharing the error messages! It's great for us to find these bugs!

Thank you for the tutorial.

My Outputs don't show the eval.

I just saw that on my end too. Already tagged an engineer on that too 😂

The predict sentiment has that too. Something with how Dicts are being handled there.

I think this is something to do with union serverless pulling in a newer version of the Flytekit sdk than it should. (beta) I should probably pin the version expected in ImageSpec for the tutorial. really should have had pinned versions of everything, but hand't run into that before for Flytekit, I just re-ran everything and this should work more reliably for everyone!

image = ImageSpec(
    packages=[
        "union==0.1.85",
        "flytekit==1.13.8",
        "transformers==4.39.1",
        "datasets==2.18.0",
        "matplotlib==3.8.3",
        "torch==2.0.1",
        "accelerate==0.27.2",
        "scikit-learn==1.5.1"
    ],
)

Looks good. Thank you.

archived the channel