jolly-florist-66728
12/04/2024, 4:29 AMs3://<my-minio-bucket>/<user-specific-prefix>/...
% pyflyte run --remote hello_world.py my_wf
Running Execution on Remote.
FlyteValueException: USER:ValueError: error=Value error! Received: 403. Request to send data
https://<my-minio-endpoint>/<my-minio-bucket>/flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE%3D%3D%3D%3D%3D%3D/fastd57844519531b1
161d2506c45b0fecd5.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<minio-username>%2F20241204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241204T
040812Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-meta-flytecontentmd5&X-Amz-Signature=d5fa6dadb90f2e5a0d73f37630e98ae6b1994d4a9e4ac760387
38ecd6e11cef0 failed.
Response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access
Denied.</Message><Key>flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE======/fastd57844519531b1161d2506c45b0fecd5.tar.gz</Key><BucketName>my-minio-bucket</BucketName><Resource>/<my-minio-bucket>/flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE======/fastd57844519531b1161d2506c45b0fecd5.tar
.gz</Resource><RequestId>180DDDCA8CF19855</RequestId><HostId>414fdc917451eb973643c2ecd05e6a7702dd399ed76ed0fbea4e978999873dac</HostId></Error>
is there any way to force Flyte to write to s3://<my-minio-bucket/<user-specific-prefix>/...
? I haven't had any luck injecting the prefix that our minIO store requires us to use.
# Answer
To resolve the issue of Flyte not writing to the specified MinIO bucket with a user-specific prefix, you need to configure the raw_output_data_config
in Flyte to include the desired prefix. According to GitHub Issue #4615, you can update the raw output data configuration using the flytectl update workflow-execution-config
command. This allows you to specify an output location prefix, which should be set to your required MinIO path.
Here's an example configuration:
domain: development
project: flytesnacks
raw_output_data_config:
output_location_prefix: s3://<my-minio-bucket>/<user-specific-prefix>
jolly-florist-66728
12/04/2024, 12:03 AMint
as argument to the workflow, but seeing TypeTransformerFailedError
.
$ pyflyte run remote-workflow <look_upservice> --var1 24
TypeTransformerFailedError: Python value cannot be None, expected <class 'int'>/Flyte Serialized object (LiteralType):
simple: 1
# Answer
The error you're encountering, TypeTransformerFailedError
, suggests that Flyte is expecting an integer value but is receiving None
instead. This could be due to a mismatch in the expected input type or a missing input value when invoking the workflow.
To resolve this issue, ensure that:
1. The workflow is defined to accept an integer input, and the input is correctly passed when invoking the workflow.
2. Check if the input variable var1
is correctly defined and passed as an integer in the command line.
3. Verify that the workflow and task definitions correctly specify the input types, and there are no discrepancies between the expected and provided types.
If the issue persists, could you provide more details about the workflow definition and how the input is being passed? This will help in diagnosing the problem more accurately.
Sources:
- Flyte GitHub Issue #4505
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1733270597.178749 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.quaint-spring-40580
12/03/2024, 7:17 PMjolly-florist-66728
12/03/2024, 6:37 PMmap_task
failed out with a very long error message (it hit a SQL error on a large insert), but the containing map_task
(Array Node
on the UI) and the overarching workflow were just stuck in a Running
state until I manually went in and terminated the workflow.
I'm guessing the cause of this issue was the size of the error message, but is there something I should be looking for in the logs to confirm this? Should Flyte be handling this sort of situation better (assuming it is the issue that I'm guessing)?
# Answer
The issue you're experiencing with the workflow being stuck in a "Running" state due to a large error message is a known problem in Flyte. According to Flyte Issue #1613, large error messages can cause failures in FlytePropeller, leading to workflows not progressing as expected. This is because the error message size can exceed the gRPC message size limit, causing the system to fail to update the workflow status properly.
To confirm if this is the issue, you should check the FlytePropeller logs for messages indicating a ResourceExhausted
error or similar, which would suggest that the error message size exceeded the allowed limit. Flyte should ideally handle such situations by truncating error messages to prevent them from exceeding size limits, as discussed in Flyte Issue #5224.
Would you like more detailed information on how to address this issue or how Flyte handles large error messages?
Sources:
- Flyte Issue #1613
- Flyte Issue #5224
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1733251055.872239 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
12/03/2024, 12:32 PMjolly-florist-66728
12/03/2024, 11:08 AM~/.flyte/local-cache/
, and changing this location is not currently configurable according to the Flyte documentation. However, a workaround using symbolic links to redirect the cache directory to another location might be possible.
For organizing directories per workflow with readable object names, Flyte does not provide explicit support for this in the available data sources. You might consider implementing a custom solution to manage cache organization externally. Would you like more details on potential workarounds or custom implementations?
Sources:
- Flyte documentation
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1733224110.006009 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
12/03/2024, 9:23 AMEcho
type has been added? Which flytekit version (and backend) is required for this? Some of my workflows could be simplified with this
# Answer
It seems likely that the Echo
task type was introduced in Flytekit version 1.13.6 and is enabled by default in Flyte 1.13.2. This task type allows for simple echo operations, where inputs are directly passed to outputs without additional processing. This feature can simplify workflows by eliminating the need for placeholder tasks in conditional branches.
Would you like more detail on how to implement or use the Echo
task in your workflows?
Sources:
- What's new in Flyte 1.13.2 and Flytekit 1.13.6
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1733217807.163399 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
12/02/2024, 3:33 PMkubectl edit cm flyte-sandbox-config
, and add some spark configuration inside it (based on the config for flyte-sandbox here). The configuration I used is attached below. Then, I use kubectl rollout restart flyte-sandbox
to restart the deployment.
When running the pyspark_pi example in flytesnacks, the workflow status stuck in queuing.
Does anyone know why this happens and how I can solve it?
Thanks a lot for any help!π
# Answer
To resolve the issue of the workflow status being stuck in "queuing" when running the pyspark_pi
example in Flytesnacks, ensure that the Spark configuration is correctly set up in the Flyte sandbox. According to the Flyte documentation, you need to ensure that the Spark operator is properly installed and configured. Additionally, verify that the necessary permissions are granted to the service account used by Flyte to create Spark resources.
Would you like more detailed steps on how to verify and configure these settings?
Sources:
- https://github.com/flyteorg/flyte/pull/3248
- https://discuss.flyte.org/t/15983401/having-some-difficulty-using-the-spark-plugin-for-flyte-in-p/
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1733153547.132949 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/29/2024, 10:32 AMjolly-florist-66728
11/29/2024, 9:57 AM{"json":{"exec_id":"***masked-exec-id***","ns":"***masked-namespace***","res_ver":"***masked-ver***","routine":"worker-2","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]","ts":"2024-11-13T08:12:16Z"}
E1113 08:12:16.842540 1 workers.go:103] error syncing '***masked-namespace***/***masked-exec-id***': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]
{"json":{"exec_id":"***masked-exec-id-2***","ns":"***masked-namespace***","res_ver":"***masked-ver-2***","routine":"worker-3","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]]","ts":"2024-11-13T08:12:42Z"}
{"json":{"exec_id":"***masked-exec-id-2***","ns":"***masked-namespace***","res_ver":"***masked-ver-2***","routine":"worker-3","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]","ts":"2024-11-13T08:12:42Z"}
E1113 08:12:42.070995 1 workers.go:103] error syncing '***masked-namespace***/***masked-exec-id-2***': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]
jolly-florist-66728
11/28/2024, 6:16 PMflytekit
Python SDK, which provides utilities for testing tasks and workflows. For unit testing, you can use the flytekit.testing
module to mock tasks and workflows, ensuring that the sequence of tasks is executed as expected. Integration and end-to-end tests can be set up by running workflows in a controlled environment and verifying the outputs.
For detailed guidance, refer to the Flyte Testing Documentation and the Flytekit Testing API Reference. These resources provide comprehensive instructions and examples for setting up various types of tests.
Would you like more detail on a specific type of test or example code?
Sources:
- Flytekit Testing API Reference
- Flyte Testing Documentation
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732817770.061369 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/28/2024, 4:24 PMjolly-florist-66728
11/28/2024, 3:15 PMvalues.yaml
file. This approach allows you to avoid hardcoding sensitive information directly in the values.yaml
file.
1. Create Kubernetes Secrets: Use kubectl
to create secrets for your OIDC and S3 credentials. For example:
kubectl create secret generic flyte-secrets --from-literal=oidc_client_secret=<OIDC_SECRET> --from-literal=s3_access_key=<S3_ACCESS_KEY> --from-literal=s3_secret_key=<S3_SECRET_KEY>
2. Reference Secrets in Helm: In your values.yaml
, reference these secrets using env
and secretKeyRef
to inject them as environment variables:
configuration:
auth:
clientSecretsExternalSecretRef: flyte-secrets
storage:
stow:
config:
access_key_id:
valueFrom:
secretKeyRef:
name: flyte-secrets
key: s3_access_key
secret_key:
valueFrom:
secretKeyRef:
name: flyte-secrets
key: s3_secret_key
Would you like more detailed guidance on any specific part of this process?
Sources:
- GitHub Pull Request on Flyte Secrets
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732806916.615999 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/28/2024, 12:41 PMmax-cache-age
parameter, which defines how far back in the past the cache can be queried. This is a global setting and not configurable at the task level (source, source).
Would you like more detail on how to configure the global TTL or explore potential workarounds?
Sources:
- Flyte Community Discussion on TTL
- Flyte Community Discussion on Caching
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732797663.363709 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/28/2024, 3:44 AMjolly-florist-66728
11/27/2024, 7:58 PMmyproject.models.alabama
, myproject.models.california
, myproject.models.texas
. Each one has the same API.
How would you all create tasks/workflows for each of these models, without having to copy/paste the same code for each model?
I've tried things like making a function/task factory, but Flyte keeps telling me ValueError: TaskFunction cannot be a nested/inner or local function.
. Using importlib to import-by-string works okay, but then pyflyte run --remote
doesn't package the model's code and its dependencies when submitting the workflow to the remote.
# Answer
It seems likely that you can address this issue by organizing your code into a shared library and using Flyte's ImageSpec
to manage dependencies. According to Flyte GitHub Issue #5492, using pyflyte run --remote
requires ensuring that all source files are present in the container running the workflow. You can achieve this by using the --copy-all
flag to include all necessary modules. Alternatively, you can structure your project to use a setup.py
file and install the package during execution.
Would you like more details on how to set up your project structure or manage dependencies with ImageSpec
?
Sources:
- Flyte GitHub Issue #5492
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732737489.392219 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/27/2024, 7:37 PMneptune_init_run
decorator in Flyte is designed to initialize a new Neptune run for each task execution, and there isn't a direct mention in the data sources about sharing a single run ID across multiple tasks. The decorator requires a project and a secret for authentication, but it doesn't explicitly mention a parameter for setting a shared run ID.
My understanding is that you might need to manually manage the run ID if you want multiple tasks to report to the same Neptune run. This could involve initializing the run outside of the tasks and passing the run object or ID to each task. However, this approach isn't explicitly documented in the provided data sources.
Would you like more detail on how to potentially implement this manually?
Sources:
- Neptune plugin example
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732736228.977539 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/27/2024, 6:16 PMpyflyte-execute
but wanted to confirm since I don't think this is documented
# Answer
Retries for user errors in Flyte are configurable and can be applied to tasks, including those using raw containers. According to the Flyte documentation, user errors are typically due to issues in user-defined code and can be retried by setting the retries
attribute in the task decorator. However, for raw containers, the retry mechanism might not behave as expected if the container does not handle exceptions in a way that Flyte can interpret as recoverable or non-recoverable.
Would you like more details on how retries are configured or how they might differ for raw containers?
Sources:
- Flyte Documentation on Retries
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732731386.825959 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/27/2024, 2:40 PMmax_concurrent_execution
parameter that would allow users to specify the maximum number of concurrent executions for a workflow (GitHub Issue #5125).
Would you like more details on potential workarounds or ongoing discussions about this feature?
Sources:
- GitHub Issue #5125
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732718388.864849 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.salmon-flower-36598
11/27/2024, 7:11 AMjolly-florist-66728
11/26/2024, 10:18 PMjolly-florist-66728
11/26/2024, 4:43 PMflytectl get execution -p flytesnacks -d development oeh94k9r2r --details
Thank you!
# Answer
Yes, you can use the FlyteRemote API to fetch execution details similar to the flytectl
command. Here's a basic example:
from flytekit.remote import FlyteRemote
from flytekit.configuration import Config
# Initialize FlyteRemote
remote = FlyteRemote(
config=Config.for_endpoint(endpoint="flyte.example.net"),
default_project="flytesnacks",
default_domain="development",
)
# Fetch execution
execution = remote.fetch_execution(name="oeh94k9r2r", project="flytesnacks", domain="development")
# Sync execution to get node details
synced_execution = remote.sync(execution, sync_nodes=True)
# Access node execution details
node_keys = synced_execution.node_executions.keys()
node_execution_output = synced_execution.node_executions["<node_key>"].outputs
This code snippet uses FlyteRemote
to fetch and sync execution details, including node executions, similar to the flytectl
command with the --details
flag.
Would you like more detail on any specific part of this process?
Sources:
- Inspecting executions
- FlyteRemote API discussion
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732639382.946299 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/26/2024, 8:17 AMTypeError: Object of type FlyteFile is not JSON serializable
I am getting the same error even if I create a JSON file locally first and then pass S3 URL as remote_path. Any help would be appreciated. Thank you
# Answer
The error you're encountering suggests that the FlyteFile
object is not being correctly serialized to JSON. This is likely because FlyteFile
is a specialized type in Flyte that handles file paths and URIs, but it doesn't directly serialize to JSON.
To resolve this, you should ensure that the FlyteFile
is being used correctly within a Flyte task or workflow. When you want to upload a file to S3, you should use the FlyteFile
type to handle the file path and let Flyte manage the upload process. Here's a basic example of how you can use FlyteFile
to upload a file:
from flytekit import task, workflow
from flytekit.types.file import FlyteFile
@task
def upload_file(local_path: str) -> FlyteFile:
# Create a FlyteFile object with the local path
file = FlyteFile(local_path)
# Flyte will handle the upload to the remote location
return file
@workflow
def my_workflow(local_path: str) -> FlyteFile:
return upload_file(local_path=local_path)
Make sure that the local path is correctly specified and that Flyte is configured to use S3 as the backend storage. If you continue to face issues, please provide more details about how you're using the FlyteFile
and any relevant code snippets.
Would you like more details on this process or any other aspect of Flyte?
Sources:
- https://github.com/flyteorg/flyte/issues/5426
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732609011.047779 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.salmon-flower-36598
11/25/2024, 6:06 PMjolly-florist-66728
11/25/2024, 1:28 PMjolly-florist-66728
11/24/2024, 6:55 AMmap_task
can be executed in parallel, leveraging all available cores. However, this parallelism is only fully realized when running on a remote Flyte deployment. Locally, it may appear serial due to the local executor's limitations (source).
2. Generic Type Hinting with Dataclasses: The data sources do not explicitly mention support for using a generic type hint with dataclasses in Flyte tasks. If you're encountering issues, it might be due to Flyte's type system not fully supporting this feature.
Would you like more detail on either of these topics?
Sources:
- Flyte Discussion on Dynamic and Map Tasks
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732431303.895879 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/22/2024, 3:51 PMfsspec
in python at all?
β’ How much of a βfirst-class-citizenβ is copilot? The βraw notesβ in its readme sound a bit like its WIP.
Thank you π
# Answer
It seems likely that when using Flyte Copilot, task inputs and outputs are managed by the copilot itself rather than being directly handled by fsspec
in Python. Flyte Copilot operates in two modes: as a downloader and as a sidecar, managing data transfer and monitoring the main container's execution, as described in the Flyte Copilot README.
Regarding the status of Flyte Copilot as a "first-class citizen," the raw notes in the README suggest that it might still be evolving, with some solutions being proposed and discussed. This could imply that while it is functional, there may be ongoing improvements and refinements.
Would you like more detail on any specific aspect?
Sources:
- Flyte Copilot README
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732290693.373909 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/21/2024, 11:15 AMoverwrite_cache=True
doesn't actually run new tasks and overwrite the cache as expected - instead Flyte tries to recover previous outputs and typically hits an OutputsNotFound
error due to some mismatch between the previous and new run.
We're using FlyteRemote to programatically create these executions as a way of relaunching previously failed executions - we do this instead of clicking 'relaunch' on the Flyte UI so that we can customise the execution name as well as relaunch executions in bulk.
Has anyone else had this issue, and is there perhaps something we're missing about our relaunching setup? I'll add a code snippet to the thread...
# Answer
This issue might be related to how the overwrite_cache
flag is being used. According to the Flyte documentation, the overwrite_cache
flag is intended to invalidate the cache and force re-execution of the task. However, if the task's inputs or configuration haven't changed, Flyte might still attempt to use the cached outputs, leading to the OutputsNotFound
error if those outputs are missing or mismatched.
To troubleshoot, ensure that the overwrite_cache
parameter is correctly set in the FlyteRemote.execute
method. Additionally, verify that the task inputs or configurations are indeed different from the previous execution to ensure that Flyte recognizes the need to re-run the task. If the issue persists, it might be beneficial to check the Flyte logs for any additional error messages or warnings that could provide more context.
Would you like more detailed guidance on setting up the FlyteRemote.execute
method or any other aspect of your relaunching setup?
Sources:
- Flyte documentation
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732187690.142099 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/21/2024, 9:15 AMflytectl
. According to discussions on Flyte's forum, users have faced challenges with callback/redirects and token parsing errors when integrating Cognito. One user resolved a specific issue by switching to using ClientSecret
in the CLI, which worked temporarily.
For flytectl
, it is suggested to use the ClientSecret
flow instead of relying on ID tokens, as mentioned in the Flyte GitHub issue. This involves configuring the client ID and secret in the Flyte configuration and ensuring that the OAuth2 provider's requirements are met.
Would you like more detailed guidance on configuring Flyte with Amazon Cognito?
Sources:
- discussions on Flyte's forum
- Flyte GitHub issue
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732180529.714059 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.jolly-florist-66728
11/21/2024, 7:39 AMmap_task
in Flyte does not support @dynamic
, @eager
, or @workflow
nodes, which includes imperative workflows. This limitation is acknowledged in various discussions and issues within the Flyte community. According to this discussion, map_task
only supports Flyte's PodPlugin tasks, such as Python tasks, container tasks, and pod tasks.
Would you like more details on the current limitations or the upcoming ArrayNode
feature?
Sources:
- Flyte Discussion on Dynamic Workflows
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732174722.731989 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.