https://flyte.org logo
Join Slack
Powered by
  • j

    jolly-florist-66728

    12/04/2024, 4:29 AM
    New RunLLM response generated: >>>*# Question* hello Flyte community, I'm taking Flyte for a test drive in a bare metal k8s environment with a flyte-binary deployment. I've been able to remotely run some basic hello_world workflows when flyte is configured to use plain AWS S3, but I'm running into problems when I switch over to minIO. our credentials are restricted to target
    s3://<my-minio-bucket>/<user-specific-prefix>/...
    Copy code
    % pyflyte run --remote hello_world.py my_wf                                                         
    Running Execution on Remote.
    FlyteValueException: USER:ValueError: error=Value error!  Received: 403. Request to send data 
    https://<my-minio-endpoint>/<my-minio-bucket>/flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE%3D%3D%3D%3D%3D%3D/fastd57844519531b1
    161d2506c45b0fecd5.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<minio-username>%2F20241204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241204T
    040812Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-meta-flytecontentmd5&X-Amz-Signature=d5fa6dadb90f2e5a0d73f37630e98ae6b1994d4a9e4ac760387
    38ecd6e11cef0 failed.
    Response: <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>AccessDenied</Code><Message>Access 
    Denied.</Message><Key>flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE======/fastd57844519531b1161d2506c45b0fecd5.tar.gz</Key><BucketName>my-minio-bucket</BucketName><Resource>/<my-minio-bucket>/flytesnacks/development/JYCYKEBONJ7A22EHVR3LYNWZQE======/fastd57844519531b1161d2506c45b0fecd5.tar
    .gz</Resource><RequestId>180DDDCA8CF19855</RequestId><HostId>414fdc917451eb973643c2ecd05e6a7702dd399ed76ed0fbea4e978999873dac</HostId></Error>
    is there any way to force Flyte to write to
    s3://<my-minio-bucket/<user-specific-prefix>/...
    ? I haven't had any luck injecting the prefix that our minIO store requires us to use. # Answer To resolve the issue of Flyte not writing to the specified MinIO bucket with a user-specific prefix, you need to configure the
    raw_output_data_config
    in Flyte to include the desired prefix. According to GitHub Issue #4615, you can update the raw output data configuration using the
    flytectl update workflow-execution-config
    command. This allows you to specify an output location prefix, which should be set to your required MinIO path. Here's an example configuration:
    Copy code
    domain: development
    project: flytesnacks
    raw_output_data_config:
      output_location_prefix: s3://<my-minio-bucket>/<user-specific-prefix>
    0
    • 1
    • 1
  • j

    jolly-florist-66728

    12/04/2024, 12:03 AM
    New RunLLM response generated: >>>*# Question* Hi Flyte Community!! I am trying to pass
    int
    as argument to the workflow, but seeing
    TypeTransformerFailedError
    .
    Copy code
    $ pyflyte run remote-workflow <look_upservice> --var1 24
    TypeTransformerFailedError: Python value cannot be None, expected <class 'int'>/Flyte Serialized object (LiteralType):
      simple: 1
    # Answer The error you're encountering,
    TypeTransformerFailedError
    , suggests that Flyte is expecting an integer value but is receiving
    None
    instead. This could be due to a mismatch in the expected input type or a missing input value when invoking the workflow. To resolve this issue, ensure that: 1. The workflow is defined to accept an integer input, and the input is correctly passed when invoking the workflow. 2. Check if the input variable
    var1
    is correctly defined and passed as an integer in the command line. 3. Verify that the workflow and task definitions correctly specify the input types, and there are no discrepancies between the expected and provided types. If the issue persists, could you provide more details about the workflow definition and how the input is being passed? This will help in diagnosing the problem more accurately. Sources: - Flyte GitHub Issue #4505 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733270597.178749 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • q

    quaint-spring-40580

    12/03/2024, 7:17 PM
    @ancient-wolf-19325, I get the following error: β”‚ β”‚ /opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:442 in β”‚ β”‚ _execute_task β”‚ β”‚ β”‚ β”‚ ❱ 442 β”‚ β”‚ _dispatch_execute(ctx, load_task, inputs, output_prefix) β”‚ β”‚ β”‚ β”‚ /opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py:215 in β”‚ β”‚ _dispatch_execute β”‚ β”‚ β”‚ β”‚ ❱ 215 β”‚ ctx.file_access.put_data(ctx.execution_state.engine_dir, output_pr β”‚ β”‚ β”‚ β”‚ /opt/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py:593 β”‚ β”‚ in put_data β”‚ β”‚ β”‚ β”‚ ❱ 593 β”‚ β”‚ β”‚ raise FlyteUploadDataException( β”‚ ╰──────────────────────────────────────────────────────────────────────────────╯ FlyteUploadDataException: SYSTEMUploadDataError error=Failed to put data from /tmp/flytefppywths/local_flytekit/engine_dir to s3:///metadata/propeller/flytesnacks-development-fksdi-c3m2/ n1/data/0 (recursive=True). Original exception: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to., cause=The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
    0
    j
    a
    • 3
    • 2
  • j

    jolly-florist-66728

    12/03/2024, 6:37 PM
    New RunLLM response generated: >>>*# Question* Hi all -- I ran into an issue this morning in our Flyte instance where an underlying (individual) task in a
    map_task
    failed out with a very long error message (it hit a SQL error on a large insert), but the containing
    map_task
    (
    Array Node
    on the UI) and the overarching workflow were just stuck in a
    Running
    state until I manually went in and terminated the workflow. I'm guessing the cause of this issue was the size of the error message, but is there something I should be looking for in the logs to confirm this? Should Flyte be handling this sort of situation better (assuming it is the issue that I'm guessing)? # Answer The issue you're experiencing with the workflow being stuck in a "Running" state due to a large error message is a known problem in Flyte. According to Flyte Issue #1613, large error messages can cause failures in FlytePropeller, leading to workflows not progressing as expected. This is because the error message size can exceed the gRPC message size limit, causing the system to fail to update the workflow status properly. To confirm if this is the issue, you should check the FlytePropeller logs for messages indicating a
    ResourceExhausted
    error or similar, which would suggest that the error message size exceeded the allowed limit. Flyte should ideally handle such situations by truncating error messages to prevent them from exceeding size limits, as discussed in Flyte Issue #5224. Would you like more detailed information on how to address this issue or how Flyte handles large error messages? Sources: - Flyte Issue #1613 - Flyte Issue #5224 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733251055.872239 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    12/03/2024, 12:32 PM
    New RunLLM response generated: >>>*# Question* Hello Flyte people. I am evaluating Flyte. One of the things that is not clear (to me) from the web is how Flyte handles overloaded kubernetes clusters. For example, if the cluster has space for 200 tasks but we have a 2000 runnable tasks, what will happen? # Answer It seems likely that Flyte does not directly handle Kubernetes cluster autoscaling. Instead, Kubernetes itself manages node scaling, and you would need to set up something like a Cluster Autoscaler to handle this. Flyte schedules tasks based on available resources, and if the cluster is overloaded, tasks will remain pending until resources are available. According to this discussion, Flyte does not automatically scale nodes, so manual configuration or external tools are necessary for autoscaling. Would you like more detail on how Flyte manages task scheduling or how to configure Kubernetes for better resource management? Sources: - Flyte discussion on scaling Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733229116.032759 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    12/03/2024, 11:08 AM
    New RunLLM response generated: >>>*# Question* Hey all, Any tips on how to make the local cache persistent and possibly more transparent to users(organize directories per wf with readable object names)? # Answer It seems likely that Flyte's local cache is stored by default in
    ~/.flyte/local-cache/
    , and changing this location is not currently configurable according to the Flyte documentation. However, a workaround using symbolic links to redirect the cache directory to another location might be possible. For organizing directories per workflow with readable object names, Flyte does not provide explicit support for this in the available data sources. You might consider implementing a custom solution to manage cache organization externally. Would you like more details on potential workarounds or custom implementations? Sources: - Flyte documentation Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733224110.006009 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    12/03/2024, 9:23 AM
    New RunLLM response generated: >>>*# Question* Good morning πŸ™‚ I have been looking in the docs and I see an
    Echo
    type has been added? Which flytekit version (and backend) is required for this? Some of my workflows could be simplified with this # Answer It seems likely that the
    Echo
    task type was introduced in Flytekit version 1.13.6 and is enabled by default in Flyte 1.13.2. This task type allows for simple echo operations, where inputs are directly passed to outputs without additional processing. This feature can simplify workflows by eliminating the need for placeholder tasks in conditional branches. Would you like more detail on how to implement or use the
    Echo
    task in your workflows? Sources: - What's new in Flyte 1.13.2 and Flytekit 1.13.6 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733217807.163399 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    12/02/2024, 3:33 PM
    New RunLLM response generated: >>>*# Question* Hi everyone~ I am new in Flyte and currently struggle on setting up Spark in sandbox (flytectl demo cluster) for local usage. I tried to edit config map with
    kubectl edit cm flyte-sandbox-config
    , and add some spark configuration inside it (based on the config for flyte-sandbox here). The configuration I used is attached below. Then, I use
    kubectl rollout restart flyte-sandbox
    to restart the deployment. When running the pyspark_pi example in flytesnacks, the workflow status stuck in queuing. Does anyone know why this happens and how I can solve it? Thanks a lot for any help!πŸ™ # Answer To resolve the issue of the workflow status being stuck in "queuing" when running the
    pyspark_pi
    example in Flytesnacks, ensure that the Spark configuration is correctly set up in the Flyte sandbox. According to the Flyte documentation, you need to ensure that the Spark operator is properly installed and configured. Additionally, verify that the necessary permissions are granted to the service account used by Flyte to create Spark resources. Would you like more detailed steps on how to verify and configure these settings? Sources: - https://github.com/flyteorg/flyte/pull/3248 - https://discuss.flyte.org/t/15983401/having-some-difficulty-using-the-spark-plugin-for-flyte-in-p/ Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1733153547.132949 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/29/2024, 10:32 AM
    New RunLLM response generated: >>>*# Question* Hello everyone! We've been scaling up our workflows recently and running into an ongoing issue with memory usage in the flyte-binary (v1.13.3), so I've been investigating. As the workflows run, the memory of the flyte-binary pod steadily increases, sometimes exceeding its limits and crashing. I think this is expected, and we can try to increase the memory available to mitigate the crashes. However, I noticed that when the workflow finishes, it looks like the memory isn't released, which means no matter how much memory we allocate to the pod, it will eventually crash. If anyone has any workarounds or fixes for this I'd be grateful. # Answer This issue may be related to a memory leak in the Flyte system. According to this discussion, users have reported similar memory utilization issues with the flyte-binary pod, where memory usage increases over time and is not released, leading to pod eviction. This behavior suggests a potential memory leak. To address this, you can try the following: 1. Increase Memory Limits: Temporarily increase the memory limits for the flyte-binary pod to prevent immediate crashes. 2. Monitor and Debug: Use monitoring tools to track memory usage patterns and identify specific workflows or tasks that may be contributing to the issue. 3. Report and Collaborate: If the issue persists, consider reporting it to the Flyte community or checking for existing issues on GitHub to collaborate on a solution. Would you like more detailed steps on how to implement these changes or further troubleshooting steps? Sources: - Flyte Community Discussion Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732876317.258609 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/29/2024, 9:57 AM
    New RunLLM response generated: >>>*# Question* Flyte propeller got stuck and stopped any execution of workflows with the following error:
    Copy code
    {"json":{"exec_id":"***masked-exec-id***","ns":"***masked-namespace***","res_ver":"***masked-ver***","routine":"worker-2","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]","ts":"2024-11-13T08:12:16Z"}
    E1113 08:12:16.842540       1 workers.go:103] error syncing '***masked-namespace***/***masked-exec-id***': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]
    {"json":{"exec_id":"***masked-exec-id-2***","ns":"***masked-namespace***","res_ver":"***masked-ver-2***","routine":"worker-3","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]]","ts":"2024-11-13T08:12:42Z"}
    {"json":{"exec_id":"***masked-exec-id-2***","ns":"***masked-namespace***","res_ver":"***masked-ver-2***","routine":"worker-3","wf":"***masked-workflow-id***:***masked-workflow-id***:map_task.my_map_workflow"},"level":"error","msg":"Error when trying to reconcile workflow. Error [[]]. Error Type[*errors.WorkflowErrorWithCause]","ts":"2024-11-13T08:12:42Z"}
    E1113 08:12:42.070995       1 workers.go:103] error syncing '***masked-namespace***/***masked-exec-id-2***': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = DeadlineExceeded desc = context deadline exceeded]
    0
    • 1
    • 1
  • j

    jolly-florist-66728

    11/28/2024, 6:16 PM
    New RunLLM response generated: >>>*# Question* Hi Team I want to write a test cases(unit test, integration and e2e) for my flyte workflows is there any rules to follow to write test cases for flyte workflows any documentaion will helpful # Answer To write test cases for Flyte workflows, you can utilize the
    flytekit
    Python SDK, which provides utilities for testing tasks and workflows. For unit testing, you can use the
    flytekit.testing
    module to mock tasks and workflows, ensuring that the sequence of tasks is executed as expected. Integration and end-to-end tests can be set up by running workflows in a controlled environment and verifying the outputs. For detailed guidance, refer to the Flyte Testing Documentation and the Flytekit Testing API Reference. These resources provide comprehensive instructions and examples for setting up various types of tests. Would you like more detail on a specific type of test or example code? Sources: - Flytekit Testing API Reference - Flyte Testing Documentation Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732817770.061369 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/28/2024, 4:24 PM
    New RunLLM response generated: >>>*# Question* Hi everyone, I’m exploring Flyte's caching mechanism and was wondering if there’s a way to configure a time-to-live (TTL) for the cache. Specifically, I’d like to set the cache to persist for a defined duration. I’d really appreciate any ideas or suggestions! Thank you! # Answer Currently, Flyte does not support setting a TTL for cache entries on a per-task basis. The TTL is configured globally in the FlytePropeller configuration, which means it applies to all tasks uniformly. This has been a topic of interest, and there are discussions about adding this feature to allow more granular control over cache expiration (source, source). Would you like more details on how to configure the global TTL or any other related information? Sources: - https://discuss.flyte.org/t/15638766/hello-wave-is-there-a-ttl-on-how-long-the-tasks-will-be-cach/ - https://github.com/flyteorg/flyte/issues/3305 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732811047.201809 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/28/2024, 3:15 PM
    New RunLLM response generated: >>>*# Question* Hi everyone! πŸ‘‹ I’m working on setting up a Flyte deployment using helm charts and need to configure OIDC secrets and S3 keys using environment variables instead of including them in the helm values.yaml file. Does anyone have experience or recommendations for achieving this? Any tips, examples, or references to existing documentation would be greatly appreciated! Thanks in advance for your help! πŸ™ # Answer To configure OIDC secrets and S3 keys using environment variables in a Flyte deployment with Helm charts, you can leverage Kubernetes secrets. According to this GitHub pull request, you can create Kubernetes secrets and reference them in your
    values.yaml
    file. This approach allows you to avoid hardcoding sensitive information directly in the
    values.yaml
    file. 1. Create Kubernetes Secrets: Use
    kubectl
    to create secrets for your OIDC and S3 credentials. For example:
    Copy code
    kubectl create secret generic flyte-secrets --from-literal=oidc_client_secret=<OIDC_SECRET> --from-literal=s3_access_key=<S3_ACCESS_KEY> --from-literal=s3_secret_key=<S3_SECRET_KEY>
    2. Reference Secrets in Helm: In your
    values.yaml
    , reference these secrets using
    env
    and
    secretKeyRef
    to inject them as environment variables:
    Copy code
    configuration:
         auth:
           clientSecretsExternalSecretRef: flyte-secrets
         storage:
           stow:
             config:
               access_key_id: 
                 valueFrom:
                   secretKeyRef:
                     name: flyte-secrets
                     key: s3_access_key
               secret_key: 
                 valueFrom:
                   secretKeyRef:
                     name: flyte-secrets
                     key: s3_secret_key
    Would you like more detailed guidance on any specific part of this process? Sources: - GitHub Pull Request on Flyte Secrets Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732806916.615999 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/28/2024, 12:41 PM
    New RunLLM response generated: >>>*# Question* Hi everyone, I’m exploring Flyte's caching mechanism and was wondering if there’s a way to configure a time-to-live (TTL) for the cache. Specifically, I’d like to set the cache to persist for a defined duration. Any insights? Thank you! # Answer Currently, Flyte's caching mechanism does not support setting a TTL on a per-task basis. The TTL is managed globally through the FlytePropeller configuration using the
    max-cache-age
    parameter, which defines how far back in the past the cache can be queried. This is a global setting and not configurable at the task level (source, source). Would you like more detail on how to configure the global TTL or explore potential workarounds? Sources: - Flyte Community Discussion on TTL - Flyte Community Discussion on Caching Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732797663.363709 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/28/2024, 3:44 AM
    New RunLLM response generated: >>>*# Question* I am not sure if this is a right place. Does anyone have a recommendation for Part time DS or any data scientist here available part time to work on text classification and recommendation models and problems. # Answer I'm here to help with questions related to Flyte, an open-source orchestrator for building data and ML pipelines. If you have any questions about Flyte or need assistance with using it for your data science projects, feel free to ask! If your question is about finding a data scientist for part-time work, you might want to try platforms dedicated to job postings or freelance work. Let me know if you have any Flyte-related questions or if you need help with something specific within that scope! Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732765483.475169 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/27/2024, 7:58 PM
    New RunLLM response generated: >>>*# Question* My team has a number of models that train, organized like
    myproject.models.alabama
    ,
    myproject.models.california
    ,
    myproject.models.texas
    . Each one has the same API. How would you all create tasks/workflows for each of these models, without having to copy/paste the same code for each model? I've tried things like making a function/task factory, but Flyte keeps telling me
    ValueError: TaskFunction cannot be a nested/inner or local function.
    . Using importlib to import-by-string works okay, but then
    pyflyte run --remote
    doesn't package the model's code and its dependencies when submitting the workflow to the remote. # Answer It seems likely that you can address this issue by organizing your code into a shared library and using Flyte's
    ImageSpec
    to manage dependencies. According to Flyte GitHub Issue #5492, using
    pyflyte run --remote
    requires ensuring that all source files are present in the container running the workflow. You can achieve this by using the
    --copy-all
    flag to include all necessary modules. Alternatively, you can structure your project to use a
    setup.py
    file and install the package during execution. Would you like more details on how to set up your project structure or manage dependencies with
    ImageSpec
    ? Sources: - Flyte GitHub Issue #5492 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732737489.392219 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/27/2024, 7:37 PM
    New RunLLM response generated: >>>*# Question* Hey all, anyone here used the neptune.ai plugin? in my workflow all my tasks should report to the same run_id, but I dont understand how can I configure that other than putting the id manually as an argument in the init_neptune_run decorator # Answer It seems likely that the
    neptune_init_run
    decorator in Flyte is designed to initialize a new Neptune run for each task execution, and there isn't a direct mention in the data sources about sharing a single run ID across multiple tasks. The decorator requires a project and a secret for authentication, but it doesn't explicitly mention a parameter for setting a shared run ID. My understanding is that you might need to manually manage the run ID if you want multiple tasks to report to the same Neptune run. This could involve initializing the run outside of the tasks and passing the run object or ID to each task. However, this approach isn't explicitly documented in the provided data sources. Would you like more detail on how to potentially implement this manually? Sources: - Neptune plugin example Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732736228.977539 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/27/2024, 6:16 PM
    New RunLLM response generated: >>>*# Question* Are retries for user errors supported for raw containers? The behavior we see seems to suggest no and I can understand since its not using
    pyflyte-execute
    but wanted to confirm since I don't think this is documented # Answer Retries for user errors in Flyte are configurable and can be applied to tasks, including those using raw containers. According to the Flyte documentation, user errors are typically due to issues in user-defined code and can be retried by setting the
    retries
    attribute in the task decorator. However, for raw containers, the retry mechanism might not behave as expected if the container does not handle exceptions in a way that Flyte can interpret as recoverable or non-recoverable. Would you like more details on how retries are configured or how they might differ for raw containers? Sources: - Flyte Documentation on Retries Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732731386.825959 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/27/2024, 2:40 PM
    New RunLLM response generated: >>>*# Question* Is there a way to limit workflow concurrency to 1? In other words, how to prevent a workflow from starting if another instance of the same workflow is already running. # Answer Currently, Flyte does not natively support workflow-level throttling to limit concurrency to one instance of a workflow. However, there is an open feature request to add this capability by introducing a
    max_concurrent_execution
    parameter that would allow users to specify the maximum number of concurrent executions for a workflow (GitHub Issue #5125). Would you like more details on potential workarounds or ongoing discussions about this feature? Sources: - GitHub Issue #5125 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732718388.864849 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • s

    salmon-flower-36598

    11/27/2024, 7:11 AM
    @ancient-wolf-19325 how do I configure logging?
    0
    j
    a
    • 3
    • 6
  • j

    jolly-florist-66728

    11/26/2024, 10:18 PM
    New RunLLM response generated: >>>*# Question* Hi all. We noticed some unexpected behavior with cloud events where details were missing for non-successful phase transitions. It looks like this code block is the cause. Was this an optimization done to reduce load on the DB? # Answer The code block you referenced is part of the Flyte cloud event publisher implementation. It appears that the logic is designed to append additional information only when the workflow execution phase is "SUCCEEDED". This could be an optimization to reduce the load on the database by avoiding unnecessary data retrieval and processing for non-successful phases. However, the specific reasoning for this implementation is not explicitly documented in the provided data sources. Would you like more detail on this? Sources: - https://github.com/flyteorg/flyte/pull/5551 - https://github.com/flyteorg/flyte/pull/4790 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732659476.114889 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/26/2024, 4:43 PM
    New RunLLM response generated: >>>*# Question* Hi All, Is there a way to get this code using FlyteRemote API? We would like to get the same information produced by the following snippet (flytectl) instead using FlyteRemote.
    Copy code
    flytectl get execution -p flytesnacks -d development oeh94k9r2r --details
    Thank you! # Answer Yes, you can use the FlyteRemote API to fetch execution details similar to the
    flytectl
    command. Here's a basic example:
    Copy code
    from flytekit.remote import FlyteRemote
    from flytekit.configuration import Config
    
    # Initialize FlyteRemote
    remote = FlyteRemote(
        config=Config.for_endpoint(endpoint="flyte.example.net"),
        default_project="flytesnacks",
        default_domain="development",
    )
    
    # Fetch execution
    execution = remote.fetch_execution(name="oeh94k9r2r", project="flytesnacks", domain="development")
    
    # Sync execution to get node details
    synced_execution = remote.sync(execution, sync_nodes=True)
    
    # Access node execution details
    node_keys = synced_execution.node_executions.keys()
    node_execution_output = synced_execution.node_executions["<node_key>"].outputs
    This code snippet uses
    FlyteRemote
    to fetch and sync execution details, including node executions, similar to the
    flytectl
    command with the
    --details
    flag. Would you like more detail on any specific part of this process? Sources: - Inspecting executions - FlyteRemote API discussion Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732639382.946299 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/26/2024, 8:17 AM
    New RunLLM response generated: >>>*# Question* Hi All, I am trying to upload a JOSN file as a FlyteFile to remote S3, but getting error -
    TypeError: Object of type FlyteFile is not JSON serializable
    I am getting the same error even if I create a JSON file locally first and then pass S3 URL as remote_path. Any help would be appreciated. Thank you # Answer The error you're encountering suggests that the
    FlyteFile
    object is not being correctly serialized to JSON. This is likely because
    FlyteFile
    is a specialized type in Flyte that handles file paths and URIs, but it doesn't directly serialize to JSON. To resolve this, you should ensure that the
    FlyteFile
    is being used correctly within a Flyte task or workflow. When you want to upload a file to S3, you should use the
    FlyteFile
    type to handle the file path and let Flyte manage the upload process. Here's a basic example of how you can use
    FlyteFile
    to upload a file:
    Copy code
    from flytekit import task, workflow
    from flytekit.types.file import FlyteFile
    
    @task
    def upload_file(local_path: str) -> FlyteFile:
        # Create a FlyteFile object with the local path
        file = FlyteFile(local_path)
        # Flyte will handle the upload to the remote location
        return file
    
    @workflow
    def my_workflow(local_path: str) -> FlyteFile:
        return upload_file(local_path=local_path)
    Make sure that the local path is correctly specified and that Flyte is configured to use S3 as the backend storage. If you continue to face issues, please provide more details about how you're using the
    FlyteFile
    and any relevant code snippets. Would you like more details on this process or any other aspect of Flyte? Sources: - https://github.com/flyteorg/flyte/issues/5426 Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732609011.047779 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • s

    salmon-flower-36598

    11/25/2024, 6:06 PM
    @ancient-wolf-19325, How do I specify the initContainer to run before deploying flyte pod?
    0
    j
    • 2
    • 1
  • j

    jolly-florist-66728

    11/25/2024, 1:28 PM
    New RunLLM response generated: >>>*# Question* Does anyone knows if this note still applicable? https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html#deployment-deployment-cloud-simple its mentioning at start of 2023 πŸ‘€ # Answer I'm not able to find any data sources to answer your question. You might want to specify which note or content from the link you are referring to. If this doesn't help, please reach out to the Flyte support team for further assistance. Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732541281.259199 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/24/2024, 6:55 AM
    New RunLLM response generated: >>>*# Question* Hi all, 2 questions: 1. When running a workflow locally, is there a way to run tasks in parallel (for example in a map_task) or all tasks essentially have to be ran in sequence? It would be very valuable to use all the available cores. 2. When using a dataclass as a task input, is it possible to type hint a generic type e.g BaseDataclass, and then pass a class that inherits from it e.g AdvancedDataclass? I'm encountering some issues with that # Answer 1. Parallel Execution in Local Workflows: When running workflows locally, tasks in a
    map_task
    can be executed in parallel, leveraging all available cores. However, this parallelism is only fully realized when running on a remote Flyte deployment. Locally, it may appear serial due to the local executor's limitations (source). 2. Generic Type Hinting with Dataclasses: The data sources do not explicitly mention support for using a generic type hint with dataclasses in Flyte tasks. If you're encountering issues, it might be due to Flyte's type system not fully supporting this feature. Would you like more detail on either of these topics? Sources: - Flyte Discussion on Dynamic and Map Tasks Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732431303.895879 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/22/2024, 3:51 PM
    New RunLLM response generated: >>>*# Question* I have two question about flyte copilot. β€’ Is my understanding correct that when using copilot, task inputs and outputs are not downloaded/uploaded using
    fsspec
    in python at all? β€’ How much of a β€œfirst-class-citizen” is copilot? The β€œraw notes” in its readme sound a bit like its WIP. Thank you πŸ™ # Answer It seems likely that when using Flyte Copilot, task inputs and outputs are managed by the copilot itself rather than being directly handled by
    fsspec
    in Python. Flyte Copilot operates in two modes: as a downloader and as a sidecar, managing data transfer and monitoring the main container's execution, as described in the Flyte Copilot README. Regarding the status of Flyte Copilot as a "first-class citizen," the raw notes in the README suggest that it might still be evolving, with some solutions being proposed and discussed. This could imply that while it is functional, there may be ongoing improvements and refinements. Would you like more detail on any specific aspect? Sources: - Flyte Copilot README Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732290693.373909 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/21/2024, 11:15 AM
    New RunLLM response generated: >>>*# Question* Hi there! We've been encountering an unusual issue where occasionally creating executions with
    overwrite_cache=True
    doesn't actually run new tasks and overwrite the cache as expected - instead Flyte tries to recover previous outputs and typically hits an
    OutputsNotFound
    error due to some mismatch between the previous and new run. We're using FlyteRemote to programatically create these executions as a way of relaunching previously failed executions - we do this instead of clicking 'relaunch' on the Flyte UI so that we can customise the execution name as well as relaunch executions in bulk. Has anyone else had this issue, and is there perhaps something we're missing about our relaunching setup? I'll add a code snippet to the thread... # Answer This issue might be related to how the
    overwrite_cache
    flag is being used. According to the Flyte documentation, the
    overwrite_cache
    flag is intended to invalidate the cache and force re-execution of the task. However, if the task's inputs or configuration haven't changed, Flyte might still attempt to use the cached outputs, leading to the
    OutputsNotFound
    error if those outputs are missing or mismatched. To troubleshoot, ensure that the
    overwrite_cache
    parameter is correctly set in the
    FlyteRemote.execute
    method. Additionally, verify that the task inputs or configurations are indeed different from the previous execution to ensure that Flyte recognizes the need to re-run the task. If the issue persists, it might be beneficial to check the Flyte logs for any additional error messages or warnings that could provide more context. Would you like more detailed guidance on setting up the
    FlyteRemote.execute
    method or any other aspect of your relaunching setup? Sources: - Flyte documentation Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732187690.142099 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/21/2024, 9:15 AM
    New RunLLM response generated: >>>*# Question* Hi! Has anyone set up Flyte with Amazon Cognito? I have trouble about the oauth client for flytectl. # Answer Setting up Flyte with Amazon Cognito has been attempted by some users, but there are known issues, particularly with OAuth client configurations for
    flytectl
    . According to discussions on Flyte's forum, users have faced challenges with callback/redirects and token parsing errors when integrating Cognito. One user resolved a specific issue by switching to using
    ClientSecret
    in the CLI, which worked temporarily. For
    flytectl
    , it is suggested to use the
    ClientSecret
    flow instead of relying on ID tokens, as mentioned in the Flyte GitHub issue. This involves configuring the client ID and secret in the Flyte configuration and ensuring that the OAuth2 provider's requirements are met. Would you like more detailed guidance on configuring Flyte with Amazon Cognito? Sources: - discussions on Flyte's forum - Flyte GitHub issue Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732180529.714059 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0
  • j

    jolly-florist-66728

    11/21/2024, 7:39 AM
    New RunLLM response generated: >>>*# Question* hey all is there a way to bind maptasks to imperative workflows? # Answer Currently,
    map_task
    in Flyte does not support
    @dynamic
    ,
    @eager
    , or
    @workflow
    nodes, which includes imperative workflows. This limitation is acknowledged in various discussions and issues within the Flyte community. According to this discussion,
    map_task
    only supports Flyte's PodPlugin tasks, such as Python tasks, container tasks, and pod tasks. Would you like more details on the current limitations or the upcoming
    ArrayNode
    feature? Sources: - Flyte Discussion on Dynamic Workflows Slack Link https://flyte.slack.com/archives/CP2HDHKE1/p1732174722.731989 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
    0