Hugo Evers
01/19/2024, 1:36 PMMlflowModelLoggerDataSet
. The error indicates that the module transformers
does not have the specified attribute LSGDistilBertForSequenceClassification
.
Below is the traceback. Any insights on how to resolve this issue would be greatly appreciated.
/home/ec2-user/PROJECT/.venv/lib/python3.9/site-packages/kedro/io/core.py:193 in load
return self._load()
...
/home/ec2-user/PROJECT/.venv/lib/python3.9/site-packages/kedro_mlflow/io/models/mlflow_model_logger_dataset.py:91 in _load
return self._mlflow_model_module.load_model(model_uri=self.model_uri, **self._load_args)
...
/home/ec2-user/PROJECT/.venv/lib/python3.9/site-packages/mlflow/transformers/init.py:875 in load_model
return _load_model(local_model_path, flavor_config, return_type, device, **kwargs)
...
/home/ec2-user/PROJECT/.venv/lib/python3.9/site-packages/mlflow/transformers/init.py:935 in _load_model
model_instance = getattr(transformers, flavor_config[_PIPELINE_MODEL_TYPE_KEY])
...
/home/ec2-user/PROJECT/.venv/lib/python3.9/site-packages/transformers/utils/import_utils.py:1375 in getattr
raise AttributeError(f"module {self.name} has no attribute {name}")
**Error**: `AttributeError: module transformers has no attribute LSGDistilBertForSequenceClassification`
This error occurred while executing a Kedro pipeline that loads a model from MLflow.
Any guidance on troubleshooting or fixing this issue would be greatly appreciated.
Thank you!Gael Ancel
01/23/2024, 12:02 PMkedro viz build
.
I am running into an error in the regex matching process, any hints on where it could stem from or if anyone ran into the same prompt? Thanks!!George Drakoulas
01/23/2024, 1:43 PMGeorge Drakoulas
01/26/2024, 10:47 AMGeorge Drakoulas
01/29/2024, 10:59 AMAdrien
02/08/2024, 5:46 PMNils Romanus
02/09/2024, 11:00 AMMark Druffel
02/10/2024, 12:05 AMkedro viz run
completes, but doesn't graph my pipeline w/ layers:
"{layer}.{dataset_name}":
type: plsure.datasets.ibis.TableDataset
filepath: data/bronze/{layer}/{dataset_name}/*.parquet
file_format: parquet
table_name: "{layer}_{dataset_name}"
connection: ${_duckdb}
save_args:
materialized: table
When I add the layers to metadata, kedro viz run
fails with the error below:
"{layer}.{dataset_name}":
type: plsure.datasets.ibis.TableDataset
filepath: data/bronze/{layer}/{dataset_name}/*.parquet
file_format: parquet
table_name: "{layer}_{dataset_name}"
connection: ${_duckdb}
save_args:
materialized: table
metadata:
kedro-viz:
layer: {layer}
Error:
...
File "~/kedro_viz/models/flowchart.py", line 238, in create_data_node
return DataNode(
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for DataNode
layer
str type expected (type=type_error.str)
Traceback (most recent call last):
File "~/kedro_viz/launchers/cli.py", line 187, in run
_wait_for(func=_check_viz_up, host=host, port=port)
File "~/kedro_viz/launchers/utils.py", line 58, in _wait_for
raise WaitForException(
kedro_viz.launchers.utils.WaitForException: func: <function _check_viz_up at 0x106898e50>, didn't return True within specified timeout: 60
If I don't use data factory at all, it works:
"raw.media_meas_campaign_info":
type: plsure.datasets.ibis.TableDataset
filepath: data/bronze/raw/media_meas_campaign_info/*.parquet
file_format: parquet
table_name: "raw_media_meas_campaign_info"
connection: ${_duckdb}
save_args:
materialized: table
metadata:
kedro-viz:
layer: raw
Iรฑigo Hidalgo
02/12/2024, 5:14 PMTakieddine Kadiri
02/15/2024, 11:34 AMboot_project
& boot_package
๐ญ
โข You need to write a sort of a Meta pipeline (can be anything from a script to a web app) that consumes your kedro project resources (pipeline runs, config loader, ...) and live inside it ? AbstractKedroBootApp
is the way to go ๐ซ. Your implementation can be declared in kedro settings.py
or directly in CLI : kedro boot run --app <http://path.to|path.to>.your.KedroBootApp <kedro_run_args>
โข You want to develop a production-ready REST API that serve your Kedro pipeline. Kedro FastAPI Server
could help. It offer a declarative way of building REST APIs using FastAPI and Kedro. Checkout the <https://github.com/takikadiri/kedro-boot?tab=readme-ov-file#consuming-kedro-pipeline-through-rest-api%7CFastAPI <-> Kedro objects mapping>. You can also develop your fastapi app explicitely and use one of the two modes cited aboves to integrate it with your kedro project/package
If you are intested, you can :
โข ๐ฆ Download the package with pip install kedro-boot
โข G Visit Kedro Boot github repo, give feedback and โญ if you like it
โข ๐งช Try & Learn with Kedro Boot Examples and give feedback !
โจ This enable using Kedro pipelines in a wide range of use cases, including model serving, data apps (streamlit, dash), statistical simulations, paralell processing of unstructured data, streaming, and more. It also streamline the deployment of your Kedro pipeline into various Data & AI Platforms ๐.
Takieddine & YolanAfaque Ahmad
02/20/2024, 10:53 AMkedro-azureml
plugin and I see that there are performance differences when I run the pipeline the pipeline locally vs on the azureml cluster. I understand that each node spins up a container and this may contribute to additional overhead.
Is there any way to make the pipelines run faster (without throwing more resources). Any optimizations that could be possibly make?Julia Makogon
02/22/2024, 10:20 AMNok Lam Chan
02/23/2024, 12:57 AMmlflow serve
any good? I guess this is question for @Yolan Honorรฉ-Rougรฉ and @Takieddine Kadiri, how much of Mlflow were you using? My guess is mlflow serve
is not good enough thus kedro-boot
has its own FastAPI serving.
btw I was reading https://github.com/Galileo-Galilei/kedro-mlflow-tutorial/ and I think it's a very nice read, especially enjoy how it puts MLOps into perspective.Iรฑigo Hidalgo
03/01/2024, 9:35 AMMark Druffel
03/11/2024, 9:13 PMspark.DeltaTableDataset
on a newer version by installing kedro-datasets w/o dependencies as mentioned here. I'm trying to load my kedro session in databricks w/ line magic:
%load_ext kedro.ipython
%reload_kedro
I got an error saying No module named 's3fs'. I'm using databricks on Azure w/ adlfs . Do I still need s3fs for credentials.yml to work? This is my catalog.yml entry causing the error:
# Just testing on vanilla spark right now to get to a version that I know works
raw.media_meas_campaign_info:
type: spark.SparkDataset
filepath: abfss://~/media_meas_campaign_info/
file_format: parquet
credentials: azure_group
load_args:
header: True
inferSchema: True
I installed s3fs
and everything works fine, I was just surprised because I assumed I would only need that if I was in AWS and I got a bunch of errors when trying to install s3fs>=2021.4,<2024.1
because databricks already had some underlying dependencies on a newer version that it wouldn't let pip uninstall.George Drakoulas
03/13/2024, 9:21 AMGeorge Drakoulas
03/15/2024, 8:50 AMGeorge Drakoulas
03/19/2024, 7:52 AMJulian Nowak
03/22/2024, 9:03 AMsome_number = (
get_kedro_boot_session("project")
._context.catalog._get_dataset("features_store")
.load()
.iloc[0][0]
)
(In the example project), is it how it should be done?Manali Kadam
03/22/2024, 1:39 PMMelvin Kok
03/26/2024, 5:01 AMkedro-mlflow
on/off in the local config?
I understand that we can deactivate tracking for certain pipelines, but on some runs I don't want MLFlow to run at all.Deepyaman Datta
03/27/2024, 4:21 PMhamiddimyati
03/28/2024, 10:59 AMkedro-mlflow
concept? In my case, I use pytorch lightning
, and they provided automatic logging using mlflow
. Using this logger, the metrics are automatically saved into a new folder. I'm wondering how to save this training metrics as artifact of experiment that can also be tracked in Kedro
catalog. Please help :)George Drakoulas
04/02/2024, 6:25 AMJulian Nowak
04/05/2024, 11:55 AMfrom kedro_boot.app.fastapi.session import KedroFastApi
Is thera a way to overcome it?
I know that KedroBootFastApi
depends on KedroBootSession
, but I guess it's not as simple as defining session=KedroFastApi(KedroBootSession(pipeline="inference"))
@Takieddine KadiriJan
04/11/2024, 7:30 AMGeorge Drakoulas
04/11/2024, 11:57 AMExecution failed. User process 'cd' exited with status code 1. Please check log file 'user_logs/std_log.txt' for error details. Error: โ 545 โ def exists(self, name: str) -> bool: โ
โ 546 โ โ """Checks whether registered data set exists by calling its `e โ
โ โ
โ /usr/local/lib/python3.10/site-packages/kedro/io/core.py:228 in save โ
โ โ
โ 225 โ โ โ raise โ
โ 226 โ โ except Exception as exc: โ
โ 227 โ โ โ message = f"Failed while saving data to data set {str(self โ
โ โฑ 228 โ โ โ raise DatasetError(message) from exc โ
โ 229 โ โ
โ 230 โ def __str__(self): โ
โ 231 โ โ def _to_str(obj, is_root=False): โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
DatasetError: Failed while saving data to data set
KedroAzureRunnerDataset(dataset_name=hull_data_merged, info=for use only within
Azure ML Pipelines,
path=<abfs://testmlkedro7563142352/kedro-azureml-temp/a0761f9e659c46a69980d54e511>
1f04d/hull_data_merged.bin).
Cannot connect to host <http://kedro-azure-storage.blob.core.windows.net:443|kedro-azure-storage.blob.core.windows.net:443> ssl:default
[Name or service not known]
Andrew Stewart
04/12/2024, 8:53 PMAndrew Stewart
04/12/2024, 9:51 PMkedro-viz
or is this obviously something outside that scope / upstream (such as something gnarly with the pipeline definition or something):
$ kedro viz build
[04/12/24 14:30:18] ERROR Failed to save node data for node ID 2aa90f47. Error: 'str' object has no attribute 'get' responses.py:430
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Traceback (most recent call last) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ /home/.venv/lib/python3.9/site-packages/kedro_viz/api/rest/responses.py:427 in โ
โ save_api_node_response_to_fs โ
โ โ
โ 424 โ for nodeId in data_access_manager.nodes.get_node_ids(): โ
โ 425 โ โ try: โ
โ 426 โ โ โ write_api_response_to_fs( โ
โ โฑ 427 โ โ โ โ f"{nodes_path}/{nodeId}", get_node_metadata_response(nodeId), remote_fs โ
โ 428 โ โ โ ) โ
โ 429 โ โ except Exception as exc: # pragma: no cover โ
โ 430 โ โ โ logger.exception( โ
โ โ
โ /home/.venv/lib/python3.9/site-packages/kedro_viz/api/rest/responses.py:336 in โ
โ get_node_metadata_response โ
โ โ
โ 333 โ โ return TaskNodeMetadata(task_node=node) โ
โ 334 โ โ
โ 335 โ if isinstance(node, DataNode): โ
โ โฑ 336 โ โ return DataNodeMetadata(data_node=node) โ
โ 337 โ โ
โ 338 โ if isinstance(node, TranscodedDataNode): โ
โ 339 โ โ return TranscodedDataNodeMetadata(transcoded_data_node=node) โ
โ โ
โ /home/.venv/lib/python3.9/site-packages/pydantic/main.py:175 in __init__ โ
โ โ
โ 172 โ โ """ โ
โ 173 โ โ # `__tracebackhide__` tells pytest and some other tools to omit this function โ
โ from tracebacks โ
โ 174 โ โ __tracebackhide__ = True โ
โ โฑ 175 โ โ self.__pydantic_validator__.validate_python(data, self_instance=self) โ
โ 176 โ โ
โ 177 โ # The following line sets a flag that we use to determine when `__init__` gets โ
โ overridden by the user โ
โ 178 โ __init__.__pydantic_base_init__ = True # pyright: โ
โ ignore[reportFunctionMemberAccess] โ
โ โ
โ /home/.venv/lib/python3.9/site-packages/kedro_viz/models/flowchart.py:790 in โ
โ set_filepath โ
โ โ
โ 787 โ @classmethod โ
โ 788 โ def set_filepath(cls, _): โ
โ 789 โ โ dataset_description = cls.dataset._describe() โ
โ โฑ 790 โ โ return _parse_filepath(dataset_description) โ
โ 791 โ โ
โ 792 โ @field_validator("run_command") โ
โ 793 โ @classmethod โ
โ โ
โ /home/.venv/lib/python3.9/site-packages/kedro_viz/models/flowchart.py:43 in โ
โ _parse_filepath โ
โ โ
โ 40 โ
โ 41 โ
โ 42 def _parse_filepath(dataset_description: Dict[str, Any]) -> Optional[str]: โ
โ โฑ 43 โ filepath = dataset_description.get("filepath") or dataset_description.get("path") โ
โ 44 โ return str(filepath) if filepath else None โ
โ 45 โ
โ 46 โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
AttributeError: 'str' object has no attribute 'get'
Laure Vancau
04/22/2024, 3:08 PMlong_params_strategy
parameter to tag
(kedro-mlflow==0.11.10 , kedro~=0.18.9)
As you can see, in our set-up, instead of going to set_tag, the parameter is sent to be logged as a parameter.
Do you have any idea as to why this is occurring?
Thank you in advance โบ๏ธ