Hi all, can someone help me with the below error? I have a custom data loader class that has applicable_types - pl.DataFrame but for some reason it loads a dataframe that isn't generated within the driver context (i.e. not being imported from modules or not passed to the function that creates the driver). A bit lost as to how is this happening, been at it for hours. Any help would be appreciated, thanks!

OK, very unclear error, but I think what’s happening is that applicable_types is returning a dataframe, not a type. It should return the type

pl.DataFrame

— the code you showed above suggests applicable_types[0] is an actual dataframes.

@classmethod
@abc.abstractmethod
def applicable_types(cls) -> Collection[Type]:
    return [pl.DataFrame]

What’s the code for your custom loader look like?

guessing here: did you ensure that your

@dataloader

decorated function is type annotated to return a

tuple

? It should be

@dataloader()
def my_loader() -> tuple[pl.DataFrame, dict]:
   return data, metadata

rather than

@dataloader()
def my_loader() -> pl.DataFrame:
  return data

this is my custom DataLoader class

@dataclass
class S3ExcelDataLoader(DataLoader):
    """Data Loader that retrieves Excel files from an S3 bucket and loads them into a DataFrame.

    It validates the S3 path and file extension to ensure proper loading of the specified Excel sheet.

    Attributes:
        s3_path (str): The S3 path to the Excel file to be loaded.
        sheet_name (str | None): The name of the sheet to load from the Excel file (optional).
        session (boto3.Session): The boto3 session used to interact with AWS services.

    Methods:
        load_data(type_): Loads the Excel data from the specified S3 path and returns it as a DataFrame along with metadata.
        applicable_types(): Returns a collection of data types that this loader can handle.
        name(): Returns the name of the data loader.

    Raises:
        ValueError: If the provided S3 path does not end with .xls or .xlsx or does not start with s3://.
    """

    s3_path: str
    sheet_name: str | None = None
    session: boto3.Session = field(default_factory=boto3.Session)

    def __post_init__(self) -> None:  # noqa: D105
        if not self.s3_path.endswith((".xls", ".xlsx")):
            msg: str = f"Excel files must end with .xls or .xlsx, got {self.s3_path}"
            raise ValueError(msg)
        if not self.s3_path.startswith("s3://"):
            msg: str = f"S3 path must start with s3://, got {self.s3_path}"
            raise ValueError(msg)

    @classmethod
    def applicable_types(cls) -> Collection[type]:
        return [pl.DataFrame]

    def load_data(self, type_: type) -> tuple[pl.DataFrame, dict[str, Any]]:  # noqa: D102
        logger.info(f"Loading Excel data from {self.s3_path}, sheet: {self.sheet_name}...")
        s3_client = self.session.client("s3")
        bucket_name, key = self.s3_path.replace("s3://", "").split("/", 1)
        obj = s3_client.get_object(Bucket=bucket_name, Key=key)
        excel_data: io.BytesIO = io.BytesIO(obj["Body"].read())
        raw_df: pd.DataFrame = pl.read_excel(
            source=excel_data, sheet_name=self.sheet_name, infer_schema_length=20000
        )
        metadata: dict[str, Any] = utils.get_dataframe_metadata(raw_df)
        logger.info("Loaded Excel data successfully!")
        return raw_df, metadata

    @classmethod
    def name(cls) -> str:  # noqa: D102
        return "s3_excel_loader"  # the name for `from_.{name}`

@sahil-shetty what is

type_: type

here?

I'm on polars == 1.8.2 I have another custom data loader that is very similar to the one above meant for the local filesystem, and that ran fine. So, this has been working for me for weeks as well. I now just orchestrate the Hamilton driver creation and execution via prefect but for some reason the above dataframe gets injected. This dataframe isn't being used in my transformations. It's generated in another module way before I instantiate my drivers or register my loaders and never gets called again so I'm not sure how even this dataframe of all the others is being passed. (Sorry for the very descriptive message but thought I'd add some more context, I'm most likely doing something very silly and not a bug in Hamilton since it's worked for me before.) this is how I invoke it:

@sahil-shetty so the stack trace seems to indicate that the error is coming from within polars.

nope, there is no

()

can you verify your

.applicable_types()

function is being called?

it is being called but contains this dataframe, rather than type

yeah — so can you validate that this is the code you think is being called:

It works perfectly fine when I run my driver function from the main.py script as opposed to importing it to another module and calling it there. I suspect there's something going on with my imports, it's firing the

applicable_types

method at some point. Debugging to figure out where.

All good @sahil-shetty. Glad to hear you resolved it. If there's documentation or better error messages that could have helped, please create an issue.