Slackbot
03/07/2024, 7:20 PMThreadScribe
03/07/2024, 7:20 PMThreadScribe
03/07/2024, 7:20 PMThreadScribe
03/07/2024, 7:20 PMThreadScribe
03/07/2024, 7:20 PMStefan Krawczyk
03/07/2024, 7:22 PMThreadScribe
03/07/2024, 7:22 PMThreadScribe
03/07/2024, 7:22 PMcheck_output
decorator in Hamilton, you can add simple data quality checks to your code by following the example provided in the documentation. Here's how you would do it:
python
import pandas as pd
import numpy as np
from hamilton.function_modifiers import check_output
@check_output(
data_type=np.int64,
range=(0,100),
)
def some_int_data_between_0_and_100() -> pd.Series:
# Your code here
pass
This decorator enables you to specify the data type and range for the output, allowing you to enforce data quality constraints. For more details, you can refer to the official documentation on the check_output
decorator in Hamilton [here](https://hamilton.dagworks.io/en/latest/reference/decorators/check_output/).Stefan Krawczyk
03/07/2024, 7:23 PM@step
decorator?ThreadScribe
03/07/2024, 7:23 PM@step
decorator?"
Let me see what I can find... 🤔ThreadScribe
03/07/2024, 7:23 PM@step
decorator is not mentioned in the provided documents, so there is no information available about it." }Stefan Krawczyk
03/07/2024, 7:23 PMStefan Krawczyk
03/07/2024, 7:24 PM@pipe
decorator?ThreadScribe
03/07/2024, 7:24 PM@pipeline
decorator?"
Let me see what I can find... 🤔ThreadScribe
03/07/2024, 7:24 PMThreadScribe
03/07/2024, 7:26 PM@pipe
decorator?"
Let me see what I can find... 🤔ThreadScribe
03/07/2024, 7:26 PM@pipe
decorator, we can refer to the Hamilton documentation. The @pipe
decorator is used to represent a chained set of transformations. It is specifically designed to solve the "node redefinition" problem and is meant to represent a pipeline of chaining/redefinitions. This can be used to display transformations as nodes in the DAG, store or visualize the result, pull in functions from an external repository, build the DAG more procedurally, and use the same function multiple times with different parameters. It is a part of the function modifiers in Hamilton, which are highly expressive and useful for writing expressive dataflows.
For more details, the documentation can be found at [Hamilton @pipe Documentation](https://hamilton.dagworks.io/en/latest/reference/decorators/pipe/).Mathew Goldsborough
03/07/2024, 8:12 PMElijah Ben Izzy
03/07/2024, 11:24 PM@pipeline
didn’t find anything but it did for @pipe
. Might be nice to prompt for more.Mathew Goldsborough
03/08/2024, 2:32 AMMathew Goldsborough
03/08/2024, 2:33 AMMathew Goldsborough
03/08/2024, 6:33 PM@pipe
decorator?ThreadScribe
03/08/2024, 6:33 PM@pipe
decorator?"
Let me see what I can find... 🤔ThreadScribe
03/08/2024, 6:33 PM@pipe
decorator in Hamilton is used to represent a chained set of transformations, providing a solution to the "node redefinition" problem and representing a pipeline of chaining/redefinitions. It is intended to be used for the following reasons:
1. To display transformations as nodes in the DAG, with the possibility of storing or visualizing the result.
2. To pull in functions from an external repository and build the DAG more procedurally.
3. To use the same function multiple times, but with different parameters.
The @pipe
decorator allows for the chaining of nodes, and it works perfectly with various types of data such as dataframes and series.
For more detailed information, the documentation for the @pipe
decorator can be found at pipe - Hamilton and Function modifiers (Advanced).Mathew Goldsborough
03/08/2024, 6:33 PM