This message was deleted.

Hey! Happy to hear. So yes, quite a few options: 1. You can always process afterwards 2. You can write a function that does this in the DAG. You just have to be careful about joining:

def hourly_results(
    period_start: pd.Series,
    revenue: pd.Series,
    price: pd.Series
) -> pd.DataFrame:
    return pd.merge(...)

def daily_results(hourly_results: pd.DataFrame) -> pd.DataFrame:
    return group_by_day(hourly_results)

Then you can call it as you want, or call out

hourly_results

or

daily_results

as an output.

Yep, it will be pretty simple. Some columns will be sum (revenue) and some would be average (price).

Ahh, so I also think it can make sense to do the grouping on a per-series basis, as you can separate out the logic that you use to group. Then you can join afterwards, on the hourly-index column:

def revenue_hourly(revenue: pd.Series) -> pd.Series:
    return ...

def price_hourly(price: pd.Series) -> pd.Series:
    return ...

dr.execute(['revenue_hourly', 'price_hourly'], ...)

You can also mess with the naming. Given that you have only two it makes sense to do them individually, but parameterization/extraction decorators can always help.

I'm also thinking about combining it with Pandera, so the input of

daily_results

is a Pandera schema. Eg:

def daily_results(hourly_results: DataFrame[HourlyPanderaSchema]) -> DataFrame[DailyPanderaSchema]:
    ...

Re; Pandera — I really like the look of that, we haven’t tried the typing-hinting API in Hamilton, although I think it should be easy to manage. What we do have is the

check_output

decorator:

@check_output(schema=DailyPanderaSchema())
def daily_results(...):
    ...

Unfortunately we don’t yet have the ability to use output type-hinting (yet), but I think we could figure that out actually. Re: renaming you have two options: 1. Change it so

revenue

is the final name, and the prior one is called

revenue_daily

2. Rename later — after the fact

df.columns = […]

. You can also make this part of a dataframe, e.g. query

hourly_results

that changes the name and joins the series, kind of like what I suggested above. You could also do a custom result builder if you want — there are a few ways to decouple the name from the exact column generated, but these two are the simplest.

Thanks! With option 1, I assume I would then combine that with the @config.when. Something like:

def revenue_daily(raw_revenue: pd.Series) -> pd.Series:
	# Sum by Day
	...

def revenue_hourly(raw_revenue: pd.Series) -> pd.Series:
	# Sum by Hour
	...


@config.when(granularity=HOURLY)
def revenue__hourly_granularity(revenue_hourly: pd.Series) -> pd.Series
	return revenue_hourly

@config.when(granularity=DAILY)
def revenue__daily_granularity(revenue_daily: pd.Series) -> pd.Series
	return revenue_daily

Yeah! Definitely a good approach. You also could have them both and just query for the one you want — depends on whether you want them to mean the same thing and go through further processing/mean the same thing to the user, or you want to query for them directly. or not down the line (E.G. whether they’re end-facing or not). Side note: for later, should you decide to run multiple granularities in the same DAG, we have tooling to repeat certain subdags (and happen to have an example almost exactly like this

Great, thanks for the help. Really appreciate that.

Yeah! Great questions. Also just created this cause you got me thinking: Integrate pandera type-hinting

Nice! Yep this seems a bit more compatible with the way Pandera does its type hinting.

Should hopefully be an easy addition — will scope though to confirm before I make promises 😆