I'm about to refactor my "E /Extract" loader written in Python, that currently loads from a web API (Google Cloud). Is my impression correct, that Hamilton provides generic adapters, but none, that already deal with rate limiting and partitioning (e.g. by date), for reliable and resumable downloading of day-wise timeline data?

Correct we don't have a vanilla Hamilton adapter for that explicitly. You have a few options: • We have the primitives for it and build it yourself • Use the dlt adapter - though I'm not sure how far it covers your needs. CC @Thierry Jean for more direction. • You delegate to another tool to kick off and provide the data and then Hamilton does the rest.

Hi @Viktor! If you're familiar with ETL (extract, transform, load), Hamilton excels at the (T)ransform layer, just like dbt. Materializers help get data in / out of the T layer, but it assumes you don't have to wrangle too much with the source (e.g., you're loading a dataset from parquet or csv). Querying APIs with Hamilton can be tricky. While we have

Parallelizable/Collect

, it's not necessarily easy to do retries or limit request rates across branches. I don't want to push you into a rabbit hole, but the tool

dlt

could be a good solution! It focuses specifically on "Extract and Load" (EL), it's open source, Python-first, and has a similar feel to Hamilton. You write

Resource

objects and then execute them in a

Pipeline

. It supports incremental loading for the "reliable and resumable" download you mention. They have a lot of built-in "Source" (e.g., Airtable, databases, Jira, Slack) but you can build your own REST API connector (see tutorial). If

dlt

sounds interesting to you, there's a Hamilton documentation page on how to use both tools. We also have a plugin that allows you to connect dlt (EL) directly to Hamilton (T) via the

to.dlt()

and

from.dlt()

materializers!

Thank you, @Thierry Jean. Things are coming together.