hello ! new Hamilton user here :wave: i’m curious ...
# hamilton-help
j
hello ! new Hamilton user here 👋 i’m curious to know how to selectively load and parse json response, in particular those complex ones with multiple levels and nested dict. I’ve been extensively using pandas json normalize in the past, but wondering if there is a “hamilton” way. Alternatively, I’d use pandas (or polars) to load/extract and using hamilton from there 🙃
👀 1
t
Welcome Jean-Paul! If I understand correctly, you have JSON files and only want to load the relevant parts in memory? 1. Normally, you can't partially read a JSON file. You would have to read it in full and then extract the relevant parts (pandas and polars are great options!). If you're concerned about the speed of reading large JSON objects, using specialized Python libraries to load JSON files and then passing the relevant data to pandas / polars could help (reference) 2. I also like
json_normalize()
quite a lot, and I use it in this tutorial! (you will have to follow the instructions to download the data). The Hamilton pattern I suggest is a. load the raw data b. clean and format the raw data (no complex transforms) c. do analysis, ML, etc. In the notebook, I use the following code
Copy code
def pdl_data(pdl_file: str, data_dir: str = "data/") -> pd.DataFrame:
    """Load raw People Data Labs data stored locally"""
    return pd.read_json(Path(data_dir, pdl_file))

def employee_count_by_month_df(pdl_data: pd.DataFrame) -> pd.DataFrame:
    """Normalized employee count data"""
    return (
        pd.json_normalize(pdl_data["employee_count_by_month"])
        .assign(ticker=pdl_data["ticker"])
        .melt(
            id_vars="ticker",
            var_name="year_month",
            value_name="employee_count",
        )
        .astype({"year_month": "datetime64[ns]"})
    )
❤️ 2
s
I like Thierry's approach better since its more readable but this is another option. https://github.com/pola-rs/polars/issues/7374#issue-1611685098
m
@Thierry Jean this might be of interest: https://github.com/simdjson/simdjson It’s the standard used by Clickhouse, Starrocks and QuestDB to partially read only certain JSON keys. Haven’t used it, but there are Python bindings too: https://github.com/TkTech/pysimdjson
t
Thanks @miek! These solutions can unlock a lot of performance when dealing with JSON data, but it requires knowing the schema / shape of the object beforehand. This can be especially useful if you're dealing with API responses, but it requires a bit of manual setup and some maintenance There's also msgspec that's available!
m
Yup, fair point. I have used “orjson” as well
j
thanks a lot for all these tips!