Elijah Ben Izzy
07/02/2022, 6:56 PMElijah Ben Izzy
07/02/2022, 6:56 PMElijah Ben Izzy
07/13/2022, 9:56 PMpip install sf-hamilton==1.9.0rc0
Elijah Ben Izzy
08/16/2022, 3:40 AMNone
for the output)
• Support for Union
type
• A refactor of the parametrize*
family -- this includes a new @parameterize
decorator that can parameterize across dependency sources/values
• Misc. bug fixes.
Reach out if you have any questions!
Installing is easy -- just run pip install sf-hamilton==1.10.0rc0
!Sarah Haskins
01/10/2023, 5:48 PMElijah Ben Izzy
12/22/2023, 9:00 PMpre-commit clean
, etc…
Thanks!
Regarding pre-commit. I recreated the venv and installed reqs and then ran pre-commit and I get the following error. That is why I indeed downgraded to an older version.
I first did this, which all worked with python 3.9.6
python -m venv ./venv
. ./venv/bin/activate
pip install \
-r ./requirements.txt \
-r ./requirements-dev.txt \
-r ./requirements-test.txt
And then when running pre-commit I get this error
pre-commit run -a
[INFO] Initializing environment for <https://github.com/ambv/black>.
An error has occurred: InvalidManifestError:
==> File /Users/roel/.cache/pre-commit/repoy7gkbpl2/.pre-commit-hooks.yaml
==> At Hook(id='black')
==> At key: stages
==> At index 0
=====> Expected one of commit, commit-msg, manual, merge-commit, post-checkout, post-commit, post-merge, post-rewrite, prepare-commit-msg, push but got: 'pre-commit'
Check the log at /Users/roel/.cache/pre-commit/pre-commit.log
Did you happen to run into this before?Konstantin Tyapochkin
03/18/2024, 8:11 PMTom Barber
03/21/2024, 2:26 PMFran Boon
04/01/2024, 4:13 PMresources={"A2":1}
)
Currently it seems that this could be pretty easily achieved by (ab)using Hamilton's Tags feature:
@tag(**{"ray.resources": json.dumps({"A2": 1}))
def my_hamilton_node_fn_which_needs_an_A2(...) -> ...:
...
RayGraphAdapter.execute_node() would be modified to:
ray_options = {tag[4:]: json.loads(value) for tag, value in tags.items() if tag.startswith("ray.")}
return ray.remote(raify(node.callable), **ray_options).remote(**kwargs)
Any concerns with taking this approach?
Any better options?Stefan Krawczyk
04/02/2024, 5:23 PMdr = driver.Builder().with_modules(data_loading, feature_engineering, model_training, model_evaluation).with_adapter(...).build()
# one sagemaker job on small machine
data_set = dr.execute(["data_set_v1"], inputs={...})
# one sage maker job on large machine with GPU
model = dr.execute(["model_v1"], inputs={...}, override={"data_set_v1": data_set})
# one sagemaker job on small machine
evaluation = dr.execute(["evaluation_v1"], inputs={...}, override={"model_v1": model})
# some ideas on config structure?
config = {
"tasks": [{"name": "data_set_v1", "sagemaker": ["machine.small"], "artifacts": ["data_set_v1"]},
{"name": "model_v1", "sagemaker": ["machine.gpu"], "artifacts": ["model1"]},
{"name": "model_v2", "sagemaker": ["machine.gpu"], "artifacts": ["model2"]},
{"name": "evaluation_v1", "sagemaker": ["machine.small"]}]
}
sagemaker_pipeline_code = SageMakerPipelineBuilder(dr, config).compile()
airflow_pipeline_code = AirflowPipelineBuilder(dr, config).compile()