wooden-spring-14915
10/01/2022, 2:02 AMlimited-solstice-12595
elegant-house-93198
for obj in self._client.list_objects_v2(
Bucket=self._s3_bucket, Prefix="great_expectations.yml"
).get("Contents"):
if obj.get("Key") == "great_expectations.yml":
What does your current file structure in S3 look like?gentle-translator-17986
10/05/2022, 11:28 PMgreat_expectations.yml
in our S3 bucket bc that's not really a requirement and we send our data context "on the fly" whenever we do our GE validations using an Airflow Operator. So essentially something like:
from great_expectations.data_context import BaseDataContext
context = BaseDataContext(project_config={
'datasources': {
'my_datasource_name': {
'module_name': 'great_expectations.datasource',
'class_name': 'Datasource',
'execution_engine': {
'module_name': 'great_expectations.execution_engine',
'class_name': 'PandasExecutionEngine',
},
'data_connectors': {
'my_dataconnector_name': {
'module_name': 'great_expectations.datasource.data_connector',
'class_name': 'InferredAssetS3DataConnector',
'bucket': 'some_bucket',
'prefix': 'some_prefix',
'default_regex': {
"pattern": "(.*)",
"group_names": ["data_asset_name"], # This name is important, not a placeholder
}
},
},
}
},
...
})
but i was wondering what do you extract from the great_expectations.yml
? because i can just put a dummy one in there to avoid the fail integration