i noticed that inside class DatasetProfile in schema_classes, there are 2 attributes, eventGranularity and partitionSpec, but they dont seem to be populated when i run a data profiling job on a postgres table. Is it actually in use, or is it "for future use"? Cos im trying to implement my own profiler using pandas-profiler with a JDBC source

@helpful-optician-78938 would probably know!

Hi @better-orange-49102, these are the optional fields available to every time series aspect. eventGranularity specifies the time window of validity of the event, such as daily/hourly chunks of data and is used with aspects like datasetUsage. PartitionSpec helps capture data about a specific partition of a table. If you have a use-case for either, you can populate those fields as well.

thanks for replying. Is there any functionality for profiling specific partitions of a table at present? My use case would be to query the latest partition and use it for profiling, since right now if i specify row limits for profiling, GE seems to always take the oldest data to profile. I took a look at GE, but it doesn't seem to have any options for that (I'm not sure about it because i find GE to be rather complicated compared to say, pandas_profiling, when applying it programmatically outside of a Jupyter notebook.)

Hi @better-orange-49102, that's a very interesting use-case. I don't see any out-of-the-box mechanism in GE for specifying partitions of a table directly for profiling. Can you try if specifying the partition via the profile_pattern's allow works?