Does a segment always consist of

columns.psf, creation.meta, index_map, metadata.properties

? I’m thinking for the s3 lazy loading, it might make sense to have separate caching settings for metadata vs

columns.psf

. Like you may want to eagerly load all or most of the metadata since it’s small and means segments can be eliminated quickly.

Yes, all segments have these file. But these are not exposed as individual files. One issue I can think of with the approach is when a segment is refreshed, the cached metadata can get out of sync, and would need some sort of invalidation/reload.

How does a segment get refreshed? I thought the idea was that data is immutable?

Having said that, I do see some merit in eager loading of metadata, Perhaps it would make sense to write down the idea and check against cases to handle.

Oh. The interface expects the full segment to be there?

I mean there is no api grtColumnPsfFile()

Added it as a comment on the lazy loading issue. I think first we do lazy loading of the whole segment. Then add this as an optimization later.

There is getSegmentMetadata() though