As a rule of thumb is it discouraged to perform any egress from Pinot servers apart from deep store commits? since it may interfere with ongoing ingestion

Yeah, no bulk egress of raw data. The only way should be via query

what about metadata? we are looking to stream segment level metadata directly from servers or even store this at a per server level

What problem are you trying to solve? There is metadata api you can call from controller

we want to store per segment metadata so we can store most of the data elsewhere and then filter to some degree based on this metadata and only then re-upload segments into Pinot. The metadata API gives only ZK metadata right? there is another one but even that does not provide full segment metadata that is needed for a metadata based upload to the controller