Hi Everyone I have added a endpoint /{org_id}/deql/{aggregate}/command It will receive DeQL commands which are commands in CQRS-ES desing. When command is successful it emits an event. This event needs to be stored in a dedicated event store. For now it has to be stored as a log in openobserve. Question : Which service is the best palace to start ingestion. Since events are structured no need of schema check. What is the fastest route to push the event into default logs. Is there a datafusion table into which it can be ingested.

Not just logs but any type of signal needs to be send to ingester nodes first where it converts data into parquet format and later store it in object storage. You can give this a read https://openobserve.ai/docs/architecture/#ingester to understand data ingestion flow better. I don't think so their is a datafusion table where we can write the records directly to be available for querying. @Hengfei @Huaijin Hao Please share if you have any further insights.

datafusion is query engine. not a storage. if you want to store data, you can check our ingestion API, how it works.

/api/{org}/_bulk

/api/{org}/{stream}/_json

receive data component is: ingester.

Is this the crate::service::logs::ingest::ingest right function to start ?

you can think, yes.

Thanks