Hi everyone. I am testing the connector of Google Workspace Admin Reports 0.1.6. Since we have billions of audit logs, it takes a long time for the first time job to fetch all the records. Moreover, it seems that it has to fetch all the records before storing the temporary or final destination, say about a BigQuery table. So, we would like to shorten the elapsed time to make sure data in destination. What do you think? • It would be nice to specify the start date to fetch records. The current version tries to fetch all logs for the maximum retention time. • It would be also nice to iterate to fetch and store records so that we can see stored ones in destination as soon as possible. Aside from that, it might be risky to have a quite long running process in a container. We potentially has to re-run the container to fetch data, if it is failed. • It would be also nice to select a strategy about how to do backfill the past from the latest logs. In my opinion, newer data tend to be more important. If we collect data from newer to older by doing backfilling, it would be awesome. https://support.google.com/a/answer/7061566

Hey yu would you mind creating an issue over github on this so that we can track these