Vitor Fortunato
05/30/2024, 7:23 AMkapa.ai
05/30/2024, 7:23 AMVitor Fortunato
05/30/2024, 7:55 AMkapa.ai
05/30/2024, 7:55 AMkapa.ai
05/30/2024, 7:56 AM{{ last_record }}
variable. This variable holds the last record extracted from the response received from the API, which typically includes the date of the last synced record. Here is an example of how you might use it in your configuration: yaml { "start_date": "{{ last_record.date_field }}" }
For more details on the available interpolation variables, you can refer to the [YAML Reference](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#variables). If you need to reference the start and end date of the interval that's currently synced, you can use the {{ stream_interval.start_time }}
and {{ stream_interval.end_time }}
placeholders. This can be particularly useful if you need to inject these values into the URL path of the current stream. For example: yaml { "url": "<https://api.example.com/data?start_date=>{{ stream_interval.start_time }}&end_date={{ stream_interval.end_time }}" }
For more information on using these placeholders, check out the [incremental sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#example). By using these variables, you can effectively manage incremental syncs and ensure that your connector only fetches new or updated data since the last sync.Vitor Fortunato
05/30/2024, 7:58 AMkapa.ai
05/30/2024, 7:58 AMkapa.ai
05/30/2024, 7:58 AMstream_interval.start_time
and stream_interval.end_time
and to review previous job history syncs, you can follow these steps: 1. Reviewing Sync Logs: The sync logs contain detailed information about each sync operation, including the values of stream_interval.start_time
and stream_interval.end_time
. You can access these logs through the Airbyte UI or by querying the logs directly if you have access to the backend. 2. Using the Airbyte UI: - Navigate to the Connections tab in the Airbyte UI. - Select the connection you are interested in. - Click on the Job History tab to view the history of sync jobs. - For each job, you can click on the Logs button to view detailed logs, which will include the values of stream_interval.start_time
and stream_interval.end_time
. 3. Example Log Entry: Here is an example of what you might find in the logs: json { "start_time": "2023-04-15T07:30:58Z", "end_time": "2023-04-16T07:30:58Z" }
4. Using the API: If you prefer to use the API, you can query the job history endpoint to retrieve detailed information about past syncs, including the state messages that contain the stream_interval
values. 5. Documentation Links: - For more information on using the stream_interval
variable, refer to the [YAML Reference](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#variables). - For details on incremental syncs and how to configure them, check out the [incremental sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#example). By following these steps, you can effectively monitor and review the values passed to stream_interval.start_time
and stream_interval.end_time
and gain insights into the history of your sync jobs.Vitor Fortunato
05/30/2024, 8:05 AMkapa.ai
05/30/2024, 8:05 AMkapa.ai
05/30/2024, 8:05 AMstart_time
and end_time
while checking the logs in the Airbyte UI, there could be several reasons for this: 1. Incorrect Configuration: Ensure that your connector is correctly configured to use incremental syncs and that the stream_interval
variables are properly set up in your configuration. Refer to the [incremental sync documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration) for detailed instructions. 2. Log Level Settings: The log level settings might not be detailed enough to capture the start_time
and end_time
values. Ensure that the logging level is set to capture detailed information. 3. Connector Limitations: Some connectors might not support logging these specific details. Check the documentation for your specific connector to see if there are any known limitations. 4. State Management: The start_time
and end_time
values are part of the state management for incremental syncs. If the state is not being managed correctly, these values might not be logged. Ensure that the state management is correctly implemented in your connector. 5. Custom Parameter Injection: If you are using custom parameter injection, make sure that the parameters are correctly set up to include stream_interval.start_time
and stream_interval.end_time
. Refer to the [custom parameter injection documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#custom-parameter-injection) for more details. 6. API Response: The API you are connecting to might not be returning the expected values. Ensure that the API response includes the necessary fields and that they are correctly mapped in your configuration. If you have verified all the above points and still can't find the start_time
and end_time
values, you might need to reach out to the Airbyte community or support for further assistance. You can post your issue on the [Airbyte community forum](https://discuss.airbyte.io/) or check the [Airbyte documentation](https://docs.airbyte.com/) for more troubleshooting tips.Vitor Fortunato
05/30/2024, 8:06 AMkapa.ai
05/30/2024, 8:07 AMkapa.ai
05/30/2024, 8:07 AMDEBUG
in a YAML configuration file: yaml logging: level: DEBUG
5. Save and Apply: - Save the changes to the configuration. - Apply the changes and run a new sync to see the more verbose logs. 6. Review the Logs: - After running the sync, go to the Job History tab for the connection. - Click on the Logs button for the latest job to review the more detailed logs. For more detailed instructions on configuring connectors and managing logs, you can refer to the [Airbyte documentation](https://docs.airbyte.com/). If you encounter any issues or need further assistance, consider reaching out to the [Airbyte community forum](https://discuss.airbyte.io/) for support.