https://linen.dev logo
Join Slack
Powered by
I need one example of how to implement the usage o...
# replication-ask-ai
m
I need one example of how to implement the usage of lookback window when I have a custom implementation of a HttpRequester due to a async behavior on the api
k
Lookback Window with a Custom HttpRequester (Async API) The Bing Ads connector is a good example ā€” it uses an 
AsyncRetriever
with a custom 
HttpRequester
and applies a lookback window via a 
DatetimeBasedCursor
. Here's how it's structured: 1. Define the DatetimeBasedCursor with a lookback_window
Copy code
incremental_sync_report_datetime_cursor:
  type: DatetimeBasedCursor
  cursor_field: TimePeriod
  cursor_datetime_formats:
    - "%Y-%m-%d"
  datetime_format: "%Y-%m-%d"
  lookback_window: P{{ config.get('lookback_window', 0) }}D   # <-- lookback applied here
  start_datetime:
    type: MinMaxDatetime
    datetime: |
      {%- set last_year = now_utc() - duration('P1Y') -%}
      {%- set default_date = last_year.strftime('%Y-01-01') -%}
      {%- set start_date = config.get('reports_start_date', default_date) -%}
      {{ start_date }}
    datetime_format: "%Y-%m-%d"
  end_datetime:
    type: MinMaxDatetime
    datetime: "{{ format_datetime(now_utc(), '%Y-%m-%dT%H:%M:%S%z') }}"
    datetime_format: "%Y-%m-%dT%H:%M:%S%z"
[Bing Ads manifest] 2. Wire the AsyncRetriever to use that cursor The 
AsyncRetriever
handles the async job lifecycle (creation ā†’ polling ā†’ download), while the cursor (with its 
lookback_window
) controls the date range passed to the `creation_requester`:
Copy code
basic_bulk_async_retriever:
  type: AsyncRetriever
  partition_router:
    $ref: "#/definitions/account_substream_partition_router"
  status_mapping:
    failed:
      - Failed
      - FailedFullSyncRequired
    running:
      - InProgress
    completed:
      - Completed
    timeout: []
  creation_requester:
    type: HttpRequester
    url_base: https://bulk.api.bingads.microsoft.com/
    path: Bulk/v13/Campaigns/DownloadByAccountIds
    http_method: POST
    request_headers:
      Content-Type: application/json
      DeveloperToken: "{{ config['developer_token'] }}"
    authenticator: "#/definitions/authenticator"
  polling_requester:
    type: HttpRequester
    url_base: https://bulk.api.bingads.microsoft.com/
    path: Bulk/v13/BulkDownloadStatus/Query
    http_method: POST
    # ...
  download_requester:
    type: HttpRequester
    url_base: "{{download_target}}"
    http_method: GET
  download_decoder:
    type: CustomDecoder
    class_name: source_declarative_manifest.components.BingAdsGzipCsvDecoder
[Bing Ads manifest] Key Takeaways The pattern separates concerns cleanly: the cursor owns the lookback logic, and the 
AsyncRetriever
 owns the async orchestration. You don't need to implement lookback inside the custom requester itself ā€” the cursor handles it before the slice is even passed to the requester. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
m
@kapa.ai I would like to have an example with a simple HttpRequester. I would like to know if for using the lookback window, is it necessary to implement somewhere?
k
Lookback Window with a Simple HttpRequester No, you do not need to implement anything special in the 
HttpRequester
itself. The 
lookback_window
is declared entirely within the 
DatetimeBasedCursor
ā€” it automatically shifts the start datetime backward before passing the date range to the requester via 
start_time_option
and 
end_time_option
. Here is a simple example directly from the Orb connector: [Orb manifest]
Copy code
requester:
  type: HttpRequester
  url_base: https://api.billwithorb.com/v1/
  http_method: GET
  authenticator:
    type: BearerAuthenticator
    api_token: "{{ config['api_key'] }}"

incremental_sync:
  type: DatetimeBasedCursor
  cursor_field: created_at
  lookback_window: P{{ config.get('lookback_window_days', 0) }}D  # <-- that's all you need
  cursor_datetime_formats:
    - "%Y-%m-%d %H:%M:%S.%f+00:00"
    - "%Y-%m-%dT%H:%M:%S.%fZ"
    - "%Y-%m-%dT%H:%M:%SZ"
  datetime_format: "%Y-%m-%dT%H:%M:%S+00:00"
  start_datetime:
    type: MinMaxDatetime
    datetime: "{{ config['start_date'] }}"
    datetime_format: "%Y-%m-%dT%H:%M:%SZ"
  start_time_option:
    type: RequestOption
    field_name: created_at[gte]
    inject_into: request_parameter   # <-- adjusted start (with lookback) injected here
  end_datetime:
    type: MinMaxDatetime
    datetime: "{{ config['end_date'] if config['end_date'] else now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
    datetime_format: "%Y-%m-%dT%H:%M:%SZ"
  end_time_option:
    type: RequestOption
    field_name: created_at[lte]
    inject_into: request_parameter
How it works ā€¢ The 
lookback_window
(e.g. 
P7D
for 7 days) is subtracted from the cursor's current 
start_datetime
automatically by the framework. ā€¢ The resulting adjusted start date is then injected into the HTTP request via 
start_time_option
ā€” no custom code needed. ā€¢ The 
HttpRequester
simply receives the date range as query parameters and knows nothing about the lookback logic. [DatetimeBasedCursor docs] In short: declare 
lookback_window
on the 
DatetimeBasedCursor
and define 
start_time_option
to inject it ā€” that's all that's required. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by