https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> i have an issue with google search ...
# airbyte-agents-ask-ai
m
@kapa.ai i have an issue with google search console source connector. My sync is a long running sync, spanning multiple hours. My token needs to be refreshed, but after the first of 20-40 concurrent workers tries to use the stale token, it get HTTP 401 error and crashes. It should be retryable.
k
Thinking completed Google Search Console: HTTP 401 During Long-Running Syncs This is a known pattern with OAuth-based connectors. During long syncs, the OAuth access token expires mid-sync, and when one of the concurrent workers hits a stale token, the resulting 401 is not currently handled as retryable in the Google Search Console connector — it falls through to a fatal error. What the connector currently handles Looking at the 
search_analytics_error_handler
in the connector's manifest, it only explicitly handles: [manifest error handler] • 
Search Analytics QPS quota exceeded
→ 
RATE_LIMITED
(60s backoff with jitter) • 
Search Analytics load quota exceeded
→ 
RATE_LIMITED
(15-min backoff) • 
User does not have sufficient permission
→ 
IGNORE
• HTTP 400 → 
FAIL
There is no 
REFRESH_TOKEN_THEN_RETRY
rule for HTTP 401, so a token expiry during sync causes an immediate crash. What the fix looks like (for reference) Other connectors handle this pattern explicitly. For example, HubSpot's connector adds: [HubSpot manifest]
Copy code
- type: HttpResponseFilter
  action: REFRESH_TOKEN_THEN_RETRY
  http_codes:
    - 401
  error_message: "Authentication token expired. Refreshing and retrying..."
This causes the CDK's 
OAuthAuthenticator
to transparently fetch a new token and retry the request, rather than failing the sync. The HubSpot connector uses this exact approach because its tokens also expire during long syncs. [HubSpot contributing guide] Recommended actions 1. File a GitHub issue on the Airbyte repository describing this behavior — the fix is a small manifest-only change adding 
REFRESH_TOKEN_THEN_RETRY
for HTTP 401 to 
search_analytics_error_handler
. 2. As a workaround, reducing the number of concurrent workers (
num_workers
config, default: 3, max: 50) may reduce the likelihood of hitting this race condition, since fewer workers mean less concurrent token usage pressure. [CONTRIBUTING.md] 3. If you're self-hosting and comfortable modifying the connector, adding the 
REFRESH_TOKEN_THEN_RETRY
filter for HTTP 401 in the 
search_analytics_error_handler
block of 
manifest.yaml
is the correct fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
I am self hosting. What should I add and where?
@kapa.ai I am self hosting. What should I add and where?
k
Thinking completed What to Add and Where File to edit Find the Google Search Console connector manifest at:
Copy code
airbyte-integrations/connectors/source-google-search-console/manifest.yaml
Where in the file Locate the 
search_analytics_error_handler
block, which currently looks like this: [manifest]
Copy code
definitions:
  search_analytics_error_handler:
    type: DefaultErrorHandler
    backoff_strategies:
      ...
    response_filters:
      - type: HttpResponseFilter
        action: RATE_LIMITED
        error_message_contains: "Search Analytics QPS quota exceeded"
      - type: HttpResponseFilter
        action: RATE_LIMITED
        error_message_contains: "Search Analytics load quota exceeded"
      - type: HttpResponseFilter
        action: IGNORE
        error_message_contains: "User does not have sufficient permission"
        error_message: "{{ response['error']['message'] }}"
      - type: HttpResponseFilter
        action: FAIL
        http_codes:
          - 400
        error_message: >-
          Invalid aggregationType ...
What to add Insert a new filter before the HTTP 400 
FAIL
entry (order matters — filters are evaluated sequentially and the first match wins): [error handling docs]
Copy code
- type: HttpResponseFilter
        action: REFRESH_TOKEN_THEN_RETRY
        http_codes:
          - 401
        failure_type: transient_error
        error_message: "Authentication token expired mid-sync. Refreshing token and retrying."
Result The full 
response_filters
list should look like:
Copy code
response_filters:
      - type: HttpResponseFilter
        action: RATE_LIMITED
        error_message_contains: "Search Analytics QPS quota exceeded"
      - type: HttpResponseFilter
        action: RATE_LIMITED
        error_message_contains: "Search Analytics load quota exceeded"
      - type: HttpResponseFilter
        action: IGNORE
        error_message_contains: "User does not have sufficient permission"
        error_message: "{{ response['error']['message'] }}"
      - type: HttpResponseFilter       # <-- ADD THIS
        action: REFRESH_TOKEN_THEN_RETRY
        http_codes:
          - 401
        failure_type: transient_error
        error_message: "Authentication token expired mid-sync. Refreshing token and retrying."
      - type: HttpResponseFilter
        action: FAIL
        http_codes:
          - 400
        error_message: >-
          Invalid aggregationType ...
REFRESH_TOKEN_THEN_RETRY
will instruct the CDK's 
OAuthAuthenticator
to fetch a fresh token and replay the failed request, which is the same mechanism HubSpot uses for the same problem. [HubSpot CONTRIBUTING] > Note: After editing the manifest, you will need to rebuild and redeploy the connector image for the change to take effect in your self-hosted environment. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai I have seen that 403 and 401 HTTP errors were handled in version 1.2.2 of this connector. What happened to that?
k
Thinking completed History of HTTP 401/403 Handling in Google Search Console The changelog confirms that version 1.2.2 (released 2023-08-23) explicitly added handling for both errors: [changelog] > 
1.2.2 2023-08-23 [29741] Handle HTTP-401, HTTP-403 errors
What appears to have happened The connector went through a major refactor at some point and was eventually rebuilt as a manifest-only (low-code CDK) connector. During that migration, the 401/403 handling was replaced with the current 
search_analytics_error_handler
, which only handles specific 403 message strings rather than generic HTTP status codes: [manifest] • 
"Search Analytics QPS quota exceeded"
→ 
RATE_LIMITED
• 
"Search Analytics load quota exceeded"
→ 
RATE_LIMITED
(added in 2.1.0) • 
"User does not have sufficient permission"
→ 
IGNORE
A generic HTTP 401 is not explicitly handled at all in the current manifest. What I cannot confirm The knowledge sources do not contain the details of PR #29741 (the 1.2.2 fix), so I cannot say exactly what that original handling did, nor which specific subsequent PR dropped it during the manifest migration. That context would require reviewing the git history directly. This does however reinforce why adding the 
REFRESH_TOKEN_THEN_RETRY
filter for HTTP 401 as suggested previously is a valid restoration of behavior that was once explicitly supported. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by