https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> please provide sample terraform res...
# replication-ask-ai
c
@kapa.ai please provide sample terraform resources for creating a airtable source
k
Thinking completed In Airbyte Terraform Provider v1.0+, connectors are created using the generic 
airbyte_source
resource [Migrating to 1.0]. Prerequisites for Airtable Authentication When configuring an Airtable source via Personal Access Token, ensure the token has the following scopes [Setup guide]: • 
data.records:read
• 
data.recordComments:read
• 
schema.bases:read
Method 1: Using airbyte_connector_configuration (Recommended) This approach validates configuration against the connector schema during 
terraform plan
, automatically resolves the connector 
definition_id
, and keeps sensitive tokens masked [Using the airbyte_connector_configuration data source (recommended); airbyte_connector_configuration (Data Source)].
Copy code
variable "workspace_id" {
  type = string
}

variable "airtable_api_key" {
  type        = string
  sensitive   = true
  description = "Airtable Personal Access Token"
}

data "airbyte_connector_configuration" "airtable" {
  connector_name = "source-airtable"

  configuration = {
    # Optional settings
    # num_workers                = 5
    # add_base_id_to_stream_name = false

    credentials = {
      auth_method = "api_key"
    }
  }

  configuration_secrets = {
    credentials = {
      api_key = var.airtable_api_key
    }
  }
}

resource "airbyte_source" "airtable" {
  name          = "Airtable Source"
  workspace_id  = var.workspace_id
  definition_id = data.airbyte_connector_configuration.airtable.definition_id
  configuration = data.airbyte_connector_configuration.airtable.configuration_json
}
Method 2: Using Inline JSON Configuration Alternatively, you can provide the connector definition ID directly (
14c6e7ea-97ed-4f5e-a7b5-25e9a80b8212
for Airtable) with an inline 
jsonencode
block [source-airtable metadata; Using inline JSON configuration]:
Copy code
resource "airbyte_source" "airtable" {
  name          = "Airtable Source"
  workspace_id  = var.workspace_id
  definition_id = "14c6e7ea-97ed-4f5e-a7b5-25e9a80b8212"

  configuration = jsonencode({
    credentials = {
      auth_method = "api_key"
      api_key     = var.airtable_api_key
    }
    # Optional parameters:
    # num_workers                = 5
    # add_base_id_to_stream_name = false
  })
}
_(Note: OAuth2.0 is also supported by the connector schema via 
client_id
, 
client_secret
, and 
refresh_token
, but Airtable issues single-use rotating refresh tokens that expire and rotate every 60 minutes [source-airtable: Unique Behaviors; source-airtable manifest.yaml], making Personal Access Tokens standard for declarative Terraform setups)._ 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
c
@kapa.ai can I get a sample terraform for Apple search ads
k
Thinking completed Here are examples for configuring an Apple Ads (Apple Search Ads) source using the generic 
airbyte_source
resource in the Airbyte Terraform Provider v1.0+ [airbyte_source (Resource)]. Required Configuration Fields The Apple Ads connector requires the following fields [Reference; Config fields reference]: • `org_id`: Apple Ads organization ID (integer). • `client_id`: OAuth client ID for the API user. • `client_secret`: OAuth client secret for the API user. • `start_date`: Earliest report date to sync (
YYYY-MM-DD
). • `timezone`: Reporting time zone (
UTC
or 
ORTZ
). • `token_refresh_endpoint`: OAuth token endpoint (default: 
https://appleid.apple.com/auth/oauth2/token?grant_type=client_credentials&scope=searchadsorg
) [definitions]. Method 1: Using airbyte_connector_configuration (Recommended) This approach validates the schema during 
terraform plan
, resolves the definition ID automatically, and separates sensitive secrets from plain configuration [Using the airbyte_connector_configuration data source (recommended); airbyte_connector_configuration (Data Source)].
Copy code
variable "workspace_id" {
  type = string
}

variable "apple_ads_client_secret" {
  type      = string
  sensitive = true
}

data "airbyte_connector_configuration" "apple_ads" {
  connector_name = "source-apple-search-ads"

  configuration = {
    org_id                 = 12345678
    client_id              = "SEARCHADS.00000000-0000-0000-0000-000000000000"
    start_date             = "2024-01-01"
    timezone               = "UTC"
    token_refresh_endpoint = "https://appleid.apple.com/auth/oauth2/token?grant_type=client_credentials&scope=searchadsorg"

    # Optional fields:
    # end_date        = "2024-12-31"
    # lookback_window = 30
    # backoff_factor  = 5
    # num_workers     = 2
  }

  configuration_secrets = {
    client_secret = var.apple_ads_client_secret
  }
}

resource "airbyte_source" "apple_ads" {
  name          = "Apple Ads"
  workspace_id  = var.workspace_id
  definition_id = data.airbyte_connector_configuration.apple_ads.definition_id
  configuration = data.airbyte_connector_configuration.apple_ads.configuration_json
}
Method 2: Using Inline JSON Configuration You can also supply the connector's definition ID directly (
e59c8416-c2fa-4bd3-9e95-52677ea281c1
[metadata.yaml]) along with an inline JSON payload [Using inline JSON configuration]:
Copy code
resource "airbyte_source" "apple_ads" {
  name          = "Apple Ads"
  workspace_id  = var.workspace_id
  definition_id = "e59c8416-c2fa-4bd3-9e95-52677ea281c1"

  configuration = jsonencode({
    org_id                 = 12345678
    client_id              = "SEARCHADS.00000000-0000-0000-0000-000000000000"
    client_secret          = var.apple_ads_client_secret
    start_date             = "2024-01-01"
    timezone               = "UTC"
    token_refresh_endpoint = "https://appleid.apple.com/auth/oauth2/token?grant_type=client_credentials&scope=searchadsorg"

    # Optional fields:
    # end_date        = "2024-12-31"
    # lookback_window = 30
    # backoff_factor  = 5
    # num_workers     = 2
  })
}
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
c
@kapa.ai Im getting below error when trying to create a apple search ads terraform connector
Copy code
2026-09-24 08:10:08,583 [public-api-executor-thread-5]  ERROR   i.a.c.s.e.p.AbstractThrowableProblemHandler(handle):31 - Throwable Problem Handler caught exception:
io.airbyte.api.problems.throwable.generated.UnexpectedProblem: Detail: class UnexpectedProblemResponse {
    status: 500
    type: <https://reference.airbyte.com/reference/errors>
    title: unexpected-problem
    detail: An unexpected problem has occurred. If this is an error that needs to be addressed, please submit a pull request or github issue.
    documentationUrl: null
    data: class ProblemMessageData {
        message: Something went wrong in the connector. logs:Something went wrong in the connector. See the logs for more details.
    }
}.detail, Data: class ProblemMessageData {
    message: Something went wrong in the connector. logs:Something went wrong in the connector. See the logs for more details.
}
        at io.airbyte.server.apis.publicapi.services.SourceServiceImpl.getSourceSchema(SourceService.kt:227)
        at io.airbyte.server.apis.publicapi.controllers.ConnectionsController.publicCreateConnection$lambda$2(ConnectionsController.kt:97)
        at io.airbyte.server.apis.publicapi.apiTracking.TrackingHelper.callWithTracker(TrackingHelper.kt:109)
        at io.airbyte.server.apis.publicapi.controllers.ConnectionsController.publicCreateConnection(ConnectionsController.kt:96)
        at io.airbyte.server.apis.publicapi.controllers.$ConnectionsController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
k
Thinking completed The error stack trace points to 
SourceServiceImpl.getSourceSchema()
failing during connection or schema discovery with an 
UnexpectedProblem
(500) [Github Issue (open): [OSS] Issue with Get stream properties API]:
Copy code
message: Something went wrong in the connector. logs:Something went wrong in the connector. See the logs for more details.
    at io.airbyte.server.apis.publicapi.services.SourceServiceImpl.getSourceSchema(SourceService.kt:227)
In Airbyte, this occurs when the connector fails during the 
discover
step, and the server wraps the underlying connector failure into a generic 500 error envelope [Github Issue (open): [OSS] Issue with Get stream properties API]. To resolve this issue, check the following common causes for the Apple Ads connector: 1. Check Data Types in Your Terraform Configuration Ensure your parameters in Terraform match the connector's expected types [Reference; manifest.yaml]: • 
org_id
must be an integer, not a string (e.g., 
org_id = 12345678
, not 
"12345678"
). Passing it as a string can cause the connector's schema validation or header construction to fail [manifest.yaml; Reference]. • Ensure 
timezone
 is set strictly to 
"UTC"
or 
"ORTZ"
[Reference; manifest.yaml]. • Ensure 
start_date
 matches the 
YYYY-MM-DD
pattern [manifest.yaml]. 2. Verify Credentials and Permissions The connector attempts to authenticate against the Apple Search Ads Campaign Management API during discover [Reference]. If the authentication handshake fails: • Confirm that the 
client_secret
(private-key-generated client secret) has not expired [Step 1: Create an API user and OAuth credentials in Apple Ads]. • Verify that the 
client_id
corresponds to an API user role within the specified 
org_id
[Prerequisites; Step 1: Create an API user and OAuth credentials in Apple Ads]. • Ensure the 
token_refresh_endpoint
is set to the defaulthttps//appleid.apple.com/auth/oauth2/token?grant_type=client_credentials&scope=searchadsorgunless you are routing through a custom proxy [Step 2: Set up the source connector in Airbyte; manifest.yaml]. 3. Check the Underlying Connector Logs Because the Airbyte API surfaces this generic error message when an underlying job fails [Github Issue (open): [OSS] Issue with Get stream properties API], inspect the Airbyte server and discover pod/container logs directly: • In the Airbyte UI, go to the source settings and run Test connection to see the detailed error output. • Check the server/worker logs (
airbyte-server
or 
airbyte-worker
) around the timestamp 
08:10:08
to find the exact exception returned by the connector container during the discover attempt. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
c
@kapa.ai Please see below error on discover pod
Copy code
2026-09-24 08:14:45,371 [main]  ERROR   i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):242 - ISO 8601 time designator 'T' missing. Unable to parse datetime string 'D'
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/isodate/isodatetime.py", line 51, in parse_datetime
    datestring, timestring = datetimestring.split('T')
    ^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 4, in <module>
    run()
    ~~~^^
  File "/airbyte/integration_code/source_declarative_manifest/run.py", line 307, in run
    handle_command(args)
    ~~~~~~~~~~~~~~^^^^^^
  File "/airbyte/integration_code/source_declarative_manifest/run.py", line 91, in handle_command
    handle_local_manifest_command(args)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/airbyte/integration_code/source_declarative_manifest/run.py", line 129, in handle_local_manifest_command
    launch(
    ~~~~~~^
        source=source,
        ^^^^^^^^^^^^^^
        args=args,
        ^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/entrypoint.py", line 397, in launch
    for message in source_entrypoint.run(parsed_args):
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/entrypoint.py", line 201, in run
    yield from map(
    ...<2 lines>...
    )
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/entrypoint.py", line 268, in discover
    catalog = self.source.discover(self.logger, config)
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/concurrent_declarative_source.py", line 392, in discover
    streams=[stream.as_airbyte_stream() for stream in self.streams(config=self._config)]
                                                      ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/concurrent_declarative_source.py", line 418, in streams
    self._constructor.create_component(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        (
        ^
    ...<6 lines>...
        emit_connector_builder_messages=self._emit_connector_builder_messages,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 871, in create_component
    return self._create_component_from_model(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model=declarative_component_model, config=config, **kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 888, in _create_component_from_model
    return component_constructor(model=model, config=config, **kwargs)
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 2009, in create_default_stream
    concurrent_cursor = self._build_concurrent_cursor(model, partition_router, config)
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 2283, in _build_concurrent_cursor
    return self.create_concurrent_cursor_from_datetime_based_cursor(  # type: ignore # This is a known issue that we are creating and returning a ConcurrentCursor which does not technically implement the (low-code) StreamSlicer. However, (low-code) StreamSlicer and ConcurrentCursor both implement StreamSlicer.stream_slices() which is the primary method needed for checkpointing
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        model_type=type(model.incremental_sync),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        attempt_to_create_cursor_if_not_provided=True,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py", line 1406, in create_concurrent_cursor_from_datetime_based_cursor
    lookback_window = parse_duration(evaluated_lookback_window)
  File "/usr/local/lib/python3.13/site-packages/isodate/isoduration.py", line 91, in parse_duration
    durdt = parse_datetime(datestring[1:])
  File "/usr/local/lib/python3.13/site-packages/isodate/isodatetime.py", line 53, in parse_datetime
    raise ISO8601Error("ISO 8601 time designator 'T' missing. Unable to"
                       " parse datetime string %r" % datetimestring)
isodate.isoerror.ISO8601Error: ISO 8601 time designator 'T' missing. Unable to parse datetime string 'D'
2026-09-24 08:14:45,373 [main]  INFO    i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):280 - Checking for optional control message...
2026-09-24 08:14:45,376 [main]  INFO    i.a.c.ConnectorWatcher(saveConnectorOutput):202 - Writing output of e59c8416-c2fa-4bd3-9e95-52677ea281c1_54302cab-9544-4dfc-96b8-02e31c676277_0_discover to the doc store
2026-09-24 08:14:45,451 [pool-3-thread-1]       INFO    i.a.c.HeartbeatMonitor$HeartbeatTask(run):94 - Workload successfully transitioned to running state
2026-09-24 08:14:45,543 [main]  INFO    i.a.c.ConnectorWatcher(markWorkloadSuccess):207 - Marking workload e59c8416-c2fa-4bd3-9e95-52677ea281c1_54302cab-9544-4dfc-96b8-02e31c676277_0_discover as successful
2026-09-24 08:14:45,576 [main]  INFO    i.a.c.ConnectorWatcher(exitProperly):265 - Deliberately exiting process with code 0.
2026-09-24 08:14:45,576 [pool-7-thread-1]       INFO    i.a.c.i.LineGobbler$Companion(gobble$lambda$1):108 -
2026-09-24 08:14:45,577 [pool-7-thread-1]       INFO    i.a.c.i.LineGobbler$Companion(gobble$lambda$1):108 - ----- END DISCOVER -----
k
Thinking completed This error is caused by how the 
lookback_window
property is being passed or evaluated in the connector. Root Cause In the traceback:
Copy code
File ".../model_to_component_factory.py", line 1406, in create_concurrent_cursor_from_datetime_based_cursor
  lookback_window = parse_duration(evaluated_lookback_window)
File ".../isoduration.py", line 91, in parse_duration
  durdt = parse_datetime(datestring[1:])
...
isodate.isoerror.ISO8601Error: ISO 8601 time designator 'T' missing. Unable to parse datetime string 'D'
The Airbyte CDK expects an ISO-8601 duration format (e.g. 
P{{ config['lookback_window'] }}D
). When the connector tries to evaluate the lookback duration, it receives only 
"D"
(which becomes an invalid ISO-8601 string) because the numerical value for 
lookback_window
evaluated to empty/null. How to Fix 1. Explicitly Set lookback_window as an Integer In your Terraform configuration, explicitly set 
lookback_window
to an integer (between 1 and 30, with 30 being the connector default) [Step 2: Set up the source connector in Airbyte; manifest.yaml]:
Copy code
lookback_window = 30
• Ensure you do not set 
lookback_window = ""
or pass an empty string. • Ensure it is an integer, not a string (
30
, not 
"30"
). 2. Check Other Required Integer and String Fields Make sure other optional or required fields in your Terraform resource are either assigned valid values or omitted completely rather than set to empty strings:
Copy code
configuration = {
  org_id                 = 12345678       # Integer (required)
  client_id              = "SEARCHADS..." # String (required)
  start_date             = "2024-01-01"   # String YYYY-MM-DD (required)
  timezone               = "UTC"          # String: "UTC" or "ORTZ" (required)
  token_refresh_endpoint = "https://appleid.apple.com/auth/oauth2/token?grant_type=client_credentials&scope=searchadsorg"
  lookback_window        = 30             # Integer: 1 to 30
  backoff_factor         = 5              # Integer: 1 to 20
  num_workers            = 2              # Integer: 1 to 20
}
Once 
lookback_window
is explicitly populated with an integer, the CDK interpolates it properly (e.g., 
P30D
) and the ISO parser will succeed. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Open in Slack
PreviousNext
Powered by