Hi, is it possible to pass a parameter to a job wh...
# help-connector-development
e
Hi, is it possible to pass a parameter to a job when syncing? like start date and number of days to backfill?
k
A message from kapa.ai
Yes, it is possible to pass parameters to a job when syncing. This is typically done when configuring the incremental sync settings. In the incremental sync form, you can use the "Inject start time / end time into outgoing HTTP request" option. This works for most cases, but sometimes the API has special requirements that can't be handled this way. In such cases, you can disable injection in the incremental sync form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers, and properties of the JSON body, by using jinja expressions and available variables. You can also use these variables as part of the URL path. For example, the Sendgrid API requires setting both start and end time in a
query
parameter. For this case, you can use the
stream_interval
variable to configure a request parameter with "key"
query
and "value"
last_event_time BETWEEN TIMESTAMP "{{stream_interval.start_time}}" AND TIMESTAMP "{{stream_interval.end_time}}"
to filter down to the right window in time. Here is the relevant extract from the [documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync):
Copy code
markdown The API requires adding a prefix or a suffix to the actual value Multiple values need to be put together in a single parameter The value needs to be injected into the URL path Some conditional logic needs to be applied To handle these cases, disable injection in the incremental sync form and use the generic parameter section at the bottom of the stream configuration form to freely configure query parameters, headers and properties of the JSON body, by using jinja expressions and available variables. You can also use these variables as part of the URL path.
Please note that the specific implementation might vary depending on the API and the connector you are using.
e
@kapa.ai how can I pass that parameter through Airflow?
k
A message from kapa.ai
To pass parameters through Airflow to Airbyte, you can use the Airbyte Airflow Operator. Here's an example of how to create a Directed Acyclic Graph (DAG) in Apache Airflow to trigger your Airbyte job:
Copy code
python from airflow import DAG   from airflow.utils.dates import days_ago   from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator      with DAG(dag_id='trigger_airbyte_job_example',    default_args={'owner': 'airflow'},    schedule_interval='@daily',    start_date=days_ago(1)    ) as dag:       money_to_json = AirbyteTriggerSyncOperator(    task_id='airbyte_money_json_example',    airbyte_conn_id='airbyte_conn_example',    connection_id='1e3b5a72-7bfd-4808-a13c-204505490110',    asynchronous=False,    timeout=3600,    wait_seconds=3    )
In this example, the Airbyte Airflow Operator accepts the following parameters: - `airbyte_conn_id`: Name of the Airflow HTTP Connection pointing at the Airbyte API. Tells Airflow where the Airbyte API is located. - `connection_id`: The ID of the Airbyte Connection to be triggered by Airflow. - `asynchronous`: Determines how the Airbyte Operator executes. When true, Airflow will monitor the Airbyte Job using an AirbyteJobSensor. Default value is
false
. - `timeout`: Maximum time Airflow will wait for the Airbyte job to complete. Only valid when
asynchronous=False
. Default value is
3600
seconds. - `wait_seconds`: The amount of time to wait between checks. Only valid when
asynchronous=False
. Default value is
3
seconds. You can find more details in the [Airbyte documentation](https://docs.airbyte.com/operator-guides/using-the-airflow-airbyte-operator).
e
@kapa.ai above I don't see any dynamic parameter that the stream can read from
k
A message from kapa.ai
New message text here
m
No it’s possible. You must add this to the connector specification
e
it is then possible to pass it through Airflow? I don't see any argument for that in the official operator
any example would help, thanks
m
Not really, today Airflow only has the trigger job option
e
so probably one solution is to extend Source and make a method available for reading commands stored in a DB (the storing piece will be the task before Airflwo triggering)