https://linen.dev logo
Join Slack
Powered by
how would I set a record filter if i want to only ...
# help-connector-development
a
how would I set a record filter if i want to only display the records which have the selected field to be not empthy in my manifest.yaml file
k
A message from kapa.ai
To set a record filter in your 
manifest.yaml
file to display only records with a non-empty selected field, you can use the 
record_filter
component with a condition that checks if the selected field is not empty. Here's an example:
Copy code
yaml selector:   extractor:     field_path: [...] # Your field path here   record_filter:     condition: "{{ record['selected_field'] != '' }}"
Replace 
selected_field
with the actual field name you want to check for non-empty values. This condition will evaluate to 
true
if the selected field is not empty, and only those records will be included. For more information on record filters, refer to the [RecordFilter object documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/components/RecordFilter%20object).
a
@kapa.ai this stream in my mainfest.yaml file gives me a json validation error
Copy code
message_media_stream:
    $ref: "#/definitions/base_stream_with_pagination"
    $parameters:
      name: "message_media"
      path: "/Accounts/{{stream_slice.id}}/Messages/{{stream_slice.message_id}}/Media.json"
      primary_key: "sid"
    retriever:
      $ref: "#/definitions/retriever"
      record_selector:
      extractor:
        field_path: ["parameters['name']"]
      record_filter:
        condition: "{{ stream_slice.message_id != '' }}"
      requester:
        $ref: "#/definitions/requester"
      paginator:
        $ref: "#/definitions/default_paginator"
      partition_router:
        $ref: "#/definitions/accounts_partition_router"
      incremental_sync:
        type: DatetimeBasedCursor
        cursor_field: "date_created"
        datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z"
        cursor_granularity: "PT0.000001S"
        step: "P1D"
k
A message from kapa.ai
It seems like there is a formatting issue in your 
message_media_stream
configuration. The 
record_selector
section is missing its properties, and the 
extractor
section should be indented properly. Here's the corrected configuration:
Copy code
yaml message_media_stream:     $ref: "#/definitions/base_stream_with_pagination"     $parameters:       name: "message_media"       path: "/Accounts/{{stream_slice.id}}/Messages/{{stream_slice.message_id}}/Media.json"       primary_key: "sid"     retriever:       $ref: "#/definitions/retriever"       record_selector:         extractor:           field_path: ["parameters['name']"]       record_filter:         condition: "{{ stream_slice.message_id != '' }}"       requester:         $ref: "#/definitions/requester"       paginator:         $ref: "#/definitions/default_paginator"       partition_router:         $ref: "#/definitions/accounts_partition_router"       incremental_sync:         type: DatetimeBasedCursor         cursor_field: "date_created"         datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z"         cursor_granularity: "PT0.000001S"         step: "P1D"
Make sure to update your 
manifest.yaml
file with the corrected configuration and try running the sync again.
a
@kapa.ai how do I resolve this error when i run the acceptance test
Copy code
self = <docker.api.client.APIClient object at 0x7f6be6caa740>

    def _retrieve_server_version(self):
        try:
            return self.version(api_version=False)["ApiVersion"]
        except KeyError:
            raise DockerException(
                'Invalid response from docker daemon: key "ApiVersion"'
                ' is missing.'
            )
        except Exception as e:
>           raise DockerException(
                f'Error while fetching server API version: {e}'
            )
E           docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

.venv/lib/python3.10/site-packages/docker/api/client.py:221: DockerException
k
A message from kapa.ai
This error occurs when Docker isn't running on your machine, and a running Docker daemon is required to run Airbyte. The error message you provided indicates a permission issue:
Copy code
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
To verify if Docker is running, you can run the following command:
Copy code
docker ps
If the Docker daemon is not running, you'll see a message like this:
Copy code
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
If you're using a Windows system and have just installed Docker, you might need to restart your machine. For more information, refer to the [troubleshooting guide on deploying](https://docs.airbyte.com/troubleshooting/on-deploying).
a
@kapa.ai this error is occuring despite docker running on my ubuntu system
k
A message from kapa.ai
If Docker is running on your Ubuntu system and you're still encountering the permission error, it might be due to the user not having the necessary permissions to access the Docker daemon. To resolve this issue, you can add your user to the 
docker
group. Run the following command to add your user to the 
docker
group:
Copy code
bash sudo usermod -aG docker $USER
After running the command, log out and log back in for the changes to take effect. Then, try running the acceptance test again. If the issue persists, ensure that the Docker daemon is running with the correct permissions by checking the ownership of the Docker socket:
Copy code
bash ls -la /var/run/docker.sock
The output should show that the socket is owned by the 
root
user and the 
docker
group. If it's not, you can change the ownership with the following command:
Copy code
bash sudo chown root:docker /var/run/docker.sock
After making these changes, try running the acceptance test again.
3 Views
Open in Slack
PreviousNext
Powered by