let's say i want to install a python library - can...
# help-connector-development
j
let's say i want to install a python library - can i use
pip install
in a venv or do i need to add to setup.py, and how do i add to setup.py any dependencies
k
A message from kapa.ai
Yes, you can use
pip install
within a virtual environment (venv) to install a Python library. However, if you're working on an Airbyte connector, it's recommended to add the dependencies to the
setup.py
file. This ensures that the build system handles installing all dependencies into the virtual environment. To add a dependency to
setup.py
, you can follow these steps: 1. Open the
setup.py
file in your connector's directory. 2. Locate the
install_requires
list. 3. Add your desired library as a string to the list, specifying the version if necessary (e.g.,
'library-name==1.0.0'
). After adding the dependency to
setup.py
, make sure to re-run the build command to install the new dependency in your virtual environment. For more information on setting up a virtual environment and managing dependencies, you can refer to these sources: - [Building a Python Source](https://docs.airbyte.com/connector-development/tutorials/building-a-python-source) - [Building a Python Destination](https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination) - [CDK Tutorial: Python HTTP](https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/install-dependencies)
j
@kapa.ai
source_defined_cursor
- If a source supports the
INCREMENTAL
sync mode, and it sets this field to true, it is responsible for determining internally how it tracks which records in a source are new or updated since the last sync. When set to
true
,
default_cursor_field
should also be set. how do i know if default_cursor_field should be true
k
A message from kapa.ai
The
default_cursor_field
should be set if a source supports the
INCREMENTAL
sync mode and has
source_defined_cursor
set to
true
. In this case, the source is responsible for determining internally how it tracks which records are new or updated since the last sync. The
default_cursor_field
is an array of keys to a field in the schema that will be used as the cursor if the user does not override it with the
cursor_field
attribute in the
ConfiguredAirbyteStream
[source](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#catalog). To know if
default_cursor_field
should be set to
true
, you need to check if the source supports
INCREMENTAL
sync mode and if it has
source_defined_cursor
set to
true
. If both conditions are met, then you should set the
default_cursor_field
.