https://linen.dev logo
Join Slack
Powered by
hey guys im not able to see a zoom recording - so ...
# help-connector-development
j
hey guys im not able to see a zoom recording - so hoping someone can help me out here: i made a change to one of the connectors that come with the deployment 
source-pardot
and i spun up the docker image, but having a hard time finding it on the airbyte webui
k
A message from kapa.ai
To use your modified connector in the Airbyte web UI, you need to follow these steps: 1. Build the connector locally using the following command:
Copy code
./gradlew :airbyte-integrations:connectors:source-pardot:build
2. After building the connector, you should have a Docker image with the 
dev
tag. You can check this by running 
docker images
. 3. In the Airbyte web UI, go to the Admin section, and click on 
[+ New connector]
on the top right. 4. Fill in the display name, the Docker repository name (repository + image name), tag (use 
dev
), and documentation URL for that connector. Once you've completed these steps, your modified connector should be visible in the UI, and you can use it for your data integrations. For more information on using custom connectors, refer to the [Airbyte documentation](https://docs.airbyte.com/integrations/custom-connectors). If you still can't see the connector in the UI, you can try checking the server logs (
docker logs airbyte-server
) or the developer mode in the network tab and check the API request/response as mentioned in this [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1674677116471079).
j
@kapa.ai are you still available
k
A message from kapa.ai
Yes, I am available to help you with any questions related to Airbyte. Please feel free to ask your question, and I will do my best to provide you with an informative and actionable answer based on the information extracts provided.
j
@kapa.ai can you give me a break down of all the technologies that airbyte has under the hood
k
A message from kapa.ai
👀 1
Airbyte is composed of two main parts: the platform and connectors. The platform provides horizontal services for configuring and running data movement operations, while connectors are independent modules that push/pull data to/from sources and destinations. Here's a breakdown of the technologies used in Airbyte's core backend: 1. [Java 17](https://jdk.java.net/archive/) 2. Framework: [Jersey](https://eclipse-ee4j.github.io/jersey/) 3. API: [OAS3](https://www.openapis.org/) 4. Databases: [PostgreSQL](https://www.postgresql.org/) 5. Unit & E2E testing: [JUnit 5](https://junit.org/junit5) 6. Orchestration: [Temporal](https://temporal.io) Source: [Technical Stack](https://docs.airbyte.com/understanding-airbyte/tech-stack) For developing Airbyte locally, the following technologies are required: 1. [
Java 17
](https://jdk.java.net/archive/) 2. 
Node 16
3. 
Python 3.9
4. 
Docker
5. 
Jq
Source: [Developing Locally](https://docs.airbyte.com/contributing-to-airbyte/developing-locally) Connectors are built according to the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol) and are packaged as Docker images, allowing flexibility in the technologies used to implement them. Source: [Architecture Overview](https://docs.airbyte.com/understanding-airbyte/high-level-view)
j
@kapa.ai hey where in the code do i find AirbtyeCatalog and AirbyteStream objects - i have my json shcema, wondering where that goes
k
A message from kapa.ai
New message text here
j
@kapa.aiwhen i cloned the git repo with 
git clone <https://github.com/airbytehq/airbyte.git>
it didn't pull the 
airbyte-protocol
directory...
k
A message from kapa.ai
I apologize for the confusion. The 
airbyte-protocol
directory is actually located within the 
airbyte
repository under the 
airbyte-protocol/models/src/main/resources/airbyte_protocol
path. You can find the Airbyte Protocol YAML file at this location:
Copy code
airbyte/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml
If you still can't find the 
airbyte-protocol
directory, please make sure you have successfully cloned the repository and navigated to the correct path. If you need to clone the repository again, use the following command:
Copy code
git clone <https://github.com/airbytehq/airbyte.git>
Once you have cloned the repository, navigate to the 
airbyte-protocol
directory using:
Copy code
cd airbyte/airbyte-protocol/models/src/main/resources/airbyte_protocol
You should now be able to find the 
airbyte_protocol.yaml
file in this directory.
j
@kapa.ai this is what i got:
Copy code
drwxr-xr-x 25 admin admin  4096 Jun  7 17:55 .
drwxr-xr-x 10 admin admin  4096 Jun  2 16:50 ..
-rw-r--r--  1 admin admin   586 May 30 15:48 .bumpversion.cfg
-rw-r--r--  1 admin admin    92 May 30 15:48 .dockerignore
-rw-r--r--  1 admin admin  5616 May 30 15:48 .editorconfig
-rw-r--r--  1 admin admin  4241 May 30 15:48 .env
-rw-r--r--  1 admin admin  1339 May 30 15:48 .env.dev
drwxr-xr-x  8 admin admin  4096 May 30 15:48 .git
drwxr-xr-x  6 admin admin  4096 May 30 15:48 .github
-rw-r--r--  1 admin admin  1455 May 30 15:48 .gitignore
-rw-r--r--  1 admin admin  1741 May 30 15:48 .pre-commit-config.yaml
-rw-r--r--  1 admin admin    90 May 30 15:48 .prettierignore
-rw-r--r--  1 admin admin     7 May 30 15:48 .python-version
-rw-r--r--  1 admin admin   547 May 30 15:48 .readthedocs.yaml
-rw-r--r--  1 admin admin     0 May 30 15:48 .root
-rw-r--r--  1 admin admin   109 May 30 15:48 CODE_OF_CONDUCT.md
-rw-r--r--  1 admin admin    90 May 30 15:48 CONTRIBUTING.md
-rw-r--r--  1 admin admin  6288 May 30 15:48 CONTRIBUTORS.md
-rw-r--r--  1 admin admin  5605 May 30 15:48 LICENSE
-rw-r--r--  1 admin admin    55 May 30 15:48 LICENSE_SHORT
-rw-r--r--  1 admin admin  9415 May 30 15:48 README.md
-rw-r--r--  1 admin admin  1579 May 30 15:48 THANK-YOU.md
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-api
drwxr-xr-x  2 admin admin  4096 May 30 15:48 airbyte-base-java-image
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-cdk
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-ci
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-commons
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-commons-cli
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-commons-protocol
drwxr-xr-x  4 admin admin  4096 May 30 15:48 airbyte-config-oss
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-connector-test-harnesses
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-db
drwxr-xr-x  8 admin admin  4096 May 30 23:04 airbyte-integrations
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-json-validation
drwxr-xr-x  3 admin admin  4096 May 30 15:48 airbyte-test-utils
-rw-r--r--  1 admin admin 19447 May 30 15:48 build.gradle
drwxr-xr-x  3 admin admin  4096 May 30 15:48 buildSrc
-rw-r--r--  1 admin admin   339 May 30 15:48 codecov.yml
-rw-r--r--  1 admin admin   480 May 30 15:48 connectors.md
-rw-r--r--  1 admin admin 13538 May 30 15:48 deps.toml
-rw-r--r--  1 admin admin  1886 May 30 15:48 docker-compose.debug.yaml
-rw-r--r--  1 admin admin 11064 May 30 16:43 docker-compose.yaml
drwxr-xr-x 16 admin admin  4096 May 30 15:48 docs
drwxr-xr-x  4 admin admin  4096 May 30 15:48 docusaurus
-rw-r--r--  1 admin admin   520 May 30 15:48 flags.yml
drwxr-xr-x  3 admin admin  4096 May 30 15:48 gradle
-rw-r--r--  1 admin admin   404 May 30 15:48 gradle.properties
-rwxr-xr-x  1 admin admin  8497 May 30 15:48 gradlew
-rw-r--r--  1 admin admin  2868 May 30 15:48 gradlew.bat
drwxr-xr-x  5 admin admin  4096 May 30 15:48 octavia-cli
-rw-r--r--  1 admin admin   284 May 30 15:48 publish-repositories.gradle
-rw-r--r--  1 admin admin  1844 May 30 15:48 pyproject.toml
-rw-r--r--  1 admin admin    71 May 30 15:48 pytest.ini
drwxr-xr-x  3 admin admin  4096 May 30 15:48 resources
-rwxr-xr-x  1 admin admin  5632 May 30 15:48 run-ab-platform.sh
-rw-r--r--  1 admin admin  6874 May 30 15:48 settings.gradle
-rw-r--r--  1 admin admin   918 May 30 15:48 spotbugs-exclude-filter-file.xml
drwxr-xr-x  3 root  root   4096 May 30 15:50 temporal
drwxr-xr-x 18 admin admin  4096 May 30 15:48 tools
k
A message from kapa.ai
I apologize for the confusion earlier. It seems that the Airbyte Protocol YAML file is not present in the repository you have cloned. The repository structure you provided appears to be outdated. The current Airbyte repository has a different structure, and the Airbyte Protocol YAML file is located in the 
airbyte-platform
repository. To access the Airbyte Protocol YAML file, you can find it in the 
airbyte-platform
repository at the following path:
Copy code
airbyte-platform/charts/airbyte/values.yaml
To clone the 
airbyte-platform
repository, use the following command:
Copy code
git clone <https://github.com/airbytehq/airbyte-platform.git>
Once you have cloned the repository, navigate to the 
charts/airbyte
directory using:
Copy code
cd airbyte-platform/charts/airbyte
You should now be able to find the 
values.yaml
file in this directory. Source: [airbyte-platform repository](https://github.com/airbytehq/airbyte-platform/blob/main/charts/airbyte/values.yaml)
j
@kapa.ai let's say i want to install a python library - can i use 
pip install
in a venv or do i need to add to setup.py, and how do i add to setup.py any dependencies
k
A message from kapa.ai
@Jake Kagan That's a great question, but unfortunately I am only capable of responding to a maximum of five questions per thread. Don't hesitate to create a new thread for any further questions you have though. I'm here to help!
Open in Slack
PreviousNext
Powered by