https://linen.dev logo
Join Slack
Powered by
I found an issue in facebook-marketing connector. ...
# replication-ask-ai
s
I found an issue in facebook-marketing connector. I wanted to contribute. ad insights Schema is missing the field called video_thruplay_watched_actions and I know how to add it but I am facing a problem setting up connector locally. So I am unable to test my changes. Here is my issue url: https://github.com/airbytehq/airbyte/issues/38579
k
It looks like you want to contribute to the Airbyte project by adding the 
video_thruplay_watched_actions
field to the Ad Insights schema in the Facebook Marketing connector. However, you're facing issues setting up the connector locally to test your changes. Here are some steps and resources that might help you get started: 1. Setting Up the Development Environment: - Ensure you have the necessary prerequisites installed, such as Docker and Python. - Clone the Airbyte repository from GitHub:
Copy code
bash      git clone <https://github.com/airbytehq/airbyte.git>      cd airbyte
- Follow the [Airbyte Contributor Guide](https://docs.airbyte.com/contributing-to-airbyte/) for detailed instructions on setting up your development environment. 2. Adding the New Field: - Navigate to the source code for the Facebook Marketing connector. You can find it under 
airbyte-integrations/connectors/source-facebook-marketing
. - Modify the schema to include the 
video_thruplay_watched_actions
field. 3. Testing Your Changes: - Once you've made your changes, you can run the connector locally to test it. Use Docker to build and run the connector:
Copy code
bash      ./gradlew :airbyte-integrations:connectors:source-facebook-marketing:build      docker-compose up -d
- Follow the [Airbyte Testing Guide](https://docs.airbyte.com/contributing-to-airbyte/testing-connectors/) for more details on how to test your connector. 4. Seeking Help: - If you continue to face issues, consider reaching out on the [Airbyte Slack Community](https://slack.airbyte.io/) or opening a discussion on the [Airbyte Forum](https://discuss.airbyte.io/). 5. Submitting Your Contribution: - Once you've successfully tested your changes, you can submit a pull request to the Airbyte repository. Make sure to reference your issue URL in the PR description: [Issue #38579](https://github.com/airbytehq/airbyte/issues/38579). For more specific guidance, you might find it helpful to look at similar issues and contributions, such as the ones mentioned in the knowledge sources: - [Issue #20197](https://github.com/airbytehq/airbyte/issues/20197) discusses adding a new enumeration to the Ad Insights stream class. - [Issue #31860](https://github.com/airbytehq/airbyte/issues/31860) involves investigating discrepancies in data and might provide insights into handling schema changes. By following these steps, you should be able to set up the connector locally, make your changes, and test them effectively. Good luck with your contribution!
s
@kapa.ai But following connector is python based
k
It seems like you are referring to a Python-based connector in Airbyte. Here are some key points and guidelines for working with Python-based connectors in Airbyte: 1. Common Languages for Connectors: - Connectors can be written in any language, but the most common languages are Python 3.9 or higher and Java 21. [Source](https://docs.airbyte.com/understanding-airbyte/tech-stack#connectors) 2. Python CDK: - Most REST API connectors are written in Python because many contributors are comfortable with it. Airbyte provides a [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) to accelerate development. [Source](https://docs.airbyte.com/understanding-airbyte/tech-stack#why-are-most-rest-api-connectors-written-in-python) 3. Base Image Requirement: - Python connectors must not use a Dockerfile and must declare their base image in the 
metadata.yaml
file. They should use the Airbyte Python connector base image (`docker.io/airbyte/python-connector-base`). [Source](https://docs.airbyte.com/contributing-to-airbyte/resources/qa-checks#python-connectors-must-not-use-a-dockerfile-and-must-declare-their-base-image-in-metadatayaml-file) 4. PyPi Publishing: - Python connectors must have PyPi publishing enabled in their 
metadata.yaml
file by setting 
remoteRegistries.pypi.enabled
to 
true
. This ensures that connectors can be published to PyPi and used in 
PyAirbyte
. [Source](https://docs.airbyte.com/contributing-to-airbyte/resources/qa-checks#python-connectors-must-have-pypi-publishing-enabled) 5. Example of Database Connector in Python: - If you are looking to build a database connector using Python, you might find it useful to look at examples or discussions on how to handle the 
source.read
method and whether to use streams for processing data in chunks. [Source](https://discuss.airbyte.io/t/1751) 6. Testing Python Connectors: - For testing a Python destination connector, you can start with the native Python way, which does not require Docker or involvement of Airbyte itself. [Source](https://airbyte.com/tutorials/creating-duckdb-destination-with-python) 7. HTTP API-based Connectors: - The CDK offers base classes that simplify writing HTTP API-based connectors, including features for authentication, pagination, rate limiting, and caching. These features are customizable and can be combined with other stream features. [Source](https://docs.airbyte.com/connector-development/cdk-python/http-streams) If you need more specific information or examples, please refer to the provided links or let me know if you have any other questions!
s
@kapa.ai I am contributing python based source connector
k
A message from kapa.ai
Great! Here are some key steps and guidelines to help you contribute a Python-based source connector to Airbyte: ### Steps to Contribute a Python-Based Source Connector 1. Fork the Connector Repository: - Start by forking the [Connector repository](https://github.com/airbytehq/airbyte). 2. Open a Branch for Your Work: - Create a new branch in your forked repository for your connector development. 3. Implement the Connector: - In the connector package directory, create a single Python file 
source.py
that will hold your implementation:
Copy code
bash      touch source.py
- Follow the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol) to implement your connector. 4. Write Unit Tests: - Write unit tests for each custom function you added or changed. 5. Ensure All Tests Pass: - Make sure all tests, including connector acceptance tests, pass. 6. Update 
metadata.yaml
: - Update the 
metadata.yaml
following the [guidelines](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook#semantic-versioning-for-connectors). - Ensure that your connector uses the Airbyte Python connector base image (`docker.io/airbyte/python-connector-base`):
Copy code
yaml      connectorBuildOptions:        baseImage: "<http://docker.io/airbyte/python-connector-base|docker.io/airbyte/python-connector-base>"
- Enable PyPi publishing by setting 
remoteRegistries.pypi.enabled
to `true`:
Copy code
yaml      remoteRegistries:        pypi:          enabled: true
7. Update Changelog: - Update the changelog entry in the documentation in 
docs/integrations/<connector-name>.md
. 8. Pass QA Checks: - Make sure your contribution passes the [QA checks](https://docs.airbyte.com/contributing-to-airbyte/resources/qa-checks). ### Additional Resources - Example Connectors: - You can refer to example connectors like [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/source_google_sheets/source.py) and [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py) for guidance. - Python CDK: - Utilize the [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) to accelerate your development process. - Technical Stack: - Ensure you are using Python 3.9 as recommended in the [technical stack requirements](https://docs.airbyte.com/contributing-to-airbyte/developing-locally). By following these steps and guidelines, you can effectively contribute a Python-based source connector to Airbyte. If you have any specific questions or need further assistance, feel free to ask!
s
@kapa.ai I am modifying existing connector called source-facebook-marketing which is python based
k
A message from kapa.ai
To modify the existing 
source-facebook-marketing
connector, which is Python-based, follow these steps: ### Steps to Modify the 
source-facebook-marketing
Connector 1. Fork and Clone the Repository: - Fork the [Airbyte repository](https://github.com/airbytehq/airbyte). - Clone your forked repository to your local machine:
Copy code
bash
     git clone <https://github.com/<your-username>/airbyte.git>
     cd airbyte
2. Navigate to the Connector Directory: - Navigate to the 
source-facebook-marketing
connector directory:
Copy code
bash
     cd airbyte-integrations/connectors/source-facebook-marketing
3. Create a New Branch: - Create a new branch for your modifications:
Copy code
bash
     git checkout -b modify-facebook-marketing-connector
4. Modify the Connector Code: - Open the 
source.py
file and make the necessary modifications. For example, if you need to add a new parameter or stream, you can update the relevant classes and methods. 5. Update Dependencies: - If you need to add or update dependencies, modify the 
setup.py
or 
requirements.txt
file accordingly. 6. Write Unit Tests: - Add or update unit tests to cover your changes. Ensure that all tests are located in the 
unit_tests
directory. 7. Run Tests: - Run the tests to ensure everything is working correctly:
Copy code
bash
     poetry install --with dev
     poetry run pytest unit_tests
8. Update Documentation: - Update the documentation to reflect your changes. This includes the 
README.md
and any relevant markdown files in the 
docs
directory. 9. Commit and Push Your Changes: - Commit your changes and push them to your forked repository:
Copy code
bash
     git add .
     git commit -m "Modify source-facebook-marketing connector"
     git push origin modify-facebook-marketing-connector
10. Create a Pull Request: - Go to the original Airbyte repository and create a pull request from your forked repository. Provide a detailed description of the changes you made. ### Example: Adding a New Parameter to Custom Ads Insights If you are adding a new parameter to the custom Ads Insights, you might follow a process similar to this: 1. Modify the Connector Code: - Update the 
source.py
file to include the new parameter. For example:
Copy code
python
     class FacebookMarketingStream(HttpStream):
         def __init__(self, config: Mapping[str, Any], level: str = "ad"):
             super().__init__(config)
             self.level = level

         def path(self, **kwargs) -> str:
             return f"act_{self.config['account_id']}/insights"

         def request_params(self, **kwargs) -> MutableMapping[str, Any]:
             params = super().request_params(**kwargs)
             params["level"] = self.level
             return params
To modify the existing 
source-facebook-marketing
connector, which is Python-based, follow these steps: ### Steps to Modify the 
source-facebook-marketing
Connector 1. Fork and Clone the Repository: - Fork the [Airbyte repository](https://github.com/airbytehq/airbyte). - Clone your forked repository to your local machine:
Copy code
bash      git clone <https://github.com/<your-username>/airbyte.git>      cd airbyte
2. Navigate to the Connector Directory: - Navigate to the 
source-facebook-marketing
connector directory:
Copy code
bash      cd airbyte-integrations/connectors/source-facebook-marketing
3. Create a New Branch: - Create a new branch for your modifications:
Copy code
bash      git checkout -b modify-facebook-marketing-connector
4. Modify the Connector Code: - Open the 
source.py
file and make the necessary modifications. For example, if you need to add a new parameter or stream, you can update the relevant classes and methods. 5. Update Dependencies: - If you need to add or update dependencies, modify the 
setup.py
or 
requirements.txt
file accordingly. 6. Write Unit Tests: - Add or update unit tests to cover your changes. Ensure that all tests are located in the 
unit_tests
directory. 7. Run Tests: - Run the tests to ensure everything is working correctly:
Copy code
bash      poetry install --with dev      poetry run pytest unit_tests
8. Update Documentation: - Update the documentation to reflect your changes. This includes the 
README.md
and any relevant markdown files in the 
docs
directory. 9. Commit and Push Your Changes: - Commit your changes and push them to your forked repository:
Copy code
bash      git add .      git commit -m "Modify source-facebook-marketing connector"      git push origin modify-facebook-marketing-connector
10. Create a Pull Request: - Go to the original Airbyte repository and create a pull request from your forked repository. Provide a detailed description of the changes you made. ### Example: Adding a New Parameter to Custom Ads Insights If you are adding a new parameter to the custom Ads Insights, you might follow a process similar to this: 1. Modify the Connector Code: - Update the 
source.py
file to include the new parameter. For example:
Copy code
python      class FacebookMarketingStream(HttpStream):          def __init__(self, config: Mapping[str, Any], level: str = "ad"):              super().__init__(config)              self.level = level           def path(self, **kwargs) -> str:              return f"act_{self.config['account_id']}/insights"           def request_params(self, **kwargs) -> MutableMapping[str, Any]:              params = super().request_params(**kwargs)              params["level"] = self.level              return params
2. Update Unit Tests: - Add unit tests to cover the new parameter:
Copy code
python      def test_request_params():          stream = FacebookMarketingStream(config={"account_id": "123"}, level="campaign")          params = stream.request_params()          assert params["level"] == "campaign"
3. Run Tests: - Ensure all tests pass:
Copy code
bash      poetry run pytest unit_tests
### Relevant Links - [Airbyte Repository](https://github.com/airbytehq/airbyte) -
s
@kapa.ai airbyte.SourceFacebookMarketingabstract source.py181 During the sync, the following streams did not sync successfully: ads: AirbyteTracedException("FileNotFoundError: [Errno 2] No such file or directory: 'targeting.json'")
k
A message from kapa.ai
It looks like you're encountering a 
FileNotFoundError
during the sync process for the 
source-facebook-marketing
connector. The error message indicates that the file 
targeting.json
could not be found. Here are some steps to troubleshoot and resolve this issue: ### Steps to Resolve 
FileNotFoundError
1. Check File Path: - Ensure that the 
targeting.json
file exists in the expected directory. Verify the path in your code where the file is being accessed. 2. Update File Path in Code: - If the file path is incorrect, update it to the correct path. For example, if the file should be in the same directory as your script, you can use:
Copy code
python      import os       file_path = os.path.join(os.path.dirname(__file__), 'targeting.json')      with open(file_path, 'r') as file:          data = json.load(file)
3. Add Missing File: - If the 
targeting.json
file is missing, add it to the appropriate directory. Ensure that it contains the necessary data for your connector. 4. Verify File Permissions: - Ensure that the file has the correct permissions and can be read by the script. ### Example Code Snippet Here is an example of how you might update the file path in your code:
Copy code
python import os import json  class FacebookMarketingStream(HttpStream):     def __init__(self, config: Mapping[str, Any]):         super().__init__(config)         self.file_path = os.path.join(os.path.dirname(__file__), 'targeting.json')      def read_targeting_file(self):         try:             with open(self.file_path, 'r') as file:                 data = json.load(file)             return data         except FileNotFoundError as e:             raise AirbyteTracedException(f"FileNotFoundError: {e}")      def request_params(self, **kwargs) -> MutableMapping[str, Any]:         params = super().request_params(**kwargs)         targeting_data = self.read_targeting_file()         params.update(targeting_data)         return params
### Additional Resources - [Airbyte Repository](https://github.com/airbytehq/airbyte) - [Python CDK Documentation](https://docs.airbyte.com/connector-development/cdk-python/) By following these steps, you should be able to resolve the 
FileNotFoundError
and ensure that the 
targeting.json
file is correctly accessed during the sync process. If you have any further questions or need additional assistance, feel free to ask!
Open in Slack
PreviousNext
Powered by