This is my first day using AirByte and Docker. I’v...
# replication-troubleshooting
y
This is my first day using AirByte and Docker. I’ve run into this issue https://github.com/airbytehq/airbyte/issues/20703 while testing it out. I am trying to setup source=SalesForce and Dest=MySQL and I hit that error. I found in the code where the error is coming from
File "/airbyte/integration_code/source_salesforce/source.py", line 95, in generate_streams
and would like to try and understand the problem better so I modified that file to output more debug information. How do I get the updated code to take effect so I can run the test again and get better debug information about the issue?
e
Hello Yepher! You can build docker image and change version of source to dev in airbyte settings
of course if is it on local machine
y
Yes, it is on my Mac that I am testing. I guess dumb question. How do I build the docker image 🙂
e
No problem, check this out
y
Awesome. Thank you @Efim Matytsin
e
Welcome!
y
I was able to see the change 🎉
🎉 1
URL is too long but BULK is not required. It should fall back to Bulk based on the common but because
Account
supports REST it gets confused
e
I see hardcoded 2000 constant there, maybe you have to tune it
y
Changed Code:
Copy code
@classmethod
    def _get_api_type(cls, stream_name, properties):
        # Salesforce BULK API currently does not support loading fields with data type base64 and compound data
        properties_not_supported_by_bulk = {
            key: value for key, value in properties.items() if value.get("format") == "base64" or "object" in value["type"]
        }
        properties_length = len(",".join(p for p in properties))

        rest_required = stream_name in UNSUPPORTED_BULK_API_SALESFORCE_OBJECTS or properties_not_supported_by_bulk
        # If we have a lot of properties we can overcome REST API URL length and get an error: "reason: URI Too Long".
        # For such cases connector tries to use BULK API because it uses POST request and passes properties in the request body.
        bulk_required = properties_length + 2000 > Salesforce.REQUEST_SIZE_LIMITS
        print(f"xxxxxxxxxx bulk_required: {bulk_required} properties_lengthL {properties_length} + LIMIT: {Salesforce.REQUEST_SIZE_LIMITS}")
        if rest_required and not bulk_required:
            return "rest"
        if not rest_required:
            return "bulk"
Output
Copy code
2023-02-09 05:45:12 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):125 - xxxxxxxxxx bulk_required: True properties_lengthL 14846 + LIMIT: 16384
airbyte-worker                    | 2023-02-09 05:45:12 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):163 - None: Stream Account cannot be processed by REST or BULK API.
airbyte-worker                    | Traceback (most recent call last):
airbyte-worker                    |   File "/airbyte/integration_code/main.py", line 13, in <module>
airbyte-worker                    |     launch(source, sys.argv[1:])
airbyte-worker                    |   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
airbyte-worker                    |     for message in source_entrypoint.run(parsed_args):
airbyte-worker                    |   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 116, in run
airbyte-worker                    |     catalog = self.source.discover(self.logger, config)
airbyte-worker                    |   File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 75, in discover
airbyte-worker                    |     streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
airbyte-worker                    |   File "/airbyte/integration_code/source_salesforce/source.py", line 110, in streams
airbyte-worker                    |     streams = self.generate_streams(config, stream_objects, sf)
airbyte-worker                    |   File "/airbyte/integration_code/source_salesforce/source.py", line 95, in generate_streams
airbyte-worker                    |     raise Exception(f"{api_type}: Stream {stream_name} cannot be processed by REST or BULK API.")
I will force to
bulk
and see if it works. I think this is pretty fixable if this is really the issue
👍 1
I hope my comment here makes sense: https://github.com/airbytehq/airbyte/issues/20703
e
of course! thank you!
y
Thank you. I appreciate you helping get up to speed quickly.
batman 1
e
Was glad to help!
y
👍 1