Yepher
02/09/2023, 4:59 AMFile "/airbyte/integration_code/source_salesforce/source.py", line 95, in generate_streams and would like to try and understand the problem better so I modified that file to output more debug information. How do I get the updated code to take effect so I can run the test again and get better debug information about the issue?Efim Matytsin
02/09/2023, 5:00 AMEfim Matytsin
02/09/2023, 5:01 AMYepher
02/09/2023, 5:02 AMEfim Matytsin
02/09/2023, 5:04 AMYepher
02/09/2023, 5:05 AMEfim Matytsin
02/09/2023, 5:05 AMYepher
02/09/2023, 5:30 AMYepher
02/09/2023, 5:43 AMYepher
02/09/2023, 5:43 AMAccount supports REST it gets confusedEfim Matytsin
02/09/2023, 5:46 AMYepher
02/09/2023, 5:47 AM@classmethod
def _get_api_type(cls, stream_name, properties):
# Salesforce BULK API currently does not support loading fields with data type base64 and compound data
properties_not_supported_by_bulk = {
key: value for key, value in properties.items() if value.get("format") == "base64" or "object" in value["type"]
}
properties_length = len(",".join(p for p in properties))
rest_required = stream_name in UNSUPPORTED_BULK_API_SALESFORCE_OBJECTS or properties_not_supported_by_bulk
# If we have a lot of properties we can overcome REST API URL length and get an error: "reason: URI Too Long".
# For such cases connector tries to use BULK API because it uses POST request and passes properties in the request body.
bulk_required = properties_length + 2000 > Salesforce.REQUEST_SIZE_LIMITS
print(f"xxxxxxxxxx bulk_required: {bulk_required} properties_lengthL {properties_length} + LIMIT: {Salesforce.REQUEST_SIZE_LIMITS}")
if rest_required and not bulk_required:
return "rest"
if not rest_required:
return "bulk"
Output
2023-02-09 05:45:12 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):125 - xxxxxxxxxx bulk_required: True properties_lengthL 14846 + LIMIT: 16384
airbyte-worker | 2023-02-09 05:45:12 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):163 - None: Stream Account cannot be processed by REST or BULK API.
airbyte-worker | Traceback (most recent call last):
airbyte-worker | File "/airbyte/integration_code/main.py", line 13, in <module>
airbyte-worker | launch(source, sys.argv[1:])
airbyte-worker | File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
airbyte-worker | for message in source_entrypoint.run(parsed_args):
airbyte-worker | File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 116, in run
airbyte-worker | catalog = self.source.discover(self.logger, config)
airbyte-worker | File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 75, in discover
airbyte-worker | streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
airbyte-worker | File "/airbyte/integration_code/source_salesforce/source.py", line 110, in streams
airbyte-worker | streams = self.generate_streams(config, stream_objects, sf)
airbyte-worker | File "/airbyte/integration_code/source_salesforce/source.py", line 95, in generate_streams
airbyte-worker | raise Exception(f"{api_type}: Stream {stream_name} cannot be processed by REST or BULK API.")Yepher
02/09/2023, 5:48 AMbulk and see if it works. I think this is pretty fixable if this is really the issueYepher
02/09/2023, 5:58 AMEfim Matytsin
02/09/2023, 6:00 AMYepher
02/09/2023, 6:00 AMEfim Matytsin
02/09/2023, 6:02 AMYepher
02/09/2023, 3:30 PM