Folks I m unable to get a Salesforce object `Lead` due to so Airbyte #replication-troubleshooting

Folks, I'm unable to get a Salesforce object `Lead...

Binay Jena

10/26/2022, 10:45 PM

Folks, I'm unable to get a Salesforce object

Lead

due to something w

<https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py>

. with the text search i'm getting

leadstatus

leadshare

but not

Lead

specifically. Its only happening with this particular object. When I login to sfdc with the same credentials I do see this object and able to query fine, but its the airbyte connector runs where its not working, it doesnt find this object at all. Any leads with what I could be trying next to debug?

Binay Jena

10/26/2022, 10:52 PM

i see the error in EC2 as this, referring to https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/source.py#L85

Binay Jena

10/26/2022, 10:55 PM

this class specifically, it returns None for Lead, causing the exception. but i'm kind of lost how to proceed further

Copy code

@classmethod
    def _get_api_type(cls, stream_name, properties):
        # Salesforce BULK API currently does not support loading fields with data type base64 and compound data
        properties_not_supported_by_bulk = {
            key: value for key, value in properties.items() if value.get("format") == "base64" or "object" in value["type"]
        }
        properties_length = len(",".join(p for p in properties))

        rest_required = stream_name in UNSUPPORTED_BULK_API_SALESFORCE_OBJECTS or properties_not_supported_by_bulk
        # If we have a lot of properties we can overcome REST API URL length and get an error: "reason: URI Too Long".
        # For such cases connector tries to use BULK API because it uses POST request and passes properties in the request body.
        bulk_required = properties_length + 2000 > Salesforce.REQUEST_SIZE_LIMITS

        if rest_required and not bulk_required:
            return "rest"
        if not rest_required:
            return "bulk"

Saj Dider (Airbyte)

10/27/2022, 5:20 PM

Confirming that it seems like the Lead stream when passed into the

get_api_type

function returns None. If it returns

None

, then that means that those two checks at the end of the function don’t return “rest” or “bulk” which means that for the Lead stream:

rest_required

is True AND

bulk_required

is True. If rest_required is true, then it means that

properties_not_supported_by_bulk

is not empty, which means that the Lead object has either a property with a “base64” field or a property that’s an object type. If bulk_required is True, then it means that properties_length + 2000 is greater than 16384 characters. This seems likely given that the Lead object has so many fields. It’s possible that the additional of simple case that checks if both rest_required and bulk_required are True would solve your issue but I’m not sure. Does this help?

Open in Slack

Previous Next