@kapa.ai on airbyte open source, I want to enable a new stream on a salesforce connection. It is finding the stream, however it is not finding all the fields that should be present within it, even after running the 'refresh source schema' operation. What could be the reason for this?

There are a few potential reasons why not all fields are appearing for your Salesforce stream after refreshing the schema: 1. BULK API Limitations The Salesforce connector automatically chooses between the REST and BULK APIs. Fields with data types

base64

or compound (

object

) types are not supported by the BULK API. If

force_use_bulk_api

is enabled in your config, those fields will be explicitly excluded with a warning like: > "Following properties will be excluded from stream due to BULK API limitations" If the stream is being processed via BULK API, those unsupported field types will be dropped. [Salesforce source.py] 2. Too Many Properties / URL Length Limit The connector has a limit on the number of properties it can include in a SOQL query (based on URL length). If a stream has too many fields and lacks a primary key, it may not be instantiated at all. Even if it is, some fields could be excluded if the total property length exceeds the request size limit. [Salesforce streams.py] 3. Schema Change Detection Timing For self-managed (OSS) Airbyte, schema checks happen at most every 24 hours before a sync. If the fields were recently added to Salesforce, the schema refresh may not have picked them up yet if a check was already performed within the last 24 hours. [Schema change management] 4. Field Selection / Column Deselection If field selection is configured on the connection, some fields may be deselected. By default all detected fields are synced, but if fields were previously deselected, they won't appear as active. Note that field selection only applies to top-level fields. [Configuring schemas] Suggested Steps 1. Check your connector logs for warnings about fields being excluded due to BULK API limitations. 2. Verify whether

force_use_bulk_api

is enabled in your source configuration. 3. Check if the missing fields have

base64

or

object

(compound) data types in Salesforce (e.g., address fields, geolocation fields). 4. Try switching the stream to use the REST API by disabling

force_use_bulk_api

, if applicable. 5. Confirm the fields are actually returned by the Salesforce Describe endpoint for that object. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai i have checked the salesforce describe endpoint and it doesn't show the new fields i was expecting. Does that mean that they won't be available to airbyte until they are added to this API?

Yes, that is correct. The Airbyte Salesforce connector relies entirely on the Salesforce Describe endpoint to generate the schema for each stream. As shown in the connector's code, it calls

describe()

for each Salesforce object and builds the schema exclusively from the fields returned by that endpoint:

def generate_schema(self, stream_name: str = None, stream_options: Mapping[str, Any] = None) -> Mapping[str, Any]:
    response = self.describe(stream_name, stream_options)
    schema = {"$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "additionalProperties": True, "properties": {}}
    for field in response["fields"]:
        schema["properties"][field["name"]] = self.field_to_property_schema(field)
    return schema

[Salesforce api.py] So if the fields are not appearing in the Describe endpoint response, Airbyte has no way to know they exist, and they will not be available in any sync. The most likely reasons the fields are missing from the Describe endpoint are: • Insufficient permissions — The Salesforce user/profile used by Airbyte may not have field-level security access to those fields. • Fields not yet deployed — The fields may exist in a sandbox but haven't been deployed to production yet. • Wrong object — The fields may belong to a related or child object rather than the one being queried. You would need to resolve the issue at the Salesforce level first (e.g., grant field-level access to the Airbyte integration user, or ensure the fields are fully deployed). Once the Describe endpoint returns the new fields, a Refresh Source Schema in Airbyte will pick them up. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai when i run the sync it is showing as succeeded however if i check the connection state it doesn't seem to be saving the resume token correctly. It shows this which has timestamp value of 0 and ord value of -1:

{
  "shared_state": {
    "state": {
      "[\"aslan-database-prod\",{\"server_id\":\"aslan-database-prod\"}]": "{\"sec\":0,\"ord\":-1,\"resume_token\":\"826A4DDB3C000000012B0429296E1404\"}"
    },
    "schema_enforced": false
  },
  "streamStates": [
    {
      "streamDescriptor": {
        "name": "decisioning_filter",
        "namespace": "aslan_database_prod"
      },
      "streamState": {
        "id": "65eedbb877c27ed6b339527e",
        "idType": "OBJECT_ID",
        "status": "IN_PROGRESS",
        "binarySubType": null
      }
    },
    {
      "streamDescriptor": {
        "name": "aslan_collection",
        "namespace": "aslan_database_prod"
      },
      "streamState": {
        "id": "6a4bcf221aa97f915147c49b",
        "idType": "OBJECT_ID",
        "status": "COMPLETE",
        "binarySubType": null
      }
    },
    {
      "streamDescriptor": {
        "name": "decisioning_input",
        "namespace": "aslan_database_prod"
      },
      "streamState": {
        "id": "67efb4f71e7eef0d2a69f019",
        "idType": "OBJECT_ID",
        "status": "COMPLETE",
        "binarySubType": null
      }
    },
    {
      "streamDescriptor": {
        "name": "decisioning_output",
        "namespace": "aslan_database_prod"
      },
      "streamState": {
        "id": "67efb4f71e7eef0d2a69f019",
        "idType": "OBJECT_ID",
        "status": "COMPLETE",
        "binarySubType": null
      }
    }
  ]
}

Based on the knowledge sources, the state you're seeing with

"sec":0,"ord":-1

and a short resume token is actually expected and correct behavior for the MongoDB connector. From the GitHub issues, users and contributors observed exactly this pattern: > When things work as expected, epoch is 0 and resume token is a small hash in the saved state [source-mongodb-v2 issue] This was confirmed by an Airbyte contributor: > The resume token type (short vs long) shouldn't be an issue. [source-mongodb-v2 issue] So the

sec: 0

and

ord: -1

values with a short resume token are characteristic of a healthy saved state. The state that tends to cause problems is actually the opposite — a longer resume token with a non-zero

sec

value, which was linked to the periodic

Saved offset is not valid

errors that were fixed in connector version 2.0.4+. Things to watch out for However, there is one thing worth monitoring in your state — the

decisioning_filter

stream shows:

"status": "IN_PROGRESS"

rather than

"COMPLETE"

. This means the initial snapshot for that stream has not finished yet. This is worth keeping an eye on, as: • If the initial sync for that stream never completes, it may cause issues in subsequent syncs. • You may want to check the sync logs for that specific stream to see if there were any errors or if it is still catching up. Overall though, the resume token format itself looks healthy based on what the knowledge sources describe. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.