<@U04S8PLJWMT> I installed the version 1.6.2 of ai...
# ask-ai
t
@kapa.ai I installed the version 1.6.2 of airbyte (a new instance) on my kubernetes cluster. But I don't see the connector "s3" anymore available in connectors we can install. It's like it disappeared. On the instances in the same version i had installed in the past, the connector is still available to install.
k
You’re likely running into a connector–platform compatibility / catalog issue rather than the S3 connector being removed. From the logs in one of the related issues, Airbyte 1.6.3 explicitly reports that newer S3 connector versions (for example
source-s3
4.14.1) require Airbyte
>= 1.7.0
, so they’re filtered out as “incompatible” and not exposed in the UI when running 1.6.x. [[sharepoint issue](https://github.com/airbytehq/airbyte/issues/61525)] This same compatibility check is what decides which connectors are visible/usable when a new instance starts up or after an upgrade. Key implications for your case: - On older 1.6.x installs, you may already have an older, compatible S3 connector definition in the database, so it continues to show up and work. - On a fresh 1.6.2 instance, the catalog sync only keeps connector versions whose
supported
range includes 1.6.2. If all current S3 connector versions now declare
>= 1.7.0
, none of them will be considered compatible, so S3 disappears from the list of installable connectors. [[sharepoint issue](https://github.com/airbytehq/airbyte/issues/61525)] What you can do (based only on the docs/issues we have): 1. Upgrade Airbyte to a newer 1.6 patch or 1.7+ when available The upgrade procedure for Helm-based K8s installs is documented here: modify your
values.yaml
and then run:
Copy code
bash    helm upgrade --install <RELEASE-NAME> airbyte/airbyte \      --values <VALUES.YAML> \      --version <HELM-CHART-VERSION>
[[1.6 upgrading](https://docs.airbyte.com/platform/1.6/operator-guides/upgrading-airbyte); [1.7 upgrading](https://docs.airbyte.com/platform/1.7/operator-guides/upgrading-airbyte)] Once you’re on a version that matches the S3 connector’s supported range (>= 1.7.0 per the issue), the S3 connector should reappear. 2. If you must stay on 1.6.2 The knowledge sources don’t document a supported way to pin or manually add an older S3 connector version to a new 1.6.x instance, and they don’t describe a workaround for this exact scenario. I’m sorry, but there’s no documented procedure in these sources to safely “backfill” an older S3 definition in a fresh 1.6.2 install. Given what’s documented, the only reliable, supported path I can point to is upgrading your Airbyte deployment so that the current S3 connector versions are compatible with your platform version.
t
@kapa.ai I upgraded to 1.7.8 and I still can't see the s3 connector in the sources.
k
A message from kapa.ai
Based on the docs and issues you shared, there are two relevant points: 1. S3 requires Airbyte ≥ 1.7.0 The SharePoint GitHub issue log shows that recent S3 connector versions (e.g.
source-s3
4.14.1) are only marked compatible with Airbyte
>= 1.7.0
. [[SharePoint issue](https://github.com/airbytehq/airbyte/issues/61525)] Since you are on 1.7.8, your platform version should now satisfy that requirement, so in principle S3 should not be filtered out for compatibility reasons anymore. 2. No explicit docs on S3 disappearing in 1.7.8 The available docs on upgrading (1.7 → 1.8, “Next”, etc.) just describe how upgrades work and how connectors are auto‑upgraded or left alone; they do not mention S3 being removed from the catalog or hidden in 1.7.x. [[1.7 upgrading](https://docs.airbyte.com/platform/1.7/operator-guides/upgrading-airbyte); [Upgrading Airbyte](https://docs.airbyte.com/platform/operator-guides/upgrading-airbyte)] Likewise, the S3 connector setup guide assumes S3 is present in the list of source types and does not describe any special enablement or feature‑flagging. [[S3 source setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-open-source)] Because of that, I don’t have enough information in these sources to explain why S3 is still missing on 1.7.8 or to give you a documented fix. From the docs/issues provided, the only actions I can confidently recommend are: - Confirm you actually completed a full upgrade to 1.7.x using the documented Helm/abctl procedures, and that the UI you’re using is really the upgraded instance. [[1.7 upgrading](https://docs.airbyte.com/platform/1.7/operator-guides/upgrading-airbyte)] - Check connector catalog behavior after upgrades: Airbyte may auto‑upgrade or leave connectors alone, but there is no documented mechanism here for restoring a specific connector definition (like S3) if it is missing from a fresh install. [[Upgrading Airbyte](https://docs.airbyte.com/platform/operator-guides/upgrading-airbyte)] Beyond that, the knowledge sources you shared do not describe further troubleshooting steps for a missing S3 connector on 1.7.8. To proceed, you would likely need to open an issue with Airbyte (as in the GitHub examples) or consult more detailed internal logs and catalog configuration, which are not covered in these docs.
t
ok now it's fixed but I have this issue with the connector : 'No streams are available for source SourceS3. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.'
@kapa.ai ok now it's fixed but I have this issue with the connector : 'No streams are available for source SourceS3. This is probably an issue with the connector. Please verify that your configuration provides permissions to list and read files from the source. Contact support if you are unable to resolve this issue.'
k
A message from kapa.ai
This message usually means the S3 source connector was able to authenticate, but it didn’t find any files it can turn into streams. From the docs and issues you provided, these are the concrete things to check: 1. Bucket access & policy Make sure the IAM user you configured has both object‑ and bucket‑level permissions on the bucket you set in the connector (at minimum
s3:GetObject
and
s3:ListBucket
). [[S3 source step 1](https://docs.airbyte.com/integrations/sources/s3#step-1-set-up-s3)] Example policy (replace with your bucket name):
Copy code
json    {      "Version": "2012-10-17",      "Statement": [        {          "Effect": "Allow",          "Action": [            "s3:GetObject",            "s3:ListBucket"          ],          "Resource": [            "arn:aws:s3:::your-bucket-name/*",            "arn:aws:s3:::your-bucket-name"          ]        }      ]    }
The S3 Data Lake docs confirm object‑level permissions alone are not sufficient; bucket‑level permissions are required. [[S3 Data Lake setup](https://docs.airbyte.com/integrations/destinations/s3-data-lake#setup-guide)] 2. The bucket actually defines the streams The S3 source has no predefined streams; streams are derived entirely from the contents of your bucket. [[S3 supported streams](https://docs.airbyte.com/integrations/sources/s3#supported-streams)] If your glob/pattern excludes all files, or the bucket is empty / doesn’t contain matching files, you’ll see no streams. 3. Globs and file formats in the stream configuration (Cloud) / source config When adding a stream in the S3 source, verify: [[S3 Cloud setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud)] - Bucket name is exactly the S3 bucket (no
s3://
prefix, no ARN). - File Format is one of the supported ones (CSV, Parquet, Avro, JSONL) and matches your files. - Globs (if you set them) actually match your object keys. - To sync everything, use
**
. [[S3 Cloud setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud)] - If you set a custom schema or validation policy, try leaving
Input schema
as
{}
and turning off strict validation to rule out schema‑related filtering. 4. Typical S3 auth failure pattern vs. your case Known issues like
InvalidAccessKeyId
or
Unable to locate credentials
appear when the credentials are wrong or not passed to boto at all. [[Forum 2370](https://discuss.airbyte.io/t/2370); [GH 57577](https://github.com/airbytehq/airbyte/issues/57577)] Your error is different (no streams), which aligns more with “nothing matched / nothing readable” than with raw authentication failure. If, after: - confirming the IAM policy (with
s3:GetObject
and
s3:ListBucket
on the right bucket), - verifying the bucket name, - loosening globs to
**
, and - ensuring there are files in a supported format in that bucket, you still get “No streams are available”, then the remaining guidance in these sources is to treat it as a connector bug and contact support / open an issue, as the message itself suggests. The knowledge sources don’t provide a deeper workaround beyond these configuration and permission checks.
t
@kapa.ai ok what kind of files I should put ?
k
A message from kapa.ai
For the S3 source connector, you must store files in one of the supported formats and then pick the matching “File Format” when you define the stream. Per the S3 docs for Airbyte Cloud, the supported formats are: CSV, Parquet, Avro, and JSONL. [[S3 setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud)] So in your bucket you should have objects like: -
*.csv
(if you select CSV) -
*.parquet
(if you select Parquet) -
*.avro
(if you select Avro) -
*.jsonl
(one JSON object per line, if you select JSONL) Then, when adding a stream in the S3 source: 1. Choose the correct File Format matching your files. [[S3 setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud)] 2. Set Globs so they actually match your keys (e.g.
**
for “all files” or
**/*.csv
for just CSVs). [[Globs](https://docs.airbyte.com/integrations/sources/s3#globs)] If you put files in any other format (e.g. plain JSON, gzip‑compressed CSV, etc.) and pick one of these formats, the connector won’t be able to parse them and you’ll effectively get no usable streams.
t
@kapa.ai but I put copy raw files so I shouldn't have this error right ?
k
A message from kapa.ai
You’re right to expect different behavior with Copy raw files, but there are still a couple of hard requirements that can cause the “No streams are available” error. From the docs: 1. Copy raw files is only available on specific, file-based connectors For these connectors (like S3), you must: - Choose Copy raw files as the Delivery Method on the source. [[Delivery methods](https://docs.airbyte.com/platform/using-airbyte/delivery-methods#copy-raw-files)] - Use a file-based destination (e.g. S3) and choose an Output Format for the metadata. [[Delivery methods](https://docs.airbyte.com/platform/using-airbyte/delivery-methods#configure-copy-raw-files-as-your-delivery-method)] 2. Files are still discovered via streams Even with Copy raw files, you must have at least one stream with files to sync: - For file-based sources, Airbyte syncs raw files plus structured metadata. The stream you select represents the metadata Airbyte adds to each file. [[Sync files](https://docs.airbyte.com/platform/using-airbyte/sync-files-and-records#unstructuredfile-based-sources)] - The S3 source has no predefined streams; streams come from the contents of your bucket. [[S3 streams](https://docs.airbyte.com/integrations/sources/s3#supported-streams)] That means if: - Your bucket is empty, or - Your Globs don’t match any object keys (e.g.
**/*.csv
but you only have
.pdf
files), then Airbyte won’t see any streams/files and you’ll get the “No streams are available” message, even in Copy raw files mode. 3. What kind of files are allowed with Copy raw files? With Copy raw files, Airbyte does not parse the file contents; it “copies bits” and just adds metadata. This is meant for unstructured data (text, multimedia, compressed files, etc.). [[Delivery methods](https://docs.airbyte.com/platform/using-airbyte/delivery-methods#copy-raw-files)] So you can have PDFs, images, zips, etc. The critical part is that: - The connector supports file transfers (
supportsFileTransfer: true
– S3 does). [[Sync files](https://docs.airbyte.com/platform/using-airbyte/sync-files-and-records#which-connectors-support-file-transfers)] - Your Globs actually match those files (e.g.
**
or
**/*.pdf
). So, with Copy raw files: - You do not need to stick to CSV/Parquet/Avro/JSONL contents, but - You still must: - Have objects in the bucket, - Configure at least one stream, - Use Globs that match your files. If all that is true and you still see “No streams are available,” the docs say this usually indicates a connector issue and recommend contacting support, which matches the error text you’re seeing. [[Delivery methods](https://docs.airbyte.com/platform/using-airbyte/delivery-methods)]
t
@kapa.ai can I reduce the source connector to only one folder of the bucket ?
k
A message from kapa.ai
Yes. You can limit what the S3 source reads by using Globs (path patterns) so that only keys under a specific “folder” (prefix) are matched. For S3, each glob is evaluated from the root of the bucket (do not include the bucket name). [[S3 globs](https://docs.airbyte.com/integrations/sources/s3#globs)] Examples, assuming your bucket has:
Copy code
my-bucket   └── my-folder/         ├── file1.csv         └── subfolder/file2.csv
Use one of these Globs in the stream config: - Only files directly under
my-folder/
(not subfolders):
Copy code
text   my-folder/*.csv
- Everything under
my-folder/
recursively:
Copy code
text   my-folder/**       # all files   my-folder/**/*.csv # only CSVs
In Airbyte Cloud S3 source, you set this in the Globs field when adding/editing a stream. [[S3 Cloud setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud); [S3 globs](https://docs.airbyte.com/integrations/sources/s3#globs)] This way, only files whose keys start with that folder path are discovered and synced.
t
@kapa.ai I don't see this option on my UI/
k
A message from kapa.ai
In the S3 source UI, you won’t see a generic “folder” picker; the restriction to a folder is done via the Globs (path pattern) field inside each stream’s configuration. From the docs for S3 on Airbyte Cloud, the steps are: [[S3 Cloud setup](https://docs.airbyte.com/integrations/sources/s3#for-airbyte-cloud)] 1. Edit your S3 source. 2. Under Streams, edit or add a stream. 3. In that stream, there is an (optional) Globs field. This is where you restrict to a folder. The Globs syntax is documented here: [[S3 globs](https://docs.airbyte.com/integrations/sources/s3#globs)] - Patterns are relative to the root of the bucket (do not include the bucket name). - To limit to a single folder: - All files under
my-folder/
(recursively):
Copy code
text     my-folder/**
- Only CSVs directly in `my-folder/`:
Copy code
text     my-folder/*.csv
If your UI truly has no Globs field inside the stream configuration for S3, that’s not described in the docs I have; they assume this field exists and is the way to filter by folder.
t
@kapa.ai I added a source connector successfully (the first one) and suddenly i have this error on the server :
2025-11-24 18:34:42,653 [io-executor-thread-1]	ERROR	i.a.c.s.e.h.KnownExceptionHandler(handle):34 - Known Exception: class KnownExceptionInfo {
message: Failed to access database. Check the server logs for more information
exceptionClassName: io.airbyte.commons.server.errors.ConflictException
exceptionStack: [io.airbyte.commons.server.errors.ConflictException: Failed to access database. Check the server logs for more information, 	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:45), 	at io.airbyte.server.apis.controllers.SourceApiController.listSourcesForWorkspace(SourceApiController.kt:132), 	at io.airbyte.server.apis.controllers.$SourceApiController$Definition$Exec.dispatch(Unknown Source), 	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461), 	at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4419), 	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272), 	at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38), 	at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:493), 	at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:470), 	at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92), 	at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211), 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), 	at java.base/java.lang.Thread.run(Thread.java:1583)]
rootCauseExceptionClassName: null
rootCauseExceptionStack: []
details: null
}
k
A message from kapa.ai
This error means the API layer cannot complete a DB operation and wraps it as a
ConflictException
(“Failed to access database”). It’s the same pattern reported in multiple issues: - When calling
/api/v1/instance_configuration
or other endpoints, the server throws
ConflictException: Failed to access database. Check the server logs for more information
. [[DB 409 error](https://github.com/airbytehq/airbyte/issues/54150); [Helm 1.8.0 db error](https://github.com/airbytehq/airbyte/issues/65096)] - The full stack trace looks very similar to yours: error raised in
ApiHelper.execute(...)
when listing resources (instance configuration, connections, etc.). [[DB 409 error](https://github.com/airbytehq/airbyte/issues/54150); [Helm 1.8.0 db error](https://github.com/airbytehq/airbyte/issues/65096)] From the provided sources, what we can say: 1. It’s not specific to your connector. In the referenced issues, it happens when the server tries to access DB-backed configuration (instance configuration, connections, workspace data). The error is generic and indicates a DB access problem surfaced to the API. [[DB 409 error](https://github.com/airbytehq/airbyte/issues/54150); [Helm 1.8.0 db error](https://github.com/airbytehq/airbyte/issues/65096)] 2. **The underlying cause is not shown in that exception.** The message explicitly says “Check the server logs for more information”, and the stack you pasted doesn’t contain the root cause. The related issues confirm that the real DB error is only visible deeper in the logs (e.g., type cast issues, Flyway/jOOQ issues, connectivity). [[DB 409 error](https://github.com/airbytehq/airbyte/issues/54150)] 3. In at least one case, upgrading images resolved it. A user hitting the same “Failed to access database” 409 error on
/api/v1/instance_configuration
reported fixing it by upgrading all core images from
1.3.1
to
1.5.1
(
bootloader
,
server
,
webapp
,
worker
). [[DB 409 fixed by upgrade](https://github.com/airbytehq/airbyte/issues/54150)] Given only these sources, the concrete, documented steps you can take are: - Inspect
airbyte-server
logs around the time of the error
to find the actual DB exception (for example jOOQ or Flyway complaining, or a connectivity/auth issue). The knowledge sources do not show a universal root cause or fix; they just show that the top-level
ConflictException
wraps some lower-level DB error. [[DB 409 error](https://github.com/airbytehq/airbyte/issues/54150)] - Verify the Airbyte DB is reachable and healthy (PostgreSQL service, credentials, and schema). Similar DB availability checks and failures are logged during startup in other issues. [[abctl db failure](https://github.com/airbytehq/airbyte/issues/54199)] - If you suspect a bug in the version you’re running, consider upgrading Airbyte (server/worker/webapp/bootloader) to a newer patch version, as that resolved the same class of error for another user. [[DB 409 fixed by upgrade](https://github.com/airbytehq/airbyte/issues/54150)] The provided knowledge sources do not specify an exact root cause or step‑by‑step fix for this error in your scenario, beyond checking server logs for the underlying DB problem and, if it’s a known bug, upgrading to a version where it’s resolved.