<@U04SE163WC8> If I use the airbyte oss deployed i...
# replication-ask-ai
d
@kapa.ai If I use the airbyte oss deployed in my own server, is there any data being sent to your company and server? And if so, what data are those?
k
Yes, if you use the Airbyte open-source software (OSS) deployed on your own server, some data is sent to Airbyte for telemetry purposes. This data is used to improve the CLI and measure its adoption. The telemetry sends data about: - Which command was run (not the arguments or options used). - Success or failure of the command run and the error type (not the error payload). - The current Airbyte workspace id if the user has not set the anonymous data collection on their Airbyte instance. However, you can disable telemetry by setting the
OCTAVIA_ENABLE_TELEMETRY
environment variable to
False
or using the
--disable-telemetry
flag. Please note that no actual data from the source (or destination) connectors ever transits or is retained in this internal database. The data that is stored in an Airbyte deployment includes technical logs, configuration metadata, and sensitive data. However, as soon as data is transferred from the source to the destination, it is purged from an Airbyte deployment. For more information, you can refer to the [CLI documentation](https://docs.airbyte.com/cli-documentation) and [Airbyte Security](https://docs.airbyte.com/operator-guides/security) documentation.
d
@kapa.ai does mongodb source connector now support cdc on sharded mongodb atlas cluster?
k
A message from kapa.ai
Based on the information provided in the extracts, it is confirmed that MongoDB is a supported source with Change Data Capture (CDC) in Airbyte. However, there is no specific mention of support for sharded MongoDB Atlas clusters. For more details on MongoDB source with CDC, you can refer to the [MongoDB CDC: How to Sync in Near Real-Time](https://airbyte.com/data-engineering-resources/mongodb-cdc) article mentioned in the [discussion](https://github.com/airbytehq/airbyte/discussions/34214). For the most accurate and up-to-date information, I recommend checking the official Airbyte [documentation](https://docs.airbyte.com/integrations/sources/mongodb-v2#cdc) or reaching out to the Airbyte community directly.
d
@kapa.ai does mongodb connector supports mongodb atlas sharded cluster?
k
A message from kapa.ai
Based on the information provided in the extracts, the MongoDB connector in Airbyte currently only supports the [replica set](https://www.mongodb.com/docs/manual/replication/) cluster type. There is no specific mention of support for sharded MongoDB Atlas clusters. In a [Github issue](https://github.com/airbytehq/airbyte/issues/28024), a user reported getting an error "Target MongoDB instance is not a replica set cluster" when attempting to connect to a Sharded Cluster, suggesting that sharded clusters might not be supported at the time of the discussion. For the most accurate and up-to-date information, I recommend checking the official Airbyte [documentation](https://docs.airbyte.com/integrations/sources/mongodb-v2#supported-mongodb-clusters) or reaching out to the Airbyte community directly.
d
@kapa.ai getting error below with the mongodb source connector PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.'
k
A message from kapa.ai
The error message you're seeing indicates that the MongoDB operation is exceeding the memory limit. This is a common issue when dealing with large datasets in MongoDB. Based on the information provided in the extracts, there are a few potential solutions to this issue: 1. Increase the memory limit: If you're running MongoDB on your own server, you might be able to increase the memory limit. However, this might not be a viable solution if you're using a hosted MongoDB service. 2. Enable external sorting: MongoDB has an option to allow disk usage during operations that exceed the memory limit. This can be enabled by passing
allowDiskUsage: true
when setting up the connector. This was suggested in a [Github issue](https://github.com/airbytehq/airbyte/issues/31514) related to a similar problem. 3. Optimize your query: If possible, you might want to optimize your query to reduce the amount of data that needs to be sorted. This could involve filtering out unnecessary data, or restructuring your query to avoid sorting entirely. Please note that these are general suggestions and the best solution might depend on your specific use case and environment. If you continue to experience issues, I recommend reaching out to the Airbyte community for further assistance.
d
@kapa.ai how to deploy airbyte oss in Google Kubernetes Engine?
k
A message from kapa.ai
To deploy Airbyte on Google Kubernetes Engine (GKE), you can follow these steps: 1. Cluster Setup: For local testing, you can use Docker Desktop (Mac), Minikube, or Kind. For testing on GKE, you can create a cluster with the command line or the Cloud Console UI. For production, Airbyte should function on most clusters v1.19 and above. If you run into a problem starting Airbyte, please reach out on the
#troubleshooting
channel on their [Slack](https://slack.airbyte.io/) or [create an issue on GitHub](https://github.com/airbytehq/airbyte/issues/new?assignees=&amp;labels=type%2Fbug&amp;template=bug-report.md&amp;title=). [Source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#cluster-setup) 2. Configure
kubectl
: Configure
kubectl
to connect to your cluster by using
kubectl use-context my-cluster-name
. For GKE, configure
gcloud
with
gcloud auth login
. On the Google Cloud Console, the cluster page will have a
Connect
button, which will give a command to run locally that looks like
gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME
. Use
kubectl config get-contexts
to show the contexts available. Run
kubectl config use-context $GKE_CONTEXT
to access the cluster from
kubectl
. [Source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#configure-kubectl) 3. Deploy Airbyte: If you don't want to configure your own Kubernetes cluster and Airbyte instance, you can use the free, open-source project [Plural](https://www.plural.sh/) to bring up a Kubernetes cluster and Airbyte for you. Alternatively, you can deploy Airbyte on [Restack](https://www.restack.io) to provision your Kubernetes cluster on AWS. [Source](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm#quickstart) Please note that the specific steps to deploy Airbyte on GKE might depend on your specific use case and environment. If you run into any issues, I recommend reaching out to the Airbyte community for further assistance.