https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> when I try to create a connection f...
# ask-ai
w
@kapa.ai when I try to create a connection from a MS SQL source onto BQ I am getting a 502 error. The source has about 1500 tables
k
Based on the information provided in the knowledge sources, it appears that you're encountering a common issue when trying to create a connection from an MS SQL source with a large number of tables to BigQuery. This problem is likely related to the high number of tables in your source database. Here's what we know: 1. Several users have reported similar issues with MS SQL sources containing many tables (850+, 3000+, etc.) leading to connection failures or timeouts. 2. The error you're seeing (502) is consistent with what others have experienced, often described as "Server temporarily unavailable" or similar messages. 3. The root cause seems to be related to the time it takes to discover and process the schema for all these tables, which can exceed default timeout limits. To address this issue, you have a few potential options: 1. Try the latest version of the source-mssql connector. As mentioned in [this comment](https://github.com/airbytehq/airbyte/issues/30287#issuecomment-1879759461), the connector has been recently revamped and might handle large numbers of tables better now. 2. If you're running Airbyte on your own infrastructure, you might need to adjust timeout settings. As suggested in [this discussion](https://github.com/airbytehq/airbyte/discussions/37798), increasing the timeout to 600 seconds or more has helped in some cases. 3. Consider using the new certified source-mssql connector, which is designed to handle terabyte-sized tables and has improved reliability. Information about this can be found in the [Request for Beta Testers: Source MSSQL](https://github.com/airbytehq/airbyte/discussions/35340) discussion. If these suggestions don't resolve your issue, you may need to reach out to Airbyte support for further assistance, as the problem could be related to your specific setup or configuration.
w
When installing airbyte with abctl where is the files actually stored ? @kapa.ai
k
A message from kapa.ai
Based on the knowledge sources provided, when installing Airbyte using abctl, the files are stored in specific locations depending on your operating system and configuration. Here's what we know: For Mac and Linux users: 1. By default, Airbyte stores data in the 
/tmp/airbyte_local
directory on the host machine. This is mentioned in the [Browsing logs documentation](https://docs.airbyte.com/operator-guides/browsing-output-logs#csv-or-json-local-destinations-check-local-data-folder). 2. For abctl specifically, it creates a data directory at 
~/.airbyte/abctl/data
. This is evident from a debug log in one of the GitHub issues: 
DEBUG   Creating data directory '/opt/airbyte/.airbyte/abctl/data'
[GitHub Issue #41938](https://github.com/airbytehq/airbyte/issues/41938). 3. For local JSON or CSV destinations, you might find the data in 
/tmp/airbyte_local/YOUR_PATH/_airbyte_raw_YOUR_STREAM_NAME.jsonl
, where YOU_PATH is the path you chose during destination setup [Browsing logs documentation](https://docs.airbyte.com/operator-guides/browsing-output-logs#csv-or-json-local-destinations-check-local-data-folder). For Windows users: The documentation doesn't provide specific information about file storage locations for Windows installations using abctl. However, it mentions that for Windows users using Airbyte with WSL2 and Docker, there's a guide to locate the replicated folder and file, which you can find in the [locating files local destination guide](https://docs.airbyte.com/integrations/locating-files-local-destination). It's important to note that the exact location might vary depending on your specific configuration and version of abctl. If you're having trouble locating the files, you might need to check the abctl documentation or reach out to Airbyte support for more specific guidance related to your setup.
2 Views
Open in Slack
PreviousNext
Powered by