https://linen.dev logo
Join Slack
Powered by
URGENT!!!!! Hi I am facing issue with source sync ...
# replication-ask-ai
p
URGENT!!!!! Hi I am facing issue with source sync ... my source is dynamodb ... but when airbyte is fetching tables it doesn't list all the tables .. some of them are missing ... is there a limitation to this ?? how can this be solved ?
k
Yes, there is a limitation on the number of streams (tables in your case) that can be returned by a source in a discover call. The maximum number of streams that can be returned by a source in a discover call is 1K. This could be the reason why some of your tables are not being listed. To solve this issue, you might need to reduce the number of tables in your DynamoDB source or split your tables across multiple DynamoDB sources to ensure that the number of tables in each source does not exceed the limit. For more details, you can refer to the [Airbyte Cloud limits](https://docs.airbyte.com/cloud/managing-airbyte-cloud/understand-airbyte-cloud-limits) in the documentation.
p
@kapa.ai I am still not able load all tables on airbyte ... how can i do that ?
k
To load all available tables and bases in Airbyte, you can use the 
Metadata API
for a given authenticated user. If you rename or add a column to any existing table, you will need to recreate the source to update the Airbyte catalog. Here is the extract from the [Airbyte documentation](https://docs.airbyte.com/integrations/sources/airtable) that explains this:
Copy code
This source allows you to pull all available tables and bases using `Metadata API` for a given authenticated user. In case you rename or add a column to any existing table, you will need to recreate the source to update the Airbyte catalog.
If you are setting up Airbyte with a custom database for the first time, make sure the database exists in the server, the user has both read and write permissions to the database, and the database is empty. If the database is not empty, and has a table that shares the same name as one of the Airbyte tables, the server will assume that the database has been initialized, and will not copy the data over, resulting in server failure. If you run into this issue, just wipe out the database, and launch the server again. You can find more details in the [Airbyte operator guide](https://docs.airbyte.com/operator-guides/configuring-airbyte-db).
p
@kapa.ai i am getting below error for all my pipelines now ... ERROR creating table model user.webinar.......................................................................... [ERROR in 0.10s] 348 Database Error in model webinar (models/generated/airbyte_tables/user/webinar.sql) 349 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '__dbt__cte__webinar_ab1 as ( 350 351 -- SQL model to parse JSON blob stored in a single' at line 8 352 compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/user/webinar.sql 353 1 of 1 ERROR creating table model user.webinar.......................................................................... [ERROR in 0.10s] 354 Database Error in model webinar (models/generated/airbyte_tables/user/webinar.sql) 355 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '__dbt__cte__webinar_ab1 as ( 356 357 -- SQL model to parse JSON blob stored in a single' at line 8 358 compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/user/webinar.sql,retryable=<null>,timestamp=1698838259071,additionalProperties={}], io.airbyte.config.FailureReason@2d178514[failureOrigin=normalization,failureType=system_error,internalMessage=1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '__dbt__cte__webinar_ab1 as (,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@415d9336[additionalProperties={attemptNumber=1, jobId=28, from_trace_message=true}],stacktrace=AirbyteDbtError: </null> 359 1 of 1 ERROR creating table model user.webinar.......................................................................... [ERROR in 0.10s] 360 Database Error in model webinar (models/generated/airbyte_tables/user/webinar.sql) 361 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '__dbt__cte__webinar_ab1 as ( 362 363 -- SQL model to parse JSON blob stored in a single' at line 8 364
k
A message from kapa.ai
The error message you're seeing indicates a syntax error in your SQL code. This error is occurring during the normalization process with dbt, which is a step in the Airbyte pipeline where data is transformed into a more usable state. The error message suggests that there's an issue with the SQL syntax near '__dbt__cte__webinar_ab1 as ('. This could be due to a variety of reasons such as a missing or misplaced keyword, incorrect data type, or a missing closing parenthesis. To debug this issue, you can retrieve the file created by dbt to check what is causing the problem. Here are the steps to do that as suggested in the [Airbyte forum](https://discuss.airbyte.io/t/3693): 1. Run 
docker exec -ti airbyte-server bash
to get a bash shell in the container. 2. Navigate to the workspace directory by running 
cd /tmp/workspace
. 3. Navigate to the job directory by running 
cd <JOB_ID>/0/
. 4. View the SQL file by running 
cat …/build/run/airbyte_utils/models/generated/airbyte_tables/vested_airbyte/pg_stat_statements.sql
. Replace 
<JOB_ID>
with the actual job id. If you're not sure what the job id is, you can find it in your logs. Once you have the SQL file, you can inspect it to see what might be causing the syntax error. If you're still having trouble, you might want to reach out to the Airbyte community on their [forum](https://discuss.airbyte.io/) or [slack](https://airbytehq.slack.com) for further assistance.
p
@kapa.ai i think there is a limit that airbyte fetches max 100 tables ... is there a way to increase that limit
k
A message from kapa.ai
Airbyte does have certain limitations when it comes to data integration. However, the extracts provided do not mention a specific limit on the number of tables that can be fetched. The limitations mentioned in the [Airbyte Cloud documentation](https://docs.airbyte.com/cloud/managing-airbyte-cloud/understand-airbyte-cloud-limits) include: - Max number of workspaces per user: 3 - Max number of instances of the same source connector: 10 - Max number of destinations in a workspace: 20 - Max number of consecutive sync failures before a connection is paused: 100 - Max number of days with consecutive sync failures before a connection is paused: 14 days - Max number of streams that can be returned by a source in a discover call: 1K - Max number of streams that can be configured to sync in a single connection: 1K - Size of a single record: 20MB - Shortest sync schedule: Every 60 min - Schedule accuracy: +/- 30 min If you're facing issues with fetching more than 100 tables, it might be due to other factors such as API rate limits of the source system, network issues, or configuration errors. If you believe you're hitting a limit that's not documented, I recommend reaching out to the Airbyte support team or community for further assistance.
Open in Slack
PreviousNext
Powered by