https://linen.dev logo
Join Slack
Powered by
Hi all, I was asked to create a report that shows ...
# replication-troubleshooting
l
Hi all, I was asked to create a report that shows the daily ELT upload. which ones succeeded/ failed, and count of new rows. Does someone have an example to take ideas?
a
@laila ribke for this you need to join following 3 table • connection: is contains list of all connections • jobs: sync status • attempts: number attempt for each sync from the 
job
-> 
status
you can find the sync status to find the number of row synced, in succeeded attempt you can find the following data 
attempts
-> 
output
-> 
standardSyncSummary
has following data ->
Copy code
{
"status": "completed",
"endTime": 1629320102908,
"startTime": 1629319299994,
"bytesSynced": 10, //10 byte synced
"recordsSynced": 2, 2 record synced
}
l
@Akilesh V Thanks for answering! Where do I find those tables?
a
@laila ribke you can find it in Airbyte config database you might have enter config db information here https://github.com/airbytehq/airbyte/blob/master/.env#L43
l
From airbyte UI I can´t access to these tables, right?
Basically I want to create a table with the source name, destination name and a row for each stream. status and rows synced. Then I will visualize data from this table in a dashboard
a
no, you cant access it from the airbyte ui
l
@Akilesh V, so, how should I do it? I would like to have the data in my destination data warehouse..
can you detail a bit?
a
you can create connection to pull data from airbyte config db to destination data warehouse
@laila ribke you can do it in multiple way, i think above one is easy solution
l
sounds amazing. How do I know where is the airbyte config db? 😁 Sorry for the stupid question..
a
you can find it in Airbyte .env file
Copy code
DATABASE_USER=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
DATABASE_DB=
https://github.com/airbytehq/airbyte/blob/master/.env#L43
l
Thank you!
hi @Akilesh V, I managed to create the connection. But I get a sync failure on the attempts stream: process: query2_2023_80006428 [pid=18875] ----------------------------------------------- compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/airbyte_etl/attempts_scd.sql 4 of 9 ERROR creating incremental model airbyte_etl.attempts_scd........................................................ [ERROR in 0.90s] Database Error in model attempts_scd (models/generated/airbyte_incremental/scd/airbyte_etl/attempts_scd.sql) Invalid input DETAIL: ----------------------------------------------- error: Invalid input code: 8001 context: CONCAT() result too long for type varchar(65535) query: 80006428 location: string_ops.cpp:110 process: query2_2023_80006428 [pid=18875] ----------------------------------------------- Any ideas on how to fix/work around it?
a
@laila ribke it seems like issue with one of data type, you might need to change the data type to text you can debug DBT by following this doc
l
Hi @Akilesh V, I downgraded the redshift destination to 0.2.25 and it worked. I hope it won´t mess with my other connections. But thanks!!
👍 1
@Akilesh V, I started to invesitage on the tables and the connection table contains only the new airbyte db connection, and not the other ones I have and wanted to monitor.. any ideas?
may it be due to the host definition in the source? I tried to set it to db but failed. It only succeeded when I configured it to 0.0.0.0
a
@Ilia Slesarev not sure, if you are pointing to right db it should be present there
can you check it through any data management tool and confirm that required data present at the database
l
Hi, the problem was when I set it as incremental refresh. When I´ve set to full refresh it found all the connections. Thank you
a
Great to hear!
l
@Akilesh V, Hi, had to roll back to latest version because the version that works with the normalization doesn´t have the S3 staging. And it´s crucial for us.. So I still have the same problem..
a
this link might help you to understand issue better
5 Views
Open in Slack
PreviousNext
Powered by