Hey <@U035ZCRAH7W> and <@U01MPEM22RK>, clicking on...
# questions
r
Hey @nutritious-easter-92546 and @orange-computer-53672, clicking on the force full refresh does trigger the schema sync as well. However the Astra Airtable architecture is different to classic as it's built to be more scalable, and is therefore more async. Can you please share more details here or with support on which fields are not getting synced for you? I'll be looking into the logs to pinpoint the issue as well.
n
Here's a video showing: • Schema changes don't sync after 5-10 minutes • Refreshing doesn't help • Force full re-sync of all data brings in the data but schema doesn't change • Refreshing shows the data with the new schema and all is as expected I hadn't realised about the full re-sync + refresh until now so that may be the solution but obvious not ideal. Right now in this base I have very few records but soon it will have hundreds of thousands of records.
Also another record syncing issue: • Record added to Registration table Airtable 8 mins ago • Record still not in Stacker in All Registration (no filters)
It just showed up 11 minutes later
r
Thank you for the details Ryan!
• Refreshing shows the data with the new schema and all is as expected
This is exactly one of the components I suspected. I believe depending on where you are and which mode you're in, you may not see the update automatically and need to refresh the page. I'll double check that this is still the expected behaviour on our side and this is not a new regression
Regarding that record taking 11 minutes to sync in, can you please shares its airtable id so I can trace it in the logs?
n
recemtFU1kiwe6Kh9
r
That record was found and saved at
14:58:02.183
BST - so very close to around when you reported it being created. Just to confirm, did you refresh the page before the 11 minutes when the record showed up? I'm wondering if the record was synced but the FE was not showing it
n
Yes, refreshed multiple times
rect1XhQ70Yupb6r2
Another example. Notes table. Refreshed and nothing.
r
Can you please confirm the exact time this record was created? (might be shown in Airtable if you've got the created date field) I can't yet see this record ID in any of our logs - so Airtable has not given us a webhook for it at all yet
n
CleanShot 2025-04-29 at 23.48.22@2x.png
thank you 1
And to clarify, if I want to see schema changes, is it expected that I do a full sync?
r
We have some support for realtime updates for schemas too so a manual re-sync should not always be needed, but if for some reason the webhook was not sent by Airtable, or there were issues processing then a manual sync would fix it.
In this case, I think both of the issues you've reported are related - it look like our processing of the webhooks is not working as fast as expected for some reason, so you're ending up waiting for the full sync to run on the schedule or when you press the sync button yourself. I'm escalating this internally now and will keep you updated
n
Thanks! FYI I'm going to force sync now so I can keep working.
👌 1
r
Your connection syncs are taking about 45 seconds, so sadly that's the minimum time for any records or schema changes to show up. I'm looking into the exact details of this now as well to see if we'd expect it to be faster for you
n
Not sure what a connection sync is?
r
My bad for using the technical terms, it's just our internal term for your airtable connection (aka entry in data sources)
It looks like the import process for you is optimal. You have 11 tables that are being imported. One of the tables,
tblm6r3xhTA7gY6Po
, seems to have a fair number records in it each time which ends up taking 30 seconds to import. The bottleneck is the duration of the Airtable API calls which is out of our hands
We're looking into why the webhooks are not working correctly though as that should help you not need to wait this amount of time for any new changes
🙌 1
n
Do you know the name of
tblm6r3xhTA7gY6Po
? We have one table with 1,300 records. But that's nothing, in classic we had ten of thousands of records.
r
That's probably it, but I can find the table name in bit if that helps. One of the main difference to classic experience from a few months ago is the recent Airtable rate limit + performance changes. Let me know if you're still using classic and if it performs better over there. (There are some intentional architecture differences but I can also look into the exact details to make sure Astra is not worse due to any bugs or unintentional decisions)
btw we've found the issue for the webhooks - I'm starting to work on the fix now
🙌 1
n
Yes, we have a Stacker Classic app linked too and it performers much better - sync is almost instant.
r
The webhook bug should now be fixed. Can you please try creating a new test record and we measure how long it takes to show up
Yes, we have a Stacker Classic app linked too and it performers much better - sync is almost instant.
Do you mean the time to get new data is almost instant or when you click on
Data sync
it completes almost instantly? These two things are working in two different ways. It's expected for the equivalent of clicking on
Data sync
to not be as fast as in classic as we now import all table together instead of their own But getting new data changes should be the same as in classic (of course wasn't the case due to the bug we discovered now)
n
Almost instant refers to new data/record being added in Airtable and then it showing in Stacker without intervention
Is there an ETA on fixing that bug?
o
HI @rich-pager-92385! There is definately a workflow sort of issue for adding new fields from airtable back into stacker astra. Last week at a client I added probably 10 new fields in their database (including a few tables) and for the life of me I couldn't get them sycning to astra. The new tables would show up, but when you toggled them on (under data sources) it just never did anything and the source never appears in your tables view. Same with new fields, they seem to take about 24-48 hours to appear, even clicking force refresh
I understand the new astra has a different syncing backend, and it's in beta
but my experience so far , is it's quite hard to find out the status of the syncing (compared to classic).
r
Hello @nutritious-easter-92546 , is the bug not fixed for you? A fix was released yesterday as mentioned here (I expect a FE refresh still being needed in some cases, but new Airtable records should show up quite quickly)
Hey @orange-computer-53672, were those tables quite large by any chance? One of the changes we made in Astra's Airtable integration is do sync the tables and data for them sequentially to help us keep under the Airtable rate limits - this has had the unfortunate implication that schema changes only show up once all the new data has been synced too. Please let me know if this is case and I can expand on the reasoning if you useful - if we confirm this as the cause, I'll raise it internally too and we can start on other solutions to avoid this
To give more context, we used to sync each table on its own in classic. But as Airtable has gotten very strict about rate limits and performs very badly if we make too many API calls, we've had to sync the tables sequentially. A consequence of this is that sometimes the sync performance is much worse than an aggressive strategy which we had in Classic
n
@rich-pager-92385 seems to be syncing much much faster. But here's a strange issue which I haven't seen before where it partially sync and then takes a while to fully sync: https://share.cleanshot.com/s3NfHqkG
r
Glad to hear it's faster @nutritious-easter-92546! I've just taken a look at why some of the updates are taking a bit longer - all the processing on our side finished under 5 second after Airtable informed us there was a change (which involves more API calls to Airtable and that's again the bottlenck) I think what's going on is that Airtable is taking longer to inform us of a change which is where the delay comes in. Are you consistently seeing this issue? (I'm wondering if it was a temporary increase in pressure on Airtable side hence why them pinging us took longer for second half of changes)
If you're able to reproduce, and if easy enough, it would be brilliant to have a recording that shows the current time. I can use that and correlate it with the logs to be more certain where the delay is coming from
@nutritious-easter-92546 I just remembered another aspect of the system which can explain the behaviour you're seeing: we limit how often we refresh the FE automatically. So it might be that the updates are actually applied but we didn't refresh the FE automatically for you until a certain amount of time has passed since last automatic refresh If you're able to reproduce this consistently, can you try to refresh the page manually if updates don't come in after ~5-10 seconds?
o
@rich-pager-92385 see 2 x screenshots. I have been trying to add these two tables for over a week now, every day I try to "add" and they just never do. always says gathering data, even if I force refresh
Screenshot 2025-05-01 at 8.38.32 PM.png,Screenshot 2025-05-01 at 8.38.37 PM.png
q
Hey Simon, thanks for the specific examples. We'll have a look into it but we have a couple of team members out today and tomorrow so it likely won't be until Monday, just for your expectation setting. Thanks again for persevering with the testing!
o
All good mate cheers Ben
r
Hey @orange-computer-53672, I've just checked these two tables and they're both imported now and with data. Was the issue somehow resolved on its own?