https://linen.dev logo
Join Slack
Powered by
# announcements
  • s
    ^this link is wrong. Here is the correct link for office hours: https://us02web.zoom.us/j/83142558458?pwd=b2cydzJ5UXJRSnhhM3ZwTmxEVm9Sdz09
  • m
    set up a reminder to “This is office hour time!! Join now: https://us02web.zoom.us/j/86004276586?pwd=eXVkL01TWE16eWErcldqNkxhTzN4Zz09” in this channel at 10:59AM every Friday, Pacific Standard Time.
  • j
    <!channel> Hey everyone 👋! here are the updates for the past week: airbyte rocket We are currently looking for some SENIOR SOFTWARE ENGINEERS and a FOUNDING DEV ADVOCATE. If you know anyone that would be interested, we would love to talk to then. airbyte rocket Fixes / features on connectors: • Google Analytics: add the ability to sync custom reports • Apple Appstore: bug fix to correctly run incremental syncs • Exchange rates: UI now correctly validates input date pattern • File Source: Support JSONL (newline-delimited JSON) format • Freshdesk: Enable controlling how many requests per minute the connector makes to avoid overclocking rate limits
    💯 2
    airbyte growth 2
    airbyte rocket 4
    🥨 3
    octavia loves 3
    🎉 8
  • m
    set the channel topic: General discussion about Airbyte. Community news and announcements! https://github.com/airbytehq/airbyte
  • j
    it seems we're on Betapage.co today! don't hesitate to upvote and leave your feedback 🙂
    airbyte growth 2
  • j
    a lot of things are happening this week. our new tutorial "*Save and search through your Slack history on the free plan*" is on Hackernews. In it, we explain how you can use Airbyte and Meilisearch to save all your Slack history, and search through it without limits, even on a free plan. something useful when you have a large community!
    🔥 1
    meilisearch 1
    S 2
    airbyte star 1
    airbyte rocket 4
    🙌 5
  • u
    Hey guys, who is waiting to use the Airbyte Operator in Airflow? 🎉 I'm already working on it, but expected some feedback from users who want to use it (please send msg). I've already code the AirbyteTriggerSyncOperator (accepted name suggestions 😛) and the AirbyteHook. Is very simple at the moment. You need an airbyte_conn_id to send requests for your Airbyte server and the connectionId to trigger the Airflow DAG, also an optional parameter for timeout. Tonight I'l finish unit tests and integration tests, probably be able to submit the PR in Airflow project between today and tomorrow. You can see the code here please add comments and ideas! I'm also working on an introductory tutorial between Airbyte and Airflow. The DAG below represents the idea. The focus is on teaching how to trigger a task in Airflow that starts the job on Airbyte. I put the DummyOperator just as an illustration of what the flow would look like in a more realistic scenario. At this point I reused the example of Getting Started as source(money) / destination(json). But I think it would be better to think of a new scenario so that it doesn't get repetitive. Any idea? I didn't want anything complex because the focus would be more on building DAG on Airflow ... In the future I plan to make a more detailed and complex blog / tutorial for anyone looking for something like that. What do you think? The link to Airflow Airbyte Tutorial
    🔥 5
    🙌 3
    octavia loves 1
    airbyte growth 2
    airbyte heart 1
  • s
  • r
    also I turn off the basic normalization with no success.
  • u
    👋 I created the channel #C01PCV47CTY to discuss about the integration, features, bugs, etc. Also I submit a PR to added Airbyte as a community providers in Airflow today (waiting for their feedback and approval)
    👌 2
  • c
    I run gradle build and still BaseConnector can't be imported. Previously it wen't well. Slack Conversation
  • j
    <!channel> Hey everyone 👋! We just released version 0.16.0-alpha. Here is what changes: • We made some progress to address nested tables in our normalization. Previously, basic normalization would output nested tables as-is and append a number for duplicate tables. For example, Stripe’s nested address fields go from:
    Copy code
    Address
address_1
    To
    Copy code
    Charges_source_owner_755_address
customers_shipping_c70_address
    After the change, the parent tables are combined with the name of the nested table to show where the nested table originated. This is a breaking change for the consumers of nested tables. Consumers will need to update to point at the new tables. • New native Hubspot connector with schema folder populated (#2150) • Other fixes / features: ◦ Add connector version to bug report ◦ Facebook Marketing connector: add option to include deleted records
    airbyte 4
    airbyte star 2
    👍 7
    airbyte growth 3
    airbyte rocket 3
    🤩 1
  • s
    Congratulations Airbyte Team (family).
    airbyte heart 2
    👍 4
  • p
    Hi all - seems like a great tool
  • j
    set up a reminder to “This is office hour time!! Join now: https://us02web.zoom.us/meeting/register/tZcod-2rpzgvHtYBIUHq6f2u5Npy-Q4Hhh3o” in this channel at 10:59AM every Friday, Pacific Standard Time.
  • s
    any suggestation will work
  • s
    any on there
  • s
    one*
  • j
    @Sumit Srivastava responses are in a thread for your original question
  • s
  • j
    👋 Hi everyone! We have been investigating both Meltano and Airbyte and are landing on Airbyte (your progress has been amazing). While our initial use cases are for data replication and data pulls from systems like FB. Ive been thinking about another potential use cases. We have use-cases where people are loading data from excel files that people need to manually upload. We create template excel files for people to use and then need to validate what was uploaded and then push into a db/warehouse. A team has been using R & Shiny for this, but Im wondering if this might be another killer use case for Airbyte. Basically a self-service mechanism to upload files, validate (via dbt) then perform an actual load into a DB. This would require some work on the UI side to support, but I would imagine this is something many folks could use where folks are slinging around excel files.
    🤔 1
  • j
    Hello everyone, Thank you for an amazing product! I am not sure if this is good chanell but if somebody could assist me, I am trying for last two days to setup HTTP request as source
  • j
    and I am stuck at getting: {"detail":"JSON parse error - Expecting value: line 1 column 1 (char 0)"}
  • a
    Copy code
    2021-03-08 16:03:11 INFO (/tmp/workspace/6/0) LineGobbler(voidCall):69 - 2021-03-08 16:03:11 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):148 - {} - Executing finalization of tables.
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - Exception in thread "main" net.snowflake.client.jdbc.SnowflakeSQLException: SQL execution canceled
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionSub(SnowflakeUtil.java:124)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowException(SnowflakeUtil.java:64)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:434)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:338)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:501)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:229)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:167)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.core.SFStatement.execute(SFStatement.java:749)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:292)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:342)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.db.jdbc.JdbcDatabase.lambda$execute$0(JdbcDatabase.java:50)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.db.jdbc.DefaultJdbcDatabase.execute(DefaultJdbcDatabase.java:61)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.db.jdbc.JdbcDatabase.execute(JdbcDatabase.java:50)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.jdbc.DefaultSqlOperations.executeTransaction(DefaultSqlOperations.java:100)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.jdbc.JdbcBufferedConsumerFactory.lambda$onCloseFunction$3(JdbcBufferedConsumerFactory.java:149)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer$OnCloseFunction.accept(BufferedStreamConsumer.java:213)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.close(BufferedStreamConsumer.java:174)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.FailureTrackingConsumer.close(FailureTrackingConsumer.java:65)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:137)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:115)
2021-03-08 16:08:12 ERROR (/tmp/workspace/6/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.snowflake.SnowflakeDestination.main(SnowflakeDestination.java:59)
2021-03-08 16:08:14 ERROR (/tmp/workspace/6/0) DefaultSyncWorker(run):104 - Sync worker failed.
io.airbyte.workers.WorkerException: target process wasn't successful
at io.airbyte.workers.protocols.airbyte.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:105) ~[io.airbyte-airbyte-workers-0.16.1-alpha.jar:?]
at io.airbyte.workers.DefaultSyncWorker.run(DefaultSyncWorker.java:103) [io.airbyte-airbyte-workers-0.16.1-alpha.jar:?]
at io.airbyte.workers.DefaultSyncWorker.run(DefaultSyncWorker.java:48) [io.airbyte-airbyte-workers-0.16.1-alpha.jar:?]
at io.airbyte.workers.wrappers.OutputConvertingWorker.run(OutputConvertingWorker.java:44) [io.airbyte-airbyte-workers-0.16.1-alpha.jar:?]
at io.airbyte.workers.wrappers.JobOutputSyncWorker.run(JobOutputSyncWorker.java:32) [io.airbyte-airbyte-workers-0.16.1-alpha.jar:?]
at io.airbyte.scheduler.worker_run.WorkerRun.lambda$new$0(WorkerRun.java:53) [io.airbyte-airbyte-scheduler-0.16.1-alpha.jar:?]
at io.airbyte.scheduler.worker_run.WorkerRun.call(WorkerRun.java:61) [io.airbyte-airbyte-scheduler-0.16.1-alpha.jar:?]
at io.airbyte.scheduler.worker_run.WorkerRun.call(WorkerRun.java:42) [io.airbyte-airbyte-scheduler-0.16.1-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.16.1-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.16.1-alpha.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
  • g
    @Cody K.
  • g
    @Nate
  • v
    can someone please help me with this error?
  • j
    <!channel> Hey everyone 👋! A LOT OF UPDATES this past week! • Airbyte raised a $5.2M seed round with Accel. Here’s an article about our vision with it. • Integration with Airflow is here. Thanks to @[DEPRECATED] Marcos Marx, you can now run Airbyte jobs from Airflow directly. A tutorial is on the way and should be coming this week! • MongoDB’s source connector is also coming, thanks to Yury! • Add a prefix for tables, so that tables with the same name don't clobber each other in the destination • Although our progress was not always visible, February was a very productive month on our side. It was a stabilization month that will help us accelerate better in the future. Here’s an article describing what was under the hood in February. • Our very first community call: it’s a 45-min call that will feature one of our prominent users on how they leverage Airbyte. The team will also provide updates on our roadmap. For this 1st time, our guest will be @Jins Kadwood Kadwood , CTO of AgriDigital, who will talk about his leveraging of Airbyte for his caching strategy.  • New contest launched today, until 04/30: 3 winners for $500 worth of prizes each (winner can choose between a list of devices, cash, and donations). Contribute sources during that time, and you will qualify to win. We will send swag gifts to all contributors.  • Other fixes: ◦ Google Analytics: Support chunked syncs to avoid sampling ◦ AppStore: fix bug where the catalog was displayed incorrectly • We published a few new tutorials:  ◦ How to save and search through your Slack history with a free plan, using Airbyte and MeiliSearch ◦ How to build a Slack Activity dashboard, using Airbyte and Apache Superset ◦ How you can add incremental to a source
    airbyte growth 6
    💰 1
    airbyte star 9
    👍 6
    💯 14
    airbyte rocket 9
    airbyte 8
    ❤️ 22
  • r
    Congratulations!
    airbyte heart 2
  • j
    YEs
1...171819...24Latest