anyone want to share some db credentials with me on slack? 😉

are the BQ credentials that we have not for the BQ instance in the sandbox project? is there another GCP project that i should be added to?

can someone remind me why we are doing both of these steps in the master build?

- name: Build
        run: ./gradlew build --no-daemon -g ${{ env.GRADLE_PATH }}

and

- name: Build Core Docker Images (Master branch only)
        if: success() && github.ref == 'refs/heads/master'
        run: docker-compose -f docker-compose.build.yaml build

isn’t the latter a super set of the former?

@s, per our MVP checklist, you’re working on making sure “it works” as part of the testing you’re doing, right. i’m struggling to get an E2E test to run myself, but if it’s something you’re already working on then, I’m going to pivot and focus on something else so we’re not solving the same problems. But if no one is focusing on the “it works” thing already, then I’ll take a look!

we mentioned changing the label for our MVP from 1.0 to 0.X. what should X be? 0.7?

check it out: https://github.com/singer-io/tap-postgres we're on the tap-postgres page!

Let me know if you have a docker id, I will add you to our docker repo

one thing that could be good is to have exchangerateapi_io as a tap

what do you think: https://docs.dataline.io/deployment/deploying-dataline/on-aws-ec2?

i’ve updated the developing locally section of our docs. at your convenience take a look and lmk if you have feedback (either itt or just edit the page).

@s is there anything i can do to help on the “it works” front?

Are you good with me merging this: https://github.com/datalineio/dataline/pull/227 ?

GCP docs: https://docs.dataline.io/deployment/deploying-dataline/on-gcp-compute-engine

Should we put

$

in the doc to signify the terminal prompt?

going over the docs. do we use yarn or npm?

what is default_target_schema for the bigquery integration?

is there a principle that we follow around retry policy of jobs? seems like jobs are currently infinitely retrying

I’m having a hard time this morning getting intellij treat our generated pojos as source. i even went back and started from scratch, so it probably means someone new to your project may run into the same issue. anyone able to screenshot the project structure of

dataline-config:models

so i can see what a working version looks like?

Found a few issues that I am fixing: • we don't check the return code of the output process to decide on success of failure • we are too agressive when we close the process • we still have a problem with schema validation when we check the schema on a jsonNode in the stream so we just skip the schema message sometime and the target doesn't like it

for testing the app internally (specifically the case where we want john to be able to test the ui) seems like there are two options: 1. just write up instructions for how to set it up and run it locally (with docker, etc) 2. host it on our gcp or aws infra so that john can interact with it just via some url has anyone thought about this already? probably just going to do 1 to get john up and running this afternoon 2, but i think 2 might be a good thing for us to set up in the future.

end to end?

regarding https://github.com/datalineio/dataline/pull/259

Heads up: We are going to start showing the

description

field for integration properties in the UI. In most cases we should be putting something in there that would make sense to an end user. We will not make them required, but my guess is in most cases it’s going to be helpful to have some text there. Jared did a nice job with the BigQuery destination, so use this as an example of what things should look like. Update: We are also going to display the first element of the

examples

field (if it exists in the UI) as well. Both

description

and

examples

are part of the json schema spec.

Do we want to log the stdout of the target in a file? and do we want to log the Tap non-singer protocol messages in a file?

@Jared Rhizor (Airbyte) isn't gradle build supposed to build the integration tests as well?

I will work on redirecting stdout to files + adding more unit tests following my changes from yesterday

draft for integration template. comments welcome https://docs.google.com/document/d/18NALB6EGo4v2jBRGWn7_1Qsax6HX13Sq1PDgWFLY5ZA/edit?usp=sharing

What are breadcrumbs used for?