Lijo Lawrance
02/15/2022, 7:49 AMLijo Lawrance
02/15/2022, 8:16 AMNathan Gille
02/15/2022, 10:24 PMicon:
come from in source_definitions.yaml?Christopher Wu
02/16/2022, 8:25 PMChristopher Wu
02/16/2022, 11:27 PM_scan_schema
because the subschema is being read as a list. Not sure if the required key should be included but it is.
subschema = [{'properties': {'Connector': {'type': 'string'}, 'Current': {'type': 'string'}, 'Description': {'type': ['null', 'str... ...}, 'required': ['portNumber', 'stationName', 'Geo', 'Description', 'Reservable', 'Status', ...], 'type': 'object'}]
path = '/Port/[]'
def _scan_schema(subschema, path=""):
if "oneOf" in subschema or "anyOf" in subschema:
if annotate_one_of:
return [
_scan_schema({"type": "object", **s}, path + f"({num})")
for num, s in enumerate(subschema.get("oneOf") or subschema.get("anyOf"))
]
return [_scan_schema({"type": "object", **s}, path) for s in subschema.get("oneOf") or subschema.get("anyOf")]
> schema_type = subschema.get("type", ["null"])
E AttributeError: 'list' object has no attribute 'get'
../../bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py:194: AttributeError
Is this an issue with my configured catalog?Nathan Gille
02/17/2022, 5:13 AMpython main.py read --config secrets/config.json --catalog source_bubble/schemas/bub.json
it gives the following error:
{"type": "LOG", "log": {"level": "FATAL", "message": "1 validation error for ConfiguredAirbyteCatalog\nstreams\n field required (type=value_error.missing)
Any idea what's wrong ?Samkit Shah
02/17/2022, 5:34 AMstate
table in the airbyte
database? My understanding was that this happens every state_checkpoint_interval
, however, this does not appear to be the case.
• I can see in the logs that the state gets set every `state_checkpoint_interval`;
• If I check the state in the database after this log appears, no update is made;
• Several STATE
logs appear, yet the database is still not updated. It seems to only happen on job completion, or after some very large number of records have been processed.
Can you please provide some further information on how this exactly works, and if this can be configured?Giunio De Luca
02/17/2022, 10:59 AM{
"success": false,
"error": {
"code": 101,
"type": "missing_access_key",
"info": "You have not supplied an API Access Key. [Required format: access_key=YOUR_ACCESS_KEY]"
}
}
I didn't find in the toturial any information about the access_key for this API, how this could be?Aakash Kumar
02/17/2022, 12:14 PMerrors: $.type: is missing but it is required
2022-02-17 11:25:10 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:14 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.
errors: $.type: is missing but it is required
2022-02-17 11:25:14 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:18 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.
errors: $.type: is missing but it is required
2022-02-17 11:25:18 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {}
2022-02-17 11:25:22 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.
errors: $.type: is missing but it is required
I have attached the logs for reference. I have already validated the schema file and it is a valid json. Also there is no field as test. Please have a look,Giunio De Luca
02/17/2022, 3:54 PMclass SourcePythonHttpTest(AbstractSource):
def check_connection(self, logger, config) -> Tuple[bool, any]:
return True, None
def streams(self, config: Mapping[str, Any]) -> List[Stream]:
auth = HTTPBasicAuth(username=config["username"], password="password")
return [DerivedStream(authenticator=auth)]
Should I use any airbyte module for that?Giunio De Luca
02/17/2022, 8:45 PMlatest
, does Airbyte always check for a new latest
image before each sync?Oleh Zorenko
02/18/2022, 9:24 AM/test connector=connectors/source-chargebee
in this PR but I keep on getting an error in :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFiles
(see details on screenshot). I pulled lates changes from master
but still getting this error. Here is the link to failed check.
Also I have few checks failing on that PR:
• https://github.com/airbytehq/airbyte/runs/5245257875?check_suite_focus=true
Execution failed for task ':airbyte-cdk:python:generateProtocolClassFiles'.
3727
> Process 'command 'bin/generate-protocol-files.sh'' finished with non-zero exit value 1
• https://github.com/airbytehq/airbyte/runs/5245259204?check_suite_focus=true
Execution failed for task ':buildDockerImage-reporter'.
1312
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist
• https://github.com/airbytehq/airbyte/runs/5245259013?check_suite_focus=true
Execution failed for task ':buildDockerImage-reporter'.
940
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist
• https://github.com/airbytehq/airbyte/runs/5245262023?check_suite_focus=true
E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 3660 (unattended-upgr)
29
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
• https://github.com/airbytehq/airbyte/runs/5245262222?check_suite_focus=true
Execution failed for task ':buildDockerImage-reporter'.
933
> Could not build image: ADD failed: file not found in build context or excluded by .dockerignore: stat bin/airbyte-metrics-reporter-0.35.30-alpha.tar: file does not exist
Oleh Zorenko
02/18/2022, 10:13 PMNathan Gille
02/19/2022, 4:37 AMSamir Bhattarai
02/20/2022, 12:22 PMSamir Bhattarai
02/21/2022, 6:44 PMOrlando Burli Junior
02/21/2022, 9:43 PMJordan Fox
02/22/2022, 2:37 PMMarcos Cunha
02/22/2022, 5:01 PMJoão Jönk
02/22/2022, 8:26 PMHuib
02/23/2022, 8:29 AMHuib
02/23/2022, 1:02 PMtls=true
and the full connection string needs to have this format: mongodb://{username}:{password}@{host}:{port}/{database}?tls=true&tlsAllowInvalidHostnames=true&tlsAllowInvalidCertificates=true
I’ve tried to test my theory by creating JDBC source, but in the end I could dockerize the container. Seems like the dependency on airbyte/integration-base-java:dev
image is not permitted.
So I guess question/request, is this possible to add to the existing MongoDB connector; and if yes, what would be the expected time frame for this? It would also be great to add ability to use SSH with this connecotr.yevgen
02/23/2022, 1:56 PMMadhu Prabhakara
02/23/2022, 6:23 PMChristopher Wu
02/23/2022, 6:43 PMMayank Malu
02/23/2022, 7:03 PMVictor
02/24/2022, 9:17 AMFred Reimer
02/24/2022, 4:24 PMManav
02/24/2022, 5:43 PMEric Santulli
02/24/2022, 5:49 PM