premsurawut
07/24/2024, 7:06 AM${NAMESPACE}/${STREAM_NAME}/loaddt=${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}/
Marco Brea
07/24/2024, 7:47 AMAyushi Haria
07/24/2024, 8:02 AMÇağdaş Akkaş
07/24/2024, 8:23 AMKevin Lefevre
07/24/2024, 8:36 AMÇağdaş Akkaş
07/24/2024, 8:40 AMPranay Dhokia
07/24/2024, 8:47 AMfailure to invoke API
error. With error sending request: Post "<http://localhost:8001/api/public/v1/sources#Hubspot>": dial tcp [::1]:8001: connectex: No connection could be made because the target
│ machine actively refused it.
When attempting to access http://localhost:8001/api/public/v1/ via my web browser I am met with localhost refusing to connect. Might you please be able to advise?Kevin Lefevre
07/24/2024, 8:47 AMmanifest.yaml
• integration_tests/configured_catalog.json
=> screenshot 1
• schemas/xxx.json
files for each stream describing its schema => screenshot 2
My case is, how can I ensure the output data will be aligned with the declared schema in .json
files ?
Some tests showed that id property is not considered as an integer
as declared, but as a number
(leading to double
in AVRO GCS destination)
How can I fix it ?Ted McFadden
07/24/2024, 8:49 AMD Ra
07/24/2024, 8:52 AMMalvic Leroux
07/24/2024, 9:29 AMKevin Lefevre
07/24/2024, 9:36 AM"reviewInfo": {
"type": ["object", "null"],
"properties": {
"lastUpdated": {
"type": ["null", "string"],
"format": "date-time"
},
"reviewStatus": {
"type": ["string", "null"]
},
"rejectionReasons": {
"type": ["array", "null"],
"items": {
"type": ["string", "null"]
}
}
}
and I have below API output example:
"reviewInfo": {
"rejectionReasons": [],
"lastUpdated": 1720710761453,
"reviewStatus": "AUTO_APPROVED"
},
but the Airbyte connector fails with below error:
Stack Trace: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field reviewInfo is expected to be one of these: NULL, RECORD. If this is a complex type, check if offending field (path: reviewInfo.lastUpdated) adheres to schema: {rejectionReasons=[], lastUpdated=1720710761453, reviewStatus=AUTO_APPROVED}
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:131)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:120)
at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToGenericDataRecord(JsonAvroConverter.java:95)
at io.airbyte.integrations.destination.s3.avro.AvroRecordFactory.getAvroRecord(AvroRecordFactory.java:46)
at io.airbyte.integrations.destination.s3.avro.AvroSerializedBuffer.writeRecord(AvroSerializedBuffer.java:52)
at io.airbyte.integrations.destination.record_buffer.BaseSerializedBuffer.accept(BaseSerializedBuffer.java:106)
at io.airbyte.integrations.destination.record_buffer.SerializedBufferingStrategy.addRecord(SerializedBufferingStrategy.java:74)
at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.acceptTracked(BufferedStreamConsumer.java:205)
at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.accept(FailureTrackingAirbyteMessageConsumer.java:65)
at io.airbyte.integrations.base.Destination$ShimToSerializedAirbyteMessageConsumer.consumeMessage(Destination.java:114)
at io.airbyte.integrations.base.Destination$ShimToSerializedAirbyteMessageConsumer.accept(Destination.java:90)
at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:234)
at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:214)
at io.airbyte.integrations.base.IntegrationRunner.lambda$runInternal$0(IntegrationRunner.java:153)
at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:272)
at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:157)
at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
at io.airbyte.integrations.destination.gcs.GcsDestination.main(GcsDestination.java:46)
Caused by: org.apache.avro.AvroTypeException: Could not evaluate union, field reviewInfo is expected to be one of these: NULL, RECORD. If this is a complex type, check if offending field (path: reviewInfo.lastUpdated) adheres to schema: {rejectionReasons=[], lastUpdated=1720710761453, reviewStatus=AUTO_APPROVED}
at tech.allegro.schema.json2avro.converter.AvroTypeExceptions.unionException(AvroTypeExceptions.java:16)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.readUnion(JsonGenericRecordReader.java:304)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:198)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:172)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.lambda$readRecord$0(JsonGenericRecordReader.java:154)
at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:721)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.readRecord(JsonGenericRecordReader.java:143)
at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:129)
... 17 more
How can I fix it ?Çağdaş Akkaş
07/24/2024, 9:36 AMIshan Anilbhai Koradiya
07/24/2024, 10:32 AMPrayrit Dahiya
07/24/2024, 10:57 AMBread Pitt
07/24/2024, 11:15 AMUdit Chaudhary
07/24/2024, 11:44 AMDeepak Jaiswal
07/24/2024, 11:57 AMKaritham
07/24/2024, 12:06 PMMalvic Leroux
07/24/2024, 12:26 PMAbhinav Pandey
07/24/2024, 12:31 PMFabrizio Bulleri
07/24/2024, 1:13 PMDeepak Jaiswal
07/24/2024, 1:37 PMPravin Singh
07/24/2024, 2:13 PMWanderson P. Medeiros
07/24/2024, 3:10 PMWanderson P. Medeiros
07/24/2024, 3:25 PMWanderson P. Medeiros
07/24/2024, 3:31 PMJason Alexander
07/24/2024, 3:39 PMSilvia Giammarinaro
07/24/2024, 4:13 PMBenjamin Trollope
07/24/2024, 4:16 PM