https://linen.dev logo
a

Ankur Adhikari

07/25/2021, 8:31 PM
Hi team i found error "WORKER_ENVIRONMENT not found or empty, defaulting to DOCKER" with multipal sources like mailchimp, slack, facebook. Anyone have any idea about this error.
u

user

07/25/2021, 8:31 PM
version: 0.28.0-alpha
u

user

07/26/2021, 12:52 AM
are you sure that is the real error? i know we log that line, but I it is just meant as information.
u

user

07/26/2021, 12:53 AM
it might help if you share all of the related logs. likely there is problem in there somewhere. i just don't think the line you're mentioning is probably the issues.
a

Ankur Adhikari

07/26/2021, 5:36 AM
Plz check this log, thank you
u

user

07/26/2021, 10:37 AM
@charles i did some testing on the issue occurred with multiple sources and here is the summary: when i choose destination as s3-jsonl all source works fine but issue occurs when i select s3-parquet format
u

user

07/26/2021, 10:37 AM
Please let me know if you would require logs for each source
u

user

07/26/2021, 10:37 AM
thank you
u

user

07/26/2021, 2:13 PM
the real error seems to be:
Copy code
2021-07-25 20:26:05 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: email_activity 
2021-07-25 20:26:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-25 20:26:06 [33mWARN[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):78 - {} - Airbyte message consumer: failed.
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - Exception in thread "main" java.lang.IllegalStateException: Field type has no type
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getTypes(JsonToAvroSchemaConverter.java:82)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getNonNullTypes(JsonToAvroSchemaConverter.java:70)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getNullableFieldTypes(JsonToAvroSchemaConverter.java:223)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getAvroSchema(JsonToAvroSchemaConverter.java:162)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.writer.ProductionWriterFactory.create(ProductionWriterFactory.java:58)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.S3Consumer.startTracked(S3Consumer.java:103)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.start(FailureTrackingAirbyteMessageConsumer.java:54)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:131)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:117)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.S3Destination.main(S3Destination.java:49)
u

user

07/26/2021, 4:26 PM
@Chris (deprecated profile) i check email_activity json schema as well on source-mailchimp but every column has already mapped with their type.
u

user

07/26/2021, 4:55 PM
here is the source catalog of mailchimp
u

user

07/26/2021, 4:57 PM
i am also sharing slack log
u

user

07/27/2021, 4:18 PM
The root cause is that currently our Json -> Avro schema converter does not support
$ref
keyword, nor retrieving schema from a URL. Fields with
$ref
led to this failure:
Copy code
"type": {
    "$ref": "<https://us1.api.mailchimp.com/schema/3.0/Definitions/CampaignType.json>"
},
"status": {
    "$ref": "<https://us1.api.mailchimp.com/schema/3.0/Definitions/CampaignStatus.json>"
}
We are slowly patching the schema converter to support more keyword. I will create a ticket to track it.
3 Views