Hi team i found error "WORKER_ENVIRONMENT not found or empty, defaulting to DOCKER" with multipal sources like mailchimp, slack, facebook. Anyone have any idea about this error.

version: 0.28.0-alpha

are you sure that is the real error? i know we log that line, but I it is just meant as information.

it might help if you share all of the related logs. likely there is problem in there somewhere. i just don't think the line you're mentioning is probably the issues.

Plz check this log, thank you

@charles i did some testing on the issue occurred with multiple sources and here is the summary: when i choose destination as s3-jsonl all source works fine but issue occurs when i select s3-parquet format

Please let me know if you would require logs for each source

thank you

the real error seems to be:

2021-07-25 20:26:05 INFO () DefaultAirbyteStreamFactory(internalLog):110 - Syncing stream: email_activity 
2021-07-25 20:26:06 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-25 20:26:06 [33mWARN[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):78 - {} - Airbyte message consumer: failed.
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - Exception in thread "main" java.lang.IllegalStateException: Field type has no type
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getTypes(JsonToAvroSchemaConverter.java:82)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getNonNullTypes(JsonToAvroSchemaConverter.java:70)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getNullableFieldTypes(JsonToAvroSchemaConverter.java:223)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.avro.JsonToAvroSchemaConverter.getAvroSchema(JsonToAvroSchemaConverter.java:162)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.writer.ProductionWriterFactory.create(ProductionWriterFactory.java:58)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.S3Consumer.startTracked(S3Consumer.java:103)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.start(FailureTrackingAirbyteMessageConsumer.java:54)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:131)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:117)
2021-07-25 20:26:06 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.destination.s3.S3Destination.main(S3Destination.java:49)

@Chris (deprecated profile) i check email_activity json schema as well on source-mailchimp but every column has already mapped with their type.

can you share the catalog.json ? https://docs.airbyte.io/operator-guides/browsing-output-logs#reading-the-content-of-the-catalog-json-file

here is the source catalog of mailchimp

i am also sharing slack log

The root cause is that currently our Json -> Avro schema converter does not support

$ref

keyword, nor retrieving schema from a URL. Fields with

$ref

led to this failure:

"type": {
    "$ref": "<https://us1.api.mailchimp.com/schema/3.0/Definitions/CampaignType.json>"
},
"status": {
    "$ref": "<https://us1.api.mailchimp.com/schema/3.0/Definitions/CampaignStatus.json>"
}

We are slowly patching the schema converter to support more keyword. I will create a ticket to track it.