I fail some of the integration test locally because `java ni Airbyte #contributing-to-airbyte

I fail some of the integration test locally becaus...

P.VAD

06/10/2021, 2:51 AM

I fail some of the integration test locally because

java.nio.file.NoSuchFileException: secrets/config.json

where should I specify and what content to be put? I running test on

:airbyte-integrations:connectors:destination-s3:integrationTest

user

06/10/2021, 6:59 AM

Hi, this should be placed in a

secrets

directory inside

destination-s3

user

06/10/2021, 7:08 AM

What the content should be?

user

06/10/2021, 7:11 AM

the config.json should match what the destination requires in it's spec.json

user

06/10/2021, 7:13 AM

Understood thanks.

user

06/10/2021, 7:13 AM

the

config.json

file should look like (not all the fields are filled up here)

Copy code

{
  "password": "b",
  "s3_bucket_name": "a",
  "s3_bucket_region": "us-west-2",
  "access_key_id": "key",
  "secret_access_key": "secret",
.... 
}

Davin Chia (Airbyte)

06/10/2021, 7:13 AM

yes this is so helpful, thank you

user

06/10/2021, 7:34 AM

i got null pointer exception, does it mean i miss some parameter?

user

06/10/2021, 7:48 AM

probably

user

06/10/2021, 7:49 AM

can I see the exception?

user

06/10/2021, 7:58 AM

Copy code

Task :airbyte-integrations:connectors:destination-s3:integrationTestJava

S3CsvDestinationAcceptanceTest > testCheckConnectionInvalidCredentials() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncVeryBigRecords() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [1] exchange_rate_messages.txt, exchange_rate_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [2] edge_case_messages.txt, edge_case_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [1] exchange_rate_messages.txt, exchange_rate_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [2] edge_case_messages.txt, edge_case_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testIncrementalDedupeSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testGetSpec() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSecondSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCustomDbtTransformationsFailure() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testIncrementalSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCheckConnection() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncWriteSameTableNameDifferentNamespace() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testLineBreakCharacters() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncUsesAirbyteStreamNamespaceIfNotNull() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCustomDbtTransformations() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

user

06/10/2021, 7:59 AM

for your information, this test is done after modification on S3-Destination connector complete. Which is now Support both AWS S3 and Alternative S3 like Minio S3

Davin Chia (Airbyte)

06/10/2021, 8:00 AM

oh nice!

user

06/10/2021, 8:00 AM

does that fail because of new feature is not write in acceptance test file?

user

06/10/2021, 8:00 AM

I don't think so. it looks like it's because the format field in the config isn't present

user

06/10/2021, 8:01 AM

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/destination-s3/src/main/resources/spec.json#L76

user

06/10/2021, 8:02 AM

Copy code

{
  "s3_endpoint": "aws",
  "s3_bucket_name": "sokhen...",
  "s3_bucket_path": "...",
  "s3_bucket_region": "ap-southeast-1",
  "access_key_id": "AKIA4...",
  "secret_access_key": "DJv0cvPls.....",
  "format": "CSV"
}

user

06/10/2021, 8:02 AM

still fail, try multiple time already

Davin Chia (Airbyte)

06/10/2021, 8:03 AM

what if i do a merge request then team can give feedback ?

user

06/10/2021, 8:20 AM

let me see

Davin Chia (Airbyte)

06/10/2021, 8:20 AM

yes! you can start a PR and we'll be happy to give ya feedback

user

06/10/2021, 8:23 AM

https://github.com/airbytehq/airbyte/pull/4019

user

06/10/2021, 8:25 AM

ah it's because you need something like

Copy code

{
  "s3_bucket_name": 
  "s3_bucket_path": 
  "s3_bucket_region": 
  "access_key_id": 
  "secret_access_key": 
  "format": {
    "format_type": "CSV",
    "flattening": "Root level flattening"
  }
}

2 Views

Open in Slack

Previous Next