https://linen.dev logo
Channels
Powered by
p

P.VAD

06/10/2021, 2:51 AM
I fail some of the integration test locally because 
java.nio.file.NoSuchFileException: secrets/config.json
where should I specify and what content to be put? I running test on 
:airbyte-integrations:connectors:destination-s3:integrationTest
u

user

06/10/2021, 6:59 AM
Hi, this should be placed in a 
secrets
directory inside 
destination-s3
u

user

06/10/2021, 7:08 AM
What the content should be?
u

user

06/10/2021, 7:11 AM
the config.json should match what the destination requires in it's spec.json
u

user

06/10/2021, 7:13 AM
Understood thanks.
u

user

06/10/2021, 7:13 AM
the 
config.json
file should look like (not all the fields are filled up here)
Copy code
{
  "password": "b",
  "s3_bucket_name": "a",
  "s3_bucket_region": "us-west-2",
  "access_key_id": "key",
  "secret_access_key": "secret",
.... 
}
d

Davin Chia (Airbyte)

06/10/2021, 7:13 AM
yes this is so helpful, thank you
u

user

06/10/2021, 7:34 AM
i got null pointer exception, does it mean i miss some parameter?
u

user

06/10/2021, 7:48 AM
probably
u

user

06/10/2021, 7:49 AM
can I see the exception?
u

user

06/10/2021, 7:58 AM
Copy code
Task :airbyte-integrations:connectors:destination-s3:integrationTestJava

S3CsvDestinationAcceptanceTest > testCheckConnectionInvalidCredentials() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncVeryBigRecords() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [1] exchange_rate_messages.txt, exchange_rate_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [2] edge_case_messages.txt, edge_case_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [1] exchange_rate_messages.txt, exchange_rate_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > [2] edge_case_messages.txt, edge_case_catalog.json FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testIncrementalDedupeSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testGetSpec() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSecondSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCustomDbtTransformationsFailure() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testIncrementalSync() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCheckConnection() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncWriteSameTableNameDifferentNamespace() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testLineBreakCharacters() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testSyncUsesAirbyteStreamNamespaceIfNotNull() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)

S3CsvDestinationAcceptanceTest > testCustomDbtTransformations() FAILED
    java.lang.NullPointerException
        at io.airbyte.integrations.destination.s3.S3FormatConfigs.getS3FormatConfig(S3FormatConfigs.java:36)
        at io.airbyte.integrations.destination.s3.S3DestinationConfig.getS3DestinationConfig(S3DestinationConfig.java:63)
        at io.airbyte.integrations.destination.s3.S3CsvDestinationAcceptanceTest.setup(S3CsvDestinationAcceptanceTest.java:213)
u

user

06/10/2021, 7:59 AM
for your information, this test is done after modification on S3-Destination connector complete. Which is now Support both AWS S3 and Alternative S3 like Minio S3
d

Davin Chia (Airbyte)

06/10/2021, 8:00 AM
oh nice!
u

user

06/10/2021, 8:00 AM
does that fail because of new feature is not write in acceptance test file?
u

user

06/10/2021, 8:00 AM
I don't think so. it looks like it's because the format field in the config isn't present
u

user

06/10/2021, 8:01 AM
u

user

06/10/2021, 8:02 AM
Copy code
{
  "s3_endpoint": "aws",
  "s3_bucket_name": "sokhen...",
  "s3_bucket_path": "...",
  "s3_bucket_region": "ap-southeast-1",
  "access_key_id": "AKIA4...",
  "secret_access_key": "DJv0cvPls.....",
  "format": "CSV"
}
u

user

06/10/2021, 8:02 AM
still fail, try multiple time already
d

Davin Chia (Airbyte)

06/10/2021, 8:03 AM
what if i do a merge request then team can give feedback ?
u

user

06/10/2021, 8:20 AM
let me see
d

Davin Chia (Airbyte)

06/10/2021, 8:20 AM
yes! you can start a PR and we'll be happy to give ya feedback
u

user

06/10/2021, 8:23 AM
u

user

06/10/2021, 8:25 AM
ah it's because you need something like
Copy code
{
  "s3_bucket_name": 
  "s3_bucket_path": 
  "s3_bucket_region": 
  "access_key_id": 
  "secret_access_key": 
  "format": {
    "format_type": "CSV",
    "flattening": "Root level flattening"
  }
}
2 Views
Powered by