*Is this your first time deploying Airbyte*: No
*O...
# troubleshootingd
Divya (Proximity)
02/16/2022, 9:17 AMIs this your first time deploying Airbyte: No
OS Version / Instance: Ubuntu 20.04.3 LTS
Memory / Disk: 16Gb
Deployment: Docker
Airbyte Version: 0.35.10-alpha
Source name/version: Custom connector
Destination name/version: CSV
Description: After upgrading Airbyte from 0.19.0-alpha to 0.35.10-alpha version i can't add custom connector to platform. All test from that connector successfully passing, i can add it to Airbyte platform, but when i add it to yml file for platform - i have trouble like
Internal Server Error: Get Spec job failed.2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeywordu
user
02/16/2022, 11:21 AMHey can you try this command
docker run --rm <image>:<version> specu
user
02/16/2022, 11:29 AM@Harshith (Airbyte) do you mean version of airbyte or airbyte-scheduler?
u
user
02/16/2022, 12:05 PMIt's basically the connector which you are trying to add
u
user
02/16/2022, 12:14 PM@Harshith (Airbyte) this container is inside yml file, i up all via docker-compose
k
Krzysztof Karski
02/17/2022, 2:52 AM@Roxy Trofymchuk please share the logs from the server, there we should be able to see the error with more details. Or please open the network tab with developer mode in the webbroswer and track the request when creating the new connector
m
Maish Maseeh
02/17/2022, 8:53 AM@Harshith (Airbyte) @Marcos Marx (Airbyte) thats all that i have in logs
2022-02-17 08:21:35 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'2022-02-17 08:21:35 ERROR i.a.s.ServerApp(main):277 - Server failedjava.lang.IllegalArgumentException: Env variable not defined: WORKSPACE_ROOTat io.airbyte.config.EnvConfigs.getPath(EnvConfigs.java:615) ~[io.airbyte.airbyte-config-models-0.35.10-alpha.jar:?]at io.airbyte.config.EnvConfigs.getWorkspaceRoot(EnvConfigs.java:239) ~[io.airbyte.airbyte-config-models-0.35.10-alpha.jar:?]at io.airbyte.server.ServerApp.getServer(ServerApp.java:152) ~[io.airbyte-airbyte-server-0.35.10-alpha.jar:?]at io.airbyte.server.ServerApp.main(ServerApp.java:275) [io.airbyte-airbyte-server-0.35.10-alpha.jar:?]j
Jorin
02/17/2022, 10:15 AMHey looks to me like the env WORKSPACE_ROOT is causing the issue can you check the value of this in .env
Jorin
02/17/2022, 10:36 AM# Workspace storage for running jobs (logs, etc)WORKSPACE_ROOT=/tmp/workspaceWORKSPACE_DOCKER_MOUNT=airbyte_workspaceJorin
02/17/2022, 10:37 AM@Harshith (Airbyte) you meant about that?
p
Pras
02/17/2022, 12:31 PMYeah can you share the complete log of server/scheduler here?
Pras
02/17/2022, 12:48 PM@Harshith (Airbyte) its complete log from airbyte-scheduler:
2022-02-16 08:52:14 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'2022-02-16 08:52:14 INFO i.a.s.a.SchedulerApp(main):224 - workspaceRoot = /tmp/workspace2022-02-16 08:52:14 INFO i.a.s.a.SchedulerApp(main):227 - temporalHost = airbyte-temporal:72332022-02-16 08:52:16 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:18 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:20 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:22 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:24 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:26 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:28 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:30 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:32 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:34 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:36 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:38 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:40 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...2022-02-16 08:52:42 INFO i.a.s.a.SchedulerApp(main):232 - Creating Job DB connection pool...2022-02-16 08:52:42 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...2022-02-16 08:52:42 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...2022-02-16 08:52:43 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!2022-02-16 08:52:43 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):48 - Database available!2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable CONFIG_DATABASE_USER: 'docker'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:<postgresql://db:5432/airbyte>'2022-02-16 08:52:43 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):65 - Waiting for database to become available...2022-02-16 08:52:43 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):27 - Testing if airbyte_configs has been created and seeded...2022-02-16 08:52:43 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):90 - Database available!2022-02-16 08:52:43 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):48 - Database available!2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable MINIMUM_WORKSPACE_RETENTION_DAYS: '1'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable MAXIMUM_WORKSPACE_RETENTION_DAYS: '60'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable MAXIMUM_WORKSPACE_SIZE_MB: '5000'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'2022-02-16 08:52:43 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'2022-02-16 08:52:43 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}2022-02-16 08:52:43 INFO i.a.w.t.TemporalUtils(waitForTemporalServerAndLog):115 - Waiting for temporal server...2022-02-16 08:52:43 WARN i.a.w.t.TemporalUtils(waitForTemporalServerAndLog):120 - Waiting for default namespace to be initialized in temporal...2022-02-16 08:52:50 INFO i.a.w.t.TemporalUtils(waitForTemporalServerAndLog):134 - Found temporal default namespace!2022-02-16 08:52:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable PUBLISH_METRICS: 'false'2022-02-16 08:52:50 INFO i.a.s.a.SchedulerApp(main):275 - Launching scheduler...2022-02-16 08:52:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'2022-02-16 08:52:50 INFO i.a.s.a.SchedulerApp(start):143 - Start running the old scheduler2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword2022-02-16 08:52:51 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeywordPras
02/17/2022, 12:49 PMWould be great if you can put this In a text file a share 😐
Pras
02/17/2022, 1:06 PM@Harshith (Airbyte) here it is😅
j
Justin Reynolds
02/18/2022, 12:05 AMbtw @Roxy Trofymchuk did you first upgrade to 0.32-path-1 version? https://docs.airbyte.com/operator-guides/upgrading-airbyte#mandatory-intermediate-upgrade
Justin Reynolds
02/18/2022, 7:55 AM@Marcos Marx (Airbyte) no, i just recently saw, that i first needed to upgrade to 0.32-path-1, and then to 35. could that be what causes this behavior?
Justin Reynolds
02/18/2022, 7:58 AMYeah can you try that as it is a mandatory one
c
Collin Lenon
02/21/2022, 6:12 AM@Harshith (Airbyte) @Marcos Marx (Airbyte) our main devops told me, that he tried 32 version and it's also didn't work for us
s
Sania Zafar
02/21/2022, 6:37 AMSo you have upgraded to 0.32 and then tried the same?
Sania Zafar
02/21/2022, 6:38 AM@Harshith (Airbyte) i didn't, but our devops have tried it. and we also have same issue
v
Vinod Varma
02/21/2022, 8:54 PM@Roxy Trofymchuk would you test running the spec function locally for your custom connector with latest version of Airbyte core code?
c
Carlos Alberto Cabrera Quiroga
02/22/2022, 6:17 AM@Marcos Marx (Airbyte) i just tried and it's also has same "spec job failed" error. i've tried it with latest version(0.35.31-alpha)
u
user
02/22/2022, 8:36 AM@Harshith (Airbyte) @Marcos Marx (Airbyte) So maybe something else what i can try also?
h
Harshith (Airbyte)
02/22/2022, 9:28 AM
Hey may I know the source and version you are trying ?
u
user
02/22/2022, 9:29 AM@Harshith (Airbyte) its a custom source
h
Harshith (Airbyte)
02/22/2022, 9:45 AM
Do you mind doing the same I have suggested before
docker run --rm <image>:<version> specn
Nathan Gold
02/22/2022, 10:00 AM@Harshith (Airbyte)
2022-02-22 09:59:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'Exception in thread "main" java.lang.IllegalArgumentException: Env variable not defined: WORKSPACE_ROOTat io.airbyte.config.EnvConfigs.getPath(EnvConfigs.java:782)at io.airbyte.config.EnvConfigs.getWorkspaceRoot(EnvConfigs.java:269)at io.airbyte.scheduler.app.SchedulerApp.main(SchedulerApp.java:224)d
Disi Koa
02/22/2022, 9:22 PMCan you make sure you’d copy the .env file from master?
o
Ofek Katriel
02/23/2022, 9:29 AM@Marcos Marx (Airbyte) i copied all from .env file to our, and it's still has this issue
j
Jason Ofua
02/23/2022, 7:49 PM@Roxy Trofymchuk confirming you see this env variable in your .env file, this is correct?
a
Alexander D
02/23/2022, 8:07 PM@Marcos Marx (Airbyte) yeah, i see it
Alexander D
02/23/2022, 8:15 PMYour deployment is in a EC2 instance?
Alexander D
02/23/2022, 8:25 PM@Marcos Marx (Airbyte) no, on docker
Alexander D
02/23/2022, 8:27 PMlocally?
Alexander D
02/23/2022, 9:01 PM@Marcos Marx (Airbyte) yeah
2 Views