https://linen.dev logo
Join Slack
Powered by
Hi! I’m a developer working on adding a new Python...
# connector-development
k
Hi! I’m a developer working on adding a new Python HTTP API source connector following the guide. I’ve got things working up to this step of adding the connector to UI and somehow can’t add the source connector locally following the instruction. From the docker server log, it seems like it cannot get spec. Can someone help?
p
Copy code
java.lang.IllegalStateException: Get Spec job failed.

at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.0.1-jre.jar:?]

at io.airbyte.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at io.airbyte.server.handlers.SourceDefinitionsHandler.getSpecForImage(SourceDefinitionsHandler.java:174) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at io.airbyte.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:137) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at io.airbyte.server.apis.ConfigurationApi.lambda$updateSourceDefinition$9(ConfigurationApi.java:306) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:733) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at io.airbyte.server.apis.ConfigurationApi.updateSourceDefinition(ConfigurationApi.java:306) ~[io.airbyte-airbyte-server-0.35.11-alpha.jar:?]

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2
server error log, and the docker image was built locally with command 
docker build . -t airbyte/source-[example]:dev
d
Hi @Phoebe Yang, what's the output of 
docker run airbyte/source-[example]:dev spec
Copy code
{
  "type": "SPEC",
  "spec": {
    "documentationUrl": "<https://test.com>",
    "connectionSpecification": {
      "$schema": "<http://json-schema.org/draft-07/schema#>",
      "title": "Test Source Spec",
      "type": "object",
      "required": [
        "client_id",
        "client_secret",
        "access_token"
      ],
      "additionalProperties": false,
      "properties": {
        "client_id": {
          "type": "string",
          "title": "Client Id",
          "description": "Client Id for Basic Auth Header when sending requests.",
          "airbyte_secret": true,
          "order": 1
        },
        "client_secret": {
          "type": "string",
          "title": "Client Secret",
          "description": "Client Secret for Basic Auth Header when sending requests.",
          "airbyte_secret": true,
          "order": 2
        },
        "access_token": {
          "type": "string",
          "title": "Access Token",
          "description": "The access token associated with the data being requested for.",
          "airbyte_secret": true,
          "order": 3
        },
        "start_date": {
          "type": "string",
          "title": "Start Date",
          "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z$",
          "description": "UTC date and time in the format 2017-01-25T00:00:00.000Z. Any data before this date will not be replicated.",
          "examples": [
            "2017-01-25T00:00:00Z"
          ],
          "order": 4
        }
      }
    }
  }
}
I also saw this in the worker error log
Copy code
airbyte-worker      | Caused by: io.temporal.failure.ApplicationFailure: message='Cannot run program "/tmp/scripts17475122944268089990/image_exists.sh": error=0, Failed to exec spawn helper: pid: 130, exit value: 1', type='java.io.IOException', nonRetryable=false
airbyte-worker      | 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143) ~[?:?]
airbyte-worker      | 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
airbyte-worker      | 	at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:173) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:92) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:44) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:48) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:23) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:174) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
airbyte-worker      | Caused by: io.temporal.failure.ApplicationFailure: message='error=0, Failed to exec spawn helper: pid: 130, exit value: 1', type='java.io.IOException', nonRetryable=false
airbyte-worker      | 	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method:0) ~[?:?]
airbyte-worker      | 	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314) ~[?:?]
airbyte-worker      | 	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[?:?]
airbyte-worker      | 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
airbyte-worker      | 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
airbyte-worker      | 	at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:173) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:92) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:44) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:48) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
airbyte-worker      | 	at io.airbyte.workers.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:23) ~[io.airbyte-airbyte-workers-0.35.11-alpha.jar:?]
seems like an error with worker finding the docker image?
I guess you are on Apple M1 computer 😄 ?
If you are indeed working on M1, we usually solve this by adding the following environment variable to the airbyte worker in the docker-compose file: 
JAVA_OPTS: "-Djdk.lang.Process.launchMechanism=vfork"
yes its M1
ahh i see i’ll give it a try!
this is the docker-compose under the root directory?
You'll find updates about our current M1 support here: https://github.com/airbytehq/airbyte/issues/2017
thanks I got it working now 🙂
a follow up question, if I deploy airbyte on ec2 can I run my test connector just by pulling the docker image from a public docker repository?
Open in Slack
PreviousNext
Powered by