Hi how can i run an airbyte connector locally without docker Airbyte #replication-troubleshooting

Join Slack

Hi, how can i run an airbyte connector locally wit...

# replication-troubleshooting

Lukasz Sowa

12/13/2022, 11:03 AM

Hi, how can i run an airbyte connector locally without docker?

Eli Sigal

12/13/2022, 11:41 AM

Hi. this can be done via regular python OOP though I strongly do not recommend this since docker is best practice these days and makes it much faster and easier to manage

Lukasz Sowa

12/13/2022, 11:47 AM

My connector is written in typescript, i just want to be able to locally debug the connector

Eli Sigal

12/13/2022, 11:48 AM

you can debug it using the airbyte_cdk while running the check, discover and read commands

Lukasz Sowa

12/13/2022, 11:49 AM

I'm running the connector with the read command but it doesnt return any records (when in airbyte it works fine)

Lukasz Sowa

12/13/2022, 11:50 AM

I suppose the "correct" way to attach a debugger is somehow doing it with docker

Eli Sigal

12/13/2022, 11:50 AM

correct

Lukasz Sowa

12/13/2022, 11:51 AM

It seems a bit complicated though, I just want to fetch data without dealing with airbyte for now

Eli Sigal

12/13/2022, 11:51 AM

if you dont have any records read when running the read command then you probably have in the parse_response function some thing like :

yield from []

Eli Sigal

12/13/2022, 11:53 AM

or in TS some thing like:

Copy code

if array is string
return [] as string[]

Lukasz Sowa

12/13/2022, 11:54 AM

Where is this function supposed to be located? My entry point looks like this:

Lukasz Sowa

12/13/2022, 11:55 AM

I assume the issue may be located in the library code

Eli Sigal

12/13/2022, 11:55 AM

it is supposed to be located "usually" in the main parent object of your connector

Lukasz Sowa

12/13/2022, 12:01 PM

I dont have anything like this, stream reading methods look like this

Eli Sigal

12/13/2022, 12:02 PM

got to be honest didnt build a connector with TS yet. can you show me the logs you get when you run read command?

Lukasz Sowa

12/13/2022, 12:03 PM

Copy code

{"log":{"level":"INFO","message":"Config: {\"access_token\":\"REDACTED\",\"organization\":\"AllegroTechies\",\"project\":\"VAbank\",\"api_version\":\"6.0\",\"graph_version\":\"4.1-preview.1\"}"},"type":"LOG"}
{"log":{"level":"INFO","message":"Catalog: {\"streams\":[{\"stream\":{\"name\":\"repositories\",\"jsonSchema\":{\"type\":\"object\",\"$schema\":\"<http://json-schema.org/draft-04/schema#>\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"size\":{\"type\":\"number\"},\"tags\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"properties\":{\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"commit\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"message\":{\"type\":\"string\"},\"objectId\":{\"type\":\"string\"},\"taggedBy\":{\"type\":\"object\",\"properties\":{\"date\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"}}},\"taggedObject\":{\"type\":\"object\",\"properties\":{\"objectId\":{\"type\":\"string\"},\"objectType\":{\"type\":\"string\"}}}}},\"creator\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"_links\":{\"type\":\"object\",\"properties\":{\"avatar\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"imageUrl\":{\"type\":\"string\"},\"descriptor\":{\"type\":\"string\"},\"uniqueName\":{\"type\":\"string\"},\"displayName\":{\"type\":\"string\"}}},\"objectId\":{\"type\":\"string\"}}}}},\"sshUrl\":{\"type\":\"string\"},\"webUrl\":{\"type\":\"string\"},\"project\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"state\":{\"type\":\"string\"},\"revision\":{\"type\":\"number\"},\"visibility\":{\"type\":\"string\"},\"description\":{\"type\":\"string\"},\"lastUpdateTime\":{\"type\":\"string\"}}},\"branches\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"commits\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"author\":{\"type\":\"object\",\"properties\":{\"date\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"}}},\"comment\":{\"type\":\"string\"},\"commitId\":{\"type\":\"string\"},\"committer\":{\"type\":\"object\",\"properties\":{\"date\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"}}},\"remoteUrl\":{\"type\":\"string\"},\"changeCounts\":{\"type\":\"object\",\"properties\":{\"Add\":{\"type\":\"number\"},\"Edit\":{\"type\":\"number\"},\"Delete\":{\"type\":\"number\"}}}}}},\"aheadCount\":{\"type\":\"number\"},\"behindCount\":{\"type\":\"number\"},\"isBaseVersion\":{\"type\":\"boolean\"}}}},\"remoteUrl\":{\"type\":\"string\"},\"isDisabled\":{\"type\":\"boolean\"},\"defaultBranch\":{\"type\":\"string\"}}},\"supportedSyncModes\":[\"full_refresh\"],\"defaultCursorField\":[],\"sourceDefinedPrimaryKey\":[[\"id\"]]},\"config\":{\"syncMode\":\"full_refresh\",\"cursorField\":[],\"destinationSyncMode\":\"overwrite\",\"primaryKey\":[[\"id\"]],\"aliasName\":\"repositories\",\"selected\":true}},{\"stream\":{\"name\":\"pull_requests\",\"jsonSchema\":{\"type\":\"object\",\"$schema\":\"<http://json-schema.org/draft-04/schema#>\",\"properties\":{\"url\":{\"type\":\"string\"},\"title\":{\"type\":\"string\"},\"status\":{\"type\":\"string\"},\"commits\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"author\":{\"type\":\"object\",\"properties\":{\"date\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"}}},\"comment\":{\"type\":\"string\"},\"commitId\":{\"type\":\"string\"},\"committer\":{\"type\":\"object\",\"properties\":{\"date\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"email\":{\"type\":\"string\"}}},\"changeCounts\":{\"type\":\"object\",\"properties\":{\"Add\":{\"type\":[\"number\",\"null\"]},\"Edit\":{\"type\":[\"number\",\"null\"]},\"Delete\":{\"type\":[\"number\",\"null\"]}}}}}},\"isDraft\":{\"type\":\"boolean\"},\"mergeId\":{\"type\":\"string\"},\"threads\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"number\"},\"_links\":{\"type\":\"object\",\"properties\":{\"self\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"repository\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"status\":{\"type\":\"string\"},\"comments\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"number\"},\"_links\":{\"type\":\"object\",\"properties\":{\"self\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"threads\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"repository\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"pullRequests\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"author\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"_links\":{\"type\":\"object\",\"properties\":{\"avatar\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"imageUrl\":{\"type\":\"string\"},\"descriptor\":{\"type\":\"string\"},\"uniqueName\":{\"type\":\"string\"},\"displayName\":{\"type\":\"string\"}}},\"content\":{\"type\":\"string\"},\"usersLiked\":{\"type\":\"array\",\"items\":{}},\"commentType\":{\"type\":\"string\"},\"publishedDate\":{\"type\":\"string\"},\"lastUpdatedDate\":{\"type\":\"string\"},\"parentCommentId\":{\"type\":\"number\"},\"lastContentUpdatedDate\":{\"type\":\"string\"}}}},\"isDeleted\":{\"type\":\"boolean\"},\"publishedDate\":{\"type\":\"string\"},\"lastUpdatedDate\":{\"type\":\"string\"}}}},\"createdBy\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"_links\":{\"type\":\"object\",\"properties\":{\"avatar\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"imageUrl\":{\"type\":\"string\"},\"descriptor\":{\"type\":\"string\"},\"uniqueName\":{\"type\":\"string\"},\"displayName\":{\"type\":\"string\"}}},\"reviewers\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"vote\":{\"type\":\"number\"},\"isFlagged\":{\"type\":\"boolean\"},\"uniqueName\":{\"type\":\"string\"},\"displayName\":{\"type\":\"string\"},\"hasDeclined\":{\"type\":\"boolean\"},\"reviewerUrl\":{\"type\":\"string\"},\"supportsIterations\":{\"type\":\"boolean\"}}}},\"closedDate\":{\"type\":[\"string\",\"null\"]},\"repository\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"url\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"project\":{\"type\":\"object\",\"properties\":{\"id\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"state\":{\"type\":\"string\"},\"visibility\":{\"type\":\"string\"},\"lastUpdateTime\":{\"type\":\"string\"}}}}},\"description\":{\"type\":\"string\"},\"mergeStatus\":{\"type\":\"string\"},\"codeReviewId\":{\"type\":\"number\"},\"creationDate\":{\"type\":\"string\"},\"pullRequestId\":{\"type\":\"number\"},\"sourceRefName\":{\"type\":\"string\"},\"targetRefName\":{\"type\":\"string\"},\"lastMergeCommit\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"commitId\":{\"type\":\"string\"}}},\"supportsIterations\":{\"type\":\"boolean\"},\"lastMergeSourceCommit\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"commitId\":{\"type\":\"string\"}}},\"lastMergeTargetCommit\":{\"type\":\"object\",\"properties\":{\"url\":{\"type\":\"string\"},\"commitId\":{\"type\":\"string\"}}}}},\"supportedSyncModes\":[\"full_refresh\",\"incremental\"],\"sourceDefinedCursor\":true,\"defaultCursorField\":[\"creationDate\"],\"sourceDefinedPrimaryKey\":[[\"pullRequestId\"]]},\"config\":{\"syncMode\":\"incremental\",\"cursorField\":[\"creationDate\"],\"destinationSyncMode\":\"append_dedup\",\"primaryKey\":[[\"pullRequestId\"]],\"aliasName\":\"pull_requests\",\"selected\":true}},{\"stream\":{\"name\":\"users\",\"jsonSchema\":{\"type\":\"object\",\"$schema\":\"<http://json-schema.org/draft-04/schema#>\",\"properties\":{\"url\":{\"type\":\"string\"},\"_links\":{\"type\":\"object\",\"properties\":{\"self\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"avatar\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"storageKey\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"memberships\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}},\"membershipState\":{\"type\":\"object\",\"properties\":{\"href\":{\"type\":\"string\"}}}}},\"domain\":{\"type\":\"string\"},\"origin\":{\"type\":\"string\"},\"originId\":{\"type\":[\"string\",\"null\"]},\"descriptor\":{\"type\":\"string\"},\"displayName\":{\"type\":\"string\"},\"mailAddress\":{\"type\":\"string\"},\"subjectKind\":{\"type\":\"string\"},\"principalName\":{\"type\":\"string\"}}},\"supportedSyncModes\":[\"full_refresh\"],\"defaultCursorField\":[],\"sourceDefinedPrimaryKey\":[[\"principalName\"]]},\"config\":{\"syncMode\":\"full_refresh\",\"cursorField\":[],\"destinationSyncMode\":\"overwrite\",\"primaryKey\":[[\"principalName\"]],\"aliasName\":\"users\",\"selected\":true}}]}"},"type":"LOG"}
{"log":{"level":"INFO","message":"Syncing AzureRepoSource"},"type":"LOG"}
{"log":{"level":"INFO","message":"Syncing repositories stream in full mode"},"type":"LOG"}
{"log":{"level":"INFO","message":"Started processing repositories stream slice undefined"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished processing repositories stream slice undefined. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished syncing repositories stream. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Syncing pull_requests stream in full mode"},"type":"LOG"}
{"log":{"level":"INFO","message":"Started processing pull_requests stream slice undefined"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished processing pull_requests stream slice undefined. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished syncing pull_requests stream. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Syncing users stream in full mode"},"type":"LOG"}
{"log":{"level":"INFO","message":"Started processing users stream slice undefined"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished processing users stream slice undefined. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished syncing users stream. Read 0 records"},"type":"LOG"}
{"log":{"level":"INFO","message":"Finished syncing AzureRepoSource"},"type":"LOG"}

Lukasz Sowa

12/13/2022, 12:03 PM

I should have included a file 😆

Eli Sigal

12/13/2022, 12:04 PM

is the log what you get directly from airbyte UI or local?

Lukasz Sowa

12/13/2022, 12:04 PM

local

Lukasz Sowa

12/13/2022, 12:04 PM

in airbyte UI it just works

Eli Sigal

12/13/2022, 12:05 PM

did you create the configured_catalog.json file ?

Lukasz Sowa

12/13/2022, 12:05 PM

I got it from the airbyte ui

Lukasz Sowa

12/13/2022, 12:05 PM

basically just copied it from the UI xhr request

Lukasz Sowa

12/13/2022, 12:06 PM

Is there a better way to generate a default catalog.json?

Eli Sigal

12/13/2022, 12:07 PM

got it - this does sound like you are missing some thing locally though I am not sure as to what lets wait for some one from airbyte team or some one who created with TS before.

👍 1

Eli Sigal

12/13/2022, 12:07 PM

you can create it by going to an online tool json to jsonschema

Lukasz Sowa

12/13/2022, 12:09 PM

The connector also comes with a catalog json used for testing but its basically the same

Lukasz Sowa

12/13/2022, 12:09 PM

and gives the same results

Eli Sigal

12/13/2022, 12:09 PM

try taking the json response and creating it as a schema

4 Views

Open in Slack

Previous Next