Hi Team,
I am in process of building singer based ...
# ask-community-for-troubleshootingr
Rahul Patel
05/25/2022, 3:16 PMHi Team,
I am in process of building singer based connector.
I am trying to run below command
./gradlew airbyte integrationsconnectorssource oracle py singerbuild
and getting the below error.
Plugin [id: 'com.gradle.enterprise', version: '3.4.1'] was not found in any of the following sources:
- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.gradle.enterprisecom.gradle.enterprise.gradle.plugin3.4.1')
Searched in the following repositories:
Gradle Central Plugin Repository
m
Marcos Marx (Airbyte)
05/25/2022, 3:45 PMWhat OS are you? any chances to move the singer code to Airbyte CDK? 😛
r
Rahul Patel
05/25/2022, 4:38 PMUsing Ubuntu on wsl, I am trying to build oracle db based connector based on pipelinewise-tap-oracle.
m
Marcos Marx (Airbyte)
05/25/2022, 5:55 PMThe Source Oracle of Airbyte doesn’t attempt what you need?
r
Rahul Patel
05/25/2022, 6:27 PMit does work, but performance is not good. 20 million records table took 15 hr. table size has 300 million records. Hoping singer based connector might give good performance. Apart from this, wanted to add start_date feature (means instead of full load, will load via state file) and batch size. Airbyte's oracle connector does not have any feature for modify read batch_size, default batch size 1000.
m
Marcos Marx (Airbyte)
05/25/2022, 6:46 PM@Rahul Patel latest version of connector should use dynamic batch fetching
Marcos Marx (Airbyte)
05/25/2022, 6:47 PMabout the problem: running the python commands did it work?
r
Rahul Patel
05/25/2022, 6:49 PMyes, python cmd works fine, read, discover tables works fine. Dynamic batch fetching is calculating max 35k records in a batch based on row size which still gives low performance for same table.
Rahul Patel
05/26/2022, 12:53 PMI did resolve above by adding cacert of site in java11 cacerts. but same error is starting to throw all dependencies.
Rahul Patel
05/26/2022, 12:54 PMI had to upgrade java 8 to 11 (since add-exports parameter is not supported) then added cacerts
m
Marcos Marx (Airbyte)
05/26/2022, 5:11 PMAirbybte uses Java 17
r
Rahul Patel
05/26/2022, 5:17 PMdoes it means that for successful gradle build, I need java 17 ? @Marcos Marx (Airbyte)
m
Marcos Marx (Airbyte)
05/26/2022, 5:29 PMProbably helps use the right version of the dev stack 😅
😅 1