Does anybody knows here how can we establish sourc...
# replication-troubleshooting
k
Does anybody knows here how can we establish source connection between Google Cloud Storage via Airbyte? I tried using File connection, but not able to make connections
Logs: Failed to load gs// FileNotFoundError('') Traceback (most recent call last): File "/airbyte/integration_code/source_file/client.py", line 89, in open self._file = self._open() File "/airbyte/integration_code/source_file/client.py", line 99, in _open return self._open_gcs_url() File "/airbyte/integration_code/source_file/client.py", line 191, in _open_gcs_url file_to_close = smart_open.open(self.full_url, transport_params={"client": client}, **self.args) File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 224, in open binary = _open_binary_stream(uri, binary_mode, transport_params) File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 400, in _open_binary_stream fobj = submodule.open_uri(uri, mode, transport_params) File "/usr/local/lib/python3.9/site-packages/smart_open/gcs.py", line 108, in open_uri return open(parsed_uri['bucket_id'], parsed_uri['blob_id'], mode, **kwargs) File "/usr/local/lib/python3.9/site-packages/smart_open/gcs.py", line 141, in open fileobj = Reader( File "/usr/local/lib/python3.9/site-packages/smart_open/gcs.py", line 227, in init raise google.cloud.exceptions.NotFound('blob %s not found in %s' % (key, bucket)) google.api_core.exceptions.NotFound: 404 blob csvfiles_incoming/WDICountry-Series.csv not found in storage.cloud.google.com The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/airbyte/integration_code/source_file/source.py", line 106, in check with client.reader.open(): File "/airbyte/integration_code/source_file/client.py", line 91, in open raise FileNotFoundError(self.url) from err FileNotFoundError:
t
Hi When you mean GCS are you talking about bigquery?
k
Google Cloud Storage (File System) Standard Storage (Not Bigquery)
n
Hey Karan, could you please clarify the connectors you are using? Are you using GCS as a source or destination?
k
As a source. Since there's no source for GCS, I'm using file as GCS source.
m
Best way is to use the S3 connector, as GCS is compatible. Just change the endpoint to point to GCS
k
@Marcelo Pio de Castro Can you send me steps for same? Any blog or article would suffice.
m
You just need to setup the HMAC Key, and add the endpoint https://storage.googleapis.com