Hello, I have encountered problems while using sou...
# help-connector-developmentj
Jakin L
05/10/2023, 9:58 AMHello, I have encountered problems while using source-s3. I hope to get some help with this
k
kapa.ai
05/10/2023, 9:58 AMA message from kapa.ai
kapa.ai
05/10/2023, 9:59 AMI'm here to help! To better assist you, please provide more information about the issue you're facing with the source-s3 connector. In the meantime, you can refer to the following resources for guidance: 1. [Amazon S3 source connection setup guide](https://docs.airbyte.com/integrations/sources/s3) 2. [Path Patterns in S3 source](https://docs.airbyte.com/integrations/sources/s3#path-patterns) 3. [Troubleshooting guide](https://docs.airbyte.com/troubleshooting) If you still need help, please provide more details about your issue, and I'll do my best to assist you.
j
Jakin L
05/10/2023, 10:13 AMThe folder I specified on S3 has about 100,000 files, with each file being approximately 2KB in size. When I configure the source and call the check_connection function, the program takes a very long time to execute. After about 15 minutes, the program throws an error and it always shows the following message.
Copy code
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - --- Logging error ---
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - Traceback (most recent call last):
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - msg = self.format(record)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - return fmt.format(record)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/logger.py", line 62, in format
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - message = super().format(record)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/logging/__init__.py", line 663, in format
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - record.message = record.getMessage()
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/logging/__init__.py", line 367, in getMessage
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - msg = msg % self.args
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - TypeError: not all arguments converted during string formatting
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - Call stack:
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/main.py", line 13, in <module>
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - launch(source, sys.argv[1:])
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 156, in launch
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - for message in source_entrypoint.run(parsed_args):
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 91, in run
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.check(source_spec, config))
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 112, in check
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - check_result = self.source.check(self.logger, config)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 84, in check
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - check_succeeded, error = self.check_connection(logger, config)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/source_files_abstract/source.py", line 69, in check_connection
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - slice_ = next(stream.stream_slices(sync_mode=SyncMode.full_refresh), None)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 430, in stream_slices
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - yield from super().stream_slices(sync_mode=sync_mode, cursor_field=cursor_field, stream_state=stream_state)
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 230, in stream_slices
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - for file_info in self.get_time_ordered_file_infos():
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 174, in get_time_ordered_file_infos
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - return sorted(
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 157, in pattern_matched_filepath_iterator
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - for file_info in file_infos:
2023-05-10 09:21:35 ERROR i.a.c.i.LineGobbler(voidCall):114 - File "/airbyte/integration_code/source_s3/stream.py", line 72, in filepath_iterator2 Views