Hi 👋, I'm having some issues with the Netsuite connector. I can set up the source ok, but when I set up a connection it runs but with error messages and then creates empty tables. Has anyone else seen this or have ideas on how to resolve? thanks!

DATE FORMAT exception. Cannot read using known formats ['%m/%d/%Y', '%Y-%m-%d']
Search error occurred: Parse of date/time "2022-12-31" failed with date format "dd/MM/yy" in time zone America/Los_Angeles

Thread sent successfully to Harvestr

just seen this: https://github.com/airbytehq/airbyte/pull/19798 do I need to do an Airbyte upgrade? I'm on 0.40.12

Yes! Please upgrade to the latest version!

Hi Chloe, can you please let us know if upgrading solve the problem?

Hi both, I've upgraded to 0.40.27, and Netsuite source version 0.1.3 and am still experiencing the same issue

Thanks for your help, Chloe. @Oleksandr Bazarnov [GL] can you take a look at this and see if you have any suggestions, please?

I’ll check it @Yowan Ramchoreeter (Airbyte) @Chloe Connor @Nataly Merezhuk (Airbyte)

Hi @Oleksandr Bazarnov [GL], did you get a chance to take a look at this? any ideas appreciated!

I did. Do we have the full logs from the failed sync?

airbyte_netsuite_salesorder_error_logs

Have you selected the 1 stream for the sync,

salesorder

? It's odd and we should investigate this. @Chloe Connor do we have the opportunity to do the replication locally using your credentials in order to catch the bug, please?

correct, I've selected one stream otherwise I get schema validation failed. unfortunately I won't be able to share the netsuite credentials. Is there anything I could try and share the result with you?

Please provide the validation error you've faced with? Also, you've got the error with validation during the sync or selecting the streams?

If I don't select specific streams I get a schema error (see below), when I select just salesorder I get the error I shared in the logs above (note this happens with all other streams I've tried too) .

It seems like the permissions / configuration issues on Netsuite side, have you followed the updated guide of how to set up the Netsuite for the Airbyte Integration? Official Airbyte connector's guide

This is the doc that @Oleksandr Bazarnov [GL] was referring to: https://docs.airbyte.com/integrations/sources/netsuite/

thanks, I'll double check this. If I only want salesorder, and I can see for salesorder I have full permissions, should this stream work?

It should, but please check the other guide parts to be sure that everything is set correctly, if - yes, it should work

Ive run through the setup guide again and still get the date error. I updated the start date to earlier and now get a failed sync with full stack trace, but it still points to a date issue:

023-01-24 17:35:02 [44msource[0m > Syncing stream: salesorder 
2023-01-24 17:35:07 [44msource[0m > Stream `salesorder`: cannot read using date format `%m/%d/%Y
2023-01-24 17:35:07 [44msource[0m > Stream `salesorder`: retry using next date format `%Y-%m-%d
2023-01-24 17:35:08 [44msource[0m > Stream `salesorder`: cannot read using date format `%Y-%m-%d
2023-01-24 17:35:08 [44msource[0m > DATE FORMAT exception. Cannot read using known formats ['%m/%d/%Y', '%Y-%m-%d']
2023-01-24 17:35:08 [44msource[0m > Stream `salesorder`: INVALID_PARAMETER error occured, full error message: Invalid search query. Detailed unprocessed description follows. Search error occurred: Parse of date/time "2022-07-31" failed with date format "dd/MM/yy" in time zone America/Los_Angeles
Caused by:
	java.text.ParseException: Unparseable date: "2022-07-31".
2023-01-24 17:35:08 [44msource[0m > Encountered an exception while reading stream salesorder
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 287, in _read_full_refresh
    for _slice in slices:
  File "/airbyte/integration_code/source_netsuite/streams.py", line 280, in stream_slices
    slice_start = start.strftime(self.default_datetime_format)
  File "/airbyte/integration_code/source_netsuite/streams.py", line 54, in default_datetime_format
    return NETSUITE_INPUT_DATE_FORMATS[self.index_datetime_format]
IndexError: list index out of range
2023-01-24 17:35:08 [44msource[0m > Finished syncing salesorder
2023-01-24 17:35:08 [44msource[0m > SourceNetsuite runtimes:
Syncing stream salesorder 0:00:05.709623
2023-01-24 17:35:08 [44msource[0m > list index out of range
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 122, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 129, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 287, in _read_full_refresh
    for _slice in slices:
  File "/airbyte/integration_code/source_netsuite/streams.py", line 280, in stream_slices
    slice_start = start.strftime(self.default_datetime_format)
  File "/airbyte/integration_code/source_netsuite/streams.py", line 54, in default_datetime_format
    return NETSUITE_INPUT_DATE_FORMATS[self.index_datetime_format]
IndexError: list index out of range
2023-01-24 17:35:08 [32mINFO[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):381 - Source has no more messages, closing connection.

@Chloe Connor could you please check the

salesorder

object in Netsuite to understand what is the format of

cursor_field

, in our case it could be either

lastModifiedDate

or

lastModified

but it could be something else, we need to understand this moment. Simply share the value example of what is stored in one of these fields. Thank you!

FYI that this seems like a related issue and someone seems to have a fix: https://github.com/airbytehq/airbyte/issues/21871. Maybe this will help us move forward here?

@Oleksandr Bazarnov [GL] I don't know how to check this in the Netsuite application, but the error message suggests it's of format

%Y-%m-%d

? setting to incremental mode suggests this is

lastModifiedDate

. Is there another stream I could try, would that help?

If you are willing to create the submit the PR - this will be awesome!

I've had a go at submitting this PR, would you mind taking a look please? https://github.com/airbytehq/airbyte/pull/21971 I still need to figure out why the checks are failing but the code can be reviewed

Does this fix allow you to read the data from salesorder object? Do you receive some records?

I could see I was getting records with one of the date formats but my initial change still didn't loop through the formats. So I think the retries is the wrong place to implement this and instead I've updated the PR: https://github.com/airbytehq/airbyte/pull/21971 and with this change I get data!! 🎉 could you please review when you get a chance? thanks!

Will review, thanks @Chloe Connor