ahsen m
03/22/2022, 5:04 PMahsen m
03/22/2022, 5:08 PMCaused by: java.io.FileNotFoundException: /root/.kube/config (No such file or directory)
Oliver Meyer
03/22/2022, 8:37 PMJorge Torres
03/22/2022, 8:49 PMAlpana Shukla
03/23/2022, 9:10 AMERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {"type":"RECORD","record":{"stream":"sensorhistory","data":
logs and json schema attached for reference. Any idea what's wrong? Thanks :)Rishabh Srivastava
03/23/2022, 9:31 AMJaime Farres
03/23/2022, 11:28 AMAlejandro Rosano
03/23/2022, 2:38 PMJyothi
03/23/2022, 7:27 PMBrian Leonard
03/23/2022, 8:23 PMAlpana Shukla
03/24/2022, 11:49 AMAlpana Shukla
03/24/2022, 11:53 AMChance Barkley
03/24/2022, 9:30 PMIvica Taseski
03/24/2022, 11:33 PMavailable fixtures: api, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, common_params, config, creds_with_wrong_permissions, doctest_namespace, fake_properties_list, monkeypatch, oauth_config, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, some_credentials, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
use 'pytest --fixtures [testpath]' for help on them.Some initial investigation implies that the requests mock decorator is not recognized in python 3 which seem pretty weird since according to the docs the supported Python version for Airbyte is > 3.7 On my local machine python3 --version resolves to Python 3.7.5
Narender Kumar
03/25/2022, 6:18 AMDavid Copeland
03/25/2022, 8:09 AMMijbel Alqattan
03/27/2022, 4:12 PMEnrico Tuvera Jr.
03/28/2022, 4:55 AMMariusz Wojakowski
03/28/2022, 6:42 AMlastmodified
as a query param and returns rows without any information about modification time. My 2 concerns:
• because I don’t have timestamp information in rows I need to take into consideration potential problems with upstream -> overlapping (e.g. grab 2 days instead of 1 during pull) is my only idea how to minimize - is that correct/‘the way’ with that API?
• I want to emit state after all slices are processed - is it idiomatic to yield airbyte state message in AbstractSource.read
?
thanks in advance! have a nice day ☺️Alexandra Goryacheva
03/28/2022, 2:37 PMDavid Hadaller
03/28/2022, 2:52 PMMatthew Brandt
03/28/2022, 4:38 PMNathan Gille
03/28/2022, 5:40 PMget_updated_state
for incremental streams, or should we be using IncrementalMixin
?Ivica Taseski
03/28/2022, 6:41 PMdef read_records(self, sync_mode: SyncMode, cursor_field: List[str] = None, stream_slice: Mapping[str, Any] = None,
stream_state: Mapping[str, Any] = None) -> Iterable[Mapping[str, Any]]:
cursor_name = cursor_field[-1]
stream_state["cursor_key"] = cursor_name
return super().read_records(sync_mode, cursor_field, stream_slice, stream_state = stream_state)
After this I will be able to access the cursor in the parse_response() and get_updated_state() methods in order to do the necessary filtering on the records.
Is there a better approach to achieve this or is this the only way?Ramon Vermeulen
03/29/2022, 9:49 AMEric
03/29/2022, 3:54 PMChristo Olivier
03/29/2022, 5:48 PMMarcos Marx (Airbyte)
03/29/2022, 8:30 PMDipankar Bhattacharyya
03/30/2022, 1:44 AMRamon Vermeulen
03/30/2022, 4:38 PM@property
def supports_incremental(self) -> bool:
return True
@property
def source_defined_cursor(self) -> bool:
return False
And in the UI I am able to select a cursor_field, for the rest I have no deceleration in my code of cursor_field (I want it to be user defined).
I have a log row in def request_params()
containing <http://self.logger.info|self.logger.info>(f"Cursor Field is = {self.cursor_field}")
my console output is as following:
2022-03-30 16:28:27 source > Syncing stream: XXX_Financial_Journalrules
2022-03-30 16:28:27 source > Cursor Field is = []
2022-03-30 16:28:27 source > Doing request with params {'take': 10000, 'skip': 0}