Caleb Hattingh
06/30/2023, 10:45 AMstate parameter passed to the read method (and what is emitted in state messages during read)?kapa.ai
06/30/2023, 10:45 AMkapa.ai
06/30/2023, 10:45 AMstate parameter passed to the read method in a Source implementation, or what is emitted in state messages during read. The state is treated as a black box by the Airbyte protocol and is modeled as a JSON blob. It is used to allow a Source to pick up where it left off and to checkpoint data replication. The Source can track state however makes most sense to it. The state for an actor is emitted as a complete black box and is wrapped in the AirbyteStateMessage. The contents of the data field is what is passed to the Source on start up. However, it's important to note that while there's no specified size limit in the extracts, practical limitations may be imposed by the specific data store or system you're working with. For more details, you can refer to the [Airbyte Protocol](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol) documentation.Caleb Hattingh
06/30/2023, 11:12 AMstate parameter is provided as a defaultdict. Under the hood is the serialization of the state blob implemented using pickle?kapa.ai
06/30/2023, 11:13 AMkapa.ai
06/30/2023, 11:14 AMpickle for serialization of the state blob in the Python source connector CDK. However, they do mention that the state attribute helps the CDK figure out the current state of sync at any moment and that the setter typically deserializes state saved by CDK and initializes the internal state of the stream. The getter should serialize the internal state of the stream. Here is an example of how the state getter and setter are implemented: python @property def state(self) -> Mapping[str, Any]: return {self.cursor_field: str(self._cursor_value)} @state.setter def state(self, value: Mapping[str, Any]): self._cursor_value = value[self.cursor_field] For more details, you might want to check the [Airbyte documentation](https://docs.airbyte.com/connector-development/cdk-python/incremental-stream).