https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
d

David Mattern

05/09/2022, 7:43 PM
Hi everyone. I am having a bit of trouble with the incremental sync for a stream. My end goal is to have my cursors be in a nested Python dictionary. I ran this code setting a simple dictionary and saving the state with this command from the tutorial: # Save the latest state to sample_files/state.json python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json | grep STATE | tail -n 1 | jq .state.data > sample_files/state.json https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/tutorials/cdk-tutorial-python-http/6-read-data.md The dictionary was successfully saved in the json file. For the next run, I commented out the dictionary and just tried to read it from the json file, and I only get an empty dictionary for stream_state. # Run a read operation with the latest state message python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json --state sample_files/state.json Here is this portion of my code:
Copy code
def update_state(self, state):
        "sends an update of the state variable to stdout"
        output_message = {"type":"STATE","state":{"data":state}}
       
        print(json.dumps(output_message))

    def parse_response(self,
        response: requests.Response,
        stream_state: Mapping[str, Any],
        #stream_state,
        **kwargs) -> Iterable[Mapping]:
     
        if stream_state is not None:

            #stream_state['a'] = 1
            print (stream_state)

            for key, value in stream_state.items():
                print (key)
                print (value)

        else:
            #stream_state['a'] = 1
            print (stream_state)
					
		#self.update_state(stream_state)
Here is this portion of my configured_catalog:
Copy code
"stream": {
        "name": "readings",
        "json_schema": {
          "properties": {
            "column_name": {
              "type": "string"
            }
          },
          "type": "object",
          "additionalProperties": false
        },
        "supported_sync_modes": ["incremental"]
      },
      "sync_mode": "incremental",
      "destination_sync_mode": "append"
    }
When printing the state, I only get an empty dictionary, whereas I think it should be the simple key value of 'a' and 1.