Hi, I’m using the low code approach to building a connector for Decipher (a survey app). I’m having trouble with the schema for one of the endpoints. The schema of the response is dynamic and varies by survey. Different surveys have different questions and each question name/id is returned as a key in the response for the GetSruevyData endpoint. As per the documentation, the (simplified) response body looks like this:

[
	{
		"status": 1,
		"uuid": "jwy3kwppp4yuqg86",
		...
		"Q1r1": "8",
		"Q2a": "I thought it was great.",
		"Q2b": "I thought it was not so great."
	}
]

Note that the

status

and

uuid

keys are returned in every response while the

Q*

keys are dynamic and vary depending on the survey. Ideally I think I’d restructure the response to something like this. It would then be easy to represent in the schema and should normalise how I want.

[
	{
		"status": 1,
		"uuid": "jwy3kwppp4yuqg86",
		...
		"answers": [
			{ "question": "Q1r1", "answer": "8" },
			{ "question": "Q2a", "answer": "I thought it was great." },
			{ "question": "Q2b", "answer": "I thought it was not so great." }
		]
	}
]

How should the be handled? Can/Should I restructure the JSON Schema as above? Do I need to move to the Python CDK instead to do this? Can I get away with just ingesting the full raw responses and do the normalisation?

you could do this by using a

CustomRecordExtractor

instead of a

DpathExtractor

. It'll allow you to write Python code to extract the records from the response without having to moving everything to the Python CDK. You can write a new Python class directly in your connector module. It'll need to implement the RecordExtractor interface. Here's an example of a different custom component for reference.

Thanks @Alexandre Girard (Airbyte), this looks like exactly what I need. I’ll give this a crack!