Bug/Issue found during configuration migration. If user export the configuration and import it other server then cursor among all the connections are lost. (on and above v0.34.3-alpha) Description: I am trying to shift my configuration from EC2 instance to EKS. Hence performing configuration migration. If I try to export configuration and try to import on other server (assuming on both airbyte is running same version and are above and equal to v0.34.3-alpha ) then after import all the connections restore, but all the existing log and execution are lost, along with cursor). Log are not important but cursor is needed. Otherwise, all the old data will be repeated again. I tried same process with v0.34.2-alpha but airbyte is able restore all the configuration perfectly. is this because of this issue ? I experimented this issue with following version:

0.32.0-alpha-patch-1	working fine
0.32.11-alpha	        working fine
0.34.2-alpha	        working fine
0.34.3-alpha	        cursor lost
0.34.4-alpha	        cursor lost
0.35.1-alpha	        cursor lost
0.35.18-alpha	        cursor lost

Sample for cursor lost

Hey may I know how you have upgraded to the latest version? If it manual upgrading with the image that doesn't support this usecase. Can you try doing automated upgrading which we provide which means you can create a EKS with the same version and then upgrade it

Hi @Harshith (Airbyte), I am doing the automated upgradation. Currently I am using 0.32.6-alpha EC2 and need to move to latest stable EKS (lets say 0.35.18-alpha). Since I cannot directly dump and restore configuration from 0.32.6 to 0.35.18, so, either 1. I have to update me EC2 to 0.35.18, then dump conf and restore it to EKS (0.35.18) OR 2. downgrade EKS to 0.32.6 and then restore the conf. and upgrade back 0.35.18 With point 1. I am able to dump and restore the conf but cursor are not getting restore Now I will go ahead with point 2. Now, my concern is. If I need to restore the configuration in future then I will make a dump it from 0.35.12 then will I able to properly restore it ? (assuming that I am not doing this migration on same server, then this will be a problem)

Yes, cursor is available on EC2 after upgrade. Everything is working fine on existing system. The issue comes when I need it to move to other machine. For that I need to export conf and import on other machine (EKS). After importing I am not getting the cursor. Only connection gets restore.

Yes I have opted that route, but since with new version of airbyte, conf import is not working properly. It might become difficult to move application later on.

@Harshith (Airbyte) it’s possible to test locally using docker if happens the same situation? If yes, please open an issue, if not reproducible we need to check if we’re going to investigate further.

Hi @Marcos Marx (Airbyte) , @Harshith (Airbyte) I tried the same process with latest version. That is, I clone my EC2(0.32.6) and upgraded it to latest version(0.35.32) then i export conf and import it on Kubernetes(0.35.32). New we have new error

2022-02-22 14:36:05 ERROR i.a.s.h.ArchiveHandler(importInternal):136 - Import failed
java.lang.RuntimeException: io.airbyte.validation.json.JsonValidationException: json schema validation failed when comparing the data to the json schema. 
Errors: $.id: is missing but it is required, $.catalog: is missing but it is required, $.catalogHash: is missing but it is required 
Schema: 
{
  "$schema" : "<http://json-schema.org/draft-07/schema#>",
  "$id" : "<https://github.com/airbytehq/airbyte/blob/master/airbyte-config/models/src/main/resources/types/AttemptFailureSummary.yaml>",
  "title" : "ActorCatalog",
  "description" : "Catalog of an actor.",
  "type" : "object",
  "additionalProperties" : false,
  "required" : [ "id", "catalog", "catalogHash" ],
  "properties" : {
    "id" : {
      "type" : "string",
      "format" : "uuid"
    },
    "catalog" : {
      "type" : "object",
      "existingJavaType" : "com.fasterxml.jackson.databind.JsonNode"
    },
    "catalogHash" : {
      "type" : "string"
    }
  }
}

Hey @Akshay Saini If I assume it right The upgrade from 0.32 -> 0.35 inside just EC2 is working fine right? Then once the data is moved to db can you just take the dump and create a new instance with latest version and add the dump

@Harshith (Airbyte)/@Marcos Marx (Airbyte) I was able to migrate data using method 2 in my case as mentioned here. But the problem is with airbyte configuration export and import. Since my airbyte is in lower version, so this functionality is not hampering my work as of now, but I believe someone other will face this issue sooner or later. I have document it and attached in attachment. Can you please check at your end whether you are also getting same error ?

One small correction to above doc and statement. I am facing cursor error but with other version

0.35.35-alpha  json schema validation failed
0.35.32-alpha  json schema validation failed
0.35.18-alpha          cursor lost
0.35.1-alpha	        cursor lost
0.34.4-alpha	        cursor lost
0.34.3-alpha	        cursor lost
0.32.0-alpha-patch-1	working fine
0.32.11-alpha	        working fine
0.34.2-alpha	        working fine

Here is the updated doc

@Akshay Saini the second method worked to retrieve the cursors? the problem it’s throwing json schema validation errors?

Yup @Marcos Marx (Airbyte) second method (in which i lowered the version of kubernetes airbyte and migrated conf from docker airbyte and then upgraded kubernetes airbyte to latest) that one worked !!. With first method ( where docker Airbyte is upgraded first and then conf migration is done to kubernetes airbyte), there we have some problem. with latest version (0.35.35-alpha) we are having

json schema validation failed.

As of now Airbyte team can focus of this error, if someone faces cursor related issue (0.34.3 - 0.35.18), then you can ask him/her to upgrade airbyte to latest version (0.35.35)

@Octavia Smartie create issue "Upgrade from 0.32 -> 0.35 is not persisting cursors"