Hey all. I am migrating target-postgres from the t...
# singer-targets
h
Hey all. I am migrating target-postgres from the transferwise to meltanolabs variant but during the load process it is converting a column from jsonb to jsonb[]. Has anyone faced this issue before? Looks like I am missing something. More info on đź§µ
âś… 1
Sample input from my tap-postgres:
Copy code
{"type":"STATE","value":{}}
{"type":"SCHEMA","stream":"public-assets_timeline","schema":{"properties":{"id":{"type":["string"]},"asset_id":{"type":["string","null"]},"event_id":{"type":["string","null"]},"author_id":{"type":["string","null"]},"author_name":{"type":["string","null"]},"note":{"type":["string","null"]},"created_at":{"format":"date-time","type":["string","null"]},"updated_at":{"format":"date-time","type":["string","null"]},"deleted_at":{"format":"date-time","type":["string","null"]},"event_data":{"type":["string","number","integer","array","object","boolean","null"]},"event_name":{"type":["string","null"]},"timestamp":{"format":"date-time","type":["string","null"]},"attachment_id":{"type":["string","null"]}},"type":"object","required":["id"]},"key_properties":["id"]}
{"type":"RECORD","stream":"public-assets_timeline","record":{"id":"cc1831da-6d79-44ec-adab-128b6735b95b","asset_id":"d327cf21-9eab-47f1-9764-75be4c23452c","event_id":null,"author_id":"user_2URB2ajknHDiNdZsKCX4gTxiAkf","author_name":"TEST NAME","note":"Equipamento voltou para funcion\u00e1rio","created_at":"2023-10-24T13:27:30.196000","updated_at":"2023-10-24T13:27:30.196000","deleted_at":null,"event_data":null,"event_name":null,"timestamp":"2023-10-24T13:27:30.195000","attachment_id":null},"time_extracted":"2024-07-27T12:24:43.615023+00:00"}
{"type":"RECORD","stream":"public-assets_timeline","record":{"id":"62d9c13e-fdd4-41ae-bb82-4421e6709d21","asset_id":"36b5f8d6-51f5-4f22-bf44-6f122fb9b1f5","event_id":"c3d683b3-e94e-43f0-857e-875003557234","author_id":null,"author_name":null,"note":null,"created_at":"2024-07-15T17:47:36.042000","updated_at":"2024-07-15T17:47:36.042000","deleted_at":null,"event_data":{"asset":{"id":"36b5f8d6-51f5-4f22-bf44-6f122fb9b1f5","brand":"Dell","model":"S2421HN","status":"available","category":"monitor","companyId":"dcababdf-b19d-4764-9f4b-23dab622154b","nfeNumber":"123123","priceCents":111968,"vendorName":"DELL COMPUTADORES DO BRASIL LTDA","enrollmentId":"1282","configuration":{},"acquisitionDate":"2021-07-23T03:00:00.000Z","nfeAttachmentId":"a8fa4f38-cf16-466b-9e5c-05c6a2b9f2ce"}},"event_name":"asset-created","timestamp":"2024-04-24T17:34:19.468000","attachment_id":null},"time_extracted":"2024-07-27T12:24:43.774995+00:00"}
{"type":"STATE","value":{"bookmarks":{"public-assets_timeline":{}}}}
Loader config in meltano.yaml:
Copy code
loaders:
    - name: target-postgres
      variant: meltanolabs
      pip_url: meltanolabs-target-postgres
      config:
        host: db-warehouse
        user: meltano
        database: data_warehouse
Resulting table:
Copy code
select asset_id, event_data from melty.assets_timeline;

| asset_id                             | event_data |
| ------------------------------------ | ---------- |
| d327cf21-9eab-47f1-9764-75be4c23452c |            |
| 36b5f8d6-51f5-4f22-bf44-6f122fb9b1f5 | asset      |
Copy code
select column_name, data_type from INFORMATION_SCHEMA.COLUMNS where table_schema = 'melty' and table_name = 'assets_timeline';

| column_name          | data_type                   | --row-header |
| -------------------- | --------------------------- | ------------ |
| id                   | text                        |              |
| asset_id             | text                        |              |
| event_id             | text                        |              |
| author_id            | text                        |              |
| author_name          | text                        |              |
| note                 | text                        |              |
| created_at           | timestamp without time zone |              |
| updated_at           | timestamp without time zone |              |
| deleted_at           | timestamp without time zone |              |
| event_data           | ARRAY                       |              |
| event_name           | text                        |              |
| timestamp            | timestamp without time zone |              |
| attachment_id        | text                        |              |
| _sdc_extracted_at    | timestamp without time zone |              |
| _sdc_received_at     | timestamp without time zone |              |
| _sdc_batched_at      | timestamp without time zone |              |
| _sdc_deleted_at      | timestamp without time zone |              |
| _sdc_sequence        | bigint                      |              |
| _sdc_table_version   | bigint                      |              |
| _sdc_sync_started_at | bigint                      |              |
I run with
meltano invoke target-postgres --input tap_output.json
The issue is with the
event_data
column that is an object in the input json file but is converted to
ARRAY
in the postgres table
e
Hey @hudo_assenco! This was merged and shipped with v0.0.14 🙂 I also added you to the contributors file.
❤️ 1
c
awesome I was just coming here to ask this same question, had the same problem - fortunately only impacted my staging environment so far.
h
Hi @Edgar RamĂ­rez (Arch.dev), thanks for the help. I found a somewhat related issue, but this time in tap-postgres and wal2json. Could you give a look whenever possible? Thanks! https://meltano.slack.com/archives/C06A1MD6A6L/p1724619858505739
e
Will take a look!