Hi @flarco We are in the process of doing some BQ -> BQ work and have run into an interesting issue. We don't seem to be able to get nested schemas working. Usually we do

dataset_name.*

but this time we need to something like

dataset_group.dataset_name.*

and we are getting all kinds of trouble.

sling run \
                    --src-conn <company>-bigquery-prod \
                    --src-stream "<dataset_group>.events_${CURRENT_DATE}" \
                    --tgt-conn <project>-bigquery-prod \
                    --tgt-object "<app>_analytics_${CURRENT_YEAR}.sftp_logs_${CURRENT_DATE}" \
                    --mode full-refresh

<company>-bigquery-prod:
        type: bigquery
        project: <project>
        location: EU
        gc_bucket: <company>-sling-temp-bucket
        key_body: 'redacted'

and this gives us

fatal:
~ Could not unload.
~ Error in Export Task
{Location: ""; Message: "Operation cannot be performed on a nested schema. Field: event_params"; Reason: "invalid"}

hey @MOBY can you try with parquet format? bq export uses csv by default.

bash
sling run \
  --src-conn <company>-bigquery-prod \
  --src-stream "<dataset_group>.events_${CURRENT_DATE}" \
  --tgt-conn <project>-bigquery-prod \
  --tgt-object "<app>_analytics_${CURRENT_YEAR}.sftp_logs_${CURRENT_DATE}" \
  --mode full-refresh \
  --src-options '{"format": "parquet"}'

or in in env.yaml:

yaml
<company>-bigquery-prod:
  type: bigquery
  project: <project>
  location: EU
  gc_bucket: <company>-sling-temp-bucket
  format: parquet     # <-- export via Parquet so nested columns survive the unload
  key_body: 'redacted'

Added

paraquet

to src and tgt options and now getting

fatal:


~ Error in Import task from GCS
{Location: ""; Message: "Unsupported field type: JSON"; Reason: "invalid"}

~ Error importing to BigQuery
~ Error in Import task from GCS
{Location: ""; Message: "Unsupported field type: JSON"; Reason: "invalid"}

Yea, that's a BQ limitation (surprising TBH), it doesn't export json to parquet. What you can do is use custom sql to convert the json to string or use [column casting](https://docs.slingdata.io/concepts/replication/columns)