Cross posting here as I am struggling to understand why date | Advice-data-transformation

Channels

advice-data-architecture

advice-data-ingestion

advice-data-orchestration

advice-data-privacy

advice-data-quality

advice-data-transformation

advice-data-visualization

advice-data-warehouses

advice-reverse-etl

airbyte-cloud

airbyte-for-power-users

cloud-master-build-failure

community-strategy

connector-development

contributing-to-airbyte

databricks-airbyte316

events-and-conferences

ext-torch-cube

feedback-and-requests

hacktoberfest-2022

help

infra-dev-alerts

infra-dev-alerts-webhook

marcos-test-discourse

mos-client

motleyfool

movedata-a-different-way-to-work

movedata-airbyte-connection-management-scale

movedata-airbyte-orchestration-in-gcp

movedata-better-data-testing-with-the-data-error-generating-process

movedata-bring-your-own-infra

movedata-building-a-real-time-user-facing-dashboard-with-airbyte-redpanda-and

movedata-building-connectors-in-minutes

movedata-cicd-for-data-building-devtest-data-environments-with-lakefs

movedata-data-analysts-are-setup-to-fail-and-its-our-fault

movedata-data-engineering-is-software-engineering-and-software-engineering-is-da

movedata-data-orchestration-is-not-just-running-jobs-on-a-schedule

movedata-dataops-on-the-open-modern-data-stack

movedata-dev-first-open-source-data-observability-solution

movedata-ducky-data-crunching-on-the-laptop---the-pendulum-swings

movedata-five-causes-of-data-quality-issues

movedata-future-evolution-of-the-data-stack

movedata-general

movedata-guardrails-not-stop-signs

movedata-if-you-build-it-will-they-come-how-to-activate-your-modern-data-stack

movedata-in-2023-data-trust-will-be-more-important-than-ever-before

movedata-ingesting-data-with-airbyte-into-a-high-speed-data-lakehouse-using-trin

movedata-keynote-building-the-foundations-of-data-movement

movedata-let-your-data-team-choose-their-own-tools

movedata-mobilize-the-worlds-data

movedata-modern-data-management---how-to-achieve-data-discovery

movedata-moving-data-reliably-ingestion-observability-are-better-together

movedata-navigating-the-modern-data-stack-with-open-source-headless-bi

movedata-open-source-communities-shape-modern-data-stacks

movedata-prefect-ing-self-hosted-airbyte

movedata-prep-your-pipelines---reverse-etl-and-the-coming-great-flood

movedata-speakers

movedata-streaming-made-modern

movedata-the-best-data-warehouse-is-a-lakehouse

movedata-the-data-ecosystem-is-ready-for-etl-to-be-dead

movedata-the-end-of-the-pipeline

movedata-traditional-data-catalogs-will-be-replace-by-active-metadata-platforms

movedata-using-airbyte-on-day-1of-our-startup

movedata-using-airbyte-to-build-your-dremio-open-data-lakehouse

movedata-who-needs-a-traditional-data-warehouse-when-you-can-get-real-time-analy

movedata-why-bi-is-not-enough

octavia-cli

office-hour

office-hour-12-october

oss-master-build-failure

p0-instagram-incompatible-metrics

partner-productpair-hightouch

understanding-airbyte

write-for-the-community

Feed

Channels

Title

#advice-data-transformation

Antonia Villa

08/22/2022, 8:06 PM

Cross posting here as I am struggling to understand why dates are not being transformed correctly in the salesforce connector

8:25 PM

Looking through the JSON Schema for the salesforce connector, it appears that anything that appears with the definition of

{"anyOf":[{"type":"string","format":"date-time"}

does not translate to a date in BigQ. In the subsequent logs, there are warnings for all of these fields:

2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column CloseDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column CreatedDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column LastViewedDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column SystemModstamp at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column LastActivityDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column LastModifiedDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Checkback_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column LastReferencedDate at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Contract_End_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Contract_Start_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column POC_Expiration_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Trade_Negotiation_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Potential_Interest_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Diagnosis_Discovery_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Discovery_Meeting_Set_On__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Discovery_Meeting_Set_For__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Proof_of_Concept_End_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Perscription_Proposal_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Techincal_Demo_Review_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Proof_of_Concept_Start_Date__c at Opportunity
2022-08-22 18:09:55 [42mnormalization[0m > WARN: Unknown type for column Last_Activity_Completed_Date__c at Opportunity

View count: 1

Join thread in Slack