Lucas Abreu
04/10/2023, 8:07 PMThe Postgres source performs best on small databases (under 100GB).
I was wondering how worse are we talking when using it with larger db's, say 1 TBMasly Alexandra Velasquez Monsalve
04/10/2023, 8:15 PMKonstantin Lackner
04/10/2023, 9:27 PMsessionSource
field of the traffic_sources
stream contains an empty entry (pos. 3 in the table), which makes up ~14% of our total traffic. Thus it's important for us to know which source that is. However, looking at the Google Analytics Dashboard directly, after the "(direct)" traffic source follows "bing", which makes me think that the empty field is a bug of the GA4 connector. Additionally, there is a discrepancy of the number of sessions by traffic source.
Airbyte version: 0.42.0
GA4 Source Connector version: 0.1.3
BigQuery Destination version: 1.2.18Trung Luong
04/10/2023, 10:09 PMGonzalo Bottari
04/10/2023, 11:57 PMMartin Jung
04/11/2023, 12:58 AMSachit Khanna
04/11/2023, 1:39 AMSrinidhi krishnamurthy
04/11/2023, 4:09 AMUUBOY scy
04/11/2023, 5:28 AMyuan sun
04/11/2023, 5:46 AMKristina Ushakova
04/11/2023, 6:04 AMSENTRY_ENVIRONMENT=production
JOB_ERROR_REPORTING_STRATEGY=sentry
Default values are āā and āloggingā accordingly.
From Airbyte env configs in EnvConfigs.java it looks like the strategy defaults to logging:
@Override
public JobErrorReportingStrategy getJobErrorReportingStrategy() {
return getEnvOrDefault(JOB_ERROR_REPORTING_STRATEGY, JobErrorReportingStrategy.LOGGING, s -> {
try {
return JobErrorReportingStrategy.valueOf(s.toUpperCase());
} catch (final IllegalArgumentException e) {
<http://LOGGER.info|LOGGER.info>(s + " not recognized, defaulting to " + JobErrorReportingStrategy.LOGGING);
return JobErrorReportingStrategy.LOGGING;
}
});
}
I may be wrong, not a pro in Java unfortunately. Would be grateful for any help on this issueToan Doan
04/11/2023, 6:35 AMAnuj Shirgaonkar
04/11/2023, 7:25 AMvismaya Kalaiselvan
04/11/2023, 9:29 AMAbdullah Alhabshan
04/11/2023, 10:41 AMShreepad Khandve
04/11/2023, 10:55 AMSushant
04/11/2023, 11:05 AMIvan Brcko
04/11/2023, 11:38 AM/api/v1/sources/discover_schema
call with a 502 Bad Gateway response, and an nginx āAn error occurred.ā error.
While inspecting the logs, I noticed the following error:
upstream prematurely closed connection while reading response header from upstream, client: 192.168.96.9, server: localhost, request: "POST /api/v1/sources/discover_schema HTTP/1.0", upstream: "<http://192.168.96.4:8001/api/v1/sources/discover_schema>"
I would really appreciate any help or guidance on where to look next or what to take into consideration. Thanks šShreepad Khandve
04/11/2023, 12:18 PMTim Josefsson
04/11/2023, 12:35 PMMatej Hamas
04/11/2023, 12:56 PMLalit Kumar Nagwar
04/11/2023, 1:48 PM--2023-04-11 13:47:49-- <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/.env>
Resolving <http://raw.githubusercontent.com|raw.githubusercontent.com> (<http://raw.githubusercontent.com|raw.githubusercontent.com>)... failed: Temporary failure in name resolution.
wget: unable to resolve host address '<http://raw.githubusercontent.com|raw.githubusercontent.com>'
--2023-04-11 13:47:49-- <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/flags.yml>
Resolving <http://raw.githubusercontent.com|raw.githubusercontent.com> (<http://raw.githubusercontent.com|raw.githubusercontent.com>)... failed: Temporary failure in name resolution.
wget: unable to resolve host address '<http://raw.githubusercontent.com|raw.githubusercontent.com>'
--2023-04-11 13:47:49-- <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/docker-compose.yaml>
Resolving <http://raw.githubusercontent.com|raw.githubusercontent.com> (<http://raw.githubusercontent.com|raw.githubusercontent.com>)... failed: Temporary failure in name resolution.
wget: unable to resolve host address '<http://raw.githubusercontent.com|raw.githubusercontent.com>'
Thiago Villani
04/11/2023, 2:53 PMSandhya Manimaran
04/11/2023, 3:28 PMKrzysztof Sikora
04/11/2023, 4:31 PMRaphael Pacheco
04/11/2023, 4:57 PMDominik Mall
04/11/2023, 5:18 PM2023-04-11 17:13:49 DEBUG i.a.w.RecordSchemaValidator(validateSchema):75 - feature flag disabled for workspace 80d44133-1d1f-4709-9ea9-95248a689b77
1. Does anyone know what this actually means/how to resolve it? (The connectors otherwise work fine)
2. Aside from increasing the log-level from DEBUG to something else, can this message be suppressed?
(Using Airbyte OSS 0.41.0 on GKE)Walker Philips
04/11/2023, 5:21 PMKevin Phan
04/11/2023, 6:24 PM2023-04-10 19:08:05 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/042ab8f0-40a1-43b3-a39c-7c97ea41c034/0/logs.log
2023-04-10 19:08:05 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.40.0-alpha
2023-04-10 19:08:05 INFO i.a.c.i.LineGobbler(voidCall):83 - Checking if airbyte/destination-postgres:0.3.21 exists...
2023-04-10 19:08:05 INFO i.a.c.i.LineGobbler(voidCall):83 - airbyte/destination-postgres:0.3.21 not found locally. Attempting to pull the image...
2023-04-10 19:09:32 INFO i.a.c.i.LineGobbler(voidCall):83 - Pulled airbyte/destination-postgres:0.3.21 from remote.
2023-04-10 19:09:32 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 042ab8f0-40a1-43b3-a39c-7c97ea41c034
2023-04-10 19:09:32 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/042ab8f0-40a1-43b3-a39c-7c97ea41c034/0 --log-driver none --name destination-postgres-check-042ab8f0-40a1-43b3-a39c-7c97ea41c034-0-qqosw --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_CONNECTOR_IMAGE=airbyte/destination-postgres:0.3.21 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.0-alpha -e WORKER_JOB_ID=042ab8f0-40a1-43b3-a39c-7c97ea41c034 airbyte/destination-postgres:0.3.21 check --config source_config.json
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Class path contains multiple SLF4J bindings.
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Found binding in [jar:file:/airbyte/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
2023-04-10 19:09:33 ERROR i.a.c.i.LineGobbler(voidCall):83 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 INFO i.a.i.d.p.PostgresDestination(main):90 - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json}
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 INFO i.a.i.b.IntegrationRunner(runInternal):124 - Command: CHECK
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 INFO i.a.i.b.IntegrationRunner(runInternal):125 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-04-10 19:09:35 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2023-04-10 19:09:35 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Thiago Villani
04/11/2023, 7:25 PM