RudderStack #support

lemon-megabyte-87173

12/14/2020, 4:35 PM

currently Rudder JS SDK doesn't support this. Only identify traits are persisted and send along with every event body.

best-holiday-32430

12/15/2020, 3:00 AM

found this error in rudder-server when sending event to webhook, net/http: request canceled (Client.Timeout exceeded while awaiting headers).

polite-megabyte-42230

12/15/2020, 10:44 AM

When creating clickhouse destination from the repo config-generator, in the final step the next button cannot be clicked. Any ideas ?

narrow-rain-89253

12/15/2020, 11:16 AM

@polite-megabyte-42230 This seems to be a bug in the current release. As a temporary workaround change the defaultOption name and value below to GCS (or whichever object storage you want to use) and run the config-generator again. https://github.com/rudderlabs/config-generator/blob/master/src/components/destination/destinationList/CLICKHOUSE.json#L93-L96

white-autumn-96862

12/15/2020, 11:47 AM

Hi, I am trying to integrate Rudder with Clickhouse as destination. I got the following error from Rudder

Copy code

Error in fetching schema from clickhouse destination:1emQpmUoKL9hUQySclLOgsY2KRB, query: select table, name, type from system.columns where database = 'ruddercdp'

polite-megabyte-42230

12/15/2020, 12:40 PM

Any way to enable logging with rudderstack node js SDK? The code below does not seem to do anything

Copy code

const Analytics = require('@rudderstack/rudder-sdk-node')

const WRITE_KEY = '1ldtoh5I4vEyaBnsn025wYoW32H'
const client = new Analytics(WRITE_KEY, 'localhost:8000/v1/batch');

(async function(){
  console.log('a');
  client.identify({
    userId: '123456',
    traits: {
      name: 'Name Username',
      email: '<mailto:name@website.com|name@website.com>',
      plan: 'Free',
      friends: 21
    }
  });
  await new Promise(resolve => setTimeout(resolve, 50000));

})();

polite-megabyte-42230

12/15/2020, 5:46 PM

According to the documentation,

source_name

is used as a prefix when creating tables. However this doesn't seem to be the case. I have several sources all configured to go into clickhouse, however all the tables are created under

default

schema. Am I missing something? Said documentation: https://docs.rudderstack.com/data-warehouse-integrations/warehouse-schemas

quick-judge-58386

12/15/2020, 5:52 PM

@polite-megabyte-42230 Source_name is not used as prefix, Schemas are created with name

source_name

quick-judge-58386

12/15/2020, 5:53 PM

Clickhouse doesnt have the concept of Schemas, Namespace. So all tables goes into same database which is default in your case

polite-megabyte-42230

12/15/2020, 5:58 PM

I see. Is there any way to get the source id from events collected by rudderstack?

polite-megabyte-42230

12/15/2020, 5:59 PM

this

SOURCE ID

quick-judge-58386

12/15/2020, 6:13 PM

you can add source_id in context property via user transformations https://docs.rudderstack.com/how-to-guides/adding-a-new-user-transformation-in-rudderstack#accessing-metadata

polite-megabyte-42230

12/15/2020, 6:21 PM

Is this feature not available on the OSS version?

white-autumn-96862

12/16/2020, 4:53 AM

Hi again, I am still trying to get Clickhouse 20.8.9.6 working with Rudderstack. Now, I get error from Rudder server

Copy code

2020-12-16T04:35:33.545Z        ERROR   [WH] Failed in handle Upload jobs for worker: %!w(*errors.errorString=&{could not load time location: unknown time zone Etc/UTC})

I've set

set date_time_input_format = 'best_effort';

in my Clickhouse. and

SELECT timezone()

returns

Etc/UTC

Can anyone help?

important-petabyte-88844

12/16/2020, 10:05 AM

Hi ! I am trying to do the "developer machine setup" given in the documentation. At the last step when I run

go run -mod=vendor main.go

I get

build context: cannot load context: open ~/RUDDER_STACK/rudder-server/vendor/context: no such file or directory

and when I run

go run main.go

I get

go: downloading <http://github.com/jeremywohl/flatten|github.com/jeremywohl/flatten> v1.0.1

go: extracting <http://github.com/jeremywohl/flatten|github.com/jeremywohl/flatten> v1.0.1

go: finding <http://github.com/jeremywohl/flatten|github.com/jeremywohl/flatten> v1.0.1

build command-line-arguments: cannot load archive/zip: malformed module path "archive/zip": missing dot in first path element

Can you help me running the rudder server ?

important-petabyte-88844

12/16/2020, 10:42 AM

Another question: The native setup is working for me. But when I make a request (basic Identify request from the node-SDK documentation) the rudder server tells me that

bugsnag/sessions/publisher.publish invalid API key: ''

. Where can I fix this ?

polite-megabyte-42230

12/16/2020, 11:54 AM

is there a way to generate

workspaceConfig.json

through an API?

polite-megabyte-42230

12/16/2020, 11:55 AM

also, are write keys inside

workspaceConfig.json

persisted anywhere in the postgres database?

narrow-rain-89253

12/16/2020, 2:46 PM

@polite-megabyte-42230 Currently we do not have API support yet for creating sources, destinations. So it would not be possible currently to generate the config json

polite-megabyte-42230

12/17/2020, 6:26 AM

Is there any documentation/guideline for loadtesting rudder-server? Will postgres be a bottleneck when processing high number of events?

handsome-terabyte-47419

12/17/2020, 6:43 AM

Hi. I am seeing the following error while starting the rudderstack server in a docker:

Copy code

backend_1        | ERROR: No .env file found

This is further crashing the server:

handsome-terabyte-47419

12/17/2020, 6:43 AM

Copy code

backend_1        | 2020-12-17T06:40:17.051Z	INFO	RUDDER_TMPDIR not found, falling back to /tmp
backend_1        | 
backend_1        | 2020-12-17T06:40:17.064Z	INFO	RUDDER_TMPDIR not found, falling back to /tmp
backend_1        | 
backend_1        | 2020-12-17T06:40:17.306Z	INFO	RUDDER_TMPDIR not found, falling back to /tmp
backend_1        | 
backend_1        | 2020-12-17T06:40:17.312Z	INFO	RUDDER_TMPDIR not found, falling back to /tmp
backend_1        | 
backend_1        | 2020-12-17T06:40:17.565Z	ERROR	Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 2020-12-17T06:40:17.565Z	ERROR	goroutine 1289 [running]:
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/logger.(*LoggerT).Fatal(0xc0001575c0|github.com/rudderlabs/rudder-server/utils/logger.(*LoggerT).Fatal(0xc0001575c0>, 0xc0005d3490, 0x1, 0x1)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/logger/logger.go:265 +0x10b
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460|github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460>, 0xc0019c8c00)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:42 +0x37f
backend_1        | panic(0x154c4c0, 0xc0005d3220)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c|github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c>, 0xb, 0x27324c0, 0x0)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/config/config.go:198 +0xea
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)|github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/misc/misc.go:794
backend_1        | <http://github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0|github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0>, 0x0, 0xc000931640, 0x1b)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/services/destination-connection-tester/destination_connection_tester.go:61 +0x6d
backend_1        | <http://github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()|github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/warehouse/warehouse.go:268 +0xd0
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc0004db990)|github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc0004db990)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:46 +0x81
backend_1        | created by <http://github.com/rudderlabs/rudder-server/rruntime.Go|github.com/rudderlabs/rudder-server/rruntime.Go>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:32 +0x3f
backend_1        | 
backend_1        | 2020/12/17 06:40:17 notifying bugsnag: Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 2020/12/17 06:40:17 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: ''
backend_1        | 2020/12/17 06:40:17 bugsnag/sessions/publisher.publish invalid API key: ''
backend_1        | 2020-12-17T06:40:17.565Z	ERROR	Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 2020-12-17T06:40:17.565Z	ERROR	goroutine 1375 [running]:
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/logger.(*LoggerT).Fatal(0xc0001575c0|github.com/rudderlabs/rudder-server/utils/logger.(*LoggerT).Fatal(0xc0001575c0>, 0xc001b46eb0, 0x1, 0x1)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/logger/logger.go:265 +0x10b
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460|github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460>, 0xc001afb140)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:42 +0x37f
backend_1        | panic(0x154c4c0, 0xc0004b4ca0)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c|github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c>, 0xb, 0x27324c0, 0x0)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/config/config.go:198 +0xea
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)|github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/misc/misc.go:794
backend_1        | <http://github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0|github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0>, 0x0, 0xc000930c80, 0x1b)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/services/destination-connection-tester/destination_connection_tester.go:61 +0x6d
backend_1        | <http://github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()|github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/warehouse/warehouse.go:268 +0xd0
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc001b467c0)|github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc001b467c0)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:46 +0x81
backend_1        | created by <http://github.com/rudderlabs/rudder-server/rruntime.Go|github.com/rudderlabs/rudder-server/rruntime.Go>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:32 +0x3f
backend_1        | 
backend_1        | 2020/12/17 06:40:17 notifying bugsnag: Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 2020/12/17 06:40:17 bugsnag.Notify: bugsnag/payload.deliver: invalid api key: ''
backend_1        | panic: Fatal error, no required environment variable: INSTANCE_ID [recovered]
backend_1        | 	panic: Fatal error, no required environment variable: INSTANCE_ID [recovered]
backend_1        | 	panic: Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 
backend_1        | goroutine 1289 [running]:
backend_1        | <http://github.com/bugsnag/bugsnag-go.AutoNotify(0xc000121ba8|github.com/bugsnag/bugsnag-go.AutoNotify(0xc000121ba8>, 0x3, 0x3)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/vendor/github.com/bugsnag/bugsnag-go/bugsnag.go:109 +0x2bc
backend_1        | panic(0x154c4c0, 0xc0005d3220)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460|github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460>, 0xc0019c8c00)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:43 +0x39a
backend_1        | panic(0x154c4c0, 0xc0005d3220)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c|github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c>, 0xb, 0x27324c0, 0x0)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/config/config.go:198 +0xea
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)|github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/misc/misc.go:794
backend_1        | <http://github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0|github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0>, 0x0, 0xc000931640, 0x1b)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/services/destination-connection-tester/destination_connection_tester.go:61 +0x6d
backend_1        | <http://github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()|github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/warehouse/warehouse.go:268 +0xd0
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc0004db990)|github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc0004db990)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:46 +0x81
backend_1        | created by <http://github.com/rudderlabs/rudder-server/rruntime.Go|github.com/rudderlabs/rudder-server/rruntime.Go>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:32 +0x3f
backend_1        | panic: Fatal error, no required environment variable: INSTANCE_ID [recovered]
backend_1        | 	panic: Fatal error, no required environment variable: INSTANCE_ID [recovered]
backend_1        | 	panic: Fatal error, no required environment variable: INSTANCE_ID
backend_1        | 
backend_1        | goroutine 1375 [running]:
backend_1        | <http://github.com/bugsnag/bugsnag-go.AutoNotify(0xc000a2bba8|github.com/bugsnag/bugsnag-go.AutoNotify(0xc000a2bba8>, 0x3, 0x3)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/vendor/github.com/bugsnag/bugsnag-go/bugsnag.go:109 +0x2bc
backend_1        | panic(0x154c4c0, 0xc0004b4ca0)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460|github.com/rudderlabs/rudder-server/rruntime.Go.func1.1(0x1bbd460>, 0xc001afb140)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:43 +0x39a
backend_1        | panic(0x154c4c0, 0xc0004b4ca0)
backend_1        | 	/root/.goenv/versions/1.13.8/src/runtime/panic.go:679 +0x1b2
backend_1        | <http://github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c|github.com/rudderlabs/rudder-server/config.GetRequiredEnv(0x17d237c>, 0xb, 0x27324c0, 0x0)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/config/config.go:198 +0xea
backend_1        | <http://github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)|github.com/rudderlabs/rudder-server/utils/misc.GetNodeID(...)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/utils/misc/misc.go:794
backend_1        | <http://github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0|github.com/rudderlabs/rudder-server/services/destination-connection-tester.UploadDestinationConnectionTesterResponse(0x0>, 0x0, 0xc000930c80, 0x1b)
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/services/destination-connection-tester/destination_connection_tester.go:61 +0x6d
backend_1        | <http://github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()|github.com/rudderlabs/rudder-server/warehouse.(*HandleT).backendConfigSubscriber.func2()>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/warehouse/warehouse.go:268 +0xd0
backend_1        | <http://github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc001b467c0)|github.com/rudderlabs/rudder-server/rruntime.Go.func1(0xc001b467c0)>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:46 +0x81
backend_1        | created by <http://github.com/rudderlabs/rudder-server/rruntime.Go|github.com/rudderlabs/rudder-server/rruntime.Go>
backend_1        | 	/codebuild/output/src878603763/src/github.com/rudderlabs/rudder-server/rruntime/goroutine-factory.go:32 +0x3f
rudderserver_backend_1 exited with code 2

handsome-terabyte-47419

12/17/2020, 6:43 AM

Can anyone help me out with this? The server was working fine 1 week back. OS: Ubuntu

polite-megabyte-42230

12/17/2020, 8:50 AM

Is there a way to add a prefix to table names when inserting events?

important-petabyte-88844

12/17/2020, 9:06 AM

Hi, I am doing the developer machine setup, when I run

npm i

I get multiple warnings of

/home/john/.cache/node-gyp/10.19.0/include/node/v8.h:9502:16: warning: cast between incompatible function types from 'v8::WeakCallbackInfo<ivm::ClassHandle>::Callback' {aka 'void (*)(const v8::WeakCallbackInfo<ivm::ClassHandle>&)'} to 'Callback' {aka 'void (*)(const v8::WeakCallbackInfo<void>&)'} [-Wcast-function-type]

9502 |                reinterpret_cast<Callback>(callback), type);

which is weird and after I run

node destTransformer.js

and I get

Master 19347 is running []

Worker 19354 is online []

Worker 19356 is online []

Worker 19363 is online []

Worker 19355 is online []

Worker 19357 is online []

Worker 19364 is online []

Worker 19382 is online []

Worker 19375 is online []

Worker 19356 started []

Worker 19363 started []

Worker 19354 started []

Worker 19364 started []

Worker 19357 started []

Worker 19355 started []

Worker 19382 started []

Worker 19375 started []

In this output there is no info that rudder transformer server started I set up the

.env

file with the token and finally I run

go run -mod=vendor main.go

and I get the error

build context: cannot load context: open /home/arsenic/tmp/rudder-server/vendor/context: no such file or directory

Can I get some help please ?

polite-megabyte-42230

12/17/2020, 12:15 PM

According to docs, events routed to warehouse destinations are sent into blob storage first (staging files) before actually going into the warehouse. Events are also stored in postgres on rudder-server I have a couple of questions: 1. How many max rows are in a file? 2. When a job to upload files to the warehouse starts, how many files / rows are loaded at once? 3. After the job finishes, are events deleted from postgres? Would love a detailed explanation of this, and how one should configure config parameters under high load, say 50k inserts / sec

narrow-rain-89253

12/17/2020, 12:43 PM

@polite-megabyte-42230 1. Max rows would be around 100000 2. Some of these details can be found here https://docs.rudderstack.com/data-warehouse-integrations/warehouse-faqs#how-can-i-speed-up-my-warehouse-uploads 3. Events are deleted in postgres after they are written to object storage

glamorous-shoe-15731

12/17/2020, 12:57 PM

Re: performance, I don't think a single node Rudder would be able to handle 50K/sec. Rudder was designed more for flexibility (e.g. transforming events via javascript transformations etc) and that has performance implications.

polite-megabyte-42230

12/17/2020, 1:06 PM

How many nodes do you suggest?

glamorous-shoe-15731

12/17/2020, 1:09 PM

Dumping into S3, a single node can do 1.5K/sec average throughput (and 4-5x peak). If you want consistent throughput of 50K then I don't think Rudder makes sense. I would probably setup something like Writers->Kafka->ClickHouse