https://linen.dev logo
Join Slack
Powered by
Hi, I'm using Rudderstack self-hosted data plane i...
# support
f
Hi, I'm using Rudderstack self-hosted data plane in Kubernetes and hosted control plane, Rudderstack transformer keeps crashing and this is process info. I removed resource request and limit and still it get crashed even though enough memory is available on nodes.
Copy code
Process info:  {
  pid: 25,
  ppid: 18,
  mem: {
    rss: 483594240,
    heapTotal: 424116224,
    heapUsed: 371040440,
    external: 4213829,
    arrayBuffers: 2397577
  },
  cpu: { user: 39577542, system: 7520616 },
  cmd: '/usr/local/bin/node /usr/local/bin/node /home/node/app/dist/src/index.js'
}
this is my values for transformer, for older version 
"07042022.165711"
it is working fine, but newer versions stuck in 
CrashLoopBackop
Copy code
transformer:
  replicaCount: 4
  service:
    port: 9090
  image:
    repository: rudderlabs/rudder-transformer
    #version: "07042022.165711"
    version: "1.28.1"
    pullPolicy: IfNotPresent
    #resources:
    #  requests:
    #    memory: 4096Mi
    #    cpu: 4
    #  limits:
    #    memory: 10240Mi
    #    cpu: 6
  extraEnvVars:
    - name: NUM_PROCS
      value: "1"
g
Did you upgrade it recently? What's the version for rudder-server?
f
@gentle-petabyte-80785 rudder-server version is 1.5.7
g
Can you please try upgrading to v1.7
I can't see what could be the issue here but trying to upgrade might help with latest optimizations
f
@gentle-petabyte-80785 thanks for your reply. I upgraded rudder server, but nothing has been changed. It uses only 45% of CPU and 7% of memory of the pod allocated resource, but it enters in crashing loop state.
👀 1
g
I see. And there are no errors in logs?
f
@gentle-petabyte-80785 I added hpa for transformer in kubernetes and set cpu utilization to 30%, its working now. But i think it's not the case. Currently it's working fine... Thanks.
It happend again, all 3 replica are in crash loop
Copy code
SIGTERM signal received. Closing workers...
Process info:  {
  pid: 18,
  ppid: 7,
  mem: {
    rss: 105984000,
    heapTotal: 32817152,
    heapUsed: 30295248,
    external: 1893594,
    arrayBuffers: 69028
  },
  cpu: { user: 2078651, system: 646416 },
  cmd: 'node /usr/local/bin/node /home/node/app/dist/src/index.js'
}
Sent kill signal to worker 1 (pid: 25)
SIGTERM signal received in the worker
Process exit event received
Redis client disconnected
Worker (pid: 25) was gracefully shutdown
Process info:  {
  pid: 25,
  ppid: 18,
  mem: {
    rss: 249954304,
    heapTotal: 196640768,
    heapUsed: 116786248,
    external: 3257441,
    arrayBuffers: 1441060
  },
  cpu: { user: 19910474, system: 4000239 },
  cmd: '/usr/local/bin/node /usr/local/bin/node /home/node/app/dist/src/index.js'
}
g
I see. Sorry, I missed your latest message. Let me follow up with the team
f
@gentle-petabyte-80785 Since configuring the hpa for transformer, just one crashing loop incident happend and it has been OK since then. It seem that was from lack of sufficient resources. Thanks.
Open in Slack
PreviousNext
Powered by