Hi team. We have an ongoing issue with Redshift cl...
# all-things-deployment
p
Hi team. We have an ongoing issue with Redshift cluster ingestion We faced the error
Copy code
'/usr/local/bin/run_ingest.sh: line 40:  2722 Killed ( datahub ${debug_option} ingest run -c "${recipe_file}" '
several times. Each time, I’ve extended the memory & CPU for
acryl-datahub-actions
pod which came out finally to this enormous config:
Copy code
acryl-datahub-actions:
  enabled: true
  image:
    repository: acryldata/datahub-actions
    tag: "v0.0.7"
  resources:
    limits:
      memory: 120Gi
    requests:
      cpu: 32Gi
      memory: 100Gi
But still failing with similar error I’m using, from the UI, the CLI version
0.10.1.2
I’ve checked also the memory usage for this pod & I don’t see any issue Any idea? Cc: @modern-garden-35830 @icy-controller-68116
1
a
@brainy-tent-14503 might be able to provide some guidance here!
p
@astonishing-answer-96712 hi Is anyone else that can help?
b
Can we see a
describe
on the failed pod and any logs from the actions container?
We might need to find someone on the ingestion side to talk about the redshift connector specifically. I believe some work was done to trade memory with disk space for very large ingestions — @famous-waitress-64616 might have some additional details around this.
f
Hi Aviv, can you try with CLI version 0.10.2.3 instead? If that doesn't work, let's see if we can get a very minimal ingestion to run by altering your recipe with the following config options:
Copy code
include_tables: false
include_views: false
include_table_lineage: false
include_copy_lineage: false
include_usage_statistics: false
profiling:
    enabled: false
p
@brainy-tent-14503 all pods are running. See img above
@famous-waitress-64616 hi I’ve executed my ingestion now with
0.10.2.3
My current ingestion recipe looks as follows:
Copy code
source:
    type: redshift
    config:
        host_port: 'xxxxxx:xxxx'
        database: xxxxx
        username: xxx
        password: 'xxxxxx'
        include_tables: true
        include_views: true
        options:
            connect_args:
                sslmode: prefer
                sslrootcert: null
        profiling:
            enabled: true
            max_workers: 3
            profile_table_level_only: false
            include_field_sample_values: false
Another issue I’m facing is the password field. When trying to use the secret service the UI provides, it’s failing with an authentication error. Is any special template needed here?
a
Did the version upgrade fix the issue? If not, can you try with profiling disabled? And if that still doesn't work, try with all the settings I describe above. Not sure what's up with the password field. Be careful if you're copy pasting a password with any special characters like
\n
, and don't include any quotes.
p
@famous-waitress-64616 YES 🙂 We also had a permission issue for the user but looks good now. I’m scaling down the resources for the datahub-action pod
I’m creating a password with your secret service It contains letters(lower & upper), digits & special characters
@famous-waitress-64616 @brainy-tent-14503 @astonishing-answer-96712 let’s say my pwd is something like this:
Copy code
password: 'A!R2jyg543mjy$vd'
Any reason that DH secret service won’t work?
f
Not that I know of -- you're pasting in the password into the
value
field as
A!R2jyg543mjy$vd
?
p
@famous-waitress-64616 yes but then getting authentication error in the ingest log
b
From what I see in the docs, creating a secret in the UI, for example
REDSHIFT_PASSWORD
[1], would then mean setting the recipe to use
password: ${REDSHIFT_PASSWORD}
in the recipe [2]. Its not clear from your shared recipe whether this is matching your setup.
p
@brainy-tent-14503 hi When using the secret service, I’m facing the following error:
Copy code
ERROR    {datahub.entrypoints:195} - Command failed: {'S': 'FATAL', 'C': '28000', 'M': 'password "
           'authentication failed for user
When putting the password explicitly in the YAML recipe, it not failing with this error What’s the issue here?
@astonishing-answer-96712 @famous-waitress-64616 ☝️
@astonishing-answer-96712 @famous-waitress-64616 @brainy-tent-14503 guys I need pls. resolution on that as my prod ingestion is using explicit pwd in the YAML file
a
Unfortunately this is hard to resolve without being able to reproduce the issue. Can you send the exact recipe you're trying to run when using a password secret? And can you try running ingestion with a secret for a different value, rather than password (e.g. database name) that doesn't have any special characters? I'd like to figure out if secrets are bugged entirely for your setup, or just for your specific password.
b
I wonder if it is just a matter for quoting the password, like it includes characters that are not liked by yaml. Maybe try
password: "${REDSHIFT_PASSWORD}"
in the recipe.
p
@brainy-tent-14503 @famous-waitress-64616 got it 😮 It doesn’t like hyphens in the secret name. Changing to underscore solved the issue