https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
k
Add more information of the UI pages, here, if I choose Access key ID type as none and Secret key type as none, I got the following error:
Copy code
Error: Cannot construct instance of `org.apache.druid.data.input.s3.S3InputSourceConfig`, problem: accessKeyId cannot be null if secretAccessKey is given at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 128]
If I choose default for both, I have following error.
Note, here, in my AWS_CREDENTIAL_PROFILES_FILE file, it contains also session token.
Copy code
[default]
aws_access_key_id =xxxxxxx
aws_secret_access_key = xxxxxx
aws_session_token = xxxxx
If I only add aws_access_key_id and aws_secret_access_key value (not place to add session token, I have following error
environment
does not seem to apply here, as I don't use environment variable here.
l
If I only add aws_access_key_id and aws_secret_access_key value (not place to add session token, I have following error
I have seen the error if the access key id and value or the bucket are incorrect. Can you access the file via the aws cli using the access keys that you have provided in the console?
Druid uses https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html 
DefaultAWSCredentialsProviderChain
class to fetch the credentials. The locations higher in the list takes a higher priority over the others. Can you please check if there is a config that is provided in your environment that has more priority over the default credential profiles file (number 4).
k
> If I only add aws_access_key_id and aws_secret_access_key value (not place to add session token, I have following error
I have seen the error if the access key id and value or the bucket are incorrect. Can you access the file via the aws cli using the access keys that you have provided in the console?
@Laksh Singla, I just tried, the credential files from AWS_CREDENTIAL_PROFILES_FILE env can be used to list the bucket
Druid uses https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html 
DefaultAWSCredentialsProviderChain
class to fetch the credentials. The locations higher in the list takes a higher priority over the others. Can you please check if there is a config that is provided in your environment that has more priority over the default credential profiles file (number 4).
Here are the checkings:
Copy code
bash-4.2$ echo $AWS_ACCESS_KEY_ID

bash-4.2$ grep aws.accessKeyId /tmp/conf/druid/cluster/_common/common.runtime.properties 
bash-4.2$ env | grep AWS 
AWS_CREDENTIAL_PROFILES_FILE=/secrets/serviceaccount/credentials
bash-4.2$ 
bash-4.2$ ls ~/.aws
ls: cannot access /opt/druid/.aws: No such file or directory
bash-4.2$
1/ Now env of AWS_ACCESS_KEY_ID set 2/ now aws.accessKeyId set in runtime properties 3/ web token. I don't think there will be any, not sure what to check 4./ no ~/.aws directory. However, I do have AWS_CREDENTIAL_PROFILES_FILE set. Since this pod has no 
aws
command line tool installed. I copied the credential file to another pod with aws tools and it worked over there.
@Laksh Singla, my question to you is that given the credential file containing also session_id, how should I configure the UI to connect to S3 bucket? Should I use 
none
, 
default
or 
environment
here?
l
It should have been working IMO. I was going through the code, and here are some alternatives that I think might be worth trying: 1. Can you try and move the credentials file to a default location like ``~/.aws/credentials` and check if it is working. 2. Alternatively, you can create a properties file and supply it as a cluster property to Druid using 
druid.s3.fileSessionCredentials
in the config.
These are mere workarounds for now. In the working cluster that you mentioned, can you please repeat the steps that you followed in https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1665552907375799?thread_ts=1665537932.673459&cid=C0309C9L90D and share the results. Thanks!
Also, does supplying the correct keys in the data loader UI doesn’t work? That’s weird 🤔
k
Also, does supplying the correct keys in the data loader UI doesn’t work? That’s weird
Here, in my case I have session_keys in the credential files. Also, can you be specific as which options should I use in UI? 
default
or 
none
?
l
default since you are providing the keys via the UI
k
default since you are providing the keys via the UI
In my case, we have to have session keys, which is not doable from UI. Let me repeat the steps in the working pod one more time.
l
Hmm, ahh I thought that you meant while testing it out from the data loader
If this doesnt’t work can you try if 
Alternatively, you can create a properties file and supply it as a cluster property to Druid using druid.s3.fileSessionCredentials in the config
this suggestion works?
k
So repeating the steps, here is what I found:
Copy code
[datagen@datagen query_data_gen]$ mkdir ~/.aws
[datagen@datagen query_data_gen]$ cat > ~/.aws/config <<EOF
> [default]
> region = us-west-2
> EOF
[datagen@datagen query_data_gen]$ 
[datagen@datagen query_data_gen]$ export AWS_CREDENTIAL_PROFILES_FILE=/secrets/serviceaccount/credentials
[datagen@datagen query_data_gen]$ aws s3 ls <s3://monc-ra-common-lab1-monitoring-dev1-uswest2-s3/perf_data>

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
using $AWS_CREDENTIAL_PROFILES_FILE itself does not work
however, if I copy the file to ~/.aws directory, it would work.
Copy code
[datagen@datagen query_data_gen]$ cp /secrets/serviceaccount/credentials ~/.aws
[datagen@datagen query_data_gen]$ aws s3 ls <s3://monc-ra-common-lab1-monitoring-dev1-uswest2-s3/perf_data>
                           PRE perf_data/
2022-10-07 00:33:05      37926 perf_data
Actually, I don't why setting 
AWS_CREDENTIAL_PROFILES_FILE=/secrets/serviceaccount/credentials
along in my config, the deep storage s3 storing the segment files would work. But it does work.
l
Hmm, so changing the file path to default ~/.aws/credentials is working as opposed to setting it custom using the env variable?
k
Let me try this suggestion: 
Alternatively, you can create a properties file and supply it as a cluster property to Druid using druid.s3.fileSessionCredentials in the config
So if I understand correctly, I will need to add this line to common properties, and redeploy the cluster, right? 
druid.s3.fileSessionCredentials=/secrets/serviceaccount/credentials
l
I think the properties file has a different format as well. Something like:
Copy code
prop1=val1
prop2=val2
k
Hmm, so changing the file path to default ~/.aws/credentials is working as opposed to setting it custom using the env variable?
Right! Here, I copied the one to default ~/.aws/credentials
I think the properties file has a different format as well. Something like:
Do you mean the common property files for druid, or the credential file for aws?
my aws credential file is like the following:
Copy code
[default]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = yyyyyyyy
aws_session_token = zzzzzz
aws_expiration = 2022-10-12T16:43:52Z
There are spaces between 
=
, does it matter?
my druid common.runtime.properties file has something like this:
Copy code
druid.storage.type=s3
druid.storage.bucket=monc-ra-common-lab1-monitoring-dev1-uswest2-s3
druid.storage.baseKey=druid_moncloud_events/segments

#to add
druid.s3.fileSessionCredentials=/secrets/serviceaccount/credentials
So you suggest me to add a line like above and redeploy, right?
@Laksh Singla, while doing the common.runtime.properties change, one more question is that how does region get set in this case?
@Laksh Singla, I redeployed the cluster with 
druid.s3.fileSessionCredentials=/secrets/serviceaccount/credentials
in all common config files and it is still not working.
In fact, the error messages are still the same. I strongly suspect it is a UI flow problem. For example, if I set access key Id type as none and secret key type as none, the error code is something like:
Copy code
Error: Cannot construct instance of `org.apache.druid.data.input.s3.S3InputSourceConfig`, problem: accessKeyId cannot be null if secretAccessKey is given at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 128] (through reference chain: org.apache.druid.indexing.overlord.sampler.IndexTaskSamplerSpec["spec"]->org.apache.druid.indexing.common.task.IndexTask$IndexIngestionSpec["ioConfig"]->org.apache.druid.indexing.common.task.IndexTask$IndexIOConfig["inputSource"]->org.apache.druid.data.input.s3.S3InputSource["properties"])
The errors are still the same.
@Laksh Singla, eventually the issue is resolved. In fact, if I just use edit spec and ensure nothing is added in input resource, it would just work.
Copy code
"inputSource": {
                                        "type": "s3",
                                        "prefixes": [ "<s3://monc-ra-common-lab1-monitoring-dev1-uswest2-s3/perf_data/>"]
                                },
g
interesting finding! i cross-posted this in #C030K0Z7S1Z, as it may be a UI problem
2 Views
Open in Slack
PreviousNext
Powered by