so, i'm the worst at java and shaded jars. i've be...
# troubleshooting
r
so, i'm the worst at java and shaded jars. i've been trying for an embarrassingly long amount of time to get a simple flink job that consumes from kinesis to work. i saw this conversation but i was already using the BOM stuff to make sure everything stays aligned but I'm still having a similar issue, i think. Essentially I'm trying to use my local
~/.aws/config
file that has several profiles, including a
default
. these all use an
sso-session
. everything works with the CLI and other languages/SDKs. Also works in Java if I use the non-shaded SDK stuff. However, if I try to run a job with
flink run
or I try to use the shaded references in an example,
CredsTest.java
then I get the same errors. I've put together this github repo that hopefully helps with reproduction. I created the java project with the flink archetype, if that matters. I imagine I could "fix" this by reshading the dependencies to match the flink-shaded stuff, but that seems like a nightmare. Hoping some smarter folks can help me with this pain. I also really hope I'm just doing something dumb.
My assumption is that there is some funkiness happening with the shading. Here is the source that tries to dynamically load the SSO factory I think that the reference is properly shaded and renamed in the shaded jar to point to the flink-shaded class. However, if I peak inside my shaded jar, I only see the non-shaded classes and not the shaded ones. So I think that's why the
To use Sso related properties in the 'default' profile, the 'sso' service module must be on the class path.
error is happening. Because the "'sso' service module"(factory) can't be found?
any thoughts on if adding `sso`(and maybe
ssooidc
,
profiles
) to this list of dependencies in the parent pom would be a way to fix this? or, we tell maven shade to not replace the
SSO_PROFILE_CREDENTIALS_PROVIDER_FACTORY
value with the shaded value?
or make the classloading stuff configurable(env or properties...) or something else?
not certain if anyone is reading this, but i think a fix is in this fork/branch. i've tested this locally but would like to get this in to the upstream. or have someone suggest a better solution. i have a pending JIRA request or I'd create the ticket myself and make a PR... but I can't.
k
I only see the non-shaded classes and not the shaded ones. So I think that's why the
To use Sso related properties in the 'default' profile, the 'sso' service module must be on the class path.
error is happening. Because the "'sso' service module"(factory) can't be found?
Yes, classloading that uses constant string as package/classname usually breaks when shading is done. Why are you shading?
However, if I try to run a job with
flink run
Are you running the job with a different user?
r
flink run
locally is using the same user, profile, and sso settings, correct. as far as "why are you shading?"... i'm not entirely certain how to answer that. to my knowledge the
flink-aws-connector
jars are all shaded. i'm just trying to leverage those shaded jars. as to why i'm shading in my application... this is because both the python example from the AWS folks and the archtype-generated pom both use shading. the "classloading that uses constant strings" is from the aws sdk for java in the ProfileCredntialsUtils.class. when i add
sso
and
ssooidc
to the
flink-connector-aws-base/pom.xml
dependencies then everything works.
@Keith Lee, are you thinking i shouldn't shade an application jar? i think this is an issue in the flink aws connector and has an "easy" solution as outlined in this PR.
k
Hey Ryan, thank you for raising the PR. I'd like to sync on this to understand the problem better. Would you be OK for a huddle today?
Left a comment on the PR, again thank you for raising it!
r
doh, missed this. let me know if/when you want to huddle.