This message was deleted.

I don’t see any reasons why we shouldn’t do it, so please feel free to open a PR on this. My only 2 concerns are 1. Dependencies. I am not sure how you are going to implement this but if we are going to bring AWS SDK and if some dependencies are not approved internally by LinkedIn, then we probably cannot use it so your own fork is preferred. 2. Testing. We are trying to enforce 33% branch coverage for every new code commit so please add some new unit tests or integration tests there to help us discover any potential issues. It may take you a while to get familiar with that so please let us know if you are having any issues. But anyways, S3 is very popular and I think we’d love to have it integrated with VenicePushJob

In that case, let us prepare a PR and we can have a more in-depth discussion there.

Let’s not worry about dependencies for now, there is always a way. Worst case scenario, even if the dependencies were really obnoxious, we could still organize the build to take them in in one way or another. In any case, thanks for continuing to push on this! Looking forward to it!

Hi everyone, I've been collaborating with @Dejan Mijic on enabling s3 data sources for the

VenicePushJob

and successfully managed to get an MVP going. In the process I encountered 2 issues maybe someone can give me an insight on this. 1. The hadoop version in the project is 2.3. Is there anything preventing us from bumping this? I bumped it to 2.6 to get

hadoop-aws

to help with the s3 access. Eventhough 2.3 has support, it didn't seem to work as expected. If we can bump, what will be the latest hadoop version ? 2. I wanted to run the test suite with

./gradlew check --continue

locally to check the hadoop bump, but some tests fail and I run out of heap space even on the main branch. Is there any settings that i should use to make it work? I have 32GB of memory if that makes a difference.

Thanks for sharing your progress, that sounds great. Updating Hadoop should be fine… I am not sure what is the best version to jump to. Perhaps the latest in the 2.x series (2.10.2)? It is still fairly recent (2022, from what I can tell). Feel free to give it a try with 3.x also, but IDK if there are more breaking changes.

Hi people, I opened a GitHub issue so that we can track the progress, discuss, and associate a pull request with it.

Created PR to bump hadoop to 2.7.2 this is the minimum bump that i could do higher than 2.6 as there is an incompatible usage of guava.

Thanks! I kicked off the CI on your PR (future PRs from you should run it automatically from now on), and if it passes I’ll merge it.

There is one more to add runtime dependency with a minor change to the configuration. I wanna clenup the code and run some tests for that. Hopefully next week I’ll have it ready

Shall we go to 2.10.2 or not worth it?

It's always better to have the newer versions. I stopped at 2.7.2 becuase that was the minimum version that had

hadoop-aws

with the guava dependancy fix. If you don't have any internal blockers we should bump to 2.10.2. Some of the newer authentication methods are only for hadoop 3.x. for instance

WebIdentityTokenCredentialsProvider

(sts) comes with`hadoop:3.3.1`

I just checked internally and we are recommended to upgrade to 2.10.2 Feel free to include that in your PR. Or we can do it separately if you prefer…

You are welcome. I’m glad that the PR was less invasive and allows the push job to read from other file systems not just S3. Sure we will keep you posted on how it goes when we test with production workloads.