I’m trying to use the map-reduce job to build segments. In HadoopSegmentGenerationJobRunner.packPluginsToDistributedCache, there’s this code:

File pluginsTarGzFile = new File(PINOT_PLUGINS_TAR_GZ);
      try {
        TarGzCompressionUtils.createTarGzFile(pluginsRootDir, pluginsTarGzFile);
      } catch (IOException e) {
        LOGGER.error("Failed to tar plugins directory", e);
        throw new RuntimeException(e);
      }
      job.addCacheArchive(pluginsTarGzFile.toURI());

This creates a

pinot-plugins.tar.gz

file in the Flink distribution directory, which is on my server. But as the Hadoop DistributedCache documentation states, “The 

DistributedCache

 assumes that the files specified via urls are already present on the 

FileSystem

 at the path specified by the url and are accessible by every machine in the cluster.”

what is the fix?

I think the tar file (in snippet above) should be generated in a temp dir, and then uploaded to the staging directory. and the staging dir URI is what’s added to the distributed cache

what do you mean by upload to staginging directory

As part of the job spec file, you include a

stagingDir

configuration.

so stagingDir should be on HDFS?

And yes, the addCacheArchive() gets called on the server where you start the job. Which is why it has to be provided a URI to a file that’s available on every slave server. So it can’t be a file://xxx path.

we thought that happens automatically

And yes, stagingDir should be on HDFS (when running distributed). And if you don’t specify it as such, the job fails (as it should) because it’s not using the same file system as the input/output directories.

got it! is this how it was from day one?

It will work if you run locally, of course, because the file://path is accessible to the mappers

what you are saying makes sense, but I thought job launcher pushes this to worker nodes

It’s a bit confusing…if you use the standard Hadoop command line

-files

parameter (as an example), then the standard Hadoop tool framework will copy the file(s) to HDFS first, before adding them to the JobConf as

<hdfs://blah>

paths. In the Pinot code, you need to do this first step (of copying to HDFS) yourself.

then the standard Hadoop tool framework will copy the file(s) to HDFS first

that what I thought would happen when we do it via code, do you know which staging directory will it copy it to?

Each Hadoop job has a “staging” directory in the cluster

thanks

Thanks @Ken Krugler

I just filed https://github.com/apache/incubator-pinot/issues/6492, looking at a fix now.

Thanks!

Funny, looks very similar to what I’ve done:

protected void packPluginsToDistributedCache(Job job, PinotFS outputDirFS, URI stagingDirURI) {
    File pluginsRootDir = new File(PluginManager.get().getPluginsRootDir());
    if (pluginsRootDir.exists()) {
      try {
        File pluginsTarGzFile = File.createTempFile("pinot-plugins", ".tar.gz");
        TarGzCompressionUtils.createTarGzFile(pluginsRootDir, pluginsTarGzFile);
        
        // Copy to staging directory
        Path cachedPluginsTarball = new Path(stagingDirURI.toString(), SegmentGenerationUtils.PINOT_PLUGINS_TAR_GZ);
        outputDirFS.copyFromLocalFile(pluginsTarGzFile, cachedPluginsTarball.toUri());
        job.addCacheArchive(cachedPluginsTarball.toUri());

👍

I’ve also got a change to

addDepsJarToDistributedCache

, which has the same issue

I would expect the staging dir to be cleaned up

OK, I’ll fix that too…

Many thanks!

@Xiang Fu what’s the “push type” when doing a Hadoop batch segment generation? Asking because it looks like

NormalizedDateSegmentNameGenerator

thinks it’s not APPEND, so you don’t get any timestamp in the generated filename.

PushType is the way to push a segment to pinot, we have 3 right now,

TAR

upload the segment tar file to pinot controller;

URI

, save segment to deep store, notify controller th uri and let controller download and validate it .

METADATA

push both uri and metadata to avoid controller download

Hi @Xiang Fu I generated a PR, see https://github.com/apache/incubator-pinot/pull/6506

Thanks! I will take a look!