https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • m
    @User do we not support a hybrid situation where index is built in one system (say HDFS), but controller data dir is Gcs?
  • j
    it's never been tested so there may be some problems with that
  • j
    but it should be supported
  • j
    if there are issues, feel free to reach out
  • m
    Ok, so we derive scheme from upload uri, right?
  • j
    yes, basically where the segment is currently located
  • j
    Copy code
    SegmentFetcherFactory.getInstance().getSegmentFetcherBasedOnURI(currentSegmentLocationURI)
        .fetchSegmentToLocal(currentSegmentLocationURI, tempEncryptedFile);
  • m
    @User You can use the same endpoint to push the segment from Gcs. You just need to specify the uri with upload.
  • m
    This is assuming there's a PinotFs implementation for Gcs.
  • e
    There is... almost:) - I am working on the feature, that's why I have all these questions :)
  • a
    @User documentations has an example:
  • a
    Copy code
    curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:<hdfs://nameservice1/hadoop/path/to/segment/file.gz>" -H "content-type:application/json" -d '' localhost:9000/segments
  • a
    what ios DOWNLOAD_URI: part?
  • j
    current location of segment in deep storage
  • a
    and ”
    Copy code
    :<hdfs://nameservice1/hadoop/path/to/segment/file.gz>
    ?
  • j
    DOWNLOAD_URI is a static field
  • j
    the second portion is where your segment is 🙂
  • a
    ah, got it 🙂
  • a
    just a thought, maybe less confusing to have an api like: POST localhost:9000/segments/uri with a header that indicates whether the destination is compressed (and which compressions) or just a plain folder .
    👍 1
  • e
    FYI, got everything working!
  • e
    Can upload a tgz file from gcs
    🎉 2
  • e
    Will add a docs and update pull request
  • e
    So next question:) How do we store segments on the controller, i.e. that "deep store"? Is there a way to back up offline segments to gcs also?
  • k
    yes.
    👍 2
  • h
    Master seems broken. Sent a diff hoping to fix it: https://github.com/apache/incubator-pinot/pull/4920
  • h
    set the channel topic: Community-wide announcements and work-based matters
  • m
    @User ^^
  • h
  • h
    Should be good now. Thanks!
  • p
    i'm looking for "like" expression... I've found "regexp_like", we have others functions to handle string "contains" operation?
1...106107108...160Latest