Hello there, After ingesting a file into pinot, I am able to see all the segments, but in a bad state. There were no errors while ingesting the file. However, the table is still empty. I can see the segment in the controllerUI also, they are properly created in the output folder. I see this error in the zooKeeper browser>instances>server>errors

{
  "id": "100056e21a30006__lineorder_star_OFFLINE",
  "simpleFields": {},
  "mapFields": {
    "HELIX_ERROR     20220402-222348.000619 STATE_TRANSITION 631528bd-ca2b-4b60-b66e-01cfa58a44ff": {
      "AdditionalInfo": "Exception while executing a state transition task lineorder_star_OFFLINE_1992-01-01_1998-08-02_1java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts\n\tat org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61)\n\tat org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:72)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:148)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:142)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:164)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:158)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:406)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:393)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:385)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:372)\n\tat org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:355)\n\tat org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162)\n\t... 12 more\n",
      "Class": "class org.apache.helix.messaging.handling.HelixStateTransitionHandler",
      "MSG_ID": "4dab2e7f-f693-4459-9cd6-6e837fa32c5b",
      "Message state": "READ"
    },
    "HELIX_ERROR     20220402-222348.000679 STATE_TRANSITION bca29fad-4930-48f8-b626-3013f465b1c8": {
      "AdditionalInfo": "Message execution failed. msgId: 4dab2e7f-f693-4459-9cd6-6e837fa32c5b, errorMsg: java.lang.reflect.InvocationTargetException",
      "Class": "class org.apache.helix.messaging.handling.HelixStateTransitionHandler",
      "MSG_ID": "4dab2e7f-f693-4459-9cd6-6e837fa32c5b",
      "Message state": "READ"
    }
  },
  "listFields": {}
}

Can you share the server side logs?

Sure, here are the server logs

What are you using as deep-store for controller? And is that configured correctly?

I am using the local file system.

className: org.apache.pinot.spi.filesystem.LocalPinotFS

How much storage do you have in the controller data dir? Check if that is full, or somehow deleted

aah yes it is not the /tmp directory, that is just the logical dir name used by docker. the segments are in a directory inside the /home directory. After the ingestion, I have 115GB left on my system. I had uploaded 75 segments each of about 450mb. and I see 75 zipped segments in the output directory which, on extracting would be about 700mb each. So seems like the segments were created properly but somehow they remain in the bad state.

docker run --rm -ti \
      --name pinot-ingestion-job \
      --network=pinot-demo_default \
      --mount type=bind,source=/home/shailee/projects/DS/data/star,target=/tmp \
      apachepinot/pinot:latest LaunchDataIngestionJob \
      -jobSpecFile /tmp/starIngestionJobSpec.yaml

The tmp here is

type=bind,source=/home/shailee/projects/DS/data/star,target=/tmp

lineorder_star_OFFLINE_1992-01-01_1998-08-02_6 for table: lineorder_star_OFFLINE from: file:/tmp/segments/lineorder_star_OFFLINE_1992-01-01_1998-08-02_6.tar.gz to: /tmp/data/pinotServerData/lineorder_star_OFFLINE/tmp/tmp-lineorder_star_OFFLINE_1992-01-01_1998-08-02_6-487fd6dd-dd9b-4a5e-a9ba-ead886a4fd1a/lineorder_star_OFFLINE_1992-01-01_1998-08-02_6.tar.gz