Hi, has anyone got any suggestion on how to make b...
# troubleshootinga
Alice
06/29/2022, 3:27 AMHi, has anyone got any suggestion on how to make backfill job easier? Any tool or doc would help. Thanks.
m
Mayank
06/29/2022, 3:30 AM
One pattern I have seen work is to have single job that takes as input start day/time + look-back period and does the build and push. Using this approach, daily job has start day as latest and look-back period as 1, where as backfill jobs can configure those values based on what needs to be back filled
a
Alice
06/29/2022, 3:48 AMThanks. Good suggestion.
Alice
06/29/2022, 3:51 AMFor backfill scenario, any method to generate and upload segments instead of through a job spec yaml file?
m
Mayank
06/29/2022, 3:49 PM
Yaml file is for convenience. What are you looking for?
a
Alice
06/29/2022, 8:53 PMI expected a Swagger API before. But I think I just found 2 controller apis, namely /ingestFromFile and /ingestFromURI, and could do the same job as a job spec. It’s ok now.
m
Mayank
06/30/2022, 3:28 AM
Note, those controller api’s are mostly to try out during PoC. Please don’t use them for large data size in produciton.