Hey! I'm using fb tail input to read data from a ...
# fluent-bit
s
Hey! I'm using fb tail input to read data from a file and using a custom output to upload the data to Azure blob storage (I don't want to use the built-in Azure blob storage output). I'm using these fb configurations:
Copy code
[SERVICE]
    Flush                 60
    HTTP_Server           Off
    Daemon                Off
    Log_Level             warn
    Log_File              <log file path>
    storage.path          <storage path>
    storage.type          filesystem
    storage.checksum      On
    storage.max_chunks_up 60

[INPUT]
    Name tail
    Tag <tag>
    Path <path>
    DB <db>
    Key Data
    Path_Key filepath
    Skip_Long_Lines On
    Read_from_Head True
    storage.type filesystem

[OUTPUT]
    Name            myoutput
    Retry_Limit     false
    Match           <tag>
I have few questions that I couldn't answer from fb docs and I would love to get your help: 1. What controls the scheduler? i.e. when does the engine decides to flush the data to output (in my case "myoutput") 2. What controls how much data is being flushed each time? 3. My goal is to have a flush one the first of these two conditions hold: 1) There's ~250Kib of records ready to be flushed. 2) The 1 minute timeout reached. Is it possible to do with FB configuration?
e
1. upon flush interval (every 60 seconds --> your default is too high) 2. how much data the input plugins has ingested and how that is appended to a chunk 3. not possible in that way
s
Appreciate the response. Thanks