Hi Team, We are using fluentbit v3.2.2. we are hit...
# fluent-bit
o
Hi Team, We are using fluentbit v3.2.2. we are hitting is the sqlite max number of bind variables SQLITE_MAX_VARIABLE_NUMBER
Copy code
Maximum Number Of Host Parameters In A Single SQL Statement

A host parameter is a place-holder in an SQL statement that is filled in using one of the sqlite3_bind_XXXX() interfaces. Many SQL programmers are familiar with using a question mark ("?") as a host parameter. SQLite also supports named host parameters prefaced by ":", "$", or "@" and numbered host parameters of the form "?123".

Each host parameter in an SQLite statement is assigned a number. The numbers normally begin with 1 and increase by one with each new parameter. However, when the "?123" form is used, the host parameter number is the number that follows the question mark.

SQLite allocates space to hold all host parameters between 1 and the largest host parameter number used. Hence, an SQL statement that contains a host parameter like ?1000000000 would require gigabytes of storage. This could easily overwhelm the resources of the host machine. To prevent excessive memory allocations, the maximum value of a host parameter number is SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.

The maximum host parameter number can be lowered at run-time using the sqlite3_limit(db,SQLITE_LIMIT_VARIABLE_NUMBER,size) interface.

so the 32k limit is there, but we have so many hosts that we are already reducing the size of the directory, but if something happens like a problem or maintenance the this would happen so
Reference: https://www.sqlite.org/limits.html I would greatly appreciate if someone could point me in the direction how I could debug it! Thanks!
s
@Pat can you help us with this?
p
I'm not sure what you need, the code is all there if you want to look
s
Can we somehow change the internal db of fluentbit? As sqlite has certain limitations mentioned in the above message
p
There is no existing configuration to do that
Is it an in-memory limitation or a per-db limitation? If it's per-db then you could split up your inputs to use separate dbs
o
This is per DB limitiation.
p
so make more DBs?