Is there anyway to set a variable dynamically to maintain some kind of state? Like, I want to associate a stack trace with the application in our APM and have the APM identifier as a field. I don't know the identifier at startup, but it doesn't change for the lifetime of the application, so I could get it from another log line. That sort of functionality doesn’t exist, does it?

You could probably vibe code up a quick lua filter for this: https://docs.fluentbit.io/manual/data-pipeline/filters/lua Imagine you could persist it to disk or in memory on first seen and then read from there for subsequent events but not sure how performance would be there

It sounds like you want record accessor syntax in some form but not sure exactly the use case you mean. If you can set a key from something then you can use that key elsewhere

+1 on using Lua although this will not work when using mulitthreading as each thread will have it's own

local

storage

pipeline:
  inputs:
    - alias: http_input
      listen: 0.0.0.0
      name: http
      port: 8084
      processors:
        logs:
          - name: lua
            call: "last_log"
            code: |
              local memory = "None"
              function last_log(tag, timestamp, record)
                record["memory"] = memory
                memory = record["message"]
                return 1, timestamp, record
              end
  outputs:
    - name: stdout
      match_regex: '.*'

+

curl -k \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello"}' \
  <http://localhost:8084>
curl -k \
  -H "Content-Type: application/json" \
  -d '{"message": "World"}' \
  <http://localhost:8084>
curl -k \
  -H "Content-Type: application/json" \
  -d '{"message": "Password"}' \
  <http://localhost:8084>

Response:

[0] http.0: [[1777380706.099247217, {}], {"message"=>"Hello", "memory"=>"None"}]
[1] http.0: [[1777380706.109482526, {}], {"message"=>"World", "memory"=>"Hello"}]
[2] http.0: [[1777380706.118885040, {}], {"message"=>"Password", "memory"=>"World"}]

Just curious, were there any reports or fixes around "increased (and growing) memory usage" btw v4.2.2 (.55.0 helm) and v5.0.6 (.57.6 helm) - we are seeing what looks to be a considerable difference and it is growing consistently.