Hi Team, I was checking if pinot really logs audi...
# troubleshooting
m
Hi Team, I was checking if pinot really logs auditing as in
who did what, how, from which source and at what time
? Seems like the code base logs only the response and the type and not the request. It will be great if request is also being logged, so audit info is fully available. In code base : ControllerResponseFilter.java > LOGGER.info("Handled request from {} {} {}, content-type {} status code {} {}", srcIpAddr, method, uri, contentType, > respStatus, reasonPhrase); If this has requestContext is also added , I believe it should add request details with payload that is initially sent by the user, or if its disabled on purpose, do you mind giving that control to log4j that enduser can choose to enable it or not. I'm no developer 🥺, I'm trying the make sense of the code and see if it can be added . Where I'm coming from is: I just added a user via controller to have read,write permissions of a particular user on all tables. All I get is below.
Copy code
2025/11/03 20:30:59.922 INFO [ControllerResponseFilter] [grizzly-http-server-15] Handled request from 192.168.13.1 PUT <http://test-phaseroundtoaudit11.ori.com:9000/users/dedactid_rw?component=BROKER&passwordChanged=false|http://test-phaseroundtoaudit11.ori.com:9000/users/dedactid_rw?component=BROKER&passwordChanged=false>, content-type text/plain;charset=UTF-8 status code 200 OK
2025/11/03 20:30:59.957 INFO [ControllerResponseFilter] [grizzly-http-server-14] Handled request from 192.168.13.1 GET <http://test-phaseroundtoaudit11.ori.com:9000/tables|http://test-phaseroundtoaudit11.ori.com:9000/tables>, content-type null status code 200 OK
2025/11/03 20:30:59.980 INFO [ControllerResponseFilter] [grizzly-http-server-12] Handled request from 192.168.13.1 GET <http://test-phaseroundtoaudit11.ori.com:9000/users|http://test-phaseroundtoaudit11.ori.com:9000/users>, content-type null status code 200 OK
But its missing read,write has been given my admin user to ALL/particular tables. There is further granularity missing which is crucial I believe. Let me know your views. Thanks!!
y
@Suvodeep Pyne has been working on adding audit logging support recently, you can check out some of the PRs here - https://github.com/apache/pinot/pulls?q=is%3Apr+audit+log.
m
@Suvodeep Pyne my request will be taken care by this PR ? [audit] Implement request body handling with payload size limits and truncation support #16747 Please see if this PR will log auditing of
who did what, how, from which source and at what time
Also, one more suggestion, I See that once auth is enabled, i dont see as which user is using this portal. Ideally i will see for admin as logged in and logout button. This is a must have i feel once auth is enabled.
s
@Mannoj yes. both request and response filters exist. the PRs that @Yash Mayya shared have all the required changes. You can simply follow the configs here to play around with it. Note that these are cluster configs that can simply be changed with a REST call on the fly. Here are some examples:
Copy code
# Enable audit logging
pinot.audit.controller.enabled=true

# Capture request payloads
pinot.audit.controller.capture.request.payload.enabled=true

# Capture response information
pinot.audit.controller.capture.response.enabled=true

# Capture specific headers
pinot.audit.controller.capture.request.headers=Content-Type,X-Request-Id,Authorization,X-Correlation-Id

# Set payload size limit to 20KB
pinot.audit.controller.request.payload.size.max.bytes=20480

# Exclude health and metrics endpoints
pinot.audit.controller.url.filter.exclude.patterns=health,metrics,swagger.*,api-docs.*

# User identification from header
pinot.audit.controller.userid.header=X-User-Email

# Fallback to JWT claim if header not found
pinot.audit.controller.userid.jwt.claim=email
m
Thats really great @Suvodeep Pyne last two questions i have for you • which version onwards this change is in effect • Is there a provisioning to exclude certain users via rest call or dynamically from audit logging? As in if application users are logging heavily i would like disable them and enable whenever audit needs it.
s
thanks @Mannoj • This is relatively recent so
1.5.0
will be the release for this. • No, I'm afraid there is no provision to filter users. There is only provision to filter endpoints. I hope that was helpful.
m
Got it thanks for your response 👍🏻
👍 1