Hi Lightdash Team :wave: I have a feature request ...
# feature-requests
h
Hi Lightdash Team 👋 I have a feature request regarding BigQuery service account impersonation. Would it be possible to support an optional impersonated service account per BigQuery connection while Lightdash itself runs with ADC? We're exploring separate BigQuery connections for: • curated business-facing access • developer access with SQL Runner enabled and this would help us avoid managing multiple static service account keys while keeping permissions scoped per connection. Related issue: https://github.com/lightdash/lightdash/issues/3520 Thanks!
✅ 1
p
Hi Hiroki! Thanks for the feature request. ADC support for BigQuery was recently shipped (#15054), but per-connection service account impersonation on top of ADC isn't supported yet. The issue you linked (#3520) tracks this - I'll link your request there so we can track the demand. In the meantime, the closest workaround is separate Lightdash projects with distinct BigQuery connections, each using ADC where the underlying compute service account has different permissions. Not as clean as per-connection impersonation, but it lets you scope access differently for business vs developer use cases.
c
I've upvoted these issues for you: 🛠️ issue #15054 - this issue has been requested by 1 other user. 🛠️ issue #3520 - this issue has been requested by 1 other user. I'll notify everyone here as soon as any of these issues are fixed.
h
Thank you 🙏
👍 1
@Tori Whaley Hi, sorry for the late follow-up. I was revisiting this and wanted to clarify one point about the workaround you suggested. If multiple Lightdash projects are running on the same Lightdash instance and each BigQuery connection uses ADC, wouldn’t they all use the same underlying compute service account? If so, separating Lightdash projects alone wouldn’t give us different IAM identities for business and developer use cases. My understanding is that we’d need either per-connection service account impersonation, or separate Lightdash deployments with different runtime service accounts. Is that correct?
👀 1
t
Hi Hiroki, you're right - with ADC, Lightdash doesn't pass any explicit credentials to the BigQuery client, so every connection on the same instance resolves to the same runtime service account. Separate projects alone won't give you different IAM identities, so my earlier suggestion doesn't solve this particular point. Sorry for the confusion there 🙏 Short of going back to static per connection keys, the options today are per connection impersonation (not supported yet, tracked in #3520) or separate Lightdash deployments each running with its own runtime service account. Your ADC use case is already captured on that issue from our conversation in May, so following it is the best way to hear when there's progress.
h
@Tori Whaley Thanks, I’d be interested in contributing this if that would be helpful. I took another look at #3520 and noticed the original request also covers service-account-key impersonation, as well as a CLI/dbt profiles.yml use case. If you’d be open to a contribution here, I was thinking of keeping the initial scope focused on: • ADC as the source credential • an optional impersonated service account per BigQuery connection • configuration through the Lightdash UI I’d leave service-account-key impersonation and CLI/dbt profile propagation out of scope initially. For the data model, I was thinking of using a generic impersonateServiceAccount field so the other authentication paths could potentially be supported later. Does that sound like a reasonable scope and approach? If you prefer, I’d also be happy to open a separate issue for the ADC-only case rather than treating it as the full scope of #3520.
g
Hey Hiroki! Thanks for thoughtfulness on these changes, scoping sounds solid, and deeply appreciate the willingness to tackle this pocket of work. We've recently decided to no longer accept new external contributions for the time being though. Agents have brought so much change, and as much as we cherish developing back-and-forth with OSS community, the tradeoffs don't make as much sense now. It's in no way a reflection on the code we expect from you or any other contributor, I'm quite certain the PR in this case would be very high quality — we've just had to institute it as a blanket policy due to the speed/security/complexity of the internal agentic systems we're now operating set against the volume and potential risks of external contributions in this new era. github.com/lightdash/lightdash?tab=contributing-ov-file We will keep you posted if/when this gets picked up and is motion, and please continue to open issues and flag improvements that you'd like to see — it remains extremely helpful to get outside perspective from people who put as much thought into the tool as you have here. Thank you again. 🙏
👍 1
h
Thanks for the context, that makes sense. I completely understand the policy, and I appreciate you taking the time to explain it. I’m glad the use case and proposed scope were helpful. I’ll keep an eye on #3520 and will continue to share feedback or additional context there if anything comes up. Thanks again!
🙏 2
neon purple heart 2
shiba purple heart 2