Sidenote, I just tried to implement the readiness probe in my k8s config, but the ready endpoint

/v1/data/system/opal/healthy

returns HTTP 200 with a content of

{"result":false}

when it's not ready - this means this endpoint can't be used as a http readinessProbe target since it treats HTTP 200 as "ready"

I've worked around this with a derived Dockerfile that installs

curl

which enables me to configure a readinessProbe like so:

readinessProbe:
  exec:
    command:
    - sh
    - -c
    - curl -s <http://localhost:8181/v1/data/system/opal/healthy> | grep 'true'
  failureThreshold: 10
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5

This doesn't solve the problem though - it just means I can now correctly identify the application pod as unsuitable for usage because the OPA data hasn't been populated.

So I think I understand the issue more now - the reason it gives up after 10 seconds is due to the default value of

OPAL_FETCHING_CALLBACK_TIMEOUT

- it doesn't care if it succeeds after this time which is why the OPA transaction fails. However, I think I have a bigger problem. As I mentioned earlier some of my services with OPAL Client sidecar containers are the source of part of their own authorization data - I have the fetcher config configured with k8s service addresses, which causes a problem for the OPAL Client sidecar that needs to fetch from the service within the same pod as it. When doing a new deployment the cluster service DNS is still pointing to the previous instance of the pod because the new one isn't ready yet. I think I might need to think about using topics to allow those services to use

localhost

instead of a cluster dns address. Perhaps I'm doing something more fundamentally wrong with this design though 😅

I've solved the problem with a hack - by using

hostAliases

I can configure a pod to resolve its own service fqdn to 127.0.0.1:

hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "service-a.multi-tenant.svc.cluster.local"

This means that OPAL Server can give out the same service FQDN to service A, service B, and service C, but service A won't be dependent on its own service fqdn (which won't be available until that pod's OPAL Clients livenessProbe succeeds). This kind of setup doesn't seem like it would be unique to what we're doing so I'd be interested to know if there's an obvious design issue here. To give a bit more context Service A is responsible for serving user roles/permissions - these are used as policy data which is fed into OPA via OPAL. Service B has ABAC-style policy data that is also fed into OPA via OPAL. And then both of those services' public APIs have policy checks using OPA from their sidecar. I have a cluster-internal API on both of these services that serves data in the correct JSON format to store into OPA via OPAL. Is this an unusual setup? It doesn't seem particularly unusual but anyone using this kind of setup would surely hit the same issue of a service needing a sidecar that needs data from itself?

@Raz Co do you know if the above issue of failed healthchecks returning HTTP 200 is planned to be fixed?

Also, sorry for missing your last message, do you still need our help with this?