Is this normal when using opal with redis as broad...
# opal
m
Is this normal when using opal with redis as broadcast? I keep receiving Redis connection made, Redis connection lost
a
Hi @Marcelo Almeida, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
Frequent "Redis connection made, Redis connection lost" messages are not normal behavior. While OPAL does handle reconnecting automatically with exponential backoff (1) , repeated connection cycling indicates an underlying connectivity issue. Here are potential causes: ​*Network/Infrastructure Issues:*​ • Unstable network between OPAL and Redis • Redis server experiencing restarts or resource constraints • Firewall or network policies intermittently blocking connections ​*Configuration Issues:*​ • Incorrect
OPAL_BROADCAST_URI
or
OPAL_REDIS_URL
settings (2) • Redis authentication problems • Connection pool exhaustion ​*OPAL Configuration:*​ The broadcast keepalive settings might need adjustment. The
OPAL_BROADCAST_KEEPALIVE_INTERVAL
defaults to 3600 seconds (1 hour) , and
OPAL_BROADCAST_CONN_LOSS_BUGFIX_EXPERIMENT_ENABLED
is set to
True
by default to help with connection loss issues . ​*Troubleshooting Steps:*​ 1. Check Redis server logs for errors or restarts 2. Verify
OPAL_BROADCAST_URI
format (e.g.,
<redis://localhost>
) 3. Ensure Redis is stable and accessible 4. Review network connectivity between OPAL and Redis 5. Check if Redis has sufficient resources (memory, connections) Once OPAL reconnects, clients will follow the data instructions in
OPAL_DATA_CONFIG_SOURCES
to reach a baseline state for data (1) .
To learn more, see the sources I considered: [[1] Data updates](https://docs.opal.ac/faq/) [[2] OPAL Configuration Guide | OPAL](https://docs.opal.ac/getting-started/configuration)
d
Hi @Marcelo Almeida, the AI gave good troubleshooting steps. Please verify and let us know if you are still running into the issue. Is this the first time you are using Opal with Redis or these are new logs of an existing configuration?
m
Its first time, i've been using postgres notifier, but when the payload is too big the broadcast fails to deliver the notify
My setup
Copy code
name: opal-example

services:
  # When scaling the opal-server to multiple nodes and/or multiple workers, we use
  # a *broadcast* channel to sync between all the instances of opal-server.
  # Under the hood, this channel is implemented by encode/broadcaster (see link below).
  # At the moment, the broadcast channel can be either: postgresdb, redis or kafka.
  # The format of the broadcaster URI string (the one we pass to opal server as `OPAL_BROADCAST_URI`) is specified here:
  # <https://github.com/encode/broadcaster#available-backends>
  broadcast_channel:
    image: redis:6
    networks:
      - broadcast_channel
    healthcheck:
      test: ["CMD", "redis-cli", "-u", "<redis://127.0.0.1:6379>", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 5s
  opal_server:
    # by default we run opal-server from latest official image
    image: permitio/opal-server:latest
    environment:
      # the broadcast backbone uri used by opal server workers (see comments above for: broadcast_channel)
      - OPAL_BROADCAST_URI=<redis://broadcast_channel:6379/0>
      # number of uvicorn workers to run inside the opal-server container
      - UVICORN_NUM_WORKERS=4
      # the git repo hosting our policy
      # - if this repo is not public, you can pass an ssh key via `OPAL_POLICY_REPO_SSH_KEY`)
      # - the repo we pass in this example is *public* and acts as an example repo with dummy rego policy
      # - for more info, see: <https://docs.opal.ac/tutorials/track_a_git_repo>
      - OPAL_POLICY_REPO_URL=<https://github.com/permitio/opal-example-policy-repo>
      # in this example we will use a polling interval of 30 seconds to check for new policy updates (git commits affecting the rego policy).
      # however, it is better to utilize a git *webhook* to trigger the server to check for changes only when the repo has new commits.
      # for more info see: <https://docs.opal.ac/tutorials/track_a_git_repo>
      - OPAL_POLICY_REPO_POLLING_INTERVAL=30
      # configures from where the opal client should initially fetch data (when it first goes up, after disconnection, etc).
      # the data sources represents from where the opal clients should get a "complete picture" of the data they need.
      # after the initial sources are fetched, the client will subscribe only to update notifications sent by the server.
      - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"<http://opal_server:7002/policy-data>","topics":["policy_data"],"dst_path":"/static"}]}}
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      # - OPAL_TELEMETRY_ENDPOINT=<http://192.168.1.113:4318>
      - OPAL_TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT:-<http://otel-collector:4318>}
      - OPAL_TELEMETRY_ENABLED=${TELEMETRY_ENABLED:-false}
      - OPAL_TELEMETRY_USERNAME=${TELEMETRY_USERNAME:-local@goapice.com}
      - OPAL_TELEMETRY_PASSWORD=${TELEMETRY_PASSWORD:-telemetry}
    ports:
      # exposes opal server on the host machine, you can access the server at: <http://localhost:7002>
      - "7002:7002"
    networks:
     - otel-network
     - broadcast_channel
    depends_on:
      broadcast_channel:
        condition: service_healthy
  opal_client:
    # by default we run opal-client from latest official image
    image: permitio/opal-client:latest
    environment:
      - OPAL_SERVER_URL=<http://opal_server:7002>
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      - OPAL_INLINE_OPA_LOG_FORMAT=http
      - OPAL_HTTP_FETCHER_TIMEOUT=60
      - OPAL_FETCHING_CALLBACK_TIMEOUT=60
      - OPAL_FETCHING_ENQUEUE_TIMEOUT=60
      # - OPAL_TELEMETRY_ENDPOINT=<http://192.168.1.113:4318>
      - OPAL_TELEMETRY_ENDPOINT=${TELEMETRY_ENDPOINT:-<http://otel-collector:4318>}
      - OPAL_TELEMETRY_ENABLED=${TELEMETRY_ENABLED:-false}
      - OPAL_TELEMETRY_USERNAME=${TELEMETRY_USERNAME:-local@goapice.com}
      - OPAL_TELEMETRY_PASSWORD=${TELEMETRY_PASSWORD:-telemetry}

      # Uncomment the following lines to enable storing & loading OPA data from a backup file:
      # - OPAL_OFFLINE_MODE_ENABLED=true
    # volumes:
    #  - opa_backup:/opal/backup:rw

    networks:
      - otel-network
      - broadcast_channel

    ports:
      # exposes opal client on the host machine, you can access the client at: <http://localhost:7766>
      - "7766:7000"
      # exposes the OPA agent (being run by OPAL) on the host machine
      # you can access the OPA api that you know and love at: <http://localhost:8181>
      # OPA api docs are at: <https://www.openpolicyagent.org/docs/latest/rest-api/>
      - "8181:8181"
    depends_on:
      - opal_server
    # this command is not necessary when deploying OPAL for real, it is simply a trick for dev environments
    # to make sure that opal-server is already up before starting the client.
    command: sh -c "exec ./wait-for.sh opal_server:7002 --timeout=20 -- ./start.sh"
o
OPAL is not designed to pass large volumes of information in updates- updates should instead push URL for opal-client data fetchers to fetch the data directly
✅ 1