https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# opal
s
This message was deleted.
a
@Raz Co can you assist here please?
r
Hey Naveen, I would love to help you with your K8s deployment! Want to share more about your needs?
n
@Raz Co As OPAL client can be loaded as sidecar. I wanted to have OPAL server instances on multi-nodes. So, I used daemon-set whenever a new node created we can spin up new instance. I need some assistance on how can we create achieve this.
r
So have you created the Daemon-set already ? Or do you need help with configuring the opal-client as a sidecar ?
Here’s an example for a Daemon-set - Pay attention that you can change the environment variables.
Copy code
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: opal-server
  labels:
    app: opal-server
spec:
  selector:
    matchLabels:
      octopusexport: OctopusExport
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: opal-server
        octopusexport: OctopusExport
    spec:
      containers:
        - name: opal-server
          image: 'permitio/opal-server:latest'
          imagePullPolicy: Always
          ports:
            - name: server
              containerPort: 7002
              protocol: TCP
          env:
            - name: NEW_ENVIRONMENT_VARIABLE
              value: VALUE_TO_REPLACE
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - web
                topologyKey: <http://kubernetes.io/hostname|kubernetes.io/hostname>
To create Opal-client as a sidecar, what you’ll have to do is basically add new container under 
.spec.template.spec.containers[]
and this will allow you to create multiple containers under one pod.
n
@Raz Co I was able to achieve multi node deployment in kubernetes without helm. But getting this error while there is a webhook update
Failed to notify subscriber sub_id=622eeb975267492fbbeaa44a5035ba13 with topic=webhook
Traceback (most recent call last): File "/usr/local/bin/gunicorn", line 33, in <module> sys.exit(load_entry_point('gunicorn==20.1.0', 'console_scripts', 'gunicorn')()) │ │ └ <function importlib_load_entry_point at 0x7f46c34a6cb0> │ └ <built-in function exit> └ <module 'sys' (built-in)> File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() └ <class 'gunicorn.app.wsgiapp.WSGIApplication'> File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 231, in run super().run() File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run Arbiter(self).run() │ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f46c3487ee0> └ <class 'gunicorn.arbiter.Arbiter'> File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 211, in run self.manage_workers() │ └ <function Arbiter.manage_workers at 0x7f46c294ad40> └ <gunicorn.arbiter.Arbiter object at 0x7f46c247f700> File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 551, in manage_workers self.spawn_workers() │ └ <function Arbiter.spawn_workers at 0x7f46c294ae60> └ <gunicorn.arbiter.Arbiter object at 0x7f46c247f700> File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers self.spawn_worker() │ └ <function Arbiter.spawn_worker at 0x7f46c294add0> └ <gunicorn.arbiter.Arbiter object at 0x7f46c247f700> File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker worker.init_process() │ └ <function UvicornWorker.init_process at 0x7f46c0f540d0> └ <uvicorn.workers.UvicornWorker object at 0x7f46c24bc460> File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process super(UvicornWorker, self).init_process() │ └ <uvicorn.workers.UvicornWorker object at 0x7f46c24bc460> └ <class 'uvicorn.workers.UvicornWorker'> File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in init_process self.run() │ └ <function UvicornWorker.run at 0x7f46c0f54280> └ <uvicorn.workers.UvicornWorker object at 0x7f46c24bc460> File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 83, in run return asyncio.run(self._serve()) │ │ │ └ <function UvicornWorker._serve at 0x7f46c0f541f0> │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f46c24bc460> │ └ <function run at 0x7f46c24f9870> └ <module 'asyncio' from '/usr/local/lib/python3.10/asyncio/__init__.py'> File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) │ │ └ <coroutine object UvicornWorker._serve at 0x7f46bdced310> │ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects> └ <uvloop.Loop running=True closed=False debug=False>
File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 220, in callback_subscribers
await self.trigger_callback(data, topic, subscriber_id, event) │ │ │ │ │ └ Subscription(id='38980dbb3b484119a7d4eee8d454a4c0', subscriber_id='622eeb975267492fbbeaa44a5035ba13', topic='webhook', callba... │ │ │ │ └ '622eeb975267492fbbeaa44a5035ba13' │ │ │ └ 'webhook' │ │ └ None │ └ <function EventNotifier.trigger_callback at 0x7f46c0789cf0> └ <fastapi_websocket_pubsub.websocket_rpc_event_notifier.WebSocketRpcEventNotifier object at 0x7f46bdccf370> File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_pubsub/event_notifier.py", line 178, in trigger_callback await subscription.callback(subscription, data) │ │ │ └ None │ │ └ Subscription(id='38980dbb3b484119a7d4eee8d454a4c0', subscriber_id='622eeb975267492fbbeaa44a5035ba13', topic='webhook', callba... │ └ <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster ob... └ Subscription(id='38980dbb3b484119a7d4eee8d454a4c0', subscriber_id='622eeb975267492fbbeaa44a5035ba13', topic='webhook', callba... File "/usr/local/lib/python3.10/site-packages/fastapi_websocket_pubsub/event_broadcaster.py", line 159, in __broadcast_notifications__ async with self._sharing_broadcast_channel: │ └ <broadcaster._base.Broadcast object at 0x7f46bd2c6290> └ <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7f46bdccf3d0> File "/usr/local/lib/python3.10/site-packages/broadcaster/_base.py", line 55, in aenter await self.connect() │ └ <function Broadcast.connect at 0x7f46c078a560> └ <broadcaster._base.Broadcast object at 0x7f46bd2c6290> File "/usr/local/lib/python3.10/site-packages/broadcaster/_base.py", line 62, in connect await self._backend.connect() │ │ └ <function PostgresBackend.connect at 0x7f46bd2c2e60> │ └ <broadcaster._backends.postgres.PostgresBackend object at 0x7f46bd2c6a10> └ <broadcaster._base.Broadcast object at 0x7f46bd2c6290> File "/usr/local/lib/python3.10/site-packages/broadcaster/_backends/postgres.py", line 15, in connect self._conn = await asyncpg.connect(self._url) │ │ │ │ └ 'postgres://postgres:postgres@broadcast-channel:5432/postgres' │ │ │ └ <broadcaster._backends.postgres.PostgresBackend object at 0x7f46bd2c6a10> │ │ └ <function connect at 0x7f46bd2025f0> │ └ <module 'asyncpg' from '/usr/local/lib/python3.10/site-packages/asyncpg/__init__.py'> └ <broadcaster._backends.postgres.PostgresBackend object at 0x7f46bd2c6a10> File "/usr/local/lib/python3.10/site-packages/asyncpg/connection.py", line 2093, in connect return await connect_utils._connect( │ └ <function _connect at 0x7f46bd2a0670> └ <module 'asyncpg.connect_utils' from '/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py'> File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 903, in _connect raise last_error └ gaierror(-2, 'Name or service not known') File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 889, in _connect return await _connect_addr( └ <function _connect_addr at 0x7f46bd2a0550> File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 781, in _connect_addr return await __connect_addr(params, timeout, True, *args) │ │ │ └ (('broadcast-channel', 5432), <uvloop.Loop running=True closed=False debug=False>, ConnectionConfiguration(command_timeout=No... │ │ └ 60 │ └ ConnectionParameters(user='postgres', password='postgres', database='postgres', ssl=<ssl.SSLContext object at 0x7f46bcfcd0c0>... └ <function __connect_addr at 0x7f46bd2a05e0> File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 833, in __connect_addr tr, pr = await compat.wait_for(connector, timeout=timeout) │ │ │ └ 60 │ │ └ <Task finished name='Task-88' coro=<_create_ssl_connection() done, defined at /usr/local/lib/python3.10/site-packages/asyncpg... │ └ <function wait_for at 0x7f46bd2031c0> └ <module 'asyncpg.compat' from '/usr/local/lib/python3.10/site-packages/asyncpg/compat.py'> File "/usr/local/lib/python3.10/site-packages/asyncpg/compat.py", line 66, in wait_for return await asyncio.wait_for(fut, timeout) │ │ │ └ 60 │ │ └ <Task finished name='Task-88' coro=<_create_ssl_connection() done, defined at /usr/local/lib/python3.10/site-packages/asyncpg... │ └ <function wait_for at 0x7f46c2556dd0> └ <module 'asyncio' from '/usr/local/lib/python3.10/asyncio/__init__.py'> File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() │ └ <method 'result' of '_asyncio.Task' objects> └ <Task finished name='Task-88' coro=<_create_ssl_connection() done, defined at /usr/local/lib/python3.10/site-packages/asyncpg... File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 692, in _create_ssl_connection tr, pr = await loop.create_connection( │ └ <method 'create_connection' of 'uvloop.loop.Loop' objects> └ <uvloop.Loop running=True closed=False debug=False> File "uvloop/loop.pyx", line 1978, in create_connection ai_remote = f1.result()
r
Hey @Naveen, look at this part of the stack -
Copy code
File "/usr/local/lib/python3.10/site-packages/asyncpg/connect_utils.py", line 903, in _connect
    raise last_error
          └ gaierror(-2, 'Name or service not known')
It means that the container can’t reach the broadcast. If you are running in k8s, you wouldn’t be able to reach the broadcast or any other service like this. Does this broadcast has a k8s service? If it does, you can put the FQDN of this service, which will look like - 
<service_name>.<namespace>.svc:<port>
.
n
containers:
- name: opal-server image: permitio/opal-server:latest ports: - containerPort: 7002 env: - name: OPAL_BROADCAST_URI value: "postgres://postgres:postgres@broadcast-channel:5432/postgres" - name: UVICORN_NUM_WORKERS value: "3" - name: OPAL_POLICY_REPO_URL value: "<<>>" - name: OPAL_DATA_CONFIG_SOURCES value: '{"config":{"entries":[{"url":"http://opal-server:7002/policy-data","topics":["policy_data"],"dst_path":"/static"}]}}' - name: OPAL_LOG_FORMAT_INCLUDE_PID value: "true" - name: OPAL_POLICY_REPO_MAIN_BRANCH value: "master" - name: OPAL_POLICY_REPO_WEBHOOK_SECRET value: "<<>>" - name: broadcast-channel image: postgres:10.4 imagePullPolicy: "IfNotPresent" ports: - containerPort: 5432 envFrom: - configMapRef: name: postgres-config restartPolicy: Always
The above one is the container configuration as I am loading this as a sidecar do we need FQDN service name?
r
Yeah, K8s can’t resolve this dns.
n
I also have a question on this as we are deploying postgres as sidecar how opal-server will work as we will also have multiple instances of postgres from the design point of view I am sensing something off here
r
Why did you decide to deploy postgres as sidecar ?
n
@Raz Co Below is the opal-server yaml file
Copy code
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: opal-server-daemonset
  namespace: opal-impl
  labels:
    app: opal-server-daemonset
spec:
  selector:
    matchLabels:
      name: opal-server-daemonset
  template:
    metadata:
      labels:
        name: opal-server-daemonset
    spec:
      containers:
      - name: opal-server
        image: permitio/opal-server:latest
        ports:
        - containerPort: 7002
        env:
        - name: OPAL_BROADCAST_URI
          value: "<postgres://postgres:postgres@broadcast-channel-service.opal-impl.svc:5432/postgres>"
        - name: UVICORN_NUM_WORKERS
          value: "3"
        - name: OPAL_POLICY_REPO_URL
          value: ""
        - name: OPAL_DATA_CONFIG_SOURCES
          value: '{"config":{"entries":[{"url":"<http://opal-server:7002/policy-data>","topics":["policy_data"],"dst_path":"/static"}]}}'
        - name: OPAL_LOG_FORMAT_INCLUDE_PID
          value: "true"
        - name: OPAL_POLICY_REPO_MAIN_BRANCH
          value: "master"
        - name: OPAL_POLICY_REPO_WEBHOOK_SECRET
          value: ""
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broadcast-channel
  namespace: opal-impl
  labels:
    app: broadcast-channel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broadcast-channel
  template:
    metadata:
      labels:
        app: broadcast-channel
    spec:
      containers:
      - name: broadcast-channel
        image: postgres:10.4
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 5432
        resources:
          limits:
            memory: 512Mi
            cpu: "1"
          requests:
            memory: 256Mi
            cpu: "0.2"
        envFrom:
          - configMapRef:
             name: postgres-config
      restartPolicy: Always
status: {}
These are the logs I am getting
logs.txt
r
Hey @Naveen, It looks like the container still can’t connect to your postgres. Can you share your service yaml ?
n
Sure
Copy code
apiVersion: v1
kind: Service
metadata:
  labels:
    app: broadcast-channel-service
  name: broadcast-channel-service
spec:
  ports:
    - name: postgres-port
      protocol: TCP
      port: 5432
      targetPort: 5432
  selector:
    app: broadcast-channel
  type: LoadBalancer
r
Now it looks like an issue with the postgres itself, as it looks like the dns resolved, but the postgres instance refuses the connection. Are you sure the credentials you passed are right ?
n
yes, That is what I have used from config map
@Raz Co I sorted the issue, Now I am able to broadcast data but in my opal client I was not receiving updated policy
2 Views
Open in Slack
PreviousNext
Powered by