Can in opal client read from a data source set wit...
# opal
n
Can in opal client read from a data source set with a timeout
a
Hi @Nishanth Mathew Joy, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
Yes, OPAL Client can read from data sources with a configurable timeout. This is controlled by the
OPAL_DATA_UPDATER_CONN_RETRY
configuration (1) . The configuration accepts a JSON object that specifies retry options when connecting to data sources :
Copy code
{
  "wait_strategy": "random_exponential",
  "max_wait": 10,
  "attempts": 5,
  "wait_time": 1
}
This configuration controls retry behavior when the OPAL Client connects to base data sources (such as external API servers that return data snapshots) . Additionally, when using the HTTPX client provider (
OPAL_HTTP_FETCHER_PROVIDER_CLIENT=httpx
) , there's now support for specifying HTTP timeouts via environment variables, which was added in OPAL v0.8.2 (2) . Previously, the default timeout of 5 seconds was always used .
To learn more, see the sources I considered: [[1] OPAL Configuration Guide | OPAL](https://docs.opal.ac/getting-started/configuration) [[2] Add option to OPAL Client for specifying HTTPX client timeout](https://github.com/permitio/opal/issues/756)
t
@Nishanth Mathew Joy AI gave the right answer. Does that make sense?
n
Do this work with a custom fetcher as well ?
a
Thanks for the feedback, glad I was helpful.
a
@Ask Inkeep
a
Thanks for tagging me @Alon Boshi. I'll look through relevant sources and get back to you shortly.
Yes, timeouts work with custom fetchers as well. When you write a custom fetch provider, you have full control over the timeout behavior since you implement the fetching logic yourself (1) . In your custom fetch provider, you handle timeouts in the
__aenter__
method where you initialize connections . For example, when using asyncio-ready libraries like
asyncpg
for database connections, you can specify connection parameters including timeouts :
Copy code
async def __aenter__(self) :
    # initializing parameters from the event/config
    self._event: PostgresFetchEvent
    dsn: str = self._event.url
    connection_params: dict = {} if self._event.config.connection_params is None else self._event.config.connection_params.dict(exclude_none=True)
    
    # connect to the postgres database
    self._connection: asyncpg.Connection = await asyncpg.connect(dsn, **connection_params)
    
    # start a readonly transaction (we don't want OPAL client writing data due to security!)
    self._transaction: Transaction = self._connection.transaction(readonly=True)
    await self._transaction.__aenter__()
    
    return self
Since custom fetch providers should use asyncio-ready libraries instead of blocking libraries , you can configure timeouts directly in whatever async library you're using for your specific data source.
To learn more, see the sources I considered: [[1] How to extend OPAL with custom Fetch Providers](https://docs.opal.ac/tutorials/write_your_own_fetch_provider)
Thanks for the feedback, glad I was helpful.