cuddly-beach-83988
04/05/2023, 7:14 PMacryldata/datahub-actions:head found the mssql-venv and phyiscally just install pyodbc, but i still require the driver file and unfortuantly the local datahub user in the container does not have sudo permissions so I can't just install it via apt-get.
sigh can anyone please point me in the right direction?lively-cat-88289
04/05/2023, 7:14 PMcuddly-beach-83988
04/05/2023, 8:17 PMcuddly-beach-83988
04/05/2023, 8:19 PMdatahub docker quickstart option. Instead locally clone the repo.
git clone https://github.com/datahub-project/datahub && cd datahub/docker && touch Dockerfile
Create an empty Dockerfile. The node that is responsible for executing the backend is datahub-actions so build a new image on top of that.
FROM acryldata/datahub-actions:head
USER root
# Install necessary tools and libraries
RUN apt-get update && \
apt-get install -y curl build-essential unixodbc-dev
# Install the Microsoft ODBC Driver 17 for SQL Server
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
apt-get update && \
ACCEPT_EULA=Y apt-get install -y msodbcsql17
USER datahub
Now in the given docker-compose.yml file you will need to make the following changes when defining the datahub-actions service:
datahub-actions:
# image: acryldata/datahub-actions:${ACTIONS_VERSION:-head}
build:
context: .
dockerfile: Dockerfile
hostname: actions
env_file: datahub-actions/env/docker.env
restart: on-failure:5
depends_on:
- datahub-gms
Finally, you can now docker-compose up and it should build your new image and replace it with an identical version of datahub-actions with the required ODBC driver on the container.
Not quite done yet. I haven't gone to deep here, but when you create an integration with MSSQL, looking at the logs, it seems to create a virtual environment for that source. I am not sure where that requirements.txt and I am sure you could go find it and embed that into this whole build process, but I found its easier (especially for a demo or something) to just exec-ed into the container and manually install pyodbc because even if you alter your new dockerfile to include pyodbc, it will be system installed and will not be visible to the venv that is used by the mssql source connector.
So I did this by simply doing the following once all containers are up and running.
docker exec -it docker_datahub_actions_1 bash
Once inside the container: cd /tmp/datahub/ingest/venv-mssql-0.10.1/ && ./pip install pyodbc && exit
And boom.. now you should be able to connect to MSSQL databases and utilize the ODBC option in source. The connector will have pyodbc installed and will find all the relevant binary files for ODBC driver.
Hopefully the team at Datahub by default already include the latest SQL drivers in ther datahub-actions images :)astonishing-answer-96712
04/07/2023, 9:46 PMrapid-airport-61849
04/19/2023, 12:22 PMastonishing-byte-5433
04/24/2023, 3:18 PMdocker exec -u 0 -i datahub-datahub-actions-1 bin/bash < <path>/install-oracle.sh
Nevertheless would be nicer to be able to install the oracle / microsoft prequerities for using UI Ingestion through the official deployment methods without customizing the container or needing to install them afterwards.
We are running in a same problem now with MSSQL and Datahub on Kubernetes and maybe need to adapt your solution to get this running, so thanks for sharing!chilly-potato-57465
06/20/2023, 2:19 PM