EDIT - Found a solution, detailed write up in thre...
# troubleshoot
c
EDIT - Found a solution, detailed write up in thread :) Hi good afternoon. We are looking into utilizing DataHub for our org, I am trying to put together a demo using our own data and integrations. I have followed the quickstart guide to spin up a dockerized version of datahub, however when trying to setup a source for Microsofft SQL Server and using ODBC, it requires both pyodbc module and the physical driver. I have connected to the docker image
acryldata/datahub-actions:head
found the mssql-venv and phyiscally just install pyodbc, but i still require the driver file and unfortuantly the local
datahub
user in the container does not have sudo permissions so I can't just install it via apt-get. sigh can anyone please point me in the right direction?
📖 1
🔍 1
💯 1
✅ 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
c
I hope this can help anyone who stumbles across the same issue.
First off, stop using the builtin
datahub docker quickstart
option. Instead locally clone the repo.
git clone https://github.com/datahub-project/datahub && cd datahub/docker && touch Dockerfile
Create an empty Dockerfile. The node that is responsible for executing the backend is
datahub-actions
so build a new image on top of that.
Copy code
FROM acryldata/datahub-actions:head

USER root
# Install necessary tools and libraries
RUN apt-get update && \
    apt-get install -y curl build-essential unixodbc-dev

# Install the Microsoft ODBC Driver 17 for SQL Server
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && \
    curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
    apt-get update && \
    ACCEPT_EULA=Y apt-get install -y msodbcsql17

USER datahub
Now in the given
docker-compose.yml
file you will need to make the following changes when defining the datahub-actions service:
Copy code
datahub-actions:
    # image: acryldata/datahub-actions:${ACTIONS_VERSION:-head}
    build:
      context: .
      dockerfile: Dockerfile
    hostname: actions
    env_file: datahub-actions/env/docker.env
    restart: on-failure:5
    depends_on:
      - datahub-gms
Finally, you can now
docker-compose up
and it should build your new image and replace it with an identical version of datahub-actions with the required ODBC driver on the container. Not quite done yet. I haven't gone to deep here, but when you create an integration with MSSQL, looking at the logs, it seems to create a virtual environment for that source. I am not sure where that
requirements.txt
and I am sure you could go find it and embed that into this whole build process, but I found its easier (especially for a demo or something) to just exec-ed into the container and manually install
pyodbc
because even if you alter your new dockerfile to include pyodbc, it will be
system
installed and will not be visible to the
venv
that is used by the mssql source connector. So I did this by simply doing the following once all containers are up and running.
docker exec -it docker_datahub_actions_1 bash
Once inside the container:
cd /tmp/datahub/ingest/venv-mssql-0.10.1/ && ./pip install pyodbc && exit
And boom.. now you should be able to connect to MSSQL databases and utilize the ODBC option in source. The connector will have pyodbc installed and will find all the relevant binary files for ODBC driver. Hopefully the team at Datahub by default already include the latest SQL drivers in ther datahub-actions images :)
a
Thank you for sharing your fix in detail!!
r
@limited-toddler-10716
a
Hey we had a similar problem with oracle and installed it after starting up the compose file, you can execute a script as root user (0) with following options:
docker exec -u 0 -i datahub-datahub-actions-1 bin/bash < <path>/install-oracle.sh
Nevertheless would be nicer to be able to install the oracle / microsoft prequerities for using UI Ingestion through the official deployment methods without customizing the container or needing to install them afterwards. We are running in a same problem now with MSSQL and Datahub on Kubernetes and maybe need to adapt your solution to get this running, so thanks for sharing!
c
@astonishing-byte-5433 Hi! Have you implemented such solution for Kubernetes. I just hit the same issue.