https://datahubproject.io logo
Join Slack
Powered by
Hi! Is there a hook anywhere in the deployment pr...
# all-things-deployment
d
Hi! Is there a hook anywhere in the deployment process to install additional pip modules? Like writing a requirements-extra.txt file that any of the bootstrap scripts or application code would look for and inject as part of the installation process?
b
For which module? The ingestion bit through the UI or something else?
d
I believe so yea. It sounds like we need to install acryl-datahub[s3] on our servers to get the s3 ingestion to show up. We deploy the datahub containers via Nomad, and were wondering if there’s any way to add requirements to be installed as part of the datahub installation process
b
Oh got it- have you tried using the custom ingestion source option?
If you provide s3 as the type it should install for you
d
hmm are there docs for that?
or can you point me in the right direction
b
No just go UI -> Ingestion -> create new source -> custom
And then configure the S3 recipe according to the S3 doc on the website
d
okay, if I recall correctly the ingestion UI didn’t work with elasticsearch as the backend when we first installed it
but I can try again
b
Oh interesting so you remember why?
It should if it doesn’t it could mean something wrong
d
ok, let me give it a whirl and report back
yea it didn’t seem to do anything, I was expecting to find something in our logs
b
Can you click "View Details"
and report what is shown?
d
where is “view details”
I’m on the sources tab
in the ingestion UI
b
In the Ingestion tab -> Click the + on the source you ran
d
oh sure
b
There should be a run
With logs
Ah!
Okay thats a different problem
It seems that your datahub-actions container is not up
Or responsive
How are you deploying DataHub
d
docker containers via Nomad
b
Ah my guess is one container is missing
d
ok hmm let me look into that
b
Or somehow unreachable on the network
d
ok yea seems like we didn’t add it to our deployment
let me add it and fire it up again
b
Sure thing
d
hey @big-carpet-38439 - is this still an issue? https://datahubspace.slack.com/archives/CV2UVAPPG/p1646345037761429
6 Views
Open in Slack
PreviousNext
Powered by