https://datahubproject.io logo
Join Slack
Powered by
Hello!! :wave: I successfully imported pipelines ...
# ingestion
a
Hello!! 👋 I successfully imported pipelines from AWS Glue (they're all custom scripts)..I'd like to delete some entries (as I changed the PATH) so now i'd like to delete either all of the object then re run the ingestion or delete the PATH that i don't need. I ran this command
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head --debug delete --env DEV --entity_type pipeline -f
for 
entity_type
i tried pipeline, dataflow, datajob, but none of them worked which is the right entity_type value i need in order to delete the entries TIA
e
It should be dataflow and datajob! Can you try removing the --env DEV ?
You can do --dry-run
to double check before you delete!
a
i get
Copy code
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Usage: datahub delete [OPTIONS]
Try 'datahub delete --help' for help.
when i execute it like this
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head --debug delete --entity_type datajob -f
or dataflow is the same
PATH looks like this • 
/Pipelines/athena/dev/
• 
/Pipelines/glue/dev/
then scripts are under 
/dev
i tried this one as well
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head --debug delete --platform glue -f
e
Are you running locally on M1 laptop by any chance?
a
M1
e
Ahh
😅 1
So for m1 we need to add more env
one sec
And the same thing happens
when you remove --debug right?
Also what does -f represent here? Can you instead use -n for dry-run?
a
-f is to confirm the deletion
i removed --debug & tried -n...didn't work either
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head  delete --platform glue -n -f
without the -f i get this
but in any case the command doesn't find any match
i tried this one as well
Copy code
docker run --network=dtahub:/home/datahub linkedin/datahub-ingestion:head  delete --entity_type dataflow -n -f
get
Copy code
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Usage: datahub delete [OPTIONS]
Try 'datahub delete --help' for help.
e
I think -f is causing it to fail, but it matching 0 entities is still not resolved
so if we do ^ without -f
what do you get?
You can ignore the warning for now, we don’t support native arm64 support for datahub-ingestion container yet.
a
without the -f i get the prompt y/n to confirm deletion, but the docker command is not interactive therefore it exits with 1
it's odd that even going "generic" ie: everything that is coming from glue is not finding anything
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head  delete --platform glue -n -f
they are the same pipelines as i changed the PATH after first ingestion
e
ah so
it seems like if entity_type is not set
it is defaulted to dataset
so if we do
--platform glue --entity_type dataflow -n -f
what do we get?
actually
--entity_type dataFlow
a
no matches still
i tried with combinations of dataflow, dataFlow datajob
e
so just --entity_type dataFlow also returns nothing right?
ah
seems like i see the problem
let’s confirm whether
just --entity_type dataFlow returns the pipelines
a
if i only do 
delete --entity_type dataFlow -n -f
it says 
Error: You must provide either an urn or a platform or an env or a query for me to delete anything
then with env or platform flags it doesn't match any entry
e
Sry for making you try so manycombinations. We will put out a fix
can you try doing --query glue ?
so --entity_type dataFlow --query glue
a
same story...no matched entries
no worries, thanks for your help @early-lamp-41924!!
m
@agreeable-thailand-43234: yes this has been fixed via (https://github.com/linkedin/datahub/pull/3619) and has been available since 0.8.17.2 (https://github.com/acryldata/datahub/releases/tag/v0.8.17.2)
a
hhmm, i just pull again the images from docker (
tag: head
both, quickstart & ingestion) and still the query "don't find" any match
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head  delete --platform glue -n -f
also tried
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head  delete --entity_type dataflow --platform glue -n -f
if I get into the shell of 
datahub-gms
to check 
datahub --version
it says "datahub not found" any idea?
@mammoth-bear-12532 ^^
BTW i'm running on a Mac M1 (not sure if that has to do with anything)
e
datahub-gms should not have the datahub cli (python-based) installed.
just to confirm
if you search platform:glue in the search bar, you get results right?
a
OMG!!! i'm so sorry @early-lamp-41924 i added this line in my ingestion 
underlying_platform: athena
so everything was under platform=athena 🤦‍♂️ so i changed to
Copy code
docker run --network=datahub_network -v /Users/edgar.valdez/Documents/datahub:/home/datahub linkedin/datahub-ingestion:head  delete --entity_type dataflow --platform athena -n -f
my bad 😞
e
ahhhhh got it! now worries!
did ^ work?
a
yup, it worked, i'm so sorry...i forgot about that setting plus having 
type: glue
made me think it should be found as glue 😅
e
Right. This does seem very misleading
What do you think will make this more clear? We are open to any suggestion here!!
a
i have to say it was my fault...documentation states it well
Copy code
Override for platform name. Allowed values - glue, athena
I guess once users are working on a daily basis with the platform it's more clear
e
Got it. Let me look into how the override is stored
2 Views
Open in Slack
PreviousNext
Powered by