👋

@billions-rose-75566 here is the abfs channel

Hi all! blob wave

Hi!

has renamed the channel from "integration-abfs-support" to "integrate-abfs-support"

Hi @dazzling-judge-80093, we would like to schedule another meeting, 30 to 45 minutes, to go over a few of our findings, better understand the scope as a driver for the RFC, and a few other topics. When are you available? We would propose any time from the 24/10.

I’m available during daytime CET

Untitled.md

Hi guys, following todays meeting, our initial proposal for the RFC. I reckon its quite empty still. Please share your comments.

Hi @dazzling-judge-80093, I succeeded in setting up the local dev environment, creating a new source called

abs

and ingesting a s3 source as abs; so no need for support but tnx for the offer ...

Morning @dazzling-judge-80093, could we have a brief meeting this week to cover the path_spec implementation on top of Azure? We have a look at it but could not find a easy way to "break" it into smaller developments, maybe you can help us.

Hi Tamas, When generating the local dev datahub, when executing a command, we get:

from datahub.metadata.urns import Urn  # noqa: F401

a

ModuleNotFoundError: No module named 'datahub.metadata.urns'

If we revert to commit

feat(sdk): autogenerate urn types (#9257)

the problem no longer occurs. Are there additional steps we need to take to make the module available? (or any other way to resolve the issue?)

Hey @dazzling-judge-80093, long time since we spoke. We have been dedicating some extra time to blob ingestion and, following your advice, implemented a first solution to ingest paths without a "path_spec". Furthermore, we are now able to ingest not only the current folder but also subfolders, that said, we have been commented a check around the

allowed

and

globmatch

methods: metadata-ingestion/venv/lib/python3.9/site-packages/wcmatch/pathlib.py:136. In this context, could you tell us whats purpose of the

_wcparse.py

file?

After discussing the above behaviour with some colleagues i understand that Datahub does not "support" recursive ingestion, is that correct? As in, urls ending with

\*.*

only ingest the current folder. If thats the case the above point is a non-issue.