A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

image.png

Hi all. I try to connect my Nifi, but its impossible for me. I used docker but through custom yaml is not working and by CLI in the docker container neither. Could you help me with this? Thanks! :)

Hi <@U042ZD3KXL1>, from all the info that you have shared, it looks like you are able to connect to nifi but the ingestion job is taking too long to finish. Is this correct?
cc: <@U02G4B6ADL6>

Could you try using `process_group_pattern` to restrict the ingestion to a small test sub-set first?

ok, I'll try these option! but the last line is from about an hour and in pipelines I don't have any imported

Hi <@U042ZD3KXL1> process group allow deny pattern works against name of process group, so please use "Practica" in allow
```process_group_pattern:
    allow:
        - Practica```
Also watch out for warning messages starting with `Dropping Nifi Processor of type` , as some ingress/egress processor nodes used in your nifi flow (GetFile, PutFile) are not currently supported by datahub nifi source. Please refer nifi source doc for current limitations - <https://datahubproject.io/docs/generated/ingestion/sources/nifi/>

Another config you can use to reduce the execution time is to set config `provenance_days: 1`, which will process events for only single day to extract lineage.

Hi all, started using the NiFi ingestion and had some troubles with nested process groups and using the "process_group_pattern". I wasn't aware that the main canvas itself is also a process group and it took me a while to figure out the right way to specify the pattern.

So with the following nested process group, the right way to specify the pattern is

```source:
    type: nifi
    config:
        site_url: '<http://xxx.xxx.xxx.xxx:18080/nifi/>'
        process_group_pattern: 
            allow:
                - EADP
                - 'DEMO.*'```
Is this how it should work? Guess it would be worth mentioning in the documentation. Should I create a PR?

Hey <@U01791E3KM5>, would be great if you can contribute a *Filtering* section in nifi documentation with this example , its explanation. If you have more examples, you can also add them.