When I directly execute

datahub ingest -c  sample.yml

which is pointing to the latest checked out bootstrap_mce.json, I see only 4 records are ingested.

[user@localhost datahub]$ datahub ingest -c ./metadata-ingestion/examples/mce_files/sample.yml 
[2021-10-04 19:35:09,834] INFO     {datahub.cli.ingest_cli:57} - Starting metadata ingestion
[2021-10-04 19:35:09,858] INFO     {datahub.ingestion.run.pipeline:61} - sink wrote workunit file://./sample.json:0
[2021-10-04 19:35:09,886] INFO     {datahub.ingestion.run.pipeline:61} - sink wrote workunit file://./sample.json:1
[2021-10-04 19:35:09,907] INFO     {datahub.ingestion.run.pipeline:61} - sink wrote workunit file://./sample.json:2
[2021-10-04 19:35:09,964] INFO     {datahub.ingestion.run.pipeline:61} - sink wrote workunit file://./sample.json:3
[2021-10-04 19:35:09,964] INFO     {datahub.cli.ingest_cli:59} - Finished metadata ingestion

Source (file) report:
{'failures': {},
 'warnings': {},
 'workunit_ids': ['file://./sample.json:0', 'file://./sample.json:1', 'file://./sample.json:2', 'file://./sample.json:3'],
 'workunits_produced': 4}
Sink (datahub-rest) report:
{'failures': [], 'records_written': 4, 'warnings': []}

Pipeline finished successfully

My sample.yml is :

source:
  type: "file"
  config:
    filename: "./sample.json"

# see <https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub> for complete documentation
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

However, when I execute the datahub docker ingest-sample-data I see there are 82 records ingested:

...
                  'file:///var/folders/89/c3061_k547b_g6tgh77dbxsm0000gp/T/tmpnczwmxw4.json:79',
                  'file:///var/folders/89/c3061_k547b_g6tgh77dbxsm0000gp/T/tmpnczwmxw4.json:80',
                  'file:///var/folders/89/c3061_k547b_g6tgh77dbxsm0000gp/T/tmpnczwmxw4.json:81'],
 'workunits_produced': 82}
Sink (datahub-rest) report:
{'failures': [], 'records_written': 82, 'warnings': []}

What is the difference between manually ingesting against the bootstrap_mce.json vs the

ingest-sample-data

command? @mammoth-bear-12532 @big-carpet-38439

@loud-island-88694 Pasting full json here in the reply thread clutters it. I'd request not to paste entire file contents.