And try to reproduce reliably

So there was no harm done in our case! 😅

Actually, I take that back. This is a partial snippet from Kibana:

Nov 30, 2021 @ 08:35:03.146	File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/source/mongodb.py", line 367, in __init__
	Nov 30, 2021 @ 08:35:03.146	--> 367 self.mongo_client.admin.command("ismaster")
	Nov 30, 2021 @ 08:35:03.146	self = MongoDBSource(ctx=<datahub.ingestion.api.common.PipelineContext object at 0x7fd1a5aa6b20>, config=MongoDBConfig(connect_
	Nov 30, 2021 @ 08:35:03.146	uri='<mongodb+srv://archive.qv7fz.mongodb.net/?readPreference=secondary>', username='<removed>', password='
	Nov 30, 2021 @ 08:35:03.146	<removed>', authMechanism='DEFAULT', options={}, enableSchemaInference=True, schemaSamplingSize=50, useRand
	Nov 30, 2021 @ 08:35:03.146	, deny=[], ignoreCase=True, alphabet='[A-Za-z0-9 _.-]'), collection_pattern=AllowDenyPattern(allow=['.*'], deny=[], igno
	Nov 30, 2021 @ 08:35:03.146	reCase=True, alphabet='[A-Za-z0-9 _.-]')), report=MongoDBSourceReport(workunits_produced=0, workunit_ids=[], warnings={}
	Nov 30, 2021 @ 08:35:03.146	'sink': {'type': 'datahub-kafka',

Still trying to replicate with a simple script

okey, yeah. Confirmed. Steps to reproduce: • spin up a random mongodb using docker • Set up a mongo ingestion job with wrong creds (I assume other criteria might trigger this as well, like temporary loss of connection):

source:
  type: "mongodb"
  config:
    # Coordinates
    connect_uri: <mongodb://localhost:27017>

    # Credentials
    username: bob
    password: pass
    authMechanism: "DEFAULT"

    # Options
    enableSchemaInference: True
    useRandomSampling: True
    schemaSamplingSize: 50

sink:
  type: "file"
  config:
    filename: output.json

• run the ingestion job with the recipe above • check the logs In this case I got :

[2021-11-30 21:11:56,850] ERROR    {datahub.entrypoints:101} - File "/home/fredrik/miniconda3/envs/datahub-dev/lib/python3.8/site-packages/pymongo/pool.py", line 1278, in _get_socket
    1246  def _get_socket(self, all_credentials):
 (...)
    1274  
    1275          while sock_info is None:
    1276              try:
    1277                  with self.lock:
--> 1278                      sock_info = self.sockets.popleft()
    1279              except IndexError:
    ..................................................
     self = <pymongo.pool.Pool object at 0x7f62dbaf76a0>
     all_credentials = {'admin': MongoCredential ('DEFAULT', 'admin', 'bob', 'pass', None, <pymongo.auth._Cache object at 0x7f62dbaf7070>, )}
     sock_info = None
     self.lock = <unlocked _thread.lock object at 0x7f62dbaf7ab0>
    ..................................................

IndexError: pop from an empty deque
---------<snip>-----------
     config = PipelineConfig(source=SourceConfig(type='mongodb', config={'connect_uri': '<mongodb://localhost:27017>', 'username': 'bob'
               , 'password': 'pass', 'authMechanism': 'DEFAULT', 'enableSchemaInference': True, 'useRandomSampling': True, 'schemaSampl
               ingSize': 50}, extractor='generic'), sink=DynamicTypedConfig(type='file', config={'filename': 'output.json'}), transform
               ers=None, run_id='mongodb-2021_11_30-21_11_56', datahub_api=None)
-------<snip>----------
     all_credentials = {'admin': MongoCredential ('DEFAULT', 'admin', 'bob', 'pass', None, <pymongo.auth._Cache object at 0x7f62dbaf7070>, )}

I'll create an issue

https://github.com/linkedin/datahub/issues/3645 🐛