Airbyte team: Do we have anybody who has extensive experience in ssh key formats (especially in java) and/or TLS protocol negotiation familiarity? My ssh tunnel work using open source libraries is hitting some very technical obstacles, for which the “just google it” solution doesn’t work.

sorry. i do not.

is it easy to state the specific question in slack? or does it require a lot of context?

If I can’t find anyone, I’ll just keep grinding and figure it out myself eventually; it just might be slow. It requires a lot of context and very precise knowledge of how these protocols work.

We’re into “there are probably jars conflicting with other jars causing protocol mismatch” territory… but if it’s not that, it could be other similar things.

Would that help?

I’m doubtful that will help. Historically, that tends to slow me down on figuring out hard technical areas, rather than speed it up, because it stops me from browsing code and online docs that ultimately inform the insights necessary to a solution.

I can’t read and talk to a pair at the same time.

hahaha. i totally get that.

I don’t mind talking with the team about where I’m at and where I’m stuck. But unless we have an expert somewhere, I’m still just going to be grinding through experiments until I learn enough to solve it.

Our other option is: Don’t initiate the tunnel from java at all. Use java to make a subprocess call out to the OS and let the ssh command line handle it. Then instead of closing at the end, kill the tunnel process. That might get us to a solution faster, by entirely bypassing java. Downside, we will have trouble intercepting errors.

oh. i kinda like that. since it's running in a docker container we can at least control the env for calling

ssh

from the OS.

what will be the trouble intercepting errors?

Right. But it means we have to persist the private key on disk somewhere, which I don’t like for safety reasons. But maybe it’s unavoidable. Just make sure that disk image itself is kept safe.

Well, I’m not entirely certain how well we can monitor a backgrounded OS process from java while the java is doing other unrelated things. Either we have a multi-threaded situation, or we have no visibility.

fair. we do some of this already.

Of course, if the tunnel fails while we’re using it, we should see an IOException of some sort, so maybe it’s not too bad.

e.g. in the

DefaultAirbyteSource

link (the class that handls starting up the source docker process, monitoring it, reading from STDOUT, and closing it.

(i won't pretend that this code is simple to follow though)

Oh hey there’s the LineGobbler I’d been curious about. :)

Good news is that I just solved 1 of my 2 problems.

The key format issue is fine now. I still have to figure out the ssl protocol mis-negotiation problem.