curious how people are feeling about caching the o...
# contributing-to-airbytec
charles
10/21/2020, 4:47 PM
curious how people are feeling about caching the output of getSpec. it feels like having to run a worker for most api actions now because it depends on getting the spec from the integration image slows stuff way down.
u
user
10/21/2020, 4:47 PMit feels early for us to be worried about caching.
u
user
10/21/2020, 4:47 PMbut it’s also slow.
u
user
10/21/2020, 4:47 PMi’m on the fence.
u
user
10/21/2020, 5:01 PMIt’s probably inevitable that we’ll want to add caching for this? If it’s slow with our internet it’s probably slower for many others
u
user
10/21/2020, 5:01 PMi don’t think it’s an internet thing, i think it’s just that the minimum latency on the worker to do anything is higher than your average API call.
u
user
10/21/2020, 5:02 PMI mean about the first time you pull an image to run the spec job
u
user
10/21/2020, 5:03 PMyeah. i just think we might be talking about two different things. 1. you’re talking about the cost of pulling down the image.
2. i’m talking about the latency of any api call that relies on the spec (because it has to wait for the worker to run)
u
user
10/21/2020, 5:04 PMit might also be something we do upon creation of the connector.
u
user
10/21/2020, 5:04 PMcaching the spec that is
u
user
10/21/2020, 5:04 PMif we assume it is fixed
u
user
10/21/2020, 5:05 PMwe just run the job once when we try to add any connector to the persistence (standard or custom) and store that
u
user
10/21/2020, 5:05 PMyeah… haven’t thought of a solution that sounds good yet. as soon as someone tries to change the spec and they receive a stale cached version we’re going to hate our solution.
u
user
10/21/2020, 5:05 PMoh. i like your last ide.a
u
user
10/21/2020, 5:06 PMthen the spec can be stored alongside the connector def if we want
u
user
10/21/2020, 5:35 PMWhat happens if the image upgrades though?
u
user
10/21/2020, 5:35 PMI guess when do we know the cache needs to be cleared?
u
user
10/21/2020, 5:37 PMthe main assumption we’d be making is that the spec is fixed for a given docker image/version. the “expected behavior” then would be to get and store the spec any time you receive a new image/version combo
u
user
10/21/2020, 5:41 PMso if we're developing on a
devu
user
10/21/2020, 5:44 PMyeah we’d need to somehow “reload” it for dev images that are changing. This being said i don’t think this is an immediate action item
u
user
10/21/2020, 5:44 PMThe caching
u
user
10/21/2020, 5:46 PMIt's generally problematic how high latency all of the docker image interactions are
u
user
10/21/2020, 5:46 PMIt makes the app feel clunky and slow
u
user
10/21/2020, 5:46 PMnot just the spec
u
user
10/21/2020, 5:46 PMit isn't close to that latency running directly on the command line
u
user
10/21/2020, 5:56 PMyeah, but that’s because you don’t have the worker polling step in there.
u
user
10/21/2020, 5:56 PMmakes sense that cli is faster.
u
user
10/21/2020, 5:59 PMwe should be able to speed that up somehow