This message was deleted.
# generals
Slackbot
03/16/2023, 6:09 PMThis message was deleted.
d
David McHealy
03/16/2023, 6:14 PMa
Abolfazl Ghahremani
03/16/2023, 6:19 PMAbolfazl Ghahremani
03/16/2023, 6:19 PMevery solution is not solution Boy
Abolfazl Ghahremani
03/16/2023, 6:24 PMyou should consider milionse of events are comming per second and we want change state of each row/user ...they are related together ......
d
David McHealy
03/16/2023, 6:27 PMGood luck.
a
Abolfazl Ghahremani
03/16/2023, 6:28 PMsuppose the ram down what happen for stream
Abolfazl Ghahremani
03/16/2023, 7:33 PMhow much is shi...t
r
Renato Santos
03/16/2023, 10:01 PMalive with million of events in the same leaderboard? that's unusual, and will require unusual software. I would recommend to enter in contact with redis enterprise
bc even clustering will not help you if you really want just one big board with a very very long tail
Renato Santos
03/16/2023, 10:04 PMif you have a lot of small leaderboards, then you can use redis with replica for failvover, and kafka for replaying when things goes bad and you need to reprocess until the last savepoint
a
Abolfazl Ghahremani
03/17/2023, 4:30 AMhi
the problem is just not (sorting)a leader board.
we should process new event consume it for past events match to rank 1 if remained more amount match it to second rank if not match add it to somewhere is best rank for it...🙂
Abolfazl Ghahremani
03/17/2023, 5:43 AMredis is nice for persisting it even we can replicate but it has not processing part ....
we should compute new board also when new orders comming always......it is a stream problem but also at same time a board
d
David McHealy
03/17/2023, 12:52 PMSounds like a moving average, but instead a moving leader board. The problem is if you are always adding and always removing you have to always be recalculating. If you know the duration of time between something entering and leaving the board, you could have one real time stream and another delayed stream and the real time stream adds and the delayed stream removes. You are probably going to have to develop something bespoke.
a
Abolfazl Ghahremani
03/17/2023, 12:59 PMi am thinking more to know whether it is streamble or not ..
r
Renato Santos
03/17/2023, 1:03 PMI'm not sure I'm following, but if it's a moving avg problem, then it's possible to use kafka-stream to emit the avg when it changes, and capture the change to update the ranking somewhere else
kafka-stream internally uses a RocksDB to store the stateful info required by the windowing and other processing, and it's somewhat crash-resilient
Renato Santos
03/17/2023, 1:05 PMI would start with ksql, instead of building direly over kafka-stream
a
Abolfazl Ghahremani
03/17/2023, 1:05 PMKafk-stream can not access to past events ....we discussed before kafka stream partioning data in network ...just spark can be model it in memory
r
Renato Santos
03/17/2023, 1:08 PMthe window is bigger than the kafka retention? because i'm pretty sure that you can read past data if it's stored in the topic
Renato Santos
03/17/2023, 1:10 PMwith this ksql code it writes a java program using kafka-stream to do the heavy part
it will keep emitting for each key at regular interval if there's changes
then you can use it to save into another topic to update the ranking in redis or other tool
a
Abolfazl Ghahremani
03/17/2023, 1:11 PMi dont know what are you taliking about
r
Renato Santos
03/17/2023, 1:13 PMksql is sort of a spark but when the source and destination is a kafka
a
Abolfazl Ghahremani
03/17/2023, 1:14 PMksql is nothing just is sql lang of kafka
d
David McHealy
03/17/2023, 1:14 PMSo I guess it would be that, but instead of avg it would be sum for this use case.
r
Renato Santos
03/17/2023, 1:14 PMyou write the instructions in sql, it compile into a program that can run in cluster mode across multiple workers, etc
d
David McHealy
03/17/2023, 1:14 PMThat's really interesting. I will have to investigate using this in the future.
a
Abolfazl Ghahremani
03/18/2023, 8:02 AMthis problem is not moving average
Abolfazl Ghahremani
03/18/2023, 8:02 AMit is matching engine
Abolfazl Ghahremani
03/18/2023, 8:03 AMmoving avg🤣