Building Banking Systems with Kafka Streams with Mateo Rojas | Ep. 28

Adi Polak talks to Mateo Rojas (LittleHorse) about his career working with Kafka Streams. Mateo’s first job: building a real-money policy management platform on early Kafka Streams. His challenge: working at LittleHorse with Kafka as a workflow engine and deciding whether it should be the source of truth.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

I mean I wouldn't call myself a veteran because that guy's really complex. Our solution back then, which is probably not the best solution, was to put like a really, really big number for that window. In my second project, I think we tried to solve that by just looking at it. Like don't saying like this is probably not gonna happen, putting some alerts on it if it does happen and move along.

SPEAKER_03 0:21

Hello everyone, I'm Adi Polak and welcome to ConfLand Developer. In this episode, I'm joined by Matteo Rojas, who started using Kafka at 20 years old and immediately got tossed into the deep end, building real money systems on tech that was still finding its footing. We're talking 2016 early days of Kafka Streams. So from early days of Kafka Stream chaos to modern workflow engines and corruption pitfalls and the eternal question of should Kafka be your source of truth? Matteo brings sharp insights and hard-earned lessons. Let's get to it. Hey Matteo!

SPEAKER_01 1:13

Oh hey Natty, how are you doing?

SPEAKER_03 1:16

I'm good. I'm super excited to have you here. It's been a while and I know you have a lot of insights for our audience.

SPEAKER_01 1:23

Oh, I'm excited too. Like I've been looking forward to this moment for since I heard about it in one or two months ago. Thank you for having me here. Really excited.

SPEAKER_03 1:32

Yeah. Absolutely. So for people who don't know you yet, shame on you. Uh maybe you want to say a couple of sentences about yourself.

SPEAKER_01 1:41

All right, I can totally introduce myself. Uh so I'm Mateo Rojas, I'm from Quito, Ecuador. I've been living here my whole life. Um I'm 28 years old. And um, I mean, I started using Kafka when I was like 20, or I started my career when I was like 20. Luckily, it was with Kafka. Um, I mean, I did have some mixed feelings about it because when I started, it was just starting, the technology was not starting. So we did encounter some not successful use cases back then. Uh but yeah, I've been working in the software industry for like eight years now, um, with as a consultant for the first six years mostly. And I'm not working at Little Course, where we are leveraging um where we are leveraging Kafka in order to build um a whole workflow engine on top of it, which has been a challenge too.

SPEAKER_03 2:31

Wow, yeah, I can imagine. So you're a Kafka veteran essentially. You saw a lot of the different evolutions of the Kafka versions.

SPEAKER_01 2:38

I did experience some of the pain of it. Yes, totally. Um, I mean, I wouldn't call myself a veteran because Kafka is really complex, you know. Uh, I've been learning about it throughout the years, um, but I can never say that I've learned enough of it. Uh, I mean, it it has so many things, and it's always evolved in.

SPEAKER_03 2:57

Yeah, it's uh it's amazing how using Kafka is very different than building or managing or expanding. Um yeah, the abilities is incredible. Um I'm curious, you mentioned that you worked on Kafka for eight years doing different things from different companies. What are some of the challenges that you know, software challenges that you faced?

SPEAKER_01 3:22

So, I mean, we can divide them in two different sorts of problems here, like uh from a maintainer perspective kind of problems and from a user perspective kind of problems. And I have been somewhat both, um, most of a user than a maintainer, but um both. So one of my first projects, which was like I don't know, seven years ago when Kafka Stream was just starting. It was 2018, Kafka Stream was just started in 2016, I think. Someone had had the amazing idea, and I'm I say that in a sarcastic way, because Kafka Stream was just starting. Someone had the amazing idea of um of doing a whole um a whole policy management platform uh with Kafka strength. Um so it was an anutic platform in which you would like uh you would like uh buy a policy, like put 10,000 bucks in it, and it would accrue interest. And um yeah, all of all of the orchestration behind it was done in Kafka. So we faced a bunch of problems in there. Like not only was Kafka, well Kafka was not getting started back then, and Kafka was somewhat established, but we didn't have like uh software service Kafka. I mean, Confluent was not completely of a thing back then. Uh AWS Kafka was not a thing back then either. So we needed to install our own Kafka in our own cluster, in our own Kubernetes and everything. And we faced a bunch of problems uh from a user's perspective. Like designing this software was like, okay, we are getting started with this new event-driven architecture thing, right? Like, what do we use? What even is event-driven architecture like? And we were like, uh, should we be sending event notifications, like uh policy created, and then have whatever microservices is are listening to that fetch the information from from the generating microservice? Or should we be using event carry the state transfer? Should we should we be using event sourcing? Uh long story short, we were using all of them and it wasn't. So yeah, it it it it uh in some of our microservices, um, we we did have a bunch of microservices. So uh we we can probably let me go back to how the system worked so that you can get an idea, right?

SPEAKER_03 5:46

Like yeah, let's let's start from the beginning.

SPEAKER_01 5:48

Yeah, let's start from the beginning. So the the whole business was that some client would go to a web page because of course a web page, uh you would buy a policy and that would trigger an event on our platform for sure. Okay, and um you could buy a policy for uh an annuity for like three years with a 3% interest rate uh at 10,000 one thousand dollars. Okay, and a bunch of things needed to happen for that. So we needed to check uh uh KYC, know your customer. We needed to check uh anti anti-money laundering, we needed to uh for the bank to we needed for the bank to clear the funds.

SPEAKER_03 6:31

So we had to coordinate with different banks to make sure that this transaction is valid, that it's not uh money laundering transactions.

SPEAKER_01 6:39

Exactly. So we needed to coordinate with a bunch of third parties in order to activate, let's call it activate this policy that was created. Okay and um we also had the wonderful idea back then to use joints in Kafka strings. And uh it was somewhat of a challenge because um as I was saying, like Kafka String was just getting started. And um the whole idea was that uh we would have a we would have a topic for, for instance, the anti-money laundering, because anti-money laundering system would just call us on a web cook at some point whenever they are done doing their anti-money laundering stuff. And the know your customer system would call us back also whenever they are done, verifying the customer identity.

SPEAKER_03 7:23

And uh we would never these are different microservices that operate separately on events that triggers them, and once they're done, they're writing back into a Kafka topic so other systems can pull out of them, right?

SPEAKER_01 7:37

Exactly, absolutely, yeah. And uh the whole thing is like we need to wait for the three of them to create three or to actually activate a policy.

SPEAKER_03 7:47

Oh, so to get the final approval of uh policy activation, we need to get three events from three different topics.

SPEAKER_01 7:55

Exactly, exactly. And uh so at that point, when all of them completed successfully, only at that point would the policy start to accrue interest, which was somewhat of like the interesting part right here, right? So we decided to use Kafka streams for that. Uh but the problem with Kafka streams as we encountered back then, like seven years ago, uh, was that it was just getting started and we decided to use joints there.

SPEAKER_03 8:21

So we're trying to join three different streams.

SPEAKER_01 8:25

Uh yeah, trying to join three different streams together. Yes.

SPEAKER_03 8:28

Do you remember what was the key back then for the joints?

SPEAKER_01 8:32

Oh my god, goodness. I think it was the policy ID. I think it was the policy ID.

SPEAKER_03 8:36

Policy ID. Okay, that makes sense. I might be wrong.

SPEAKER_01 8:39

I might be wrong here. It was so long ago.

SPEAKER_02 8:41

Oh good.

SPEAKER_01 8:42

And uh we discovered back then, we discovered, I mean, right now it's not not something new, but back then we discovered that you can only join within a specific period of time, a window in period of time. Right. So if a message happens to come outside that window period of time, then the join wouldn't wouldn't happen. Even though you have received the three messages, but one of them came outside of that joint window of that window in period of time, and the joint wouldn't happen. So our solution back then, which is probably not the best solution, was to put like a really, really big number for that window in window of time.

SPEAKER_02 9:19

Oh, okay.

SPEAKER_01 9:20

Yeah, a really, really big number. Um, it's not a solution I would suggest nowadays. Yeah.

SPEAKER_03 9:26

Was that days, weeks, dear months?

SPEAKER_01 9:28

We were talking years. We're talking years here.

SPEAKER_03 9:30

Years. So your window was years. Just in case. Okay.

unknown 9:35

Okay.

SPEAKER_01 9:35

We were just making sure. We were just making sure back then.

SPEAKER_03 9:38

Um definitely stress testing Kafka streams.

SPEAKER_01 9:42

Absolutely. Absolutely. Uh the the the fun thing about that is also that um back then we didn't have exactly what's one semantics either. So okay. It it was not a thing at all. And we are talking about a system that manages money here. So we cannot have uh we we cannot have like we cannot trigger two interest accrual processes just because there were two messages there. So back then only was semantics, what not a thing. So we had to create our own uh way of deduplicating messages. And for that, we also had the wonderful idea of using CAF strings for that, k tables at that moment. So whenever a message would come, we would start first store it in a k table. Uh and then, I mean, check first, first check if it if it's stored in the k table. If it is, then we will discard it because it means that we have already processed it. Uh if it is not, then we would save it in the gate table and then process it, which means that if we if the message were to come again for whatever reason, like right, uh, I mean, Kafka went down and sent us again, we were not able to commit the offset or whatever, it would mean that we wouldn't process the message again. So we we we kind of like used Kafka for deduplication, given that we didn't have exactly one semantics. That was that was really nice, actually. Uh so we had a bunch of gate tables all over the place because we have like 20 microservices reading absolutely everything, uh absolutely all the messages, uh and all of them needed to deduplicate.

SPEAKER_03 11:20

Yeah, I wonder if you ever entered like a you know a race condition with the writing to table and validating against the table, because it very much depends where you where the table lives, but you know, reading access and writing axes are different usually.

SPEAKER_01 11:35

So yeah, no, we absolutely hit race conditions, and not only with tables, right? Uh, but we hit race conditions all across the system because we were just starting to understand how these event-driven microservices work, how this orchestration works. And um, some some microservices would depend on others, but given that we have like eventual consistency, uh one event that got into one microservice might not have gotten already into the other microservice. And given they are depending on each other, it was fun times. It was really fun times testing for that and orchestrating all of that.

SPEAKER_03 12:14

Yeah, the adventures of eventual consistency is always uh keep on giving all the gifts.

SPEAKER_01 12:20

Absolutely. I mean, looking back then, looking back then, if you were to ask me, if you were to ask me like, would you do this system again with casket strings? I would say maybe I would say maybe not necessarily the whole system. I would say like I would say I would use, and this sounds like I'm I'm I'm kind of like doing a uh little first ad right here, but bear with me, I'm not. I'm this is what I would use. I would use a workflow engine. Um if I had to this case in which I have to orchestrate all these different things, I would I would probably try to use a workflow engine in there and probably massage the data with the strings, like if there needs to be some massage, or just for analytics, for instance. So no joints. I would try to avoid joints. Do we have try to avoid joints?

SPEAKER_03 13:11

Okay, no, that makes sense.

SPEAKER_01 13:12

I mean that that was a traumatic experience. Trying to actually, I mean, like trying to debug trying to debug joints is is hard. Like, I mean, uh being being really serious here, like you didn't know if like a message is stored in the in the I mean it's still in the window of time and it is stored in the cake table. So trying to debug that at least seven years ago, not sure if that has uh improved a lot these days, but trying to debug that in production was was was difficult. It got up to a point in which um our system was so messy in terms of like how many messages we had and how many topics and how many microservices and how many joins and stuff that we it got up to a point that um we needed to do some automatic consistency tests in which we we would run some kind of like um uh script that would check like the data in this microservice and the data in this other microservice just to check that the mainful consistency was actually eventfully consistent. Which I think kind of like be fit the purpose um at some point. Like it was very complex. Like we didn't know what we were doing back then.

SPEAKER_03 14:20

You can build two-phase commit, but it wasn't exactly a two-phase commit.

SPEAKER_01 14:24

Exactly, exactly, with extra steps. We did it with extra steps and way way harder. Way harder.

SPEAKER_03 14:30

Yeah. Interesting. I mean, it definitely from what I hear, it's definitely gave you a lot of insights into how the system operates and how things works, especially uh event-driven patterns and architecture was relatively early days. They weren't like big books where you can take and read. It's uh, you know, pre uh Martin Klipman uh known book and pre-other really, really great books in the industry. So um, you know, you kind of invented it yourself at the company, which is exciting.

SPEAKER_00 14:58

Now, a quick word from our sponsor. Confluent Developer the Podcast is brought to you by Confluent Developer, the website, which has everything you need as a developer of data streaming systems. And it's completely free. We've got curriculum, hands-on exercises, executable tutorials, the online data streaming engineer certification, also free. A way to find a meetup near you, those are free. Everything is there. I really want you to be successful in your journey as a data streaming engineer, and this is the site that has what you need. Check it out at developer.confluent.io. That's developer.confluent.io. Now back to the show.

SPEAKER_01 15:38

We did invented it. Uh, I mean, I was really excited as as the as the developer on it. Um, I'm not sure if I would have been excited as the company hired me, but it it was exciting time. It was exciting time. So, right. Uh well one thing as you as you mentioned is that um, I mean, we encounter a bunch of things, right? Like um, we also came into into into some problems about data encryption, right? Because we were using Kafka. Uh this this was a banking system, basically, because it was moving money.

SPEAKER_02 16:10

Yeah.

SPEAKER_01 16:11

And uh how do you make it so that you encrypt data in Kafka? Uh at least uh when it's not at rest, like um if you encrypt dating the topic, exactly, right? Uh so we did we didn't solve that problem, but it was a huge challenge. Um, we didn't get to solve that. It was somewhat of a huge challenge because topics and data Kafka is immutable. So if you were to, for instance, encrypt the data and put it in the topic, and then uh I mean, for sure your all your consumers would need to know how to decrypt it, and you would need to share your decryption key and whatever with between all your consumers or producers. Uh but aside from that, what happens if if the if the if the um if the key were to were to were to leak? Um what do you do with already existing data? Like, how do you uh encrypt that again? Like, do you read all the data from the topics and uh kind of like decrypt it, re-encrypt it with another key and produce it again? Um assuming that Kafka is kind of like your source of truth or and database, which is somewhat of what we were doing back then. Like we were completely relying on Kafka to store our data. Like it was our source of truth. If we were to go down, we would rebuild the state based on Kafka.

SPEAKER_03 17:30

Interesting. So, how did how did you solve the problem of security? We did not. We did not.

SPEAKER_01 17:36

Okay, so it was the most thing we did at that point was uh make sure that our disks were doing um, I think we were using AWCC2 at that point because we we had to install our brokerage manually. We uh Kafka as a service was not a thing back then. So the most we were doing, I think, was making sure that there was encryption across the data. And uh but we never managed to solve that problem. Um by by the end, uh, I mean, we had to give the whole project to the client and not sure what happened with that. So I'm still intrigued.

SPEAKER_03 18:10

Interesting. Yeah, I mean, security is always a big problem, and encryption is always a big problem, especially when you have um data that is immutable, it's like in one state, but then you move it, then are you allowed to decrypt it? Like with what stage can you bring it back to the original state? Like when would the actual customer need it, or if you need to search some things where you actually need uh the decrypted data? It's banking systems are hard.

SPEAKER_01 18:37

Banking systems are hard, yeah. I think that a way, a way around it, right, would be like probably have a short retention period, a short retention period of of the messages, right, in Kafka so that you don't need to encrypt them. And you do have you you do use TLS in order to encrypt them during in motion, I mean on the on the network and at rest. That would be somewhat of a way of solving it because your the messages won't would not live long enough to be a security risk. Uh but that would assume that you're not using Kafka as your source of truth, that you are not going to be rebuilding your whole application state based on Kafka.

SPEAKER_03 19:13

So our dedicated service that is uh highly secured and is responsible for decryption for only specific requirements and requests from authorized users and so on. So it gets it gets complex. I can see how it gets complex.

SPEAKER_01 19:29

It gets complex, yeah. I'm I'm actually curious how how confident has solved this for some customers because I'm I'm guessing that you do have banking customers over there.

SPEAKER_03 19:40

Yeah, we have a lot of our architectures and articles, actually. I recommend reading Quora. I think Cora is a very interesting article if you want to dive into it. Uh but you know, it's a whole story. Uh but I can share from previous companies that I worked for and I worked in the security area. We had a dedicated service that was responsible for decrypting and had the specific permissions, access to specific keys, salting, and so on. But it was highly, highly protected, you know, only access with specific credentials and so on. Because sometimes you do need that data decrypted, original state, uh in order to do whatever things you need to do. Um, but again, those are usually highly um unaccessible. You know, make sure there's only very specific uh tools or users uh permissions that you give. Um yeah, you can think about systems like care and so on in the cybersecurity world. It's an operational system. It's highly limited, but if you have the right permission, you are allowed to do specific activities to the system. But again, it's all based on permissions. The rest of the users are not allowed to do most of the things in cybersecurity. Um, but I hear you, I hear your pain. It's complicated and it's uh it's getting into the weeds of you know, building highly secure systems that are you know, people specialize in that for years and years. It's not like you wake up one day, you read one blog post, and here you go, you have a special system.

SPEAKER_01 21:12

I really wish it was that way, right? I mean, software if if there's something I've learned in these eight years, is that software is a matter of picking the right trade-offs. And I guess that there's never a right answer, maybe just a right trade-off. Or exactly probably less harmful trade-off.

SPEAKER_03 21:30

Yeah, no, building a system is always finding the right trade-off, the right skill, the right tools to use, uh, and so on. So it's it's fascinating. And your learning are super exciting.

SPEAKER_01 21:42

Yeah. So yeah, that that was kind of like basically um my very first experience with Kafka. And and that was like when I was just starting developing, like one year, and you could imagine, uh, I mean, it was difficult for me to. I mean, I was just getting started in this world and learning that Kafka has particularly. Has brokers, has uh partition keys, um, I don't know, replication, um uh producers, consumer, consumer groups, uh different kind of protocols. I think that transactions were not a thing back then, but yeah, it was overwhelming. But I I really loved it. Um and one thing that I mean, it did raise a lot of questions that I have not necessarily been able to solve yet. Like as you and I were talking before we started, um before we started the podcast, like should we use Kafka as a source of food for our database or sorry, or a database? Like, should our system completely rely on it to build its state? Uh and uh considering that you can it's difficult to query, uh or should just should we just use Kafka like a means of transporting data from one microservice to another with a short retention period? Like uh I'm also curious kind of like to hear uh like what kind of like uh constant customers have been doing through that. Because querying Kafka is difficult, right? Like and and you could if you were to use it like um you could argue that you could argue that you have a bunch of different microservices, right? You could argue that you have a bunch of different microservices and that you send your data to Kafka and your microservices are able to replay the messages and construct and reconstruct all the data from Kafka. But how different is that from having like a monolithic database where you like uh have a single database with tons of tables and all your microservices talk to those different tables? So those are questions that kind of like are always in my mind when I'm trying to design these event-priving architectures nowadays.

SPEAKER_03 23:48

Yeah, and I'm I'm sure you're not the only one facing those questions. And this is a little bit of a philosophical question. I won't speak to confluent customers because we have multi-thousands of customers, and each one of them is is unique in their needs and their architecture and uh the problem that they're solving for their own businesses. Um, but what I can say, uh, and you know, from my thoughts and from my usage of and knowledge of Kafka, it very much depends on the scale. I mean, if you're saving multi-year data, I'm not sure if it's going to be easy for you to search in Kafka if you think about it, because you do a full scan. You're not getting the indexes that a solid database would give you. Um so how do you expect to search it? Kafka is meant for transactional data. Right now, in the last couple of hours, in the last couple of days, you can you know pull all the data from the last couple of years, but in terms of indexing, you won't be able to pinpoint a specific key without searching through the whole thing. Um if that's if you have if you have small data relatively, perhaps it could be a good hack. But if you have large multi terabytes of data, this is where Kafka might be the best solution. Um and you want to start thinking in terms of object storage and table formats and maybe an analytics engine on top of those hopefully well-indexed um data, data lake or data lake house uh that you're keeping, because otherwise it's you're just gonna kill compute. Uh one of the greatest challenges in the analytics space, and we're kind of like moving to a different topic, but that's uh small data files, which means a lot of calling to uh creating a lot of I.O. calls uh because your data is sparsed or split across multiple small files, and you want to avoid that by having one big file that holds the right indexes. When you go to Kafka, you know, data is being saved on disk, unless you're doing a third uh third tier or you're saving it to other storage if you wish to. But again, you have to scan through the whole thing, which will be uh expensive because you have to bring it to all of that information to memory and then filter it out through memory. So I'm not sure if that's the best way to go about it.

SPEAKER_01 26:11

You know what? There is one other drawback of that. I'm thinking about it now, um, which coming back to my first project, we did not implement at all, is a schema. So you you can have like, I mean, if you if you have like a really long-living topic with tons of data from like a year or two years ago, at some point you would need to literally change the schema because software involved. And uh if you want, if you want, uh I mean, if you want other microservices to rebuild from those two years of messages, it means that you can never uh you're never gonna be able to kind of like evolve that schema in an easy way or in a breaking way, right? Because consumers will need to be able to consume messages from two years ago. So you will probably kind of like be able to remove fields, I don't know. And if you are lucky enough, you will be using a schema registry to be able to make sure that you didn't not screw up along the way. Uh but yeah, I mean having having short-lived topics might be the answer for evolving your messages and uh indexing as you were saying. Uh because I mean you wouldn't index in Kafka, you would index this in an external database.

SPEAKER_03 27:26

Yeah, I mean you're tapping into schema evolution, which is you know a big topic by itself, but essentially there are rules to what you can do with schema evolution. And even databases, when you ask to remove, let's say Postgres, for example, or MySQL. When you ask to delete a column, and the metadata, it never really deletes that column. It just points out and say, you know, that column is deleted. So if a user asks for this table, don't bring in that column. So it makes it very interesting where the engine, the query engine itself in database plays a big part into what is the metadata and what are the different rules and information that we have on the data to answer the uh the user query. Um yeah, and schema registry is, and and that's again, well, that's maybe not a philosophical, but mostly a religious uh conversation. It's like, are you schema-less or are you schema fool? Which one do you prefer when you use the paci Kafka? Uh some people depend, you know, some people want to be schema fool. They want to have the schema right in the beginning. They know exactly which data product they're building, they have a good alignment in their company. Some companies, some people and engineers don't have that alignment and the schema continuously changes and kind of data flows from one state to another, but there's no good correlation or interface between those two. And so the downstream folks that takes the data needs to make sense out of that. Uh schema registry can help with uh the rules that you can define there for evolution. So you can actually enable the data to evolve based on specific rules that you predefined for that specific topic. So that's fantastic because then the data can actually, we can get the this, you know, uh schema evolution in place uh with the guardrails uh and so on. Um and then you can, you know, even if you love schema-less worlds, uh, you can still have some schema with some rules that are more permissive that enables downstream folks to uh to make sense uh out of the data. So I think it's a whole world.

SPEAKER_01 29:32

It's a whole world, absolutely. And I think it it also kind of like depends on the scale of your company, right? Like if you have like one or two teams that can work together very well and communicate very well, you might not need all these guard rails, at least not at the first. Because adding these guardrails also kind of like um slow you down, quote unquote slow you down at the beginning, right? Because you need to understand how SteamajorST works, uh how this technology works, how to use it, how to put it on your pipeline, and whatever. Maybe at the beginning it's it's kind of like your development curve goes slower. Uh but if if there is great cohesiveness between these two or three things, you might not need it. But as your organization grows, you do need them because kind of like communication is is is starting to get I mean becomes a problem there, right? Do you do need cardrails and standards?

SPEAKER_03 30:22

Right, right. It's different when you're 10 people sitting together in a meeting and talking through things versus a thousand people.

SPEAKER_01 30:30

Completely, completely, yeah. Yeah.

SPEAKER_03 30:34

A thousand people, some, you know, even in ten people, things are gonna be missed. Even in three people, you know, people are gonna miss some stuff. But then you can always remind the person, hey, it's supposed to be so and so. But in a thousand people that needs to collaborate on that huge, you know, data streaming world uh framework and uh, you know, whatever platform people are building today. Um yeah, sometimes that Slack message just doesn't cut it.

SPEAKER_01 31:03

Yeah, I completely agree. And and you know what? Not to change the topic too much, but to go back a little bit today. I I just remember something. Yeah, to go back a little bit to our conversation about having long-line topics and Gafka as a source of food. I just realized that we as little cars uh we do use Gafka as a source of food. And we have been experiencing some interesting problems when it comes, when when it takes to um restart the basic. So uh our whole engine is built in Gafka. So Gafka is our source of food. Whenever we we get like a run workflow command, right? Uh we whenever we get like a run workflow, we we do an event, uh run workflow command event that's later closer. And uh from that event we basically store. And we store in in rocksdv with a change to a copy, so that we are able to basically rebuild the data. And the thing is that the thing is that at some at some moments, um, not sure if you if you know this part because um I mean Kafka stream it's it's it's very difficult. I don't even know all of it at all. So when when you use K tables and and when you save data into uh Rocksdb because RoxDB is the underlying storage, uh the thing is that that's a two-phase commit because uh you would you you save it into Rocksd.

SPEAKER_03 32:30

And then you and RocksDB is a consistent database. Exactly. Highly consistent, yes.

SPEAKER_01 32:36

Exactly. Uh and then you you send it into a change of topic so that you can rebuild it later. You can rebuild the state later if if if a server were to go down, right? And then you commit that Kafka uh that Kafka transaction. And the problem is that if in any of those three steps, like saving to RoxDB, uh producing to change topic and committing transaction, if some of those fails, what Gafka Stream decides to do is completely wipe out your data of completely wipe out your data in Rocksd. So it means that it needs to rebuild the whole state. And that's what's something that my team has been fighting with. Like, how can we make it so that it doesn't decide to wipe all the data that we have in Roxy B? Because what if we have like a terabyte of data and we have a stumbler so broken? Like we have a terabyte of data, it decides to wipe out all the data, and then we we're stuck there, we're stuck there uh waiting for it to reveal the state because Kafka stream wouldn't process any other message until it has revealed the whole state. So that has been an interesting challenge that we're trying to solve here.

SPEAKER_03 33:43

Yeah, that's really interesting. It makes me think a lot about Apache Flank, because Apache Flank has um snapshots and checkpointing mechanism. So you can always roll back to the last checkpoint that you had. It doesn't delete everything, um, but it saves the snapshots and it gives you. I mean, we need to look at the nuances, but I think it becomes very interesting to see how these two frameworks that kind of tackle well, Flink is a little bit more um, there's more things you can do with Flink. It's more extensive, it's it's more it's more verbose in terms of capabilities and uh as a data streaming solution. Uh it does require it, you know, you need more to manage it because it's separate from Kafka. Kafka, if you're already using Kafka, Kafka streams is a library that you add. So if you already have a managed Kafka, it's relatively straightforward. Versus if you have Flink, it's a completely different distributed system that you need to think where you're hosting it, how you're managing it, how you're scaling it, and so on. But the values that you get in similar situations is things like um snapshots and checkpointing uh mechanism. Um again, you need to look in the nuances of what you're building. Um, but definitely, definitely an interesting challenge. And I think it also could be a great opportunity for Kafka streams to evolve. So this can be, you know, uh I will say a feature. I'm not sure if it's a feature, it's probably bigger than a feature, but it can be kind of like a feature request for the open source community to say, hey, you know, we're facing this thing. Is there something we can do about it before jumping ship into a new framework?

SPEAKER_01 35:27

No, you're right, you're right. I mean, uh there is a point. I I'm not part of that team, I'm part of another team, but the team that works on that came to a point in which they are literally kind of like opening KIP skits in capital streams. Um, yeah so that and trying to implement them so that we solve this problem because it's somewhat of a major problem for our platform run.

SPEAKER_03 35:51

Yes, for people, people listening, people listen just for people listening in at home, KIPs, KIPS are essentially Jira ticket for the open source. So you can, if there's something you have a request, there's something you want to build, you can always go there, give as much details as possible. You open a ticket, and someone from the community, once they have you know the capacity, there's enough information, they'll pick it up and uh start working on it. So just FYI. Yes, continue, please.

SPEAKER_01 36:18

No, awesome, great, great clarification there. Um, where was I? Um so yeah, no, and people have come to me. Like uh when I was at current, people have gone to me after I talk about Kafka streams and how we use it and all that stuff. Like, why are you not using Flank? Yeah and I I always tell them the answer is complicated. But the thing is that one really nice thing about Kafka Streams, as you were saying, uh, is that it's embedded, it's a library, and we can literally embed it in our servers, in our code without having to deploy a whole separate thing. Uh we're we're we have encountered a bunch of problems, as I was saying, uh for instance, some other problems that we have encountered with it that we are trying to solve, uh or kind of like rebalancing tasks, because Kafka Stream has got has got tasks that it assigns to a product to an instance to process, right? Like depending on on the partition numbers that you have in your topic. And uh, if you have like three instances of that Kafka stream processor, uh it it I mean it might shuffle task around. And uh what if we want to scale our solution? Like we have from three instances to four instances. How do we do that in a way that Kafka streams uh shuffles the least amount of task into that new instance so that it doesn't have to rebuild that whole thing? So I I think that we always come back to that, to this rebuilding state thing, and uh how it might or might how it might or might not be the best idea uh when in terms of indexing and stuff. A snapshotting might be a good thing.

SPEAKER_03 37:54

Um I mean software as a whole world, there's always pros and cons for everything that we do.

SPEAKER_01 37:59

Yeah, so that that that's somewhat of like a the the interesting things we're solving at Ladel course um right now that comes to my mind. I do have some other experiences. Um I do have some uh uh some other experiences with with Kafka and some other questions, right? Uh along my my my career with different clients, right? So for instance, there's also not to completely change the conversation, or maybe maybe it's also part of this conversation that um in one instance um for for one client, uh we we would have like a microservice, right? And um we would have like a microservice that I ideally kind of like save into a database, uh kind of like get a request, save into a database, um, and that's it. And then you would need you would want to add Kafka to it, like in order to be able to orchestrate stuff or to do analytics. And then the problem comes like how do you make sure that if you commit your your transaction in the database, and uh if you send a message, how do you make sure to do that atomically? Uh because those those are two separate systems, and though we are talking into we are talking two-phase commit at that point. Um so so we have tried, I mean, in some of my projects we try to solve that. Uh in my in my second project, I think we tried to solve that by just uh uh uh looking at it, like saying, like, this is probably not gonna happen, putting some alerts on it if it did if it does happen and move along. And in other projects, I think um like uh the outbox pattern have come along, like in which the the whole idea of putting into the database and having a process, in this case Kafka Connect, for instance, reading from the database and producing to Kafka so that you only from your microservice, you only have one transaction into the database and not into another system. But yeah, that's another problem that we I have encountered along the way. I'm pretty sure a lot of people have encountered that.

SPEAKER_03 40:03

Yeah, it is it is a very known it's a it's a known challenge in uh connecting multiple systems together and having that event guarantee of exactly once across multiple systems. Uh in Flink, there's actually a protocol. I don't know if you got the chance to look at it. It's um end-to-end two-phase commit with Flink. Uh there is a protocol where you expose, yeah, there's you have some of them requires implementation, but the interface with Kafka, for example, is already implemented. So you can get exactly ones between Kafka and Flink. Um but essentially there is an interface where you'll need to implement. So if you're working against a database, you know, if it's not already implemented in the open source, something to look into. But essentially it does it gives you um exactly that two-phase commit that the Flink um job manager manages on behalf of you, and it will reach out to Kafka and to make sure it actually finished the whole transaction in between those two systems. Um but yeah, it's it's a known thing, especially for systems that requires that guarantee. It's it's a guarantee that um considered um you know strict requirements usually. Well, if people don't have, don't need that strict requirement, then you know, my advice would always be if you don't know you need it, you probably don't need it until you know that you need it. Because it just gets you into a loophole of you know, how do I make sure in between those? So I have exactly ones in Kafka, I have exactly ones in Kafka streams and Flink, but how do I make sure that I get it across my you know event-driven microservices databases and so on? Uh so it all works um in tandem. Um yeah, and uh and that's a challenge, I'm sure that I I didn't know that about Flink.

SPEAKER_01 42:00

I'm totally taking out I'm really not gonna be looking into Flink after this.

SPEAKER_02 42:06

Maybe our next podcast should be tell me all you need all you know about Flink now.

SPEAKER_01 42:13

I agree, agree. Yeah, you you you have gotten invested in Flink now.

SPEAKER_03 42:18

Oh no, no, no. You you full on Kafka streamed. You've been doing it for so many years now. I know it's uh it's in your heart. It's that's it. It's there.

SPEAKER_01 42:29

Yeah. But you know what? Uh one thing that comes to my mind about this outbox pattern that we were talking, which is one of the solutions that usually people people people do about uh about this kind of problem, right? If they don't happen to have these two-phase commits linking it. Right. Um one one one one one real problem about the outbox pattern is can it become a bottleneck? Uh because it probably will, because when at least if you're doing it with Kafka Connect, right, uh, which is a wonderful tool, and Division, which is the tool that I was using. The thing is that you only have one task in Kafka Connect to be able to read the writer headlock. Sorry, to read the writer headlock. Um you cannot read it in parallel, like you cannot parallelize the network.

SPEAKER_03 43:15

Right, because you want to make sure you have exactly once.

SPEAKER_01 43:18

Exactly. So you cannot parallelize that work, and and and the drawback there is that you probably will encounter a bottleneck if you if throughput is something you care about, this is not the way to do it. Or latency, latency too, right?

SPEAKER_03 43:34

Yeah, it's uh you know, our life as engineers is figuring out what what is important, what's not.

SPEAKER_01 43:41

But yeah, no. Alrighty.

SPEAKER_03 43:44

Mateo, thank you so much. It was so insightful. And you know, you're the way you talk about Kafka streams just makes me want to learn and use it more. So it's uh thank you for that.

SPEAKER_01 43:55

No, I I I really appreciate the invitation, Ari. It was amazing talking to you. I I hopefully I'm really hoping that I didn't change the topics too much, uh, but I really enjoyed the conversation.

SPEAKER_03 44:07

No, no, it was a lot of fun. And you know, our next episode, we can talk about how you discovered Flink and what are you building with Flink?

SPEAKER_01 44:16

I'll totally get my hands on that. You can do it. Yeah.

SPEAKER_03 44:20

Alrighty, take good care. Good to see you.

SPEAKER_01 44:22

Thank you very much, Abby. Have a great day.

SPEAKER_03 44:26

Bye.