That is fascinating, and very well articulated. So now that we've covered what it is, how it works and when to use it I've heard there is an interesting story behind how Lyra came to be. Can you tell us what made you dive into the world of full-text search engines?
Michele: Yeah. So, I started working on Lyra in July 2022, and it was for a talk at WeAreDevelopers World Congress, which is the biggest developer conference in the world with 8,000 developers altogether in Berlin. My talk got accepted, and I had to talk about how to build a full-text search engine from scratch in TypeScript.
I wanted to give a talk about algorithms in data structures, and full-text search engines are a beautiful starting point, if you want to learn how algorithms in data structure work. So, I built Lyra as a demo for my talk. And eventually, my manager told me, "Okay, you should definitely open source this. Why not? I mean, it's working, so you can publish it."
So I decided to publish it on npm. And when we hit our first stable release on the 2nd of August, if I remember correctly, in two days, we reached 2000 stars on GitHub, and in one week, we reached 3000 stars on GitHub, and since then, we have thousands of downloads on npm. So, I would say it's getting a lot of traction.
We've already made community first releases, where the community is providing fixes, new features to Lyra, and we already made two or three different releases where the core team haven't worked on any of the stuff we released, which is crazy. I mean for a young project, such as Lyra. NearForm:
Well it sounds like you got a successful demo for your talk! So do you think that because you built this as an example to show in your talk rather than a tool for production that maybe you didn't overanalyze it too much? Do you think that affected your approach to buiding it at all?
Michele: When I built it, I built it with speed in mind, and that's because one of the goals I had for my talk was to demonstrate that there is no slow programming language, just bad algorithms and data structures design.
That's Interesting. I know it's already lightning fast, and I know you're a performance guy, so are you already thinking of ways to making it faster?
Michele: Right now, I'm not thinking about making it faster, but I'm thinking of making it more optimized. One issue we have is that the data structure we are using to perform searches is quite optimized, but could be even more optimized. So, we are experimenting with different data structures to compress the index size even more, which means faster searches, faster deploys and less storage costs for your data.
So, I only see benefits except for the fact that it's incredibly painful to develop such complicated data structures, that's part of the game. And that's something we definitely want to look into.