ActiveMemory the missing ORM for ETS and Mnesia | Erin Boeger | Code BEAM America 2022

foreign [Music] [Applause] okay great I'm not used to hearing my own voice so this is a little weird but I'm glad everybody stuck around last Talk of the day before happy hour I'm sure everybody's hungry and tired so um but I'm Aaron bodger I'm with erlink Solutions I've got a pretty long fintech background U.S and across Asia former rubius like many people in the electric Community I'm an avid hiker I love hiking I love the outdoors and I'm on LinkedIn if everybody wants to connect feel free to connect um but this has already been kind of asked a little bit but it seems like a lot of people are familiar with ETS how many people are familiar with amnesia how many people are actually using it in a project see this is this is something um so it doesn't look like anybody here is pretty new to ETS or Amnesia seems like we got a lot of senior experienced people so we'll move on but why use these I'm sure many of you know the faster it adds a lot of speed reduces latency when requests come in to requests go out um you can use native Elixir data types I'm sure you all know and even functions which are really really nice um but the one I like the most is reducing database dependency and we're dealing with modern web apps and currently the way we use web apps shrinking down that database dependencies I think is is really important and also these these Technologies got many years of experience I tried to find out how old these are and I probably should ask people like Francesco and stuff who are here but I didn't before the talk so but they date back to the early 90s if not the first release of Elixir or excuse me erlang so the database dependency to me is kind of the thing that I kept coming across when I was working on various apps I kept doing the same things over and over again to kind of reduce the amount of calls I I do uml diagrams of method calls you know tracing the entire request through the app and be surprised that sometimes how often we hit that database sometimes just simple request took five six seven calls to the database for a lot of stuff and as we're developing and writing code we just kind of take for granted we don't really look at how many times we hit the database and so reducing some of those requests can make a huge difference in your app um and and make a difference in your your database capacity not in terms of storage but just throughput and and things of that nature so the kind of made me come across this active memory um the goals of it number one is to help increase awareness um a lot of us in the electric Community come from other languages and tools like ETS and Amnesia are not in those other languages and I think because of that they're not top of conscious we're not when we go to solve problems in in our in our systems and things we don't think of them naturally as much um and so I want to kind of help bring it up a little more because I think these are really good tools and I know in this room you guys have a lot of ETS experience but a lot of the apps I've been involved with even with some senior lectures developers there's almost no ETS almost no Amnesia they're very rarely used and it could be just like a small set you know just me and my experiences but it just seems to be the case um and so I also want to kind of help push the discussion and exploration of maybe getting some more packages and Libs around them because there there isn't that many I also wanted to kind of simplify the implementation and the use things like setup tables data management to basic crud stuff pulling things in putting things out or pulling things out putting things in um schema changes with things like Amnesia and initial setup familiar syntax fungal lines of active actor record Ecto a lot of us are familiar with but I didn't want to lump these in I know there's been attempts of using an extensions to Ecto for these but it just doesn't feel right because they're very different tools and I got the two two trucks here the delivery trucks and it's a I mean it's a good analogy because they both are delivery trucks but they have completely different use cases you wouldn't want to use that giant semi truck to deliver to local neighborhoods and tight streets and things of that nature small packages it's just not a practical use case for it and then the same thing is you don't want to use a tiny delivery truck to run around large cross state huge Halls of mass amount of goods it's just not practical but they both are delivery trucks and they both have specific use cases and I think maintaining a separation of that name that that thinking in our minds of how we use these tools is important and that's why I think keeping it separate from Ecto is is important so during my time with the Elixir I've come across a bunch of use cases where I found these tools ETS and Amnesia very helpful um button okay here we go um application secrets and settings this has been a good one uh provided a simple UI using live view and boot up the app it read in a default set of application settings and secrets and then the UI could go on and edit these and these came really helpful when we were in for example a staging environment we wanted to test production type scenarios we could easily go into the UI update those credentials update whatever it is for making third third-party API calls to things like AWS or payment providers was a big one so we didn't use those credentials other than obviously production and we had separate ones for sandboxes and stuff you could actually change these on the Fly and it worked really really well and it also helped clean up our config files because things just get kind of chucked into those files and environmental files kind of scattered all of our apps cleaning them up putting them in a simple Place giving a nice little UI to it was very helpful another one is one-time use tokens these are very short-lived things um you know uh password reset tokens all these kinds of stuff again put it in a nice Amnesia table sometimes we would include things that are necessary so if people come back in from a password like a magic link we would store a current user struct which is typically not a full user anyways typically when you have like a current User it's just basic small set of data subset of what an actual user is email maybe a uid an ID just a small set of data that would all get thrown in people come in from the the link they automatically get put into the app we don't need to go to the database it's all right there ready for them and then we put in a refresh that the Gen server to go in and sweep these out on a regular basis and it worked really really well another one is API Keys a lot of apps I see kind of are going this way with the back end supporting multiple front-end clients an Android app IOS app some Spas I worked with one client in Japan that had 300 of these things that are primarily WordPress sites and we were able to put all those API Keys you know more of like identity versus an actual key inside these along with some basic configuration so again when those API requests come in and they come in every single time for every single request I mean we've significantly reduced the database burden with this and it worked really really well button's not working JWT encryption keys this was another one we issued jwts we we stored both the private you know the private key and the public key we had an endpoint to publish the public key because all good consumers as little jvt should be verifying that uh authenticity and then the the private key was readily available for use encryption throughout the application and issuing of those and then they refreshed on a regular basis as well we rotated those out and this worked really well admin user management admin users are usually a very small subset of the actual users so putting all of that inside Amnesia or ETS table worked really really well to help reduce that database dependency session management this one was a great one too it was in the same map where we did the API keys for the clients the user session management stuff was able to be stored in there as well so when the user came in the API request identifying the client along with the client configuration as well as the user and the session information was all stored in these tables we had almost no hits to the database it was all really fast and worked really really well and so that's kind of what made me want to write active memory so some of the features are we have a table [Music] um try to create an easy syntax it has the ability to auto drainerate uuids and it's able to handle a lot of Amnesia and ETS basic options and things so it's kind of somewhat of a flow through on that we also I also added something called migrations which will modify the the structure where you can connect into Amnesia you add the node copy table all that kind of stuff and I had the query syntax obviously Amnesia and ETS are very different kind of Technologies each Amnesia being distributed some of the query syntax between the two is slightly different but they both use things like match specs which aren't very approachable to a lot of developers and so I created these syntax here the match syntax where you're able to use your basic operators and everything and they're agnostic it doesn't matter if you're using ETS or Amnesia you can change these table types and the queries should work just as they are and the other one's more of a straight key value key value matching and then we have What's called the store and the store is basically just backed by a gen server and it manages the table and it has your basic crud operations where you can get all delete delete all and then I added the one withdrawal which I used a lot because of things like the one-time use tokens and things of that nature you want to get the data and then delete it after you've gotten it so you can only get it once and then the basic right and I try to keep a syntax that was not the same to be confused with Ecto but at the same time familiar for everybody to be able to use and then other feature of the store seeing that's backed by gen server is during initialization when the when the app first boots you have the ability to add a seed file and it'll just read the seeds in and populate the data you can do that um what's that word you know item potently you know with the finder create or something along those lines and it worked really works really well the other thing is before you knit which is a little tough on naming this one because it's not really before it's like during in it it'll help you run some special methods that you need to get run to set things up maybe set a specific specific State you need sayings run going before you have it you can add these functions in there as well and then the initial state is another one so a lot of the the tables that I had were things that needed to be refreshed on a regular basis so I'd set up an initial state of when the last time these things were refreshed and so you can kind of go in and you can you can you can check that state had their last refresh happened and how many records were deleted all that kind of stuff so you can just set that initial State structure up how you want it within that gen server and then how you want to maintain it obviously throughout the rest of the code you write and so with all of this in mind I got experimenting a little bit more and this Amnesia manager is kind of more of a design kind of a concept that I've been toying with and basically it's an abstraction where it's just a simple General Elixir app it's just almost no code other than maybe lib cluster connect to everything so that way there's almost no reason to reboot this thing it can run 24 7. it's you know it's just you know pretty bulletproof and it abstracts away some of the responsibilities of of Amnesia and uh for example disc persistence I use this to dispersist and then heavy right tables because I know there's been some issues with consistency so before Amnesia writes an easy consistency amongst the nodes so if you have a heavy right table you can store that and use amnesia's table transparency where each node can connect but as long as the table is in that cluster it doesn't matter where that table is and which node that table is on it'll write to that it'll read from that and so the goal of this is kind of assist with horizontal scaling things spin up things spin down connect in get get the data you need things of that nature so this is kind of what it looks like so we got three Phoenix apps here and all of these Phoenix app just really have only Ram copies and then the memory manager has a ram copy as well as disk persistence and then these guys can spin up and spin down as they want and then this is the case if you have a heavy right table and then did a little homework on this one did some bench tests nothing too significant but there is a penalty for the reads there's significance no eight times slower but it's still faster than database but the rights are about the same it really isn't I think if you add more nodes into it the rights would actually probably be equivalent or maybe even faster so um I wanted to do a demo but there's just kind of so much a demo it's kind of weird so what I did instead is I created an application and so I have the Amnesia manager sample as well as a sample app and everything in here all the all of this address user user token configuration settings state code all of this here is stored in them you can configure it as to ETS or Amnesia so if you want you can go in I have this all available on the GitHub repo for active memory so if you Google active memory hex for example in the hex packages you pull it up look at the GitHub and there's links to all of this on there so you guys can pull these down there's instructions you can kind of play with them UI ux is ugly I am not a designer I barely know how to dress myself so please don't but feedback is really really hope you know really looking forward to so anybody got feedback please feel free I am looking for people to help contribute to this I would really like to see other use cases and other people's struggles with it I know there's some issues with amnesia when it comes to things like net splits and stuff of that nature and that's kind of stuff that I want to get to tackle with but I kind of want to see if other people are interested and other people want to get involved and help build some good tools and support around around Amnesia and eps and so for the future I'm kind of looking at more extensions to this being able to handle attribute changes so if you change the structure in a table where you want to add another field you want to remove a field you got to send it through a mutation of some kind so finding a nice way that that we can handle stuff like that as well as improving the the query syntaxes this you know better error handling better documentation working on I've heard that some people don't like the fact that each store is one gen server so maybe I need to create a a general server maybe call it a plaza or a mall I don't know just as a joke but something to handle multiple tables versus just one improve the net splits handling I kind of want to tackle that issue because I think out of all the issues where Amnesia has driven away people unfortunately is because I think that's what seems like everybody comes to everybody says it's the Nets Blitz and I really think there's a lot of value in these tools and I really think it's worth investing in kind of like what Maxim and them have said earlier you know instead of Reinventing a wheel let's enhance this wheel I think there's very good use for these tools and so I think we should give some attention to enhancing these improving them because I think they're really killer apps to The Elixir and erlang ecosystem that a lot of com languages just don't have and it can bring real value to our clients and our and our customers and our companies we work for and so that's pretty much it any any questions or anything put a round of applause first thank you very much [Applause] very good whistling thank you for that all right we've got a question hey great job Aaron um question you said that the store is backed by a gen server are all the read calls the one the select the withdrawal going through that gen server and if so how are you getting around the fact that essentially that makes access to the store single threaded in high concurrency are you going to run into the Gen server bottleneck there honestly I don't remember how I implemented all that I'd have to go back and read it again I don't think I implemented all of them through the Gen server I think the Gen server is just really managing State and getting things initially set up and I believe I just went straight and which I'm pretty sure not I think about I went straight to ETS and Amnesia directly I haven't I have adapters so we could add other things in there you know um other key value stores you know that if they followed the Adaptive Behavior but I'm pretty sure it went straight to those themselves but that's a good question thank you any other questions yes the trustee Lucas hey man uh first of all like a really nice talk like I had some some troubles before with ETS and I think you'll give me the path to to solve them but something that I that I still want to find out how to do it is like we we already use ets a lot in production and like we we have this issue that is horizontally like like our systems like kubernetes and something like this that we have lots of servers uh to to have only one thing and the issue of like like not sharing the ATS table I I know you shared like using Indonesia for example like to to have one Central thing but how would you recommend for someone that like it's not easy to like change all the system to to use them in Egypt for example how would you recommend like saving these ETS data like having a centralized server just to to so all the the other servers can access it or each of them like is the best option so which of them has its own table but it sounds like you really need amnesia on that one I could be misunderstanding you but again that's part of what this was it was agnostic so on that demo app you can put all the tables as ETS and then just change them all to Amnesia and none of the rest of the code changes it can all stay the same so that kind of sounds like we what you would need is is that way you could start with ETS and then and then as you grow and you need to horizontally scale you can migrate those to amnesia but if you're going to be needing you know I'm no expert in these but if you're going to be needing to just obviously share that information amongst nodes you really want to use them in Asia you'd want to use it but I mean feel free to play with this library and and I wouldn't think it would be too hard because I I backfilled some apps using this library after I made it and it wasn't too terrible to implement to be honest it wasn't too terrible but I'm a use case of one dude so look for other use cases so if you want let me you know play with it and let me know yeah thank you like this is the thing like it makes sense like now seeing your your presentation that Indonesia makes sense for our case but like maybe it would be too hard to to adapt all the system to use them in Asia right this is why I did this question to how like how to avoid Indonesia In Like A system that already uses a lot of ETS you know but I think maybe trying to go to the right path and museum in Egypt will be the correct way yeah thank you like it yeah any other questions okay cool once again thank you so much Aaron that was fantastic thank you [Music]

This video was recorded at Code BEAM America 2022 - https://codebeamamerica.com/ ActiveMemory the missing ORM for ETS and Mnesia | Erin Boeger - Senior Developer @ Erlang Solutions ABSTRACT A package to help bring the power of in memory storage with ETS and Mnesia to your Elixir application. ActiveMemory provides a simple interface and configuration which abstracts the ETS and Mnesia specifics and provides a common interface called a Store. Use ETS and Mnesia to help boost your application performance, simplify configurations and secrets, help reduce database dependency, and more. OBJECTIVES Introduce the ActiveMemory hex package and what problems it is trying to solve. Also help people better understand ETS, Mnesia, and how they can make our apps better. AUDIENCE Anyone who has interest with in memory tables and or has had previous experience with ETS or Mnesia. • Follow us on social: Twitter: https://twitter.com/CodeBEAMio LinkedIn: https://www.linkedin.com/company/27159258 • Looking for a unique learning experience? Attend the next Code Sync conference near you! See what's coming up at: https://codesync.global • SUBSCRIBE TO OUR CHANNEL https://www.youtube.com/channel/UC47eUBNO8KBH_V8AfowOWOw See what's coming up at: https://codesync.global