Mnesia as a complete production database system | Chaitanya Chalasani | Code BEAM V

hi my name is caitanya i work as an airline programmer and an architect so this is my talk today on the nisha as a complete production database system it has two parts part one is the case studies um of all the way i've been using nichia quite some since quite some time and after the second part is about my experiences i calculated them into an open source component and we'll discuss about the open source component as a second part of that so my journey with erlang began from 2003 and from 2003 to 2006. um i've been working on i was working on telecom value-added services roaming products and certain other signaling products and during these three years of period we were using nichia only as an in-memory while all the transactions and all the master data would remain in asia as long as the service is running and the persist for the persistent we were using mysql and we normally do not bring down the service often and all the updates that has been on to them in memory database we've been uh asynchronously pushing into mysql that maximum user base and peak throughput is you know not because of nichia or erlang it's the solution then had that kind of capacity and it sustained from 2006 to 2010 what we did was for those the set of products that we built during this period we moved um parts of the tables which are more needed for the development purposes and transactions and you know master data completely indonesia and it had both the disk and the memory part of it in asia while the rest of the you know completed transactions or cds or tds would remain on my scalar postgres and third evolution is we wanted to avoid a second database completely so there was um what we did was we used nisha fragm behavior to you know use the disk only copies to store more data more than 4gb while still using certain other you know data as an indexed flat files while the entire database of you know the master data and the transaction data remained in asia fortunately in 2016 klarna released a level db initia backend and followed by attending to release the roxdb backend and this this broke the limitations of the disk only copies then i started using leveldb for one of the product and then db as a complete backend and we could now store as much data as we can in in all on in the whole initial only initial part of it oh so some of the this uh you know um uh products that we did we just take a case study of that one is a call player a cloud-based call center service and it it ran with thousand tenants 75 000 extensions 25 000 calls per day 75 000 recordings per day and it had a maximum retention period of an year in the system the rest was shipped out it just ran with two oltp nodes and two lap nodes completely running on initial and using a memory of 24 gig and the disk of almost 45 gig and it's live since 2017 and that's what we haven't seen any initial based outage still yet and 2019 we we started i moved to a different company and what happened is that i had to reboot all the tooling that i did on top of nisha here and you know rewrite them here as well which i really didn't want to do that's when i thought let me write something on the open source which i can use it in every other job that i move on that's when the next part of the talk of the framework comes in and this is a connected car and smart home cloud services it's still running from neo and it does a good job and so far no nisha based our teachers or network partitions or any such things and the new service that we started is a video conferencing service all of them just uses nisha there's no secondary database other than asia now this is a well-known advantage of nisha over other databases if we are using an airline vm on airline vm-based services into an isolated you know a monolithic uh like vm is that there's a zero impedance in reading and writing um you know data indicia as long as you're not using the rocksdb copies or disk only copies while in the other hand if you use an external database and an erlang vm external layout you're now there is an impedance in terms of data serialization in the network and also you need while or the optimization there are other factors that you need to cater to is your external database the ipc's in between the networking and the airline vm itself so this convergence makes architecture you know the design easy the optimization simpler the convergence is really beneficial for that now let's establish the context here uh when to use nishia as if complete production database um it you could use it as in in memory cache otherwise too the significant consumers are erlang native and it's small and medium and moderately large user base can be accommodated within six to eight nodes if they could be powerful enough and can be accommodated in less than four to six nodes or eight nodes at the max is fine and the vms or servers are memory optimized and the cluster needed with good quality of service network we know for sure lang doesn't tolerate good network uh partitions we can work around speeds of eight parts of it and we can do some kind of management mechanism but it's completely doesn't actually uh you know you kind of completely avoid that and so when where not to use it is um you know when the us consumers are not all airline native there's no point using nisha here and if a user base is so huge something like whatsapp or facebook where it goes spanning more than eight notes then now the very thing about nation is about synchronous replication it doesn't help here and the nodes are not optimized for memory and there are pretty tiny and they couldn't actually hold then it doesn't make sense and the cluster networking in the the whole cluster is inherently low quality of service then i wouldn't suggested to use nisha here so what's this framework that aids certain parts of the nisha to be more resilient and more feature-rich that's named it as tivan from tanalithi one from marvel comics it's just a utilities wrappers and behaviors that would enrich augmentee as a complete production database system the two key components of it is that minister of records will be using maps and that would mean that i can easily evolve it there's no static pre-processor binding to modules on you know what's the structure of the tables are and then there is the json conversions becomes easy and one another point that i took as a design principle is that i don't want to paint niche schema i don't want to create a meta tables or any of the sort to manage this framework to briefly look at the design of it the bottom layer is what we we have is the eds nishia with rox tb national rocks db plug-in which is a set lang rocks tb and then rocks db as a library to that the layer above that is the one which um uses a scheme manages the schema it mana and tim nisha is the one which translates from maps to records records to maps and performs certain converts you know maps based query to [Music] initial math specs and calls the select functions and there's a time page which uses which is which ties up with the ets to provide a better pagination for tables and queries or the if you want to have sorting and things like that the layer above uses the layer below here to get the tags working and the tiwan server a custom behavior on gen server and a fragmentation is um you know what what i have missed in certain parts of the problems that i faced i tend to i mean not provided by an initial fragment itself i had to create a separate fragmentation for that now um when is when you create a schema in using tivon there are two ways to create there's a standalone once you start the one as an application it creates a standalone schema with the disk copies by default you could give a um you know system config parameter um saying first is db false which then it translates it to that it doesn't want a ram copy schema it doesn't want a display scheme might want it the schema wants to be in the ram and you could also give the next remote node for to which it can actually connect and then attach to that cluster of the niches so this way you could actually grow the cluster chain by chain instead of pre-setting the cluster and then you know specifying what remote nodes you want to connect to now how does this work is that when you start this in application it checks whether a remote node is defined and reachable if yes it tries to give an rpc call to add the to ask to add this node to the cluster if not it just proceeds and if persistent db is already is set to false but the schema is a dyscopy schema already then what you intend to give config is that you want it to be a diskless node then in that case what the one does is that it converts all local tables into ram copies and then it converts the schema to ram copies and then making them all diskless there are few other things that i wanted to add on top of the thing i used backup and restore of nisha but the problem with the backup and the store of nisha is that it with a really large tables it seems to consume a lot of memory so i had to write my own backup and restore part of it and it adds certain other you know flexibility here is that it doesn't tie up to any node that means that you can take a backup of nisha from production load it into the lab from the lab to any any production system or you took a backup from three node system and then you want to now restore it to two node system i mean it doesn't tie up to the node so you don't have to go about changing the schema removing the nodes mentioned inside the schema apart from that you can mention the tables list and while restoring you can also give a thing whether you want to recreate the tables but drop them and recreate or directly you know pump the data in another function is about inspecting the backup which gives what table what attributes what indexes what storage type and the number of rows that are there in the backup so while creating the table the tivan create table wraps around the nisha create table but you can now specify in terms of maps you can give the columns in this fashion in terms of a list and you can give when you give you know your attributes or columns in this fashion and then you provide you wrap certain columns around as a tuple they're indexed so c and d are indexed apart additional to a and when you say memory true uh it's your option whether your table wants to be in memory or not whether it has to be persistent or not and type of the table other niche options which which um you can you want to add to it whether you need transformation of the table when it needs to be created every time it creates to need the table to be auto transformed is a boolean here and the defaults here when you move from the record to map you don't have defaults here records have defaults so i had another field where you can specify the defaults for certain fields that you like to have defaults for otherwise undefined is considered the very creation of the table has the steps like storage type in the storage type based on the memory and persist the idea is that if the person's db set to false and the table requested is asking for a persist it still doesn't do it just creates a ram copy for it because the schema is false and this is different combinations of that based on the memory persist and the versus db flag based on that it either creates a remo there's one thing here is that you can set both false then it typically means that if there is a remote copy available then let it be i don't need that remote copy locally but i want that remote copy to be accessible to me then this particular request will throw an error out saying that the remote copy isn't there if the remote copy is not available this is i will discuss why we had to use this the second step is to get the table in case the table status i mean the current table doesn't exist and it needs a remote copy it throws an error if it have current table doesn't exist and whatever table you are requesting it creates a table if the current table type and the requested table type is the same it does nothing if it's a remote copy but and you're asking for some copy in locally then it makes a table copy if it's a local copy here but you i really want a remote copy it just removes the current copy and if the storage types are different then it just changes the local storage type after it gets uh creates a table then it does waits for the table to be completely loaded because once you create the table probably the table already exists and it's trying to read from the disk to the ram you know all the rows then it waits for the slabs of 50 seconds 25 seconds 12 seconds six seconds and finally five seconds and until the table is loaded it keeps waiting every five seconds now that step of the table transformation happens unless you provide it to transform to false and if there is a change in the columns that you specified and the columns that are already there in the in the table are different then it drop then it it's there's no difference in the columns and it deletes unwanted indexes in case you've specified you change the indexes um you know needed now different from the one which is already there then it adjusts to that by deleting unwanted indexes and adding additional requested indexes if the columns are different and the transform option is not set to false then it drops all indexes apply the transformation accept the default values from the you know fields and then apply new rows and then it recreates the indexes as requested in the create part these operations of write and read are also changed a bit i did change a bit because now we're going to accept maps so um this is this is how the map structure is going to be you can give all the column values that are needed and you can also specify the context we'll discuss what is context here later and one change from the initial right is that initiative input always returns the primary key you can always give a set of you know right objects to write and you can get a list of the primary keys of the objects written the get has three models here the first model is directly uh you know um you can give the keys and you'll get the keys you will always get the maps with the list of maps and you can give a multiple set of keys to as a list and you will get you know different objects out of it the second way is that you can give a start key and a row limit this one uses nisha first nature next initial read functions underneath and then it it gives you that you know set of key limit of keys that you need starting from a key and it always returns as you know a tuple with a next key and the set of the you know objects that you asked for the third model is a select and match model um in the match and select model you have um you know you can give in terms of map a pattern in terms of match you can give a field and a pattern and you can give as many fields and pattern that you want to match and then you can also give an underscore atom and then give either a binary or multiple binaries to do a full row text binary match and you know do the filtering on that select is to select what columns you actually need from this query the context is we'll discuss the context later the continue and the rows limit are a way of pagination through nisha select mechanism so an example of that is match the id with range one two two we'll discuss this thing in the next step and i selected the pin and timestamp i get the list of objects and if i happen to use any rows limit then i get a continue and opaque uh you know structure from snesha select and i can pass the same thing to get the next set of it and if i don't have anything else i'll get an end of table and then the list and these are the patterns that are available you know you can prescribe you can either give a value directly you can give multiple values you can given you can do any kind of guard evaluations here or you can give a range of um you know is it's like i want it from two to five or you know kind of list of ranges that you can give you can give a binary prefix to search you can put it three dots at the end there's another set of it which also you can you if a field is a complex way is a is a collection field with tuples inside each field then you can actually specify the size of the position of the tuple the size of the tuple and what you want to match and the same matching of all the normal values can also be done here the process the way it would it is built is that the it takes a match and select it builds a math spec then it calls an issue select picks up the record checks whether there is a full text search if not it just keeps to converting to maps if s then it does binary match it before it does that it strips all the undefined attributes and then it does memory match and then goes to converting into maps and then returns on objects an update would take the similar way like a match um in in what we seen in the get part of it and the context as well and then it takes some another map as an updates for different values that you want to update you take something like some picking up all the pins and i apply you know for them records of two and three of the ids i want to change the status to burnt then it does this and then returns those primary keys for which it's actually applied to and this is the response for that remove more or less works the same way as this but it what yeah you just have to give the key the server behavior is the one which is a custom behavior built on gen server it enriches the create function by um you know adding certain other um fields that are needed something like if you don't specify a primary key then it adds a uuid and it automatically manages the uuid value and if you have aspirin and audit fields it creates audit call fields and then creates a table and if you ask for a tags out of band tags for this particular table then it creates this relevant tags table outside of the table out of this table it enriches the put functions by um you know doing the limits checks um type checks uniqueness checks foreign key reference checks it also enriches the object by adding the uu ready for primary keys and audit values you can also do the tags management by adding and removing the tags and it also manages the context of operation in the get of it get operations it has a more liberal query parameters than the normal uh they won't get and it facilitates you know that searching on tags excluding inclusive tags and you know getting the tags for an object and it can also do a referring and expanding the foreign key reference that you have in any object in any field and it can get that particular you know reference and then include into this and it has a fast pagination and column sorting functionality through out-of-band cache module the servers typically looks like this where when a callback module usually starts and then in it causing it you provide the table definitions it gets processed and then it will cause a create of different modules a function call you know a get put update when you call into the tivan server behavior there are two ways you can either use per system and then pass it on to the internal functions and then do the operations or tunnel it through the gen server if you really want to have it serialized or queued through the gen server the table definitions uh kind of looks like this where you provide the columns in the column and the type binary list double atom integer float second millisecond any of those steps you can also specify you know the other table you can specify the primary key of other table or a list of primary keys of other table or a list of a certain other field of that particular table other than the primary key you can also specify a single field of that other table or you can also specify any or you can say that it can be any of those types in this you can limit it by saying you know max value or max length or a range of length or value of the integers or is a collection of objects to which if you specify the type as binary and you say it has to be these three objects only then it does limiting on that a key tells it that this is a primary key if you don't specify the key that typically means that it wants you you want the server to add a uuid and then manage the uid itself you can index it you can say you can specify the unit to check for the uniqueness and check for whether it is null and what kind of default values to be provided the memory persist is the same as the create the type also the audit if we specify true it adds a couple of other fields to see when the table is written when the table is modified things like that you can do an out-of-band tags table by specifying what kind of you know tag table that you need to use you can use common one or you want to use specific to each table is up to you you can also check the unique combinations um you can say that if you know column of this and this together should not should be unique across the table you can specify that you can read context and write context you can always override for this particular table you can specify any other options as well okay an example definition of something like content repository looks like this is the contents with the name and links you can also specify in the limit a regular expression it runs a regular expression on this binary there's a containment bundles which takes a type here as content so it can have multiple content ids into this etcetera in the contents column and the content packages say that it either accepts a single content or a content bundle of contents and you can specify the price and duration just an example there's a difference between the server of the put um it returns in terms of a map with the primary key it's giving the primary key as well and it also does this checks whether you know you haven't provided the pin or the pin that you're referring to doesn't exist the gate is slightly different we've seen this pattern before you're going to add a few other things the start limit sort column sort order ascending or descending you can expand if you want to expand how what level you want to expand the cache and you can you can skip the match and directly provide the columns along with the meta info you know requests and do not to collide with the actual column names you can you know prefix with underscore yes typically the way it works and the output of it if you are using pagination it works like this the object that you're referring to then the rates of the and the size of the total size and the cache of that and if you don't specify the pagination it directly gives the list of objects the context here is about passing dirty or sing dirty transactions sync transaction ets there are three levels of it if you provide in the sysconfig that's a global one and if you have a table definition that's the second level of it or if you can specify individual functions you can also pass the context to it to override the one in the global the tags is an outer band you can do exclusive inclusive the one which i used here is a bag type the pagination is an ets table which again has two methods to do it if you specify that you're gonna manage then it passes uh you want the server to manage it passes through the server gen server if you want yourself to manage it goes without passing the gen server this is something um i wanted to complete it but still hasn't completed it is we have a problem always is that what if a table can't fit into memory but i still need a good read then you can have a fragmentation of a table into ram copies and disk copies and you can specify by time space or size how what you want to split how much of it it should be in the ram and the rest should be in the disk and you can then pull out this ram copies into manager from a remote from a stateless docker container and run it there and the one which was specified there as you know the persist falls memory falls to ask saying that i need only remote only copies that's when this thing comes in the stateless docker container you can skip the disk copies part and only get the replica of the ram copies the shortfalls in the schema there's few shortfalls which i had to still work on um these are some of the beautiful third parties because of with without which i couldn't have completed it and to give a bit confidence it ain't actually walking on the tightrope it's far more than that it's way more stable yep any questions i know i we have quite a few questions so that's a good thing um yeah fantastic presentation and like personally really excited it seems like because you're using maps it will be easier to use with elixir probably yeah so i have to check that out uh but um but let's go to the question so i think we have unmuted the mics now here so let's start with gustavo v agus has a question if you want to ask it yourself uh or i can just ask it um what's the benefit of using nisha as dpms over more known technologies such as postgres or dynamo db um if your services are all predominantly an airline native services then using initia as the database makes life's a lot easier in terms of um designing optimizing and you can write because the whole data is inside the same user space it's it's gonna the impedance to read it send the query out get the data in um it's not going to be there it's all very local to you and and you can pack as much you know algorithm as complex algorithm into that and just run it it's just going to load the cpu it's not going to load any of your io if i had time i could have discussed a very complex call center algorithms that we did very very very in you know complex and thing which loads the cpu heavily but it still runs as a one monolithic piece so it's it helps us there and uh andrei fatson has a question hope i didn't butcher his name my question about um i'll just read it uh what is a typical and or maximal amnesia load you can handle in your production system uh right load especially um i haven't measured in terms of iops but um we could handle around thousand trans a thousand transactions a second and that we've seen that actually thousand and each transaction would at least have about 10 or 20 rights i think you can do the math on that um there's someone speaking let's see do you have a question um okay so do you have a do you have any problems with long startup times when asia grows in size no um yes yes yes we do um however we made a trick to that is we don't actually do bad you know we don't do a whole cluster backups we split the cluster and then we upgrade part by part and then when we do that getting um you know reading those and you know the huge database like 45 gig to a memory and then it takes a lot of time but what if i get from this neighboring machine or from a network and the ram isn't so much okay yeah and um paulus sulatto has a question does three just even help with recovering cluster from network positioning uh did you face this kind of issue um we know about the network um partitioning problem with nisha so we design it such a way or we architect it such a way that um our network is really good quality of service and so we since it's been running for three years across two racks of you know blade servers we we did see one rack completely going off a couple of times but nichia does actually recover in some situations where it really understands that the other node is not up to date it it's in very in in a very in other cases where both the things are isolates and then both are working and both are updating that's when you would have an issue but we do work around that too again as a the solution is i mean using nisha on the production is works very well when you have a small and moderately large services where you can accommodate with packed high volume high powerful um you know vertically powerful nodes something like 405 or until six so yeah um to simple answer no we haven't seen we mean we have been waiting for the such an event happened so that i could accommodate that into this framework to see how it works so far now um yeah and matthias has a question uh with remote only tables could you separate the storage node from application node and use the tibian api to communicate nisha does that actually directly you don't need an api for that especially i mean we have these dockers which starts up and then actually we have two vms running with 24 gig and both of them um are hard nodes they have disk copies and everything when docker starts then it knows to connect one of the node and then it pulls up that you know ram based session tables and it it and the rest of the tables are all again in the remote nation does this transparently whatever you are asking whatever you're putting in the query it automatically does you know it checks whether in which node this particular table is and it goes there and fetches the data for you is that something i answered it or it's something different let's see we have two more questions so for the tax functionality it seems like there's this isn't an amnesia concept but a design pattern that you use to organize dimnesha tables can you explain no i didn't understand the question i can try to read it again for the tax functionality uh it seems like this is not something that's uh natively amnesia but a design pattern that you have put on so the tags part of it it just bag the table type is bag initial bag you can have multiple keys i in for the tags i did multiple i checked multiple of things um i tried um bitwise operators bitwise way of um you know storing the tags and you know having collating multiple tags and indexing them i did um you know other a few other algorithms but finally the bag seems to be it it has a problem a airline bag has a problem that if you if the table loads beyond a certain point of time then it the rights are very slow but for some reason i tested it even for a million or 100 million um tags but still it seems to cope up it's basically back data type i don't know if it um if the person who aaron levelia would like to follow up on that hi hi can you hear me yes yes um yeah uh i was asking about the uh the tags that um there was a you showed a couple functions that said like search by tags or get tags and it seems like this was like a like a key value tag for certain tables that that tibbin does to to organize the tables you're talking about this uh yes maybe that last function call yes yes yes okay um what what i typically mean i mean these three functions i'll explain that you specify the number of tags that you want to access the entities for if somebody wants to search the movies which are both sci-fi and comedy r comedy then it's an exclusive you will get all the objects that are there which which has both the stacks when you say that you need a sci-fi and comedy which is inclusive then you search that particular movie which falls into both the genres when you say i need all the tags for this particular object that's when the last function is

This video was recorded at Code BEAM V 2020 - codesync.global/conferences/code-beam-sto/ Mnesia as a complete production database system | Chaitanya Chalasani - Senior Cloud Application Architect at Access Europe GmbH TALK LEVEL: INTERMEDIATE / ADVANCED Mnesia is often underplayed as an in-memory cache and is seldom used alone in production. The first part of the talk is to present a few case studies of using only Mnesia as the production database engine for large throughput services. The second part of the talk is to present an open-sourced framework for Mnesia (Tivan) which is a collection of utilities and behaviours that enriches its role as a prime database engine. THIS TALK IN THREE WORDS The Understated Mnesia OBJECTIVES The prime objective of the talk is to look beyond general perception of Mnesia as an in-memory cache database to an apt contender as a complete robust database system through case studies and presenting an open-source framework of utils and behaviours to enrich its features and functionality. TARGET AUDIENCE Programmers with a certain exposure in using Erlang/OTP and Mnesia. Architects who are aware of Erlang/OTP, Mnesia and its capabilities. • Follow us on social: Twitter: https://twitter.com/CodeBEAMio LinkedIn: https://www.linkedin.com/company/27159258 • Looking for a unique learning experience? Attend the next Code Sync conference near you! See what's coming up at: https://codesync.global • SUBSCRIBE TO OUR CHANNEL https://www.youtube.com/channel/UC47eUBNO8KBH_V8AfowOWOw See what's coming up at: https://codesync.global • SUBSCRIBE TO OUR CHANNEL https://www.youtube.com/channel/UC47eUBNO8KBH_V8AfowOWOwThis video was recorded at Code BEAM V 2020 - codesync.global/conferences/code-beam-sto/