The Rails Delegated Type Pattern with Jeffrey Hardy

You got to think of the recordables as being pretty dumb. You know, they're just like in the case of a message, it's literally just a title and content. That's it. They have no connection to the outside world. You can build entirely new features that should take months in like a week, 2 weeks. We've been running this architecture for 10 years. This is recordables place where the 37 signals team share some of the technical work behind Base Camp, and some of our other opensource projects. I'm Kimberly from the product team joined by Fernando who's helping out with some of the technical aspects of today's discussion and this one we're excited to kick off this series with an episode about recordables. To do that we have principal programmer Jeffrey Hardy with us today. Jeffrey, thanks for being here. >> Hey, thank you. This is great. >> Before we get started, I know this is a topic people really want to hear about, but tell us a little bit about you and what you do here at 37 Signals and how long you've been here. You've been here for a while. Yeah, I've I've been at 37 Signals for 18 years. So, I've worked on all our products, all versions of Base Camp, including the most recent version and Hay and everything we've done. It's uh it's been a really great place to spend 18 years and have the advantage of seeing the evolution of of Base Camp through all these different versions. Um, so yeah, I pro I work on the product team. >> Amazing. Well, we know people want to hear about recordables. We're excited to kick off this new series where we're sharing some of the behind the scenes stuff with this episode. And before we dive in, I think I have to ask kind of the obvious question. As the non-technical person here, I always hear people talking about about recordings and recordables. Will you just kind of break that down? What does that even mean before we deep dive into the topic? >> Right. It's a little bit abstract. Recordings and recordables and buckets and and what is this? So in Base Camp pretty much all content things like documents, messages, comments, uploads, these are modeled as a a pattern we use called um a delegated type. And so in the delegated type pattern which comes from Rails, you have a primary table. In our case, it's recordings and recordings reference recordables. And the recordables are like the concrete types. A recordable would be a message, a comment, a document, or an upload. And so with this pattern, we're able to model all the various content in Basec Camp, which is essentially all the same. There's a bunch of advantages to treating all that content in the same way. And Basec Camp's architecture like hinges on this concept. And it's that, you know, our third iteration of Base Camp. We're on we we we call um the version of base camp the internal version is three. We call that the chassis like the architecture on which it's built. It's our third attempt. And so when building the third version we came up with this pattern that we use and it's called a it's called a delegated type which I know doesn't make a lot of sense. What is that what does that mean? But I'll I can describe the the the sort of hierarchy. So you have recordings and this is a table of records that stores all the metadata and common information between recordable types like messages and documents and uploads. But it doesn't have any specific information. It doesn't know like the title of a message or the content of a message >> or the location of a file or other things. >> It delegates that that knowledge to other records. Those records have their own tables in the database. Messages, documents, uploads, comments, all separate tables. And the advantage here is that you have your your single recordings table that delegates its types. Can be lean and tight. It only has metadata. It has references to recordables. So it has a recordable ID and a recordable type. It has timestamps. It has a creator ID. It has tiny little columns, no text columns, nothing big and heavy. And so all of the specific information is contained in the recordable tables which can vary. So to back up a little bit in Rails and in in software architecture patterns in general that there are a few ways to handle this idea of like things are mostly the same but they're kind of different and how do you account for the differences without repeating yourself? And one common way is a pattern is called single table inheritance. You have a database table and you have subtypes that also use that same table you know to that's where you would have your shared metadata the things you have in common. And then you know comments might have a body and messages also have a body and so you're like great I have a body column I can use the same table to store these. I just have to keep the type in the table. Is this a message row? Is this a comment row? That works good for things that are super similar, like maybe messages and comments, but like what about things like events? They have a starts at and an ends at column. Now you got to add those to the main single table, right? And so the table keeps getting bigger widthwise. It has to have all the different kinds of columns that you might have for any type. And adding a new type means you have to modify that single table. and that table gets really really big. It's just sort of intrinsic to to having a single table that does that. The bigger the table is, the harder and slower it is to migrate. So that's like one way to do it. Another way is to have like just different tables for everything. Um and then you take the things that are the same and instead of belonging to a specific kind of record like uh lots of things for example can be commented on in a system like Basecamp. So you might have a documents table and you might have an uploads table and a messages table and you you want comments in all these things. So the comments table could instead of belonging to a you know having a reference to a a specific thing like you know having a document ID or an upload ID you know or a message ID. It can belong to a polymorphic type. It can store you know a commentable ID and a commentable type. And so now you get you know this is called like polymorphic relationships. That's another way to model this sort of everything is the same but slightly different. we need to differentiate and the delegated type pattern is neither of these but it's similar and it's something that we you know uh I I think we pioneered it at base camp maybe some other really smart people have come up with this but it's possible they haven't um but that's what I can talk about today is like how this system works it's it's um it's a really neat system and it's what's enabled us to to scale Base Camp and to use the same version of the chassis of Base Camp version 3 for multiple subsequent versions without having to do a rewrite. >> Yeah. So, just to clarify, we're currently working on Base Camp 5. We said that, but we're building it on the same platform as three was built and four was built and that is because of this recordable situation. >> Yeah. I mean, the main reason we made other versions of Base Camp was because they become too difficult to work with. they're kind of rigid and there's, you know, we also want to introduce new features or sort of take a different have a different take on how we do something and that's hard to change, you know, when existing customers are using it. But it's also been good to like sort of have a blank slate and start over because you don't have all this baggage, this big ship that's hard to turn around. Well, using this uh delegated type recording recordables pattern has made the ship less big, >> you know, like it's it's um I was talking about single table inheritance and how you've constantly got to modify that single table and migrate it every time you want to add a new kind of you know thing. You don't have that with the delegated type recordable pattern. Your recordings table basically never changes. you just add new types which are their own fresh tables. And so there's no penalty to adding a new type deciding that like you want to add comments or messages or documents or any other thing that we can imagine. You know, you just make a new table and you'll have new recording rows that reference it, but you don't have to modify recordings. So it's been much easier to scale than other versions of Base Camp. And consequently, we don't start hating, you know, like the effort of doing a small thing. You want to be able to iterate on your product easily and you know I think that in the past making a new version of Base Camp was the the best way to do that you know brand new and we we just haven't felt that pain with the with the version three architecture and it owes entirely to this recordings recordable pattern. Now there are a few things about that like it's not just the delegated type. So, for those that don't know, delegated type, it's it's available in Rails. The documentation in, you know, um, in the Rails docs is pretty good, but I think a lot of people don't quite understand like when they would use it or like why to use it. So, hopefully we can like clarify that today. Um, but there are a few other patterns that we use, you know, in addition to the de delegated type with recordings that make this content system possible. One is that the recordables themselves, we treat them as immutable. So we don't modify them in place. We create new ones. And another thing that we do is the recordings are organized in a tree like a parent child relationship. So for I'll go back to the messages example because it's the most intuitive. We have a message board and a message board's children are messages and a messages children are comments and a comments children are like attachments if you've uploaded a file to your comment or you know and so like >> what about a boost would a boost be a child? No, boosts are different. Boosts actually use that other pattern I was describing uh called polymorphic relationships where a boost belongs to a boostable thing. And that's because we want to boost things that aren't necessarily recordings. >> You might want to boost like a status notification or an event. Events are another key part of the pattern. So that that helps me tie into events. So we have the delegated type recording recordable pattern. We have the arrangement of recordings as a tree. We have immutable recordables and then we have an events system and the events system can tie a recording to a recordable at a particular moment in time. And so this is why so the the recordables themselves are immutable. So we never change them, but we can move the pointers to them around. And at any given time, a recording only points to one recordable. But if you want to have a history of like what are all the recording recordables that it ever pointed to, we track that in an event model. The event has a cop has a reference to the recording and it has a reference to a recordable. This means we can like look at the history of a recording and see all of its changes and you know look at that recordable that is immutable at any moment in time to see how it looked. >> Yeah. Here's a good example here. >> Yes. Yes. I know what you're going to going to show here. This is great. >> Do you want to walk us through this? >> Yes. So, this is a document in Base Camp and what enables this but we have the change log here and what enables this is the this combination of immutable recordables and events. So, what you're seeing here in this history of changes is really a listing of all the events for this particular recording. And so where you see like you know saved a change to this document and you can say see what changed what we're able to do is find the document recordable instance at that moment in time. This is how it looked. This is what its content was and compare it to the previous version. And we can only do that because we didn't update the document in place. We created a new one. Now we can compare version A to version B. And you'll see there's a button there like make this the current version. When you click that button, all we're going to do is update the recording record to point to this version of the document instead of the current version. >> So recordings are not immutable. >> No, recordings are fully mutable. We change them. It's the recordables, the type that we delegate to. >> But what are we changing in a recording? >> We're only changing the pointer reference to the recordable, right? >> There are a few other things that can change like one is its time stamp. Like it's updated at time stamp, >> of course. Mhm. >> Right. Or if um most thing other things won't change. We we store a color value on the recording so you can change its color, you know, um presentational things like that. >> But >> Oh, that makes sense. >> Yeah, you're right. The the recording is mutable, but many things about it don't change. All of the meat, like all of the action is in the recordables themselves. And and you mentioned like for example a document can have comments. And so the logic I have in my mind here is I want to get to this document. I have the recording. The recording has a reference to the recordable document. when I fetch that recordable that document which is a recordable uh is are the reference to its children are the references to its children recordings. Yeah. So it's only the recordings that have children. So you look up a recording by primary key and then you that would be the ID that you see in the URL and then you ask for its recordable. And the thing is you don't care often. You don't care what kind of recordable it is. >> Um if it's a document or a message. When you do care, the pattern that Rails exposes lets you ask for that type of recordable by name. So if you know you want a document, it's you're expecting a document to be the recordable. You wouldn't just ask for the recordable, the generic recordable. You would say recording.cument or recording comment. Another convenience that Rails' delegated type provides is the ability to query like and filter for the type of recordable the type of delegated type. So you would say recordings domes messages would only return recordings that have a type of message or recordings comments would only return recordings that have a type of comment. And if I do like recordings messages children, I would get back an array of recordings. >> Yes, that were all that were messages, >> right? >> No, no, no. The the children of messages, let's say comments. >> Yes. >> Okay. >> Yeah. You're always going to get recordings. So the children and the parent, they're always recordings. >> Oh, >> so yeah, this is a good point to clarify something or No, >> I I kind of can. Yeah, I can show this. Um it might help to see Yeah. how the code is organized. Let me share my screen. So, here I'm on my Mac. I have been using Omari on a Framework 13 laptop, but my camera is not working great right now. Not it's probably configuration. I don't want to say anything bad about Framework because it's an excellent device, but I I didn't want to risk it for this this presentation yet. But, okay. So, this is the recording class in Base Camp. So you know what you're seeing is a whole bunch of concerns. Basically you see there's very little like this is a for you know the main model in base camp is 42 lines long. It's kind of incredible but like you can see a few things immediately that are important about a recording. It belongs to a bucket and I'll talk about I can just introduce the bucket briefly. The bucket is just a container for recordings. Buckets are how we control access in basecamp. So you have buckets have accesses. You add people to a bucket. If you can have access to the bucket, you can see all the recordings that are in the bucket. Buckets are also a delegated type. A kind of bucket is a project, a template, a ping. Anything that can contain a distinct set of recordings >> is is a bucket. >> So is a bucket a recordable? >> No, a bucket is a delegated type which creates a bucketable. So you have the recordable is just the name we've given to the target of the delegated type right. So in hey we use the same pattern except in hey we've called them entries. Entry has an delegated type which we've named entryable and in hey those are things like messages, replies, notes on a message and these represent emails. So the naming can be whatever you want but the the pattern is the same. So buckets have bucketables. Bucket is the thing that delegates to a bucketable. >> Oh, okay. Perfect. >> A recording is a thing that delegates to a recordable and in hey an entry delegates to an entryable >> and so a bucketable can be a recordable depending on the context, right? >> I mean technically, but no, you wouldn't do that, right? Like bucketable would be a different thing. Um but there's no reason. They're just classes. They can reference any kind of thing they want. But we want all of our recordables to behave the same like or have a similar interface. And you would want that of all the types you intended to delegate to. Right. So you want all bucket-like things to quack but like buckets >> the bucket buckets. >> Yeah. Right. like as containers of recordings in this case or in hey as things that are like email messages or in base camp recordables things that are like our our chunks of content documents messages uploads you know so mixing the types while it would be technically possible would be kind of confusing right so the thing I want to point out here in the recording class is right here this little inclusion of a Ruby module called recordables and I have that up here. This is what the recordables concern looks like. And right here on line five, this is what does all this work. So we say imagine that this is mixed into recording. So we say that recording has a delegated type called recordable. All the types that it can have are named in this constant which uh is in another file which I'll show you later. Um and and like this does all of the work, right? And so these are by mixing this into recording like these are all the things that a recording can can do. But you see this is also a really small file. It's like a 100 lines long. Um it it doesn't do a whole lot. It doesn't even include a lot of like other behavior. So like where does all this come from? And like so this is sort of the key. So this is the recordable class. This is what every individual recordable class like a message, a comment or whatever includes and this is where all the types are defined. So all of these like attachment, an autol image, a bulletin message, a bulletin link, all the chat things like a chat, everything from a chat line to a chat transcript, a cloud file, a comment, a doc, a document, a door, all of these things are recordables. So like every one of these content types, which is tons, are are all modeled in the same way as far as recordings and recordables go. And then all of the things that recordables have in common are listed here. So I'll scroll down a bit. Like these are things that they can do. You know, are they autosubscribable? Are they subscribable in general? So this is where the type that we delegate to defines its capabilities. So by default no recordables are subscribable. But if we go to like a document and we look like oh it it enables subscribable here it includes the recordable class the recordable mixin and now it overrides the subscribable method to say yes I can be subscribed to I can be exported I can be commented on. I'm autoositioned when I'm placed inside a list. So all of these things you can kind of like respondable backlinking like when uh one recording references another we can create a link to that so you can navigate it easy in base camp if something's copyable or movable. Um if it's recurable or recurring if it repeats you know um these are things that you can ask of any kind of recordable. >> Jeeoff is there something in here like we have public links that we can make. Is that part of this? Yeah, it probably would be as like publishable. >> Publishable. >> Yeah, >> publicly linkable. >> Looking for it, >> right? You know, is it memorizable? Um, but I think do we have I think all recordings we've defined as are publishable. So, we tend to list only the things that might need to vary, right? So if if some things are commentable but some things aren't then then we define like a commentable capability and let subcl let other recordable types define it override it. You can see some of our like comments here too like there's life cycle events that happen. A a recordable is recorded meaning we've created a recording that references this recordable. And often you have specific behavior that isn't going to apply um unless you uh it only applies to specific types, right? Like you might want to do something with a comment being recorded that you wouldn't want to do when a document is recorded. And so this is one of the challenges of the the delegated type and type pattern in general is that how do you where do you stick that specific behavior? you're mostly working with the, you know, your type that does the delegating, the recording, but like you got specific behavior that should only apply to like certain recordable types. And in that case, we we pass in the context of the recording because recordables themselves, they don't know, they don't belong to any particular recording. They can be referenced by many recordings. And in fact, that's how copying works, right? Right. And that's how we make copying really efficient. Instead of like actually copying the content of a message, we just create a new recording row and point to the same message recordable that already exists. We don't need to copy it at all. And it's super fast and storage efficient for that reason. But it means that a message doesn't know it doesn't have any particular recording that owns it. It's own it can be owned by many or none. Um and so sometimes when you need to work with a recording we we pass it in. So here we can say like you know the comment reads overwrite to perform recordable specific actions after making a recording. So comment uses this to compose its title based on the parent of its recording. Right? So it that's the whole purpose of buckets. Then recordables have no understanding of access control. So if you're in a private project, you have your own document there and you want to copy it over to a public like it's just referencing the same recordable. The recordable change there's no change in access to that recordable because access goes through the bucket. >> Yes. >> Right. They're sort of orthogonal in that sense though. Like it's more that like the the the record the recordable is it doesn't even need to like we don't even keep things like timestamps on our recordable records because they're just chunks of content. >> Yes. >> You know what I mean? Some recordables don't even have any any row like any columns, >> right? They're just a table that has an ID. Like it has what you know their rows just have an ID. They sort of act as a placeholder if they don't have any distinct content to store >> or they might just have a title, >> you know, >> right? That makes sense. So they can be as small or as large as they need to be. >> They can be Yeah, they could be as small or as large as they need to be. And you get to modify them without having to modify all of the recordings, which is becomes the big table, the table with billions and billions of rows. >> Right. This is I know it's a little abstract, right? Even when I'm showing the code, it's kind of like >> No, no, no. I think it makes perfect sense. You have the recordings which themselves are uh just fixed point uh they're they're the latest uh pointer to a recordable. >> Yes, they reference >> usually the latest version. >> Yeah, usually. And then however if you needed to go through the history of how that recording got to a certain recordable you go to the events table right >> yes you ask the recording for its events and you could go over each event and say what was your recordable when this event was created. Now for example if I'm in a project how what is the what is the the algorithm what are the steps if I'm opening like a tool and I want to see the the the messages right so the bucket has recordings the bucket is the project you know a bucket delegates its type you know buckets are just things that have recordings and so if you were in a project we would be looking for buckets that have a type of project and then from the bucket record You can ask a bucket for all of its recordings. You would want to filter those recordings to just those that are messages. So your call would look like bucket recordings messages. Now you're going to get a list of recordings that all point to message recordables. Right? So in Base Camp, the messages themselves because of the tree structure are owned by a a message board. So every bucket has one or more message every project bucket has one or more message boards. You would find a message board bucket.recordings.mssageboards.first. This would give you a recording of type message board and you would ask for its children. Now in Basecamp the only children of a message board are going to be messages but you could have different types. So you would want to filter that down and say like just show me the messages. So you would say message board, your message board recording, you would say children do messages. Now you'll only get its children that have a type of message. Still recordings, it's recordings all the way down. When you finally get to the message recording that you want, you would ask for the recordable. That's where the title is and where the content is and where other, you know, like message specific things are. >> So how how do you enforce that? I I'm coming from a very compilery part of the programming world and all I'm hearing is like let's make a message board hold pings and chats and >> right you don't enforce it right like there's no method to enforce that it's it's just how you do it it's I guess you could say it's convention like you know um but but and that's why the the conveniences that delegated type provides like the ability to filter on type are useful probably wouldn't want to just ask for like any child, right? Because it could contain mixed types. Let's think about um in base camp you have documents, but you can have Google documents and uploads and all of these go into a container, a parent that we call a vault. A vault is a thing that holds document-like things. So it wouldn't you wouldn't want to just you know if you ask a vault for its documents, you're only going to get documents. So there is an instance where you do just want to ask for children like all of the things that this might contain you you want the mixed result. So this actually brings me to another cool thing like this is an one of the reasons we have this pattern is this mixed use case. Let's say you have a timeline. We have a timeline in base camp. We want to see like everything that's happened you know so um at a global level. How do you do that and pageionate it if all of your records are in different um tables, right? Like you have to, you know, select from the comments table and the documents table and the uploads table and the messages table, like everything. It's it's super hard. But with recordings, you're just querying for recordings. You can say recordings where recordable type is message, document, comment. you're going to get all the ones that are that type in one query that you can pageionate with a limit and an offset. Super handy. And that query itself is cheap because you've only asked for for recordings rows that are not big. They don't have any text columns. You know, as you it's an n it's sort of n plus one by design, right? But as you you iterate over those then you go and fetch the recordable records and to improve performance you can preload them you know get them all in one batch for each type you know but the the the key thing that you're able to do is work with just recordings as if they're all the same. So I'll give you another example unless this is too much but another example where this like uniformity matters. it's copying, right? Or or or any actually just think not just copying, but like any generic process that you'd like to do with a recording. We can export recordings, right? We have an HTML export feature in Base Camp. It's nice to be able to write the exporter service that doesn't care about its individual types. All of these the the stuff that would apply to exporting a recording is common to any recordable type. you know it knows how to write itself like it knows it has to write something to to you know some output and then it can delegate the like what to do to its recordable so we basically say like recordable write out your export format thank you means you don't need to change the exporter when you add new recordable types right each recordable type just has to define the format that it exports as same thing with copying and moving you write a copier that works with recording That's it. It doesn't need to care about what kinds of recordings it's copying. It as it iterates over each recording that it's copying, it sends that recording sends a message to its recordable that says I would like to you've been requested a copy has been requested of you, right? And it just does its thing. So again, adding new types, there's no penalty. You just add them and the system can can work with them. So any of those uh what we have in base camp like a bunch of controllers um these these are what you know field requests that come in over HTTP. So instead of having to have like if you want to trash a document or archive a document we don't need a separate documents controller for this we have a recordings controller that knows how to trash things and archive things and restore things. We just need one. It works with any kind of recording. Same thing that's how copying and moving works too. We don't need a document copier and a message copier and an upload copier. We have a recording copier, one controller. It takes the recording that you want to copy and a destination. And the copier itself and the controller that orchestrates it doesn't care what kinds of recordables it's working with. It just works with recordings. So I think that's one another one of the reasons why Base Camp 3, that architecture has been so easy to maintain and scale. When you decide like, oh, we're going to add hill charts. Oh, we're going to add, I don't know, templates. Like, you don't have to change or migrate the existing system. It it uh it adapts well to change. You just insert new types and the system knows how to deal with them. >> Are you at any point worried that the uh the recording table is like massive, billions of entries? >> I mean, it is billions of entries. Um not really because its size on disk is small. Um, so it doesn't take much to index it. So, but I mean, and if it did, let's say it did get too big and we had to shard it or break it into pieces, it would be much easier to do because it's an inherently lightweight table. It really just has foreign key references in it. That's it. You know, by contrast, like the messages table can have like many megabytes of text, you know, and these are big on disk. And so the way copying or making an index on disk works is that you got to copy the table. Like it's still an entity on disk that you need to copy and the bigger it is, the slower it is. So yeah, that there is a danger that recordings gets too big. But if it does, it's much easier to deal with than if it was storing content. >> Jeffrey, you've made it sound like this recordables pattern like that we're using is much simpler, like makes things a lot easier. Are there any downsides? Yeah, I think the probably the main downside is familiarity and like the with the way you would quote unquote normally do this in Rails. Like normally you would have a message class, a document class and it would have everything it knows about being a message or a document would be encoded in that class. I would I would call that a rich class. It has all the functionality in it. This is what the active record pattern is about. It's backed by a database table, but you never interact with the database table directly. You interact with your message model. And so if there's specific things that a message would do, you know, that are not generic, you would write those methods right inside the message class or the comment class, you know, and with the delegated type pattern, you kind of want all of your types to be generic and you don't want and you're not working with them directly ever. like we never look up just a message because a message on its own is it's not linked to anything. It's not useful, right? So in order for a message to implement its specific work, it often needs a reference to a particular recording. Like if a comment wants to know, you saw in that one comment when I was showing the code where it said, you know, like comments can set their title based on the parent that created them. Well, a comment doesn't have a parent. A recording can have a parent. So in order for the comment to be able to do this, right, it needs to be given a recording to work with. Now it can ask for it's that recording's parent and be like, "Oh, okay, great. Now I can, you know, change myself based on some context." You got to think of the recordables as being pretty dumb. >> You know, they're just like in the case of a message, it's literally just a title and content. That's it. They have no connection to the outside world. they don't have any associations and nothing points to them directly except for recordings and events. So I think some of that richness gets a little bit lost like you don't do things quite in the the Rails standard way you know where you tend to think more abstractly like how is this a generic concern how if I want something to be commentable do I make this work with any kind of recording so it means that you would have to add some facility to the recording class that helps it stay generic that's where more delegation comes in so you would never just like add a comment to a recording. You would ask like is this recording commentable the recording can't answer this itself but it can ask it's recordable and pass the message along. So it can say like document what do you say when I ask are you record are you commentable and document says yes right but like something like I don't know message board the message board container itself says no I'm not commentable you can't put comments on me directly so in the generic comments controller one of the preconditions is is that the recording has to be commentable now you can have one generic controller that handles comments and doesn't you can pass it any kind of recording and it it just asks that recording, are you commentable? Like, do can I work with you? And if the recording says yes, you're good. And if it says no, it's an error. >> Speaking of drawbacks, is that I mean, I think I know the answer to this, but it's that is that slower. Are you trading like speed for clarity? >> M I mean like development speed. >> Yes, >> maybe a little upfront like there's a higher learning curve cost, right? Mhm. >> So when we onboard new developers, you know, it can be non-intuitive. It's like, uh, I want to add some behavior to the messages messages. I just opened the message file and there's like nothing in it, >> you know, like where is all this behavior? And we're like, oh yes, it wouldn't go here because now you >> Yeah, but it's it's it's like no, you wouldn't put it here. You you you sort of have to define it in a more abstract, generic way. And then I think that there's then it becomes a huge net benefit. Like once you've learned the pattern that now you you start to get like, you know, because then you're like, well, I want my message to be copyable. It's like, oh, that already works. I want it to be commentable. Oh, just add a defaf commentable method and return true. Done. All the commenting works. I want it to be, you know, exportable. Done. Just add an exportable method. So you start to realize that like oh what I've sacrificed in like you know being able to modif like make changes to message directly and add some richness to like that like um I'll give you an example messages can be categorized in basecam you can make categories it's useful but it's not really useful for other types maybe documents but we don't have that feature enabled um so I think in a standard Rails app you would add your you know you would say that a message belongs to a category and cate categories have many messages. But in the delegated type recordable pattern, you've got to make it so all recordings have the potential to be categorizable, right? And only message opts opts in. So that that cost is there upfront, but then you know a month later when someone's like, "Hey, can we have comments on or categories on documents?" It's like, "Yes, that's easy. No problem. And so like we've gotten a ton of leverage out of that. You know, it allows you to build like you know a lot of things that should be hard just are not hard. They come down to a matter of configuration. You have this the system behaves in a uniform way. And where I think this works best because this won't work in all application domains and we don't use a delegated type for every kind of model but in base camp where you have basically you know the same chunks of content like you would have in a content management system or like a wiki or like something like notion you know it what these chunks of content have in common is the operations you can perform on them and with them. And so like their ability to just participate in that system just by being defined is like incredible, you know, like we we added the uh the card table like our our our conbon board, you know, system we call the card table and it was like super easy to do because we're just like all right, we have a board. It's got children. It's got like you you know like you can watch a column. Well, that's just a subscription, you know. So we'll make those you know card columns are a kind of recording their children are the cards themselves each of the cards can have comments you know that the column itself can be watched the board itself can also be watched you're sort of mixing and matching behavior that already exists and then when it's like well how would I move a column to another a card to another column it's just a move operation all recordings are movable they can move to different containers so that just works you know like how do I get a reminder about a you know a a a due date I've given myself on the card like oh oh that just works because all recordings are remindable just you know and so you're like you can build entirely new features that should take months in like a week two weeks I mean excluding the design but like the the the the software modeling part and it's all uniform so once you've figured out how this works you understand the whole system it doesn't scale up with the number of types, right? Like it behaves the same whether you have one kind of recordable or like 200. >> It kind of sounds like Jeeoff, I'm imagining as you're describing this like a whole bunch of Legos or puzzle pieces that are just being redone in a different way. Like you're taking elements that you guys already have and >> yeah, >> reconfiguring them to make a new feature. >> Yeah. And if you think Legos in particular, >> grossly simplifying that, >> but Legos are a good example. If the Legos didn't fit together, like if they didn't have the same like hole and bump pattern, it wouldn't work, right? Like what you get is being able to say like, "Okay, here's all the Legos and they're all interchangeable. Like every Lego works with every other Lego." And like that's why it works, you know? So there there were all these things that motivated the pattern and what I don't think we could have anticipated was how well it would have it worked out like in the long term, right? from being able to query efficiently to to query across types with pageionation to be able to copy and move things really efficiently because we're not having to recreate like the message content every time. You know, if a message gets copied 100 times, there's still only one message recordable. That's it. You know, whereas in earlier versions of Base Camp, we would have 100 different rows that each had a copy of that content. So things like this adds up. Like whenever you start a new base camp account, you get like a uh a demo project. So like thousands and thousands of base camp accounts that all have the same starting content and we would need thousands and thousands tens of thousands of rows to represent this. And so you get a really really big database is base camp is a the majestic monolith. It's it's we don't use microservices. It's one app with one big database. And it it there's you know a lot of efficiency like in terms of workflow that you get from just having one database. And this is one of the patterns this this delegated immutable types in particular make that possible. You know stay in the majestic monolith as long as you can. We've been running this architecture for 10 years. you know, like other versions of Base Camp didn't make it quite that long before we created a new version and then therefore like they're still growing, but it really reduced the rate that the content is growing in those systems. But Base Camp has just been going up and up and up for over 10 years, version three. So, this architecture scales really well. We're totally confident to build Base Camp 5 on it. Um, it's just a little bit different maybe than what you're used to. So once you you know and and I don't know how widely used it is because because of that. So David is going to do you know a a talk on and like a sort of like a a code demonstration of like the mechanics like but like it's that's useful but like we sort of already have that too in documentation and still people miss the big picture. So I'm hoping that that's what I I'm able to convey here is like what why would you do this? It's like why not just have messages and documents and separate records like the universe intended and like that's why I mean it is fine for a lot of purposes there there is a cost to this level of abstraction but it's not as great as you might think and you it pays dividends once you've made the investment. >> I mean I'm fascinated by this. >> Oh I agree. I'm trying I've been trying to think. >> Well, Fo, you work on the mobile team and like think about the API like we're able to expose a single recordings JSON API. >> Pretty great. Like >> you can just ask for recordings and you can provide filters like between this time and that time you're going to get all different types. But this is really easy to implement, right? And you can do things like on your your uh >> native code can do things and not like it means that things just work. So we add a new type and we don't need to issue a new uh create a new build of the mobile apps. It's just it's going to get a new kind of recording and it can deal with it in a generic way. It knows that all recordings have titles. All recordings might have a position. All recordings can be archived or active or trashed. like it's not like oh everything breaks because we introduced a new type you know and and that's one of the other ways because we also are a super small team you know like I mean super small at times there's only a handful of people working on any of this stuff you know and you would need way more people if if you had to coordinate these sorts of changes it means that like we just deploy base camp whenever and we're confident that nothing is going to break even as we add new things not not you can't do for like everything. But as far as recordings go, which is the bread and butter of Base Camp, these different kinds of content, these content blocks, you know, that are fundamentally contentoriented, all behave in the same way. And and it means that we ship new stuff and we don't have to wake Fernando up at night or anybody on his team to say you need to like do a release because we busted something. That's the kind of efficiency I think you get and you know and that's the kinds of things we're looking for so that we can stay a small team >> if someone is like yes this is a better way to work or this is a better way to do this programming with the delegated types all of this is it something that you can incorporate into something you're already doing or do you have to start from scratch >> you can incorporate it into what you're doing it is a little hard though it it's definitely better to start from scratch the migration path path would look like. Um it would be a little bit complicated. You would you would introduce your your main table, we'll call it recordings, that delegates its types and you would either create new um types, you know, so if you already had comments, you would need to name it something else like recordable comments and migrate your content into that. create recordings as you go because the way you create a recording is you create a recording record and you pass it a new recordable and then when we save the recording that's when we update the reference and persist the recordable. So you would sort of do that process and move your content into it. So we've done this before when we worked on the hey calendar. The hey calendar also uses the delegated type pattern for all of its calendary things. Think of all the things that can appear on a calendar day. >> You know you got the event itself. You might want to circle the event. You can have a countdown. You can >> You might want to name it. >> Yeah. Name it. It has a title, right? >> You can add a picture. >> Yes. All of these things are implemented as like a calendar is the bucket and the bucket has recordings and the recordings are things that are eventlike. Now, I talked about how the recordings table, the main table stores like the metadata, the stuff that all of the types have in common. So in an application like hey calendar they have everything that can appear on a calendar has to have a starts at and an ends at time stamp. >> You can't put it on a calendar unless it has knowledge of when it exists. >> Right? >> So in that case starts at and ends at aren't delegated to like an event type. They're on the recordings table. That means we can query for any kind of thing that might have be might be on the calendar in any moment in time from one place. That's what allows us to like in one query fetch all the things that we need to display on a given day or in a given m week or month. So part of it is is it's that you you find the stuff that never changes the things that all of the types would need and that becomes you you know your main class your main table and then that create that and and the the you know you've got it when you can just query that one table to get everything you need. Our first iteration of hey had us like going through the different tables. So we had to like query for events and we had to query like oh we're going to add you know other things to the calendar like I forget like even just being able to circle a certain day or something and they were all slightly different >> and there's like tasks sometimes this week task. >> That's another great example and and so we did the you know and and then we had to migrate into that. This was still while we were developing but we didn't do it by deleting everything like we were able to transition to it. But that's a great question. it it is the kind of pattern you want to start with. The single table inheritance pattern and the polymorphic pattern are similar. Yes, you can migrate into them, but it it's it's a pain. It's better to identify upfront that this is something you might want to use. And similarly, you could go the other way. You could start with a delegated type and turn these all into concrete types that have their own tables. You know, you would iterate over the recordings. For each recording, you would create a new record, you know, take its recordable content, take the recording metadata, stick them together, and insert that in a new table. So, there is migration paths to and from, but you're right. You want to you want to start with something like this. Jeff, I'm skeptical. I have to I have to say like I work at at 37 Signals. I work with base camp all the time. It is very fast. Why is this not more popular? Is it like obscurity? Is it like what? Because like come on. Like we've been talking about this for like however long the episode has been going on and it's like I keep trying to I've been thinking about poking holes at it and there's like >> Yeah. I don't I don't know. I think it's just familiarity, which is why I'm glad we're, you know, we're doing this and sort of educating people because I think it would be a lot more used if people understood its strengths. Like the Rails documentation on it is good, but it's kind of tur. It's kind of like describes, you know, like a shallow version of the benefits, but like you know, you talked about fast speed. That's another great example. Caching, we don't cash at the recordable level. >> Oh my, that's right. Cash at the recording level means that like the way caching works for any kind of content type is identical across content types across recordable types. We cash on the recording. When you want to expire the cache, you touch the recording. In the tree pattern, if the tree needs to touch its children, it can, but it's more common for a child to touch up its tree. But then we have like you know we call it the Russian doll caching pattern where it doesn't m like you can invalidate you know a child way down the tree and not need to invalidate the parent. You know the parent can still render itself and then when it gets to this child it's like oh this part you know I need to change and so but yeah you don't get uh to use the same caching scheme for everything you know when you don't have a uniform thing like a recording. So what you'll see in our views is like cache recording do end like everything is wrapped in a cache block keyed only on the recording. >> Yeah. >> So the you get to share a ton of stuff in terms of the it's not just the I've been mostly talking about the domain model but this also this uh cascades to the controllers and the views right it's it's all the same. It's recordings all the way down. You mentioned you mentioned something about Rails like is this pattern like in Rails somehow or part of Rails? >> It's part of Rails. So it's it started life in Base Camp as like you know uh just in our application code and then uh once we realized how good it was David packaged it up and you know wrote documentation for it and tests and extracted it to Rails. That's another reason we like know it works. It's not, it sounds very abstract, but it's not ivory tower sort of like theory thing like this is what we're actually using and we've proven it out over like a decade. So, it really really works, you know, and it's replaced our use of of single table inheritance and even polymorphic relationships. We almost don't like when you're going to reach for one of those, you should give delegated type a look and be like, could I do the same thing with the delegated type? because you get the same sort of um sharing, but I think that it it's sort of inverted, right? It's it's it's particularly an inversion of the polymorphic type. In a polymorphic pattern, you have a a record that can belong to any kind of record, right? And in the delegated type pattern, it's the other way. It's like, so it's not your child record that can belong to any kind of thing. It's the parent record that can have any kind of thing. But it means you get to work with those parents. You know, you're starting at the top because like if you think about something like a comment can have any kind of commentable. When are you ever working from the comment? When do when do you want to start with the comments table and find all the things? No, it's at the wrong end of the relationship. You want to start at the top, you know, or that's that's why the tree pattern works really well, too. So, I think that um yeah, I I I think that it it it just needs more people if once they try it would realize this is really cool and this is uh this can change a lot. >> Okay, Jeeoff, I have a question for you. So, you're clearly an expert at all this recordings, recordables, all of the things. What would you say if you like had to give someone advice like someone who is using this like what are do you think are like some of the biggest mistakes people have? I know I didn't prep you for that question. >> No, it's okay. I I don't know if there's any mistakes so much as like maybe the mistake is like not taking it far enough. You know what I mean? Like you you can model like you know all the things that you have in common in this way like and you can go pretty far with it. We did um like hill charts in base camp like how do you model those as recordings and recordables? But there's a way, you know, and you need other patterns, you know, or how do you take like you want document history or event history? The events stream is another thing. Not only do we want to show you events, but we want the events that we're listing to be like accurate. So you have a timeline and it's like Kimberly saved a new document called this. Well, then you went and renamed it. But when you created it, it had the original name. And so when I'm scrolling back, you know, I'm like Kimberly didn't name this, you know, new features, you know, it says new features March, but I'm reading this, you know, I'm scrolled back to February. That doesn't make sense. You know, she have a time machine. No, you you want like sort of accurate history, and that's another way to get it is with this. So I think it's it's combining the delegate pattern the dele delegate type pattern with other patterns and they really can build off of each other. So once you have this concept of like the type doesn't matter you can do interesting things with the type you know like don't change them you know keep a log of them right so that you can easily move between them move pointers around it gives you ideas for how you could copy and move things efficiently. Um, so I think that it's that people don't realize where it's not that they use it wrong. It's that they don't realize how in how many places it's actually applicable because it's not nearly as rigid as STI or polymorphism. It's it's super super flexible. >> Were you a part of like the the or was this like David's invention? It was David. It was a big part of Dave. David like came up with the original idea and it was out of the frustration of like how expensive and slow it had gotten to copy things in Base Camp 2 and how every time you introduced a new thing you also had to introduce changes to the copier and like so yeah David had the the first idea for this typical David thing like how can I invent a new you know a brand new pattern that nobody is using that's why I think like it originated here and it's sort of like it has a lot in common with STI but it's flipped on its head and and I think also a thing that only David can do is that it was unproven when he introduced it. It was like this might work. You know, it's harder to get away with the speculative let's give it a shot unless you're David but it's David and he can just do that. And then it was like let's see how this works because at the time we hadn't been designing Base Camp like that. We had our it was very base camp 2 like and uh you know and then we just from there it became you know a shared thing we could iterate on and like it it we didn't extract it for several years after it was in use maybe a year or two years uh after it was in use in base camp before it like proved itself to be like a viable pattern. It went through a bunch of iterations, but yeah, the kernel of that idea that you have one row of one table that holds, you know, most of the information about the records and but none of the specifics came from David motivated by copying efficiency and the need to pageionate, the need to like have have a timeline of of you know different types that we could pageionate with one query because the other way to do something like that is to like sort of create a table of copies, you know, that you insert, you know, different records in the table that you're going to pageionate. But now you have a copying system. And we do have a version of that in Base Camp, but it's a performance optimization. It's not like the main way that we query for recordings. You know, think of it like caching. We can relay events into another table so that we can like very efficiently be like, here's all the events that apply to just this person because everybody can see different things in Base Camp, right? My timeline doesn't look exactly like your timeline because we're on different projects. >> No, I can I could see how that would be a nightmare to implement if you had like different types. Like I I can I can imagine it in my head like a PM going like, "Okay, I want you to give me like a timeline of things that are happening and the programmer is going like wof that's going to be rough." >> And you pay that cost every time you add a new thing. >> Yeah. you know. Yeah. And and even when we're doing work in beta, you know, like that's a one time. So we do work in beta, we'll modify the database, we'll add new types, and production needs to know how to deal with these new types. And in most cases, it just can because we've programmed it in a way that like if they don't know how to deal with a particular recordable, it just deals with it in a generic way. It displays the generic icon. It asks for things that it knows that all recordables have a title, you know, possibly content, uh, subscribers, you know, and and so like nothing breaks. >> Nothing breaks. >> Yeah. >> Yeah. >> And so that was another key element of the design, like how can we have something that's flexible? What we found with Base Camp 2 is that yeah, it works, but it's slow and every time, you know, we're like, hey, we want to add a new thing, we're like, that's expensive. That's going to be a pain in the ass. How do you make it not a pain in the ass? And I think good patterns emerge when like that's your criteria. You know, you're we come up with something to make it easier. What would make this easier? You know, and then so you find the specific problems we had and then correct for those. And and David is basically an expert at that. >> And he hates when things break. >> Yeah. And he hates friction, right? Like when when the common thing is hard, the common thing should be easy. The common thing we do is introduce new features, new kinds of content. The common thing we do is you know and with those content that content we we need it to work on our mobile apps with no extra effort without having to hire new mobile developers. You know this is why we have turbo hotwired and why we have hotwire native native. You know why we have rails like make these super common things easy. The uncommon things just need to be possible. you don't do them that often. It's okay to have friction there. Look at deploying apps, right? Nobody, everyone was deploying apps with Docker before using Docker in production before we were, but David looks at it and is like, "This is a pain. Why is it like this?" And so he invents Kamal, you know, like it it's that response. He has a low tolerance for friction on the critical path, you know, and I think people just sort of um they they they become oblivious to that friction. They're just like they sort of ramp up slowly with it and they're like this is just the way it is. David never accepts that this is just the way it is. He's got a real eye for like this could be different and and he's not just going to complain about it. He'll do something, you know, he'll disappear for a week or two and then be like here I fixed it. You know, it's great. >> Jeeoff, you mentioned some documentation, some Rails documentation over delegated types. We'll link to that in the like notes for this episode. Are there any other places people should go for information about all of this? >> I mean, right now there's not a ton of places. There are a few people who have written articles, you know, that are that are pretty good and they're sort of the exploratory kind of article. They're like, hey, you know, haven't found a lot of documentation about this cool feature of Rails. And so, they've written about it. So that's really good. The Rails documentation, well TUR is also very good. Um, and I think that what we're doing right now is is adding to the growing body of documentation on this pattern and what David's going to do is is also going to help that. But yeah, if I can find any, I I'll I'll send you the links and then we can add that to the show notes because yeah, I think that's, you know, the main thing we're trying to correct by doing, you know, an episode like this is to share the benefits of this under represented pattern. >> Awesome. Well, with that, we're going to wrap it up. This has been a production of Recordables by the 37 Signals team. To hear more from our technical team about their behind-the-scenes work, visit our developers blog at dev.signals.com. 7s signals.com.

The 37signals developer blog dives deep into technical topics, but some ideas are easier to explain through conversation, a little back-and-forth, and an occasional screen share. That’s why we’re launching RECORDABLES, a new video series hosted by Kimberly Rhodes, who you may know from The REWORK podcast, alongside Fernando Olivares, Lead Programmer on our Mobile Team. We couldn’t think of a better way to kickoff RECORDABLES than exploring recordables! We’ve had many requests over the years to explain how and why we use the delegated types pattern from Rails in our products and we’re finally explaining it in-depth. Kimberly and Fernando sit down with Principal Programmer Jeffrey Hardy to unpack the recordables architecture that powers Basecamp and HEY. Jeffrey explains how delegated types made it possible to scale Basecamp for over a decade without constant rewrites. The discussion explores why this approach makes copying, version history, timelines, and mobile support dramatically easier, along with the tradeoffs and learning curve. We’re sharing a behind-the-scenes look at how 37signals builds and evolves complex products with a small team and the Rails pattern that helps make that possible. *Timestamps* 00:00 – Introduction 01:20 – What “recordables” are and why they matter 02:15 – The delegated type hierarchy 04:15 – The challenge with single-table inheritance 09:05 – Organizing recordables and recordings 11:45 – Tracking change history with recordables and events 22:40 – Copying and moving content efficiently 27:54 – Pagination and querying across content types 34:00 – The learning curve with delegated types 39:54 – Building new features faster with reusable behavior 43:03 – Long-term scalability and performance *Documentation on delegated types* https://api.rubyonrails.org/classes/ActiveRecord/DelegatedType.html https://guides.rubyonrails.org/association_basics.html#delegated-types For the full episode transcript, visit https://dev.37signals.com/the-rails-recordables-pattern/