"Clojure in live sports television" by Christoph Neumann

hello got one minute but as people are walking in I want I want you to do one thing for me real quick turn around and look at the back of the room in the back of the room we have our av crew and that's a tough job they're making this happen they're making this happen and and I've done many many many hours in Live Events and it's tough because when you make a mistake an entire room of people see it or hear it right and so you have hours of boredom followed by seconds of Terror and so anyway we appreciate what you guys are doing for us thank you very much so I'm here to talk about using closure for live sports television and really live sports production this doesn't always go out over the airwav sometimes it goes to a stream sometimes it's in a venue right but my hope is by the end of this talk you're going to have an idea of what's involved in pulling off a live sports production and then how to build a a system using closure that is low latency and reliable and and really doesn't go down or lose data and then why closure in particular I believe is just a great language for this domain and so those are some of the goals of my talk as we go in but before we talk too much about closure let me mention who I am so I'm Kristoff Newman I'm a software engineer and founder I have a consultancy called appau and we make custom software for Life Sports production among other things and I created a product that's used for situations like this when you need a remote presenter to a live audience or you need to bring a guest into a TV show and that product is called hypers signal all of that uses closure all the way up until we have to call the apis in the browser and things like that I also am a podcaster I have a podcast functional design enclosure with my good friend Nate Jones we started this a number of years back and we go through Furious spurts of episode creation and then take some breaks and so feel free to check that out too so I want to take you back to December of 2015 maybe it's not memorable for you but it's memorable for me and in December of 2015 I'm sitting there trying to make a hard Choice I'm really excited but I'm really afraid that choice is do I commit and use closure for my next project I'd been dabbling with closure for a while and I was getting more and more excited about it but the stakes seemed really high and I just wasn't so sure this would be a great idea the project was for Heroes of the Storm and I needed to create a tool for that competition as part of the game they take teams take turns picking from the myriad of Heroes that exist in that game and they have to pick which Heroes they're going to have on their team and they in order to make this fair and competitive they each take turns picking a hero and that's called Drafting and I needed to make a draft tool that the teams were going to use live on air for the competition and it needed to be real time and so the teams would get together in a huddle and there you have the uh production assistant there the tournament assistant helping and they pick a hero and then they lock that hero in and boom that hero goes up on the big screen that hero drops in this big LED panel behind the team uh there's a lighting effect there's a sound effect there's all the stuff that happens when they pick a hero and then you have commentators right you have casters and analysts and they see which hero got picked and they go ooh that was unexpected why did they pick that hero well that's interesting and then they begin to talk about what it means and so you have a room full of people you have this tool that's being used as part of the competition you have it being analyzed to death all these effects and so I'm thinking to myself oh man I really don't want to screw this up right because that mistake is going to be seen by everybody there but actually it was a little worse because this was going to be on ESP too and so this mistake Not only was it going to be seen by everyone in the room whatever screw up I had was going to be seen by everybody watching the live stream and by everybody watching the event on National Television so I was thinking to myself I really want to do this this is a great idea this is a terrible idea what am I thinking this is a terrible idea how can I use this list crazy list language but oh man close is such a great language and I'm having a battle between my brain and my heart on should I use closure for this to give you a little bit of background on why this was so hard for me is I was a PhD student in programming languages and I had gotten the full indoctrination of type systems and object-oriented programming one of my advisers literally wrote the book on object-oriented programming Timothy bud and he was a researcher in small talk one of my other advisers Martin AIG had fundamental contributions to functional programming and representing trees in a type Safe Way and so I thought multi-paradigm programming is it it's got everything it's got all the features it's the best of everything and and you know what let's add some static typing in there to make it even better and so I was in Academia for a while and I was enjoying that but after a while I got burned out on Academia and realized I loved industry better and I had gotten my Master's Degree so I went back into industry and I worked on this Behemoth I went to huet Packard and worked on an enormous printer that over the years got even more enormous this thing can print 10 feet per second and up to 10 feet wide right so you're literally printing 100 square feet double-sided every second and this is a machine that when something goes wrong it can cost a lot of money it can break things or it can hurt people and so while I was working in manufacturing I was realizing oh man I really don't want to screw this up right I don't want to cost money I don't want to hurt people for sure and so I learned a lot about making reliable systems when I was in manufacturing and so I felt comfortable in high stakes environments where reliability was key and I knew when I went to HP I brought all that uh multi-paradigm statically type programming with me and I introduced Scola to the team and I felt really good about how it was helping us make Reliable Software but after I'd been at HP for a while working in manufacturing I decided I wanted to go work on fresh new businesses and I got involved in startups again well in startups I ran into a different problem I needed to make stuff and I needed to make it fast and also be reliable at the same time and so I was feeling this friction from my language choices my tooling and I was like how can I make things fast but have it be reliable right I was experimenting with more Dynamic languages I was using groovy I was I said let's just go all in on JavaScript for a while but then I ran into the what felt like a catch 202 if I'm fast now I run into a mess later and that really hurts reliability if I'm very careful and very slow now well I'm going to have reliability later if I'm still in business right so you only got so much runway so that that was pretty rough but then meanwhile during this time I noticed that my thinking and my methods for software development had been changing but not my programming languages right I'm using the same tools I had used in grad school and Beyond to solve this however there is this one language that kept catching my eyee I bet you can't guess what that language was so this is my closure slowman so I actually became aware of closure 2008 very very shortly after it was released and you know what I thought is this a joke is this for real what what kind of list fanatic would make such a thing right and then I'm like oh I bet this is one of those like hobby languages is one of those esoteric languages like the whes space language right or the chef language where every program is a recipe or golf script where you're minimal characters but that crazy language didn't seem to go away and in 2011 I watched simple made easy and I thought well this guy doesn't sound crazy he doesn't really sound like a fanatic except maybe for Simplicity and practicality and you know he's really not all that esoteric EX except for maybe those Latin definitions are a little esoteric but all the rest was intensely practical and so I thought I have to give this a try so over the next couple of years I was having fun playing around with closure but I wasn't really ready to commit and so that brings us back to December of 2015 I was facing this new project and boom something happened does anybody remember a special thing that happened in December 2015 15 probably not Advent of code was released right and the community went crazy and and what was cool in the closure Community there was all these people sharing their Solutions right they're checking them in the GitHub they're sharing Solutions people are doing this this is a new thing it was so much fun and this happened right around the same time that I was considering starting this new project and so I'm thinking about how I want to approach this new project and I'm coding up the latest problems of the day and I'm thinking about how I want solve this and I'm coting up the lace problems of the day and so maybe it was the dopamine from all those solutions that I came up with or maybe it was just the intensity of the activity but I thought yes I am going to use closure after all and uh what's the worst that can happen and so it turned out to actually be a pretty good idea there was a whole brain flip that I had to go through and so that had me really scared for a while but then I I changed it continued to change my perspective on programming and it very much aligned with the perspective that I already been changing and I I continued to use it on a number of projects so The Heroes of the Storm project but then also scraping twitch for the top channels and creating a visualization of that live for some of my clients uh enabling The Not So secret debug log for Hearthstone and uh slurping in those Li the the debug line lines dynamically and parsing those with insta parts and creating a representation of the game State and then using that live on air um developing custom Integrations with the OverWatch server team and helping launch the OverWatch League um also doing that again for Call of Duty working on custom Integrations with the Call of Duty server team and helping launch reboot the Esports for Call of Duty and launching the Call of Duty league and then that led into working with other large equipment providers Brands like Wimbledon and Brands like the NFL and so over time I've been able to use closure in a big way in all these environments and so from that experience I can tell you that closure loves life Sports production it's a great fit but why why is a great fit well first you need to understand a little bit about live sports production so who here has done prod uction at all maybe theater or something like that production at all okay who here has done uh TV production or a or AV production okay so you know a a fraction of the room so in production it's all about capturing creating and broadcasting so we want to capture audio visual data we want to put all that together into a big live soup of information and video and audio and experience and then we want to send that out to screens and streams and the national Airwaves so we are have all kinds of equipment involved in this right we're pulling in video be and mic so we got microphones and cameras and we're going behind the scenes and we're getting little featurettes once again of audio and video of that and we're we're taking in all these sources there's so many sources this is control room you can see all these sources coming in live and you have stuff that's recorded that you're going to play out later and things like that and we even want to get the gaml itself and so we're capturing the gameplay and we're capturing the audience and we're capturing all this stuff but data is everywhere so let's take a look at this uh screenshot of part of the Call of Duty live stream so you see data lots of data we have the score well that's data we have the clock that's data well we have the map we're on that's also data we have the team names that's data we have the side the team is on that's also data right you have to keep track of that we have the match score well that's data and that's and that's from a totally different place it's not even from the game it's from some tournament system we have what this match means this is a championship match so that's some context well that's data too we have the players we have the order they're sitting in in we have to match the players up to the seats because we have cameras on all the players and so we need to know when we press you know button three which player that's going to be we also have um their jersey numbers and more information about that we have whether they're alive or dead or have special powers in the game and we even have a feed of notable events that have occurred so you can see even in this one screenshot that didn't seem like much when we first put it on there's data everywhere right and then that isn't even counting all of the gameplay statistics so this is from OverWatch and this is only a fraction of the performance statistics that are available in the game this is an example of a live user interface that's used by the casters and the analysts to see insights into the game as it's played so all of those sources come together to create a story we're taking the video we're taking the game we're taking the data we're putting it all together and we're telling a story of what you're seeing and why matters and the talent or the casters and the commentators they're telling that story and it's a team effort and so we take those audio and video sources we bring them all together we throw some data on top and we tell this story and we create this experience uh whether you're in the venue and you're seeing things on the big screen or you're outside and you're watching the broadcast we're we're weaving a story of what you're seeing and why it matters using all that and it is a live Act of production it's going to happen whether you make a mistake or not it's going to happen even if something crashes or goes down you're going to have to do it and you're going to have to keep going and it can get really really big and really really complex and so live production is really hard right you have a lot of people in moving Parts you often need to improvise and flex because you don't get to postpone when it was going to happen you need to do it anyway so you need to make do with what you have you need to make it happen and success is expected people get irritated or annoyed when things go wrong but your mistakes are recorded and watched by millions the bigger the game the more complex the production but the more complex the production the higher the chance of something going wrong right sounds great right to give you an idea of how crazy this can be I got to know an Olympics Tech manager who worked in Sochi and all these trucks come from all over the world to in order to make this production and some of those trucks uh are from the US and it turns out the US uses a different voltage than many of the rest of the world we on 110 volts right so they needed to adapt the power for these trucks in order for these trucks to be able to run and so in the spirit of live production somebody had a great idea let's take a UPS an uninterruptible power supply those things with lots of batteries that keep things on when the power goes out and let's take all the batteries out of it and let's just plug it into power and it will give us 110 coming out and we're just going to use it to downstep the power and that's going to be just fine and it was a great idea it worked perfectly UPS without batteries who would thought until it got to about 110% of its capacity and so it was a little overloaded and then the day got really hot it got up into the 90s Fahrenheit or 30s for you Celsius people and uh the production manager was walking by he looked over at this UPS power adapter shim device and he noticed something was different there is a flame coming out of the side of it well that's not good so did what any of us would do he ran and grabbed a fire extinguisher and he put out the flame but then he got some help and they posted somebody to stand there and to continue to put out the flame until they could figure out another way to get power to the truck because they weren't going to be turning the power off anytime soon and that captures the spirit of live production you're gonna figure out a way to make it happen and then when that goes wrong you're going to figure out another way to keep making it happen so this is a really different than your 247 SAS application right little bit different context live production is kind of like manufacturing but you're building the whole Factory in a couple of days but you're still expecting Perfection and then you're going to tear it all down and move on after after the event so it has some pretty brutal demands working right you have these immovable deadlines you have all these parts that aren't there all at the same time until the last minute and so you're working in these incomplete environments or maybe something did go wrong and part of that's missing you have all this integration there's so much gear involved right and you need to keep everything up and running and then it all needs to work right now you can't go hey hold on let's let's give it 10 minutes and try again right The Show Must Go On and of course there's mountains of data especially in Esports I've worked in both traditional Sports and Esports but Esports has the most data and then you need to save everything you need to write it all down you need to capture it because that is the record so in this critical period of time this 2our or 4 Hour window you need to record everything and lose nothing because you don't get to do it again so that leads to a certain amount of stress as you imagine so I know you're thinking after seeing those kind of working conditions where can I sign up I think it takes a certain kind of person you either look at at that and go yes or you look at that and go no thanks and so we pull this off with a combination of language and architecture you could say sure the architecture is the most vital but the architecture creates the groundwork for Success without the architecture you're you're no matter how good or helpful your language is you're still going to be in trouble but the language actually helps you pull this off and so what are some of the implications so when we have immovable deadlines we need not done to work anyway right we need things that aren't there yet or things that went down uh to be okay we need to handle those missing parts we have all kinds of things we got to integrate so we need to develop those Integrations quickly and easily and we need to make sure that all these things are DEC complected and isolated and redundant so there isn't the Lynch pin that brings it all down and because it has to respond now we need it to be live and multi-threaded and locked free and all that and because we have mountains of data we really want just working with the raw data and working with generic data to be productive and useful we want it to be first class and we need to be able to save everything so we need to be able to take all that raw data and write it down too in an easy way and because of stress it has to be simple because the last thing you need is to try to figure out some really complex thing when everything seems like it's falling apart and so closure as we'll get into is helpful for all of these in terms of reliability for this system I just want to address this briefly before we move on because I'm going to focus more even though the architecture is essential more on the language aspects in the architecture with enclosure but for reliability we use redundancy and clustering right we use item potency so we can retry things over and over because when something's not there you want to be able to try it again later and if it just happened to come up and you do it a couple of times you don't want to mess things up we use a technique called reconciliation where we have a record of how something should be a record of the way it is and then we use reconciliation to figure out what we're supposed to do to make it be the way it's supposed to be and then we use collocation because we don't want to lose data so if we can run that thing on the game server or directly next to it that's great because if it has to reach across the Pacific Ocean to make it into the kfka queue well you could have a problem if you have an internet route that goes down and then we use stream replication because we want to get that data as close to the point of use as possible and so we're going to capture that data as close to the point of origin and we're going to replicate it as close to the point of use so those are some general principles that we use throughout this system I'd be happy to talk about this more at Great length but just to kind of lay the groundwork for reliability there's so many Integrations so these are just the ones I could think of when I was typing the slide off the top of my head there's even more but you have the game you have the software running the tournament you have graphic systems that for the onair show graphic systems for in the room screens lighting sports information systems that have all the details biographical information team history factoids things like that you have storage for data storage for media you have data warehouses you have a whole operational infrastructure and monitoring and all that you need to handle identity and authentication because not everybody should have permissions to everything else because you don't want just anybody to be able to take the whole thing down and you have all these things that are foisted on you by sponsors and other valued partners that they have requirements on your system and so for example you know here's an example of a sports information system it's giving us the current standings in the tournament or we're going to take some of that data we're going to make a graphic like this comes from the overw OverWatch league and we're going to use this graphic to tell a story so we need to know what happened in the past because remember we saved everything and we can look at how these teams did before and now they're going to be on this map again and how we might expect them to behave or we're going to create like I showed you before a live visualization that shows you all this information and get the data out of the game we can't make this if we're not integrated with a game and a lot of these not all of them but many of them are 100% custom Integrations for example I worked with a game that shall remain nameless that had one of its versions ran on a specific console platform and we needed to get data out of that game for tournaments and of course the developers are paranoid that they don't want anyone else to be able to do this so they have a secret Port UDP Port that you have to send a special cryptographic message to after you go through the special song and dance to even be in the right game mode to even make this happen at all and then if you get that far and get the encrypted data passing back and forth It's a in-house serialization protocol that you have never heard of right to actually pass the data back and forth and so we need to integrate with that and we need to make it work perfectly in production so this is a simplified system diagram of some of these Integrations right so on the left we have these systems that we're integrated with like the game server the tournament system and sports information system for the purposes of this example but now imagine about a dozen or two of those and then let's have an app for controlling different systems and we'll have another app for authoring graphics these are two very common things that need to happen and then this live data engine which we'll look at more on the next slide is going to take all these sources and take the input from the user and it's going to produce outputs to the Integrations that go to live visualizations or broadcast graphics for on the air Graphics or stage graphics for big LED walls or screens or things like that so the way this live data engine works is this is sort of The Logical structure of it right the way we implemented it was with core async and we set up this graph of what we called workers which are little core async things they they maintain a state you could kind of think of them like a a transducer m contains a state but in this case these workers have to be able to do IO so they they have a thread so they have a local state and a thread and they're kind of in this uh recursing forever situation and so we integrate these sources via some API so we have a special kind of worker which is a collector and that worker is going to reach out via the API it's going to poll in some cases it's going to do this custom protocol in some cases it's going to be reading the file system in some cases and so on and so forth and that collector is going to produce a closure data document that describes all the things that matter for that problem domain right and we're going to have one collector for each of these Integrations and then we're going to have another kind of worker that we call an asynchronous join worker that's going to hear these records well these documents these big data documents and some of these can be huge right uh we had one uh the OverWatch game State it's like 250k per snapshot or something it was a huge amount of data and it's going to take different data sources and it's going to Pivot them and merge them and join them and produce a view of that data and that view is very specific to the needs of the outward integration and so this is nice you have this kind of uh data Mart right where you have things represented in the origin domain on the left and then you have things represented in the output domain on the right and then core async allows us to do all of this data flow instantly and concurrently and so as things change these async join workers they get a copy of all the current states of everything and they rerun their pure models and so we refactored everything well we factored everything really we didn't refactor it we factored everything so that all of the work is structured in terms of a pure model and so let's take a look at this so here's an example data flow so the game integration might tell us what players are in the game and it might tell us the what the player stats are the tournament system integration is going to tell us which teams are playing right now in the tournament because the game might just tell us red team and blue blue team well who's the red team who's the blue team right we need to know what actual teams they are and it's going to tell us what the match score is because the game has no idea about a match right it just knows about that current game and then we have the sports information that's going to tell us players hometowns and jersey numbers and things like that and team colors and team logos and things like that right and so then those are going to come in and we're going to have a view a life stack View and that live stats view is going to be made by one of those async join workers and it's going to take in this case the players in the game and their stats and then some of their details because maybe we want to put jersey numbers on it and it's going to make that document and it's going to send it to a live stats application that I showed you earlier or the stage Graphics or the broadcast graphics and then we have um the tournament I'm sorry then we have the match State coming from the tournament system and then some of the details and it's just going to send it to the broadcast stats because we don't care about that in the venue right so you see that the way it's a very data flow oriented thing and so this is great in closure because closure is a great fit because closure is data oriented right data is Central when we look at this diagram data is Central to everything right data is easy to understand visualize and say right the information isn't trapped in the apparatus of the language right sure closure has language constructs for representing those datas but it's very easy to represent and and transform into things we can visualize Behavior has been separated out right so therefore the information is Central not the behavior and so this encourages us to put everything into the data that we can everything that matters is in the data and because we have immutability closure is safe so we don't have to worry about things messing up our precious data when I grab that data from the OverWatch server and I'm holding it in memory I can know nothing can corrupt that right and it can get written down in a safe way it can get transformed in a safe way and I can do that concurrently and cheaply because it's shared in memory and this really helps make isolation and redundancy e easy because the all these different transforms are separated from each other if we do have a bug well it's not going to take down all the rest of them if it's Downstream in that join and closure is responsive because we have this buil-in concurrency with core async right and Lock Free concurrency with atoms so we can use this information quickly and get an answer back out right away and lazy evaluation helps too because we don't want to do any extra work unless we need it so a story when I did integrate with the OverWatch server I went into their office and I worked with their their server team and we weren't 100% sure what data export out of the game because there's so much data in the game so we went through this back and forth process of well what should we export and what should we not and so it's exploratory so we had a discussion what we needed and then the server Dev went and he uh spent a little bit of time in his code and he kicked off the compile meanwhile I was setting up my closure environment getting it all starting a polling Loop to the adjacent endpoint to the IP on the network and he would come back and be like okay I got the compile it's up and working I'd be great from the reppel I'd run it I'd go great I got the data and we looked at the data they're in the output and I'd go okay let's oh oh can you add this he' go sure no problem so he'd go back he spent some more time he' kick off a compile 20 minutes later he come back okay I put it in there I said I noticed you put it in there because I was already pulling the data writing the data capturing it and saving the data that whole time right and so we went through this process where 20 minutes of compile we come back two seconds of Rippling yep it works 20 minutes to compile but then we got the whole integration saved and then I had this idea like oh what happens when I leave and I don't just have an OverWatch server at my disposal or a server Dev I thought well I better save all this data so I very easily was able to serialize that this was all an Eden and just concatenate Eden to a file and then later and timestamp it all and then later I was able to read that in with the Tim stamp and play it back at the correct time with the correct timing information so now my collector if we go back to that diagram that collector I replaced with something I called a faker that gave me the same input into the data flow but it came from a file instead of coming from an actual OverWatch server and nothing else in that pipeline needed to care and this gets back to closure is live and interactive right this reppel connected editor cycle gets rid of the whole compile run right any complete expression is something you can run it's an entry point you're working with the actual thing too you're reaching into the actual system and working with it right now you can get the state of it right now you're not working just simply with a representation of it but you can call it and get the data and that allows you to experiment and discover what you need to do and you can manipulate your Source very efficiently and safely using structural editing and also because this data is incomplete closure treats incomplete and partial as first class right these Integrations sometimes I just want a little bit of it I don't want this huge giant specification or it's totally custom and so it's very easy enclosure to work with part of the data or part of the API I don't need to generate you know uh a bunch of bindings or not bindings but I don't need to generate a bunch of handles and classes and stuff for an API all these parts I don't want to use right and so generic data so using part of it as first class generic data is first class I can work on just a little part of it and get it working before I have to get it all working and so another example is in this this is an earlier version of that live visualization and this came out of having conversations with the commentators and so the commentators sit there and they look at the screen and they're watching the game on the left and they're looking at the screen on the right and they want to say something insightful and so they told me information they would like to be able to know that wasn't in the game itself and so I took the recordings of my sessions for the integration and I use that to play it back and develop a UI to visualize it then when we got together on site at the big event at Bliss con they said oh this is missing oh that is missing no problem I had them come sit down with me and I had my recordings of the data and through interactive development added more and more things for them to see and they like that's it I love it that's great let's use it and we did a few hours later we were using it live in production right and so closure really gives you this uh phenomenon of being able to grow things and start small the language F almost nothing on you itself it's very simple right it has few constructs but also few demands you can start with an expression you can grow some expressions and turn them into a function you can build those up into composable functions you can stitch those together with closure core and then you can add schemas later when they're helpful right and so closure by building this up that's the way it happens closure builds this up allows me to build this up and the same thing for lighting we integrated with a lighting system had to sit down with a lighting board and work that out and from the reppel once again the lighting gu say oh send me this midi Q send me that midi Q send them the midi q and we would experiment to see what works in this space that giant lighting ring up there we're we're manipulating it live to figure out what works based on what's happening in the game and so you end up with this meta tool for closure you have closure itself you have the repple you have your editor and you have some data visualization like a data browser like portal earlier right and through this you can eliminate all kinds of tools right everything is staying in the closure Universe you can evaluate expressions to explore and visualize you can grow that up into your actual application right and it's all just closure source and closure is robust right because it's safe because you are working on the actual thing you're not just messing around with your representation but you're interacting with the actual world because closure has a good story on time and behavior and identity and separating those you end up with this robustness in working with closure and then when the stress happens you have closure is simple right fewer language constructs fewer demands less Hoops you have to jump through no side effects way easier to think about when you've been working for 16 hours and you're really tired right um by separating time State and behavior you don't have to hold all that in your head so it's a low cognitive load the problem is Central not the language demands right not all the ceremony that you have to go through so closure really does love live production and The more I've worked with this the more I go well you know what else needs to be reliable and grow quickly what else really we want to keep things minimal for the long run what else do we need to be safe and responsive and interactive well I think closure loves startups too right because all these requirements if you think about it well they're also true for startups and well you know what they're also true for prototypes because you don't really know where you're going with prototypes either and you have to figure it out it's a new integration and and sometimes you have crazy unrealistic deadlines that don't get to move and you have to make something that works so maybe it's not just that closure loves live production or startups or prototypes maybe closure just really loves the messy real world as calls it situated programming right closure isn't here to make the world conform to its image but it's here to help you work with the real world and so I have a mission I have a personal mission I want to continue to take closure outside the closure Community I want developers to learn about how closure loves the messy real world and I want them to be able to make bulletproof software that's fun to maintain instead of a schlep a grind right I personally have been through programmer burnout and I wish that on no one and I would love I've talked to so many of you and I have heard your closure story that many of which involve burnout so are you interested in that too if you are interested in that too then let's talk I would love to talk to you afterward and I hope in this talk you've had a chance to learn about live production learn about how closure is very much suited for that and other things in the real world and how we were able to do a really robust realtime information system with no data loss in no downtime thank you

Have you ever embarrassed yourself in front of a room full of people? What if it was recorded and watched by millions? In live, televised sports, the team’s reputation is on the line, but so is the production team’s reputation. Worse yet, the bigger the game, the more complex the production with even higher odds of failure! Would you like to see behind the scenes of live sports production? Would you like to see how Clojure fits in? Learn how Clojure is an ideal language in the most demanding environments where failure is not an option. See how to build low-latency integrations with no downtime or data loss. Recorded Oct 24, 2024 at Clojure/conj 2024 in Alexandria, VA