Creating a Scripting Language in 5 Days - Live Coding in C

me what's up everybody welcome to flex harmonic i'm david wilson and unfortunately i just uh took a bite of a cookie right before i started the stream so i got a little bit delayed there uh hello to mr bill t got it gone item gone i don't really know how to pronounce your name i apologize about that but i appreciate that you're always here um so let me see uh t says sheesh is it because i was later because the audio is too loud i'm kind of curious about which it is i had to bump the audio a little bit because my daughter is currently playing with her cousin and we're in a very small apartment and it's extremely loud so i'm trying to drown out some of that noise with the background music hey bill and i go so uh i've been having a pretty fun week uh doing a little bit of hacking on flux compose behind the scenes we're gonna talk about that uh just in a second gun says uh uh gun enem okay cool i'll remember that i always appreciate it whenever i get uh pronunciation help because i like to pronounce it pronounce things correctly and i often don't so if i ever say your name wrong on any stream that i do please tell me and i'll be happy to say it uh correctly the next time hello to uh pyotr all right so let's see ah she is cheering okay cool even better thanks so um another stream where i'm going to be basically coming in saying i've changed my mind on something i wanted to do uh for this project and um i will try to explain to you why i decided to do that so uh the title of this stream or i guess the topic of the stream today is creating a scripting language in five days um and the goal for today is to replace guile with a custom scripting language that i've been working on so we're gonna give that a little bit of a try today uh so probably you're wondering why would i bother trying to create a language and in fact let me just bump this audio down a little bit the music is starting to annoy even me so let me just tweak that slightly okay maybe that's better so um over the last couple weeks when we've been doing doing these flux harmonic streams uh one of the recurring themes is the struggle with um trying to call c functions from kyle's scheme and the whole idea there is that i wanted to use a lisp language like or specifically a scheme language to drive the tools that i'm writing for my creative projects and the reason why i wanted to do that is because i like how interactive lisp languages are where you just have a text file open and you can have a reply connected and you can start evaluating forms and have stuff happening it happen in the application automatically or in in interactively in real time so um that was sort of the the thought process i've had for a while when i've been thinking about making this project and um guile schemes seem like a really good starting point just because i've been using it for you know configuring geeks and i've also started talking about it a little bit on the system crappers channel so it made sense to continue following that path in this project as well but um as as powerful as guile is and as much powers excuse me power as they give you to actually be able to invoke c functions from scheme uh it's still you know caused us a lot of trouble trying to wire those things up so it got me to think about you know maybe guile isn't the best supplementation maybe i should look into a different implementation of scheme that is more you know meant for actual embedding in a program guile itself is not necessarily meant for embedding per se it is meant for being used as an extension language to see programs uh is sort of meant to be the scripting language for the gnu project but it's not meant to be embedded per se it's sort of like a a side car to your existing code i started looking into using another schema implementation i've used before called chibi which is actually something that you embed into your program but the reality is that the um the c binding story was pretty much the same or actually i would say require even a little bit more work than what guile does guile does give you the ability to call directly into c functions but the problem is that it's harder to figure out sometimes the right way to invoke it now uh that said ashwas did actually figure out a way to invoke the function we were trying to do the last time so we were trying to write a function that renders the current scene to a file and passing the string for the file name over to the guide layer from the guide layer to the c layer seemed like it was going to be difficult um and we did not figure that out on the stream australized give me some code after the fact that he figured out um how to do it but i was still sort of already on that path of thinking how i would like to do this differently so i started thinking even more about this and it occurred to me that if i'm only using scheme as a control language that binds to the c layer then why should i use a full language and programming system for that i mean it seems kind of heavyweight to use guile scheme which is pretty big i mean it's its own language and run time even chibi scheme is a pretty full uh scheme implementation that has a lot of libraries and stuff that comes with it as well so why would i want to go and pull in all that stuff when really all i want to do is have my own functions to find and see and then invoke them from a scheme like language so this weekend i had a crazy idea i would try to create my own scheme inspired control language that would be purpose built for this project so we're not going to try to implement actual scheme or lisp we're going to use scheme or lisp styles syntax basically s expressions or what looks like s expressions where you have nested lists that comprise all of the code and the reason why i would do that is because i do feel that that syntax is very amenable to um interactive development because it's easy to know where expressions start and finish and it's easy to write code that processes those that syntax it's easier to write a parser for a lisp style language that it is to write a c set for a c style language in my opinion uh just because you can easily write a recursive descent parser that uh is able to just look at you know where parentheses start and end and everything in between is just you know symbols of the language effectively so um i'm not going to call it an actual scripting language yet because we're going to avoid adding certain things like control flow and other stuff that we'll talk about the moment as long as we can so i'm going to see how far we can get with it just being a direct portal into the c program we're not going to be doing direct c function invocation we are going to actually have a binding layer that binds c functions into the language but i feel like that is acceptable in this case we can talk a little bit more about why i think that in a bit another benefit of doing this is that i get complete control over memory management and also how code execution is integrated with the language or with the runtime i guess i should say so with the uh runtime so um scheme implementations typically you're using garbage collection i mean i would say 99 of the time you're going to use garbage collection with a schema implementation uh they all have you know different strategies for how you would do garbage collection there's a different you know a few different techniques for that um i kind of want to avoid having to do typical garbage collection i would prefer that um whenever you evaluate code in this language is a little bit more minimal to data oriented programming which may not be fully possible but uh the idea being that you know the locality of things in memory is a very effective for the cpu cache the way that i've written the parser the like the tokenizer in the parser so far actually does fall within that category but actual evaluation may be slightly different on that front but something that we will experiment with over time i think and also about the execution thread issue um at least with guile it had its own execution thread that i had to you know had to start up another thread entirely to run all the rendering code in this case i have full control over when expressions get evaluated or when they even get read in from input from the reply so in theory i could just have my single thread that is um asynchronously pulling a socket listening for input from the reply and then execute an expression only whenever it's necessary and do it within the um the actual event loop of the program so uh that means that threading issues are will be reduced so long as i don't use threads for other things and um it will be a lot easier to synchronize certain behaviors so i think that that will actually help out with some of what we're doing now obviously you could say uh this could mean that it causes slowness in the loop but really this would only happen if you were executing code all the time like i said before the purpose of this is just to transmit information from the author of the project or author of whatever creative work is being worked on at the moment to the c layer so um there is really no constant execution of code from the higher level scripting language layer it's just sort of like translating immediately from that that code into c functions and c data structures um and the most important reason for wanting to do this is that it's uh fun i like writing language implementations over the last couple years i've had two other instances of projects where i was writing my own lisp or scheme implementation for various purposes and this is another instance for me to do it but in a more constrained scope where i get to have you know take very specific decisions that work for this project and not try to write some general purpose language so i think it's kind of fun to have a very specific use case here so that we can play with language design a little bit and try to figure out some cool ways to apply this i also want to say hello to johnny walker and to john thank you both for joining i appreciate that um mr bill says this is fun i want to create a custom language for my engine too well you know maybe uh whenever we're chatting on these streams we'll have some ideas and we'll both get to implement some cool language features so let me just get a sip of this copy here and we'll move on to the next slide all right let's talk about the language design itself so since this language is actually being created specifically for this project we get to choose how it works uh we're not going to be following any existing language specifications uh we will be taking inspiration from things that make sense from different languages but this is really sort of a blank slate basically for how things are supposed to work and um more importantly we get to choose what not to implement which is very important when it comes to trying to get things done fast if you don't have to think about a lot of different use cases or edge cases or anything like that you can really quickly get something working which is basically what i've been trying to do and then be able to become productive with it quickly so i kind of like the fact that you know by saying no to certain things we actually get a result more quickly and then i think it will increase the leverage of what we can do with this project a lot more quickly than if we were to use some other language mainly because we have total control and we can um avoid some you know certain classes of problems like i was saying about you know threading and synchronization issues before so what do we really need from such a language it's like i said before this is a control language for the c engine layer so what do we really need to actually communicate from the lisp or scheme style language down to c uh we need a specific set of primitive data types obviously you're gonna need primitive data types so things like strings integers floating point numbers uh lists and also time so since we're going to be dealing with music video um potentially game programming having time be a first class language type or concept could be interesting basically to specify let's say five seconds or something like that with a specific syntax so i want to add stuff like that in so that it's easier for me to express certain things like timing for different events and whatnot also we need to be able to invoke c functions and declare bindings in a convenient way so the whole point here is that this language exists to call directly into the c layer and if we need to add new functions for this language to be able to call it needs to be easy to add them and it needs to be easy to implement the behavior in the c side of the functions so right now i think that we've got a decent or i'll show you what i've been doing i've got a decent idea for how to make it work but i think over time we'll figure out some optimizations to the pattern for describing bindings that might make it even quicker to deal with and over time i mean we could just realize oh we can call directly into c we don't have to worry about doing any translation so it's just a matter of iterating on that concept as we go but i'm going to try to be very pragmatic and just get things working first and then build it up as we go on in later streams um also we're gonna have a very limited concept of scope for now um if you have studied programming languages to the extent you sort of know how they work scope is a big concern in terms of you know how you are able to resolve variable references or function references for now we are only going to be concerned with module scope we're not going to have any kind of function scopes or anything like that just basically whatever you see in the current file is what's available in the the scope of the quote unquote program it's not really a program it's more of a description of a project later if we decide to add things like functions and whatnot we will have to deal with scoping but like i said i'm going to try to avoid that because i don't really want um the c layer to have to execute functions defined in this layer so we'll see about that maybe maybe i'll change my mind on that in the future but for now we're going to try to avoid it also we want to make sure that the language enables an interactive workflow so we need a reply when we need constructs that are evaluable and when i say that what i mean is aside from the language itself being conducive to interactive programming we also need to design the um the way that you use the language the constructs or the ways that you call functions to and even the c bindings themselves to make them amenable to re-evaluation while the uh the the program is running so we want to be able to update what we've written in the file and see those changes happen immediately inside of the the preview window uh and lastly we need to be able to send individual commands from the editor um and in this case the editor being emacs we need to be able to do things like invoke the preview window start and stop the transport for playing audio or video changing the project in you know maybe causing properties of certain things to be changed etc so aside from the need for a language to describe what should be displayed or played in audio form or whatever we also need a way for the runtime to accept commands and the commands will be sent at the same way as what you see in the documents you know it's just gonna be function calls et cetera so um i it's just it's basically gonna be like a scheme or a list where you have a reply and you're sending commands to it and that's what happens so we're just gonna continue building on that concept as we go here hello to uh ronnie and uh cool ayat cool sorry okay let's talk about what we don't need for now at least we do not need logic constructs and control flow so we don't need if statements we don't need loops um those aren't necessary because we're we're taking a declarative approach to describing describing what should be happening in the creative work or in the project so if we're rendering something to the screen we're sort of taking a declarative approach to describing what to be just what should be displayed we're not going to write loops to say i want to render 1000 things now in the future it may become necessary for generative art or other things of that nature to have loops or functions but we're gonna wait until we get to that part for now the things that i have in my head are all mostly about like describing specific things i want to display or audio that i want to create etc and i think that most of it can be done declaratively without any kind of uh functions or logical constructs and also that's the next thing functions in lexical scoping we don't really need that yet um i don't rule it out but like i said i'm going to avoid it as long as possible data structure definitions like uh complex data structures et cetera we don't need that in the in the language layer the scripting language layer because we have that already in c so the c program understands the c the data structures that are defined in c um the way that the code work on the scripting language side is that you just sort of describe what things you want and that gets boiled down into c function calls that create the requisite data structures so um the the scripting language does not need to be able to describe those things because they already exist the scripting language purely interfaces with the c layer through function calls um macros we don't need macros yet but there's a chance we will want them at some point in the future especially if we want to make the language a bit more expressive in terms of how you can declaratively define things um but the design for the language i've come up with so far should be um smooth enough for now so that we don't really need it for a while uh but it really just comes down to like if i'm trying to write this stuff in practice to create a project um do i feel like i'm having to write too much code do i want to kind of simplify the patterns and whatnot the the reason why a macro would be helpful there is because i don't have to literally follow the definition of the language because everything needs to be a function call into the c layer i can have a layer above which sort of translates some structures of code into the actual c function calls at the scripting layer so sometime later we probably will do that but for now we won't also library modules defined in the language this is something that we don't need immediately but i can already imagine a use case for library modules let's say for instance um if i wanted to reuse some assets for making thumbnails for videos or if i wanted to reuse some definitions for audio plugins or synthesizers etc whenever we start getting to the audio side of things it would be nice to define libraries of those in the scripting language itself and be able to load those in to various projects so we have like a load path and then load in modules but we don't need that at the beginning at the beginning we can just copy and paste code and just you know go on with our business so uh we're gonna try to be pragmatic pragmatic is sort of the the uh the word of the channel here which we're just trying to build things up and polish as we go so we'll deal with library modules and stuff like that later after we sort of see a direct need for it and i'll also say that this is more than just a data format since we can actually invoke functions in the c layer it allows us to automate behavior where necessary so uh what i'll show you in a second will look sort of like you know a data format to some degree um but really it is just you know supposed to be directly function calls all right so let's take an actual example or look at an example code snippet for what i have in mind for this language i think that will be better than me just talking on and on about language features so um this snippet demonstrates some of the ideas that i have in mind it's sort of my current state of where i've been thinking about all this and um let's just take a look at what i've what i've got here um so first of all we will have things like define which is what you see in scheme for defining a binding for a symbol name to some value so in this case we're defining a font called font jost and then it calls a function called font and basically says the font family is just star and the weight is medium so this is basically us loading a font into the project so we can use it to render text somewhere i actually don't have an example of the text being rendered here but it's an example of us pulling in an asset and then being able to use it later in some part of the code then we're defining a binding called moving circles and we're passing that the output of the scene function which takes uh keyword parameters so i guess that's one thing i should mention here is that i will prioritize using keyword parameters for things mainly because i think it makes it easier to read what's going on positional parameters are notoriously difficult to understand like sort of the intent of the code so having keyword parameters like this gives us the ability to have some kind of structure to what we're passing in so here we can say the members of this the the scene are a list which only contains a circle so in this case uh list is um a c function which is going to take whatever is passed into it and then basically create a list out of that so it's not like um scheme or lisp where you call list and it sort of creates uh a um a linked list of sorts this is gonna be a different strategy that we're following because we're trying to be a little bit more um memory friendly circle is another function that takes in details about a circle to be rendered on the screen then we have a timeline down here which uh you can basically describe events that you want to happen and you can see here we have at uh five seconds so this is a place where we would have some syntax where you could describe a time range we want to say five seconds um so really what you're seeing here is just a hierarchical hierarchical description of what should be happening in the scene and then at the very end we have this uh call to scene preview moving circle so we're basically passing in the scene here and then scene preview will be rendering moving circles and now um when it comes to interactivity we should be able to reevaluate this whole form for defining moving circles and then any change we make to that should be automatically visible in the preview window so we shouldn't have to restart the whole thing we should be able to just um uh see the changes in real time now what i'm not sure about is if i need to define all of these uh elements in advance like the circle maybe have a top level defined for that so that it's easier to iterate on that specific element or if there's some other ways that i can do this we're going to have to play around with that concept a bit to find the right formulation for it you know it's just some of the things we'll figure out over time but for now i'm pretty happy with this um it took a while to think about how i wanted to represent all this both in the code itself and behind the scenes in the actual language implementation but i think this is a good enough starting point where we could move forward and try to use it and then hash out some ideas so um the symbol in the first position of the list is always a function to be called so things like define or scene or list there are always functions that are going to be defined in the c layer so if you know anything about scheme or lisp implementations there's this concept of a special form where it's a function that's defined in the implementation of the language and it possibly could have its own rules for how its parameters get how and when its parameters get evaluated normal functions in scheme or lisp have a very specific evaluation order depending on which implementation you're using you they basically tell you what the evaluation order is but when it comes to things like macros or special forms the evaluation order and how the parameters get used is not doesn't necessarily follow the standard evaluation order of normal function invocation so what we're basically saying here for this language is that uh all of these uh call eva expressions they expect to have some function that you've defined in c already for now and then those functions can evaluate the parameters however and whenever they want um but you know the reality is that whenever you call one of these you're we're basically going to be evaluating recursively everything in this tree to get the values out and use them so it's not like you're going to have any kind of weird sequencing of evaluation it's prob it's going to be what you basically expect but um it is up to the implementation of a given function just to decide that and uh functions will often be called with keyword arguments but could also have positional args you'll see here that there is an rgb function that only has positional arguments and that's mainly just to keep this the code terse for a very obvious use case here rgb 255 0 0 it's pretty obvious what that means for circle it would not be so obvious if i did not have name x y and well color could be obvious but those other ones would not be as obvious so keyword arguments are going to be better for uh for all that bill asked me have you named it yet not yet um i mean i've sort of been looking around for things that are like you know flux control flux something whatever and a lot of stuff is already taken apparently flux is a pretty common and popular name for you know not even just software other other things as well so um i haven't come up with a name with it for it yet um but i'm definitely open to ideas if people have ideas for what to call it um since it's very tied to this project i would probably be something related to flux compose klux from pose that's a possibility all right so i think that's it um hopefully that looks cool and we're going to look at more about the implementation in just a moment all right let's talk about our tasks for the day uh first of all what i really would like to do ah i'll show you the code after we talk about the tasks um i'll briefly say what i've done so far i have the tokenizer working i have the parser working and what those both of those do is basically get us into a state where we have an ast for the language that we can then use for evaluation so the point we're at now is that we need to implement basic evaluation of expressions in the language which basically means we need to write some functions for evaluating different types of expressions to get the values out of them out of them and we also need to work on function registration as well so we need to be able to register functions in the runtime to be callable so we're going to need that pretty quickly to make this possible um that might be interesting we'll see how that goes um i haven't really thought about the data representation for that yet and you'll see that i hesitate to just jump right into creating data structures or you know arrays or anything like that mainly because um i'm trying to be very thoughtful about how i represent all the stuff in memory so we might i'm gonna i'm gonna talk about uh all that stuff and a gun i will tell you how i wrote the tokenizer and parser in just a moment um also i've got a weird issue like a couple hours before the stream i realized that there was a critical bug that would prevent us from being able to do any of what i wanted to do today so i had to fix that and there's one little tiny issue left it is a little bit more than tiny but we're gonna try to figure out um how to fix that and i'll talk more about that when we actually get to that part of this and we what we really want to do is evaluate call expressions we want to get the point today where we can evaluate these expressions and do something in the c layer and if we can do that then we might be able to get to porting the previous guile interface code over to the new language i don't think we'll get all the way there today but it'd be really awesome if we could so the the stuff we were working on the last time where i had guile scheme code that caused some circles to draw i would like to be able to get back there as soon as possible so i think if it doesn't happen today i might do some work over the weekend to finish up some details so that whenever we um come back on tuesday then we can just sort of get started using the language and and get back to where we were and then you know we should try to finish rewriting that uh writing that render to file functions where we actually have the ability to send the string over to the c layer uh successfully i think we can do that now i only just had to write like a whole language to uh to solve that problem so uh that's where we are uh bill says did you write a bnf for it no i did not i i am taking a fully hacker approach to this i'm just writing code straight up okay so let's take a look at uh what i've done so far um trying to think of the best place to start on this what i'll do is show you the internal header um yeah there's a few things to mention but we'll start with the data structure so the tokenizer is basically a function in the code that will take a file stream that the user has given and then um one by one look at all the characters in that and decide what type of token that character represents and maybe if there's a substring of characters that represent a token it will chop that piece out and call it something like let's say a string or a symbol or a keyword etc so we have some definitions for the different types of tokens we might have like a parenthesis a string symbol keyword integer and float source location is not something i'm not using right now but it is something that would be nice whenever i actually write error messages so it will tell you which location in the code uh the problem was found and then i'm using this uh strategy for um modeling the data where you have a struct which is a header struct for a particular family of types called token header and the most important piece here i did not use flex for the tokenizer i wrote everything by hand um the uh the kind of token is what's getting stored there and then any other struct that represents a token type starts with the token header and then adds any exist or extra fields that are necessary to model that particular token and the reason i do this is because we can exploit the way that structs are organized in memory so that we can have a token header be sort of like the generic type for all the different types of tokens but then we can cast down to the individual token types when we know what kind we're looking at so it also gives me the ability to store these things sequentially in memory so basically whenever i tokenize um or parse uh the code i'm i'm writing all these structures at linear linearly in memory uh it's sort of orthogonal i guess to the the way that i'm setting these up but it does give me the ability to sort of know what i'm looking at i can sort of jump ahead based on the size of each type of token and just look at them as a symbol a single stream um so like i said i'm thinking a lot about memory layout whenever we're dealing with this code it's probably not fully necessary because it doesn't need to be the most performant parser in the world but it's an opportunity for me to try to um improve my skills with doing this kind of thing because i haven't had to do it you know i've been using higher level languages for years and i haven't had to think about where in memory everything goes so this is a great opportunity for me to actually learn some things about that i'll talk about that more in a minute to explain more about what that really means all right so for um that's everything for tokens now we also have value types but that sort of comes later in the pipeline um and now we also have expression uh kinds so we can have a list we have a symbol we can have a keyword and we can have a value now other things might get added there later but for now um when you tokenize all the input you have a list of tokens effectively then the parser takes that list of tokens and turns it into higher level language constructs basically creates the ast for the language so now we know how to identify expressions because we know where a starting or an open parenthesis started and there's a bunch of stuff in between and then we had a closing parenthesis and then anything that's in between can get interpreted as something that's relevant to the language now obviously we're doing sort of a one-to-one mapping for everything aside from parentheses at this point but it may turn out that uh with keyword and value pairs i may have a higher level construct for that that's easier to handle so you don't have to go looking for the value so that may be something that changes soon but um this is simple language so we don't really need a whole lot of complexity in that in that part so same structure here we have a header type for the um the expression family of of types that has a kind attached and then we have the individual structs for each kind so like a symbol has a name we have to link the name we also have whether it's quoted or not so we can do symbol quoting basic quoting basically in uh in this language to use a symbol as a a way to identify something without having it be a a symbol that tries to be resolved so we want to avoid resolving those symbols we just want to use them to identify something uh keywords similarly have a name and a link but they don't have any information about whether they're quoted because we don't care about quoting keywords uh value types have a value header and this is a bit of a freaky part of this because we're sort of nesting these concepts of um having an expression header and then having a value header that could be something of various different sizes now you might be looking at this and thinking well why don't you just use a union for that the reason why is because i have value types that have character arrays in them and you can't know how big that object is going to be or how big that struct is going to be at compile time so there's no way to build a union off of that as far as i know so it's better for me to just sort of deal with this myself also a list this is a place where we have lists now we can have a an array of items that gets dynamically created based on how many things are inside the list and also the length of the the list itself also i've got this concept of cursors here where when you want to iterate over how this stuff is laid out in memory you need a cursor that can sort of say uh where you currently are and the sort of the list or the array or vector you could even call it of expressions or tokens and then we have a function that can increment that cursor it's like an iterator almost where you can sort of get to the next location so i've basically implemented iterators to some degree uh here as well um and then we just have some internal apis that we're going to be calling in the unit test which i can show you in a minute as well so basically tokenization moving to the next token using a cursor moving to the next expression using a cursor parsing using a a a vector of tokens i guess you could say and also this init function which probably is being used in tests but this is a an internal header file that's only used within the project these things are not part of the actual public api of the lib flux library mainly because uh the public api should just be simple where you just call like eval so i think down here we have uh flux script eval and flux script eval string in flux.h which is basically the public api so anyone anyone who wants to eval some script code just calls one of those methods and you're done gun says not linked lists like where you can insert delete items the reason why i don't like linked lists is because you're allocating each node of the linked list on the heap and it could just be anywhere in the heap and if you're iterating of that list you're just jumping all over the heap in memory and you're you're just getting cache misses left and right now it may be the case that the memory allocator is actually allocating all those probably sequentially in some space but you can't really depend on that and the more fragmented the process memory gets the worse that's going to be so i don't know maybe it would be fine to use lists but like i said i wanted to try this strategy out and uh and see how it went okay um the other thing that i did for this which i'll get into the code for how this stuff works in a second i wrote a unit test harness i know that there exists uh unit test harness for c but because we're writing everything from scratch in this channel i wrote a unit test harness from scratch as well so let's see test main um so the this is super simple code but it only has to be super simple code um computer says you can define your own allocators that's true i'm sort of doing that i guess but just not at the level of allocator but if you have suggestions on that i'm certainly willing to hear them um so the test harness is basically a c program and um it will run this test laying suite and i think that gets defined in test.h so in test.h i have a couple macros for uh defining a suite and and then also saying whether something has passed or failed a test is passed or failed let's see where is teslan suite defined let's find that test oh test length suite that's in the the test link file okay so um basically i just run whatever the sweets are and if i add more test suites i just add the function for those here and then it will write out the results at the end and um the language tests a little bit sloppy here on the the line endings on the macros but basically i've written a bunch of assert functions for uh asserting whether a value is what it's supposed to be and writing out messages and then i have um some helper functions for tokenizing and parsing test code and then i go into just the list of tests basically where i have a test name test link tokenize empty and i tokenize some code i just check what type of token was returned and if it passes i write pass if it fails for some reason which usually is comes from one of these asserts then it will show in the terminal that uh the test has failed and it will tell me which function it failed in so pretty useful very basic i don't really need more than that and in fact it's been very nice for uh iteratively building on what i'm doing here because i can really quickly just run the test and get the output immediately without having to like try to start the program load a script etc so having a unit testing framework is going to make things a lot faster here and i can actually try to run that right now um let's see if i just run let me just jump back to the main folder run compile um let's see make c build and uh build slash run tests let's see if that works uh yes save it okay so it's already finished compiling and running here i think that i compiled it before i've got a lot of uh test output here sorry um output we're logging from the code but we'll talk about that in a minute too so um we run the test runner and man there's just a lot of noise here from all that that output but let me see if i can take turn that off really quickly uh let's go into uh script.c i just want to show what the output looks like without all the spew in it so for these um macros we'll just uh paste this here clear that out and um i'm gonna just put in like brackets so that it's like a no-op let's just see if that works i'll be curious to find out if that actually turns off all the output all right we're compiling we're running tests again okay cool so now we can actually see the output uh correctly um and we can see that we have a suite called tesling suite and just a bunch of tests that have passed here and one that has failed and it told us why it failed so it didn't really take long to write a test harness for this um and it was very worthwhile because it's served me well so far let's see johnny says language design is so cool working my way slowly through crafting interpreters book oh yeah that book looks really good um i actually want to take a look at that at some point soon gunn says the dragon book is also really worth reading and principles of compiler designed yeah um i haven't read either of those books but uh the dragon book is kind of um funny to me because in the movie hackers which is kind of a uh recurring theme in my stuff um one of the books that they pull out as like one of the cool hacker books is the dragon book uh compiler design so i don't know it's it's funny when people bring that book up it's super old though i don't know i haven't read that before one book that i read um what was it called lisp and small pieces which basically tells you how to write a list compiler pretty useful for understanding how list compilers work um and they go pretty deep and there's another one also i was looking at can't remember what it's called compiling with continuations maybe so you know there's a lot of really cool compiler books out there if you like learning about compilers and lisps are a great language type to write a compiler for i think compiling with continuations is like a mix between writing a scheme compiler and writing an ml compiler so uh it's kind of interesting stuff in that one but it's very very um it's more difficult material than typical compiler books i would say all right so um yeah test harness that was really fun to write and very easy to write that was probably i don't know an hour hours worth of work potentially so now we are in script.c this is the file where i've written all the code um as of right now it's about 512 lines and a lot of that is logging because uh whenever you're dealing with the sort of memory mechanics i'm doing and also dealing with parsing and tokenizing you kind of need to log stuff to understand what's going on i mean you could run into bugger too but sometimes i feel like it's good to have uh statements that tell you where you're at um so first of all we can talk about the tokenize function which um the first thing it does is it actually allocates a buffer for all the token information so like i said i don't want to be allocating a much stuff randomly on the heap i actually just allocate a block of memory about 1k that will store all of the token information for the current eval so whenever you're evaling a an expression or a script we are just we have one block of memory that we're using we may have to do it deal with resizing at some point if the code files get large i don't really know yet uh we'll have to do some tests on it to see how soon it's needed but we allocate that strip of memory and then we basically go into a loop where we look at every character and decide what type of token it is like i mentioned before there's a lot of redundant code in here because i wanted to write it first and then start breaking things into functions that take away some of the repetition and also make sure that i don't have little bugs here and there if functions are out of sync but very straightforward stuff mainly because this language is not very hard to parse that's a good thing about having syntax like this it's not hard to parse at all and a lot of it is sort of the same pattern over and over especially when you're doing things like parsing strings or symbols or keywords because they're just you know strings of letters um so that function itself is not very long and like i said there's there's some deduplication that can be done in some of the patterns and once that's done you basically end up with a linear strip of token data allocated in memory so i'm not doing reallocations every time i actually i've pre-allocated this one block of memory and then for every token that i get i'm basically just stamping it out at the next location um in that buffer that i had allocated that out ahead of time so everything just gets put in place one after the other and then when i want to read back through that list i start back at the beginning the first memory location where i allocated all that and then i stepped forward using this function called let's see it's uh flux script token next so you give it a token cursor which has a location that is currently at and then it will look at the kind of token that's at that location and depending on what type of token it is it will just increment the pointer location based on the size of that type of token to get to the next location where a token would be so that gets used both for writing out the tokens and also reading them back because we need to sort of be able to jump to the right location in memory at any given point um so yeah a lot of pointer arithmetic going on here and a lot of just writing directly to memory and the danger of doing that is that if you happen to get your calculations off in one of one of the locations where you're either writing or reading the memory you could end up reading garbage and then have problems with your algorithm and this is why i've written a ton of logging that actually writes out memory locations of the things that are currently being worked on so it makes it easier to sort of trace through and see where certain things are happening to certain objects so if i turn on that logging one more time let's just uh pull these macros back in actually we'll uh we'll leave the parse log out for now if i rerun the tests you can see that um there's a bunch of little uh hex identifiers at the very beginning and these are all like the memory locations of the token that we're currently looking at and i'm writing it out in a way where it's easy for me to pick all these out so the cool thing about this is that for a given memory location like let's say this one right here that ends with ca4 if i use consult line and type in ca4 i can see all the locations where that the hash was used especially if it's used somewhere else in the line and also for whenever the parser is picking up that token and using it so this made it a lot easier for debugging having all this log information out especially with the memory locations and i can also sort of tell based on you know whenever i'm incrementing the pointer how many bytes forward the pointer is moving to tell whether my calculations are correct or not so there's a lot of reasons why if you're dealing with direct memory locations writing all of this stuff out is very helpful hey minican peter says do you plan to have special literals for time-related values might be a cool feature hmm i hadn't thought about that um i mean i'm definitely gonna have uh if what you mean is like have an actual primitive type where i can parse out like a time value yes i'm gonna do that in fact i showed that a little bit here where um right there i have like five seconds i don't know if you can see i'm trying to circle it with my mouse here but that's basically five seconds or ten seconds uh we can even have something like 150 milliseconds uh i would like to um be able to parse these things because i think it would be nice to make it very obvious when you're reading the code what time intervals are being used for certain things so uh we will definitely be trying to do some of that if that's what you meant if it's not what you meant then definitely let me know all right so that's uh tokenization uh for parsing uh parsing is a bit more complicated mainly because actually let's turn on my key cast here parsing is a bit more complicated because um you have longer strips of expressions and when you have lists especially nested lists to write it all out as a linear strip of memory it gets a little bit more involved you can definitely do it i'm doing it right now but it was fairly difficult to uh get through and fix all the issues um i'm really really enjoying writing this in c but the danger of writing this in c is that i have to deal with these kinds of strange bugs where you just get a segmentation fault or some weird behavior and you have to try to trace through it yourself but i don't know um it's frustrating but it's also like a really interesting challenge to come up with the tools necessary to make it easier to discover these problems and also it's very interesting to build that intuition about what the problems could be and just be able to sort of psychically debug the code just by looking at what the output is so uh i don't know i've talked about potentially using a different language for the native side of things but right now i'm having so much fun with c even though it is problematic at times that uh i don't think i'm going to do that anytime soon c is pretty awesome and also as you can tell it compiles super super fast and that's great all right so um parsing so we have this flux script parse function and we'll get to coding in a moment i just want to sort of explain what i did because i've been coding for a few days on this off stream uh the parser will also do the same thing where it allocates its own buffer for parsing so we have two parallel buffers now one for tokens one for expressions uh we'll probably do the same thing for eval as well but it might be different in some ways same same basic idea though and then we take the token cursor so we're getting um a token header passed in which is basically the start location for the tokens to read and then we're going to plug that into a token cursor you can see down here a bit lower in the screen so we have a token cursor we say the current location for the cursor is the start token and then we start by creating a top level list so in list languages you have this con concept at the top level where it's the um the is effectively an implicit list where you have every expression at the top level of the file and you can kind of treat it in conceptually as if it's a list but it doesn't look like a list in the file because you don't have to wrap the whole thing in parentheses so we have the top level list and we are creating that right at that first memory location of the script parse buffer so we're saying the first thing that you're going to see is the top level list and then everything in what gets evaluated is going to be contained within that top level list create a list cursor then we initialize the list which just um oh let's see i need to turn on uh eglot let's see if eagle out will work for me today all right cool um that basically just sets the length of the list of zero the the kind of the uh expression at this point in memory and also the um the first element of the list we're basically initializing it to the none expression so that we don't end up with any kind of weird behavior because it's uninitialized and that's sort of another issue with writing code in c and doing some of these things manually with memory is that in other languages they will pre-initialize things for you in many cases but in c you don't get that so you have to be very careful to make sure that if you're going to be reading values of a particular data structure you need to make sure that wherever you're creating instances of this you need to initialize them somehow at a reasonable location so that you can depend on them being what you expect them to be because otherwise you're going to end up with very strange issues so pre-initialization is very important a lot of the code that i've written is structured in a way where things sort of naturally get pre-initialized at the right places like for instance whenever you're building the list of tokens or building a list of expressions we automatically set what would be the next token or expression location to a none kind so that we don't accidentally forget in some code path to um to clean that up and then have the the iterator just run off into oblivion looking at garbage locations in memory basically um all right so then this function basically just calls directly into a recursive function called flux script parse list which will descend into the nesting of lists effectively by looking for parentheses so it just uses the the token cursor and looking at the current token and just goes into a while loop basically looking for the end of the list of tokens and um it will look for the print paren tokens to decide whether to go even one level further down in the nesting structure or you know we'll look for certain types of tokens for symbols or keywords et cetera and we'll set all those things up there's a lot of code here not really worth me walking through line by line explain it to you but if you want to look at it it's in the scripting branch of the repo at least right now i'm going to merge this into master as soon as i feel comfortable with it but if you want to look at it it is there um and we have some other helper functions here a lot of that's not really worth talking about just yet but um a lot of stuff is related to trying to uh um judge the size of elements because like i mentioned before when you have these sort of header structs and your actual struct types can extend the width of that potentially indefinitely based on or to an undefined length um for strings and whatnot you have to be able to calculate for any given element you're looking at how big it is so there's you know functions here for dealing with that uh gunn says uh pointer equals ox dead beef yeah i don't know that's that's a very somehow that popped up in my head either today or yesterday also which is pretty funny i love this backdrop so um gun also says uh bison helps a lot when doing parsers yeah you know there are definitely tools for doing this stuff it would probably have made my life a lot easier but i really just wanted to enjoy writing this code um the thing that i like about this is it doesn't require any external dependencies it's all part of the c code and it just gets compiled directly in don't have to worry about statically linking stuff or whatever obviously bison and yak they generate something you compiling to your code also but uh this is all my code and it's up to me to uh to extend it and maintain it so i kind of like that you know that's the whole impetus of this channel is we want to do things from scratch in a lot of cases so i'm not even going to bother using tools like that unless it's really really uh beneficial and in this case i feel like the the value of using those tools isn't worth it enough for me to to try to use them all right so that's one hour of me talking basically about what i've done in the last uh four or five days um and i did say in the title that i've you know writing writing is uh what did i call it writing a language in five days let me check that creating a scripting language in five days so uh the part that we're at right now is that we can successfully tokenize and parse the input for this language um not everything i haven't added times yet i will add times but you know keywords symbols uh parentheses like the general structure of the language is being parsed to where now we can actually look at those and try to uh evaluate that uh iatcool says live coding in five days yeah you know if i had tried the live code doing this thing it would have taken us like three weeks probably because there's a lot of this that really requires a lot of head scratching and thinking um so yeah i don't think we would have been able to make as much progress as i did off stream if we had been doing it on the stream so i feel like it was worthwhile to yeah that's a hackathon it definitely was a hackathon i had various reasons why i was uh uh able to spend time on that the last few days so uh i was able to make some progress it was great it's always nice every now and then to have one of those extended hacking sessions where you're really deep into some code and you're just building stuff up and like it's like the only thing you're thinking about you want to wake up early and go to bed late just because you want to work on it more that's that's sort of the situation i've been in so now we're going to take a more measured approach in the streams and try to implement some of this stuff i probably will continue to do this a little bit though like do some hacking off stream mainly because some of this stuff is not very interesting to watch in terms of live coding especially if it's the first time i started working in a certain area like if i had started writing this language probably writing the tokenizer might have been fun because that's easy but the parser part is not so easy bill says is circle missing radius yes you're right it definitely is missing radius i'll have to add that later so anyway the point of that was i probably will do some more work on the language off stream so that next week we will be able to use it for more fun stuff and i think we're gonna be able to make a lot faster progress because of that um okay so implement basic evaluation of expressions so um let's go back to script.c and we're going to jump down to the bottom here so we can assume we've already got a fully parsed file at this point or expression at this point and we need a function that can deal with um the actual list of expressions or the top level list of expressions and and decide what to do with them so i'm going to kind of go top down a little bit at first to sort of get some thoughts together and then we might have to go bottom up to do some of the lower level eval stuff and then meet at the middle a little bit so we'll see how that goes so we have this flux script eval function i think which is part of the public api so we're not gonna excuse me we're not gonna use that one for this what we're gonna do is have a we'll call it void for now we'll fix this in a minute flux script um eval expression all right we'll just say eval expert and the input to that will be an expert list well no it might be it might be any kind of expression so we're going to say expression header so we may also need a cursor because we're going to need to move forward as well and i've found that reusing cursors can be useful because it it keeps you from needing to go back and reiterate over everything to get to where you were before so we'll think about that for a minute um for now i'll pass in a cursor so expression list cursor you know actually let's let's skip that we'll go back to that we're going to start simple and try to go from there so expression header so for evaling and expression we have to figure out what type of expression it is first um so we're going to use a switch and we're going to look at expert kind and for the first case we're going to look at uh expert kind um let's see what should we try to valve first so evaling values values are kind of broken at the moment we may have to look at that too but the evaling values just basically requires returning the value which is going to be easy which that's kind of nice so uh kind value so i think what we'll end up returning here is a value header because we want to return a value as a result of evaling something in theory so here we can say return uh expert value expert and then it's going to give me any kind of completion on that uh value so what does an a lot like at this point uh returning value header from a function with incompatible result type oh okay yes and this is another thing um since these are reused pieces of memory i can't just straight up return that value i'm gonna have to create a new copy of it in memory somewhere so uh any of this eval stuff uh memory buffer of some sort so we're going to need a memory buffer of some sort for this and this is where things get more complicated because you can't just have a linear strip of memory that you're dealing with for storing the things the results of evaluation because you need to hold on to some of them so let's say we just we define a scene and a scene is using um some other thing like a circle that was defined and assigned to a variable so if you have a scene that's currently being used that's currently stored in memory and it has a reference to the uh the circle that's stored in another variable then you well this is the other way around either way you need to be able to keep a piece of thing whatever component in memory that you've already evaluated while you're sort of re-evaluating re-evaluating the other part of it and um you'll have to do garbage collection of some sort now i think that we can definitely simplify garbage collection in the sense that if we're evaling inside of the event loop then if we're not doing something on another thread then we have a point in time where we can get rid of the old object and put in the new object without there being any kind of synchronization issue and we could potentially free that memory or at least let go of that location in memory and have it be reclaimed by someone else later and if we do have basically a block of memory where we're doing these allocations where we don't have to do freeze free free like basically releasing memory then um pausing will be less of a problem i think because we don't have to worry about you know that you know any momentary pauses from freeing memory now gc is a lot more complicated of an algorithm free is not the only thing that gc is doing or garbage collection is doing is doing a lot of like you know mark and sweep is looking at is like walking the stack trying to find pointers to locations in the heap all kinds of stuff like that which is a lot more complicated process so you know freeing memory by itself is not super slow but the mechanics of checking for what can and cannot be freed is is more complicated in this case with all of the things we're going to do in this language we sort of know where everything is there's a lot more constrained scope about um what variables are current currently being used so i think that we won't have as much of a much trouble there so anyway all that is to say that um i think maybe we just use a memory buffer and just really inefficiently deal with it for now and then later try to do some kind of basic garbage collection that isn't complicated all right so we might do the same thing here where we have like a a value cursor that is able to just place something at the next location the next free location in memory that could be useful for things like lists or um mainly lists where there's a list of items that needs to be held onto that if you want to read that list and get to its stuff really quickly then you're pulling a lot all that into the cache at the same time so that does sound like a pretty good idea so we may have the same structure i actually started working on a more generalized way to do this kind of what i'm calling vector implementation um on my other computer and i did not check it in yet but i may have to do that soon because i keep doing the same pattern with everything so i probably should pull that in but we'll just write it from hand for now all right so um like i was saying before we need to copy this not uh pull the memory location all right so then what we'll do so um one other interesting aspect of this is we also kind of need a binding table as well let's see we don't need it for things that are being evaled in line so if you eval a a function that's calling circle to define a circle if it's not being stored as a value somewhere like inside of a scene object or even as a define then we could immediately just drop it in fact that's probably what we'd be doing i mean it's not gonna there's not gonna be a reference to that anywhere so we have a block of memory there's just something that got out allocated and there's just a hole there in the memory that won't be filled um hmm i'm trying to think about like we'll have a mixture in this memory block of things that have names and things that don't have names and um there's probably gonna be some way we need to deal with that but we'll need a table of binding names at some point but now i'm not gonna add that so right now we'll just call this a script value buffer something like that um value buffer initial size i'll set this to something a little bit bigger but it probably we're going to make these expandable at some point in the future where you know when you start to get to a point where you're going to exhaust it it will double the size or maybe use some other heuristic decide how much of an increase it needs in size and we can use realic for that to make it really easy to get a larger chunk of memory that's still contiguous um or seemingly contiguous based on how the memory works in the operating system so we'll start with that um yeah i'm just going to call that value buffer and then we'll also add a logger specifically for eval and the reason why i'm doing this is because it's nice to be able to have log messages that have the prefix like eval at the front so you can kind of see when messages are being written with respect to a specific part of the process so that will do for eval and then then i'll go rip off some code that i wrote here for allocating the buffer so we're just going to follow basically the same process here i'm going to change this to uh value buffer value buffer value buffer initial size value buffer value log i know i'm i'm not being very efficient with my keyboard movements but that's okay value buffer value buffer okay so i think we got everything here oh that shouldn't be all caps value there we go all right um gun says create some sort of symbol table where a symbol points at its value storage exactly that's basically what i'm going to end up doing um that will be easier because at least there well no it won't be easier because a hmm yeah we'll probably have to have the same kind of structure where because symbols can have you know any any size of name any length of name so to have a symbol table you're going to need to have sort of variably width entries so i know another usage for a potential vector implementation okay so um what we'll do we need some way to copy something we're specifically values i think values need to be copied other things will be re-created so if we're evaluating a value and the nice thing here is that eval expression will be used recursively also so when you're when you're doing evaluation of a tree of expressions at the lower levels we're going to eval something and if what we get back is a value then that value can be used directly to assign to something else or to plug in somewhere for a higher level structure of the code so i think this is the right thing we're gonna get the basically the memory location back for what's stored in the uh the value buffer all right so for copying this we want to copy the expression value we actually want to copy the value itself and that's easy we can use mem copy for that so what we can do is have another function for that same pattern yet again we're going to have a function just like what we had here for the next token i'm just going to sort of steal this pattern really quickly use it again it's what's nice whenever you write some code is that you can just copy your own code and it's not a big deal um but definitely is begging for [Music] some kind of uh abstraction i think so value header yeah i'm not going to go through too much on this value cursor i'm going to have a cursor type for that as well i think so yeah a lot of stuff that obviously needs to be um factored out so flux script value next um yep value log value cursor yeah this won't take too long in fact what if i do this let's do this i'm going to use um query replace i think that that's good enough right so um token value yes yes yes no i wonder if i can figure out the casing let's let's see oh that's amazing you see that it actually picked the right casing that's that's really good yes yes yes oh i love that okay that's gonna save me a ton of time then okay some of these things obviously need to go away but uh the fact that i can just run through this really quickly is fantastic and i'll tell you what i did in just a moment okay so i ran the query replace function in emacs which asks you for what term you want to replace and then what to replace it with and then it basically just asks you yes or no at any location for the next um match about whether you want to replace it and apparently it follows the casing of the matches which is fantastic because it just saved me a ton of time there um it didn't miss this one because i told it not to use it but i think i have a value kind none okay excuse me so we have none integer float and string currently so none integer float string there's also entity which is something that's going to come later we don't have that right now so i think we do have a basically working function for this at this point i need to go and add the uh cursor type for that so let's go down to the end here we're just going to drop this in right there yes gun says that calls for a macro yeah it definitely calls for a macro of some sort i'm going to end up writing something for that i just haven't gotten to it yet so once i get eval working in this one of the things i'll do this weekend is clean up the code so that it's much clearer to look at then hopefully you know people can understand what's going on there a little bit better okay so here we're going to put in value header and oops not that value cursor all right cool huh you know i was calling this expression list cursor but really it probably should just be called expression cursor i'll think about that another time okay so back to the script code um so now what we have is the ability to walk ahead which is gonna be necessary because if we're going to eval and expression um for certain types of expressions we're gonna need to copy those things which means we're gonna have to have a location where we can copy it to so here uh let's see i can say the assumption is you'll be able to evaluate something but really we shouldn't pre-evaluate until we get to the point where we need it so i can call value next i don't have a value cursor let's see eval expression i might need to keep one so value cursor uh value cursor and here what i'll do is flux script give me some completions no ah here we are flux script value next can you oh come on so i was using lsp mode on my other computer and now i'm using eglot and eglot isn't acting the same way so it's a little bit inconvenient at the moment so flux script script value there we go next and then we're going to put in this value cursor here and then we can save that to value header um new value and then we can use mem copy i think can you give me the signature for that signature please it doesn't like something ah there it is okay destination which is going to be new value and source which is going to be i think value here is what is that a expression value i think that's just a regular struct member so let's jump to internal value yeah so that's just value so we're going to say let's pull this in and then we can pass the new value out value header why does it not like that uh expected expression okay that doesn't make any sense i don't know if i did anything wrong in my code i'll try to compile it really quickly no references found for okay let's uh pull the compilation buffer back all right so we got some compilation issues here thanks for coming peter uh enjoy your uh your feedback on stuff all right too few arguments to function mem copy okay fine uh it's not helping me at the moment what am i missing uh size t okay excuse me so at this point i know that it's yeah okay i know what the size is the size of um value integer i need to put the of there all right so that is basically copying from the old value to the new value location that we just grabbed from the memory block and we're copying the amount of bytes that are necessary for that integer now we should be able to test this if i yeah i'll make it work so so we'll we'll test it and see if we can actually get the the value back that we expect so it's a great opportunity to write a test so let's go to test lang and then we're going to add a new test here for eval so test laying a eval integer and we'll go up to the top here we're going to add a new function for eval similarly similarly to what we have done for the other ones maybe i should sort of interleave these a little bit eval okay so input expression list cursor um yeah i don't really need that i do need the result value though value header so let's leave this off for now because i don't really need that and then input from string tokenize parse the code so the result of parsing the code let me check what i'm actually returning there parse is returning an expression list which is great so i should be able to take this expression list results hold on a second why am i doing this i'm resetting the cursor okay yeah i don't need that right now in fact i could probably just no no i need to do the eval okay so return we're just going to evaluate so flux um script eval expression result and it might want me to cast that down we don't have a list cursor here i need to get this out of here so let's see eval the resulting expression and return it all right so flux script eval expression um implicit declaration i think i need to put that in the internal header file so let me grab the signature for uh let's see eval expression let's grab this put it into the internal header down right here make it x turn clean it up a little bit and now in this file we should be able to use it whoops the test length file all right so if i just kind of clean that up it should make that marker go away okay what don't you like it needs to have two things passed to it what else you want value cursor oh great value cursor yeah do i need that didn't i say yeah that's sort of the problem i guess you need to kind of have a value cursor that remembers the last place something was put in memory so we might even need a global value cursor that is relevant to the uh the global memory allocation but that's okay we won't deal with that right now um let's do this value cursor we're just going to knit a value cursor so value cursor value cursor um current equals null now that's going to be interesting we need to do a little bit of uh initialization on that as well so no maybe maybe what i'll do is i'll just go ahead and assign it immediately so the value next we have there okay so here's what i'm gonna do for now no no no let's not do that here's what i'll do since we're evaluating the memory we know that the memory hasn't been set up yet so we're gonna we're gonna set the value cursor to start with uh current equal script value buffer now let me actually just uh leave a note for myself here to do this may not be the best place to initialize this but it's convenient for now the idea is that we need to make sure that the cursor has a starting point no matter when we called it the first time and uh here that's going to enable us to let me make sure there's nothing else let's see value cursor uh yeah just current okay so yeah basically i don't need to to initialize this here i can just say value cursor run eval with the value cursor and that should work um result it probably wants me to cast that down to expression header let's just do that to make everything happy um and now i should be able to use eval inside of a function so i'm gonna start with just evaling an integer so we're gonna get rid of that part and we're gonna say uh 311. so eval i think i only take the the code right we'll double check that value header uh result equals eval 311 right okay i think that's right eval care input yes okay so now what we can do is uh check the output and i'm gonna rip off something i did up here we're gonna take a look at the oh no let me go right back to where i was ant value let's just grab one of these here and i'm going to assert that the what we get back is a value integer so [Music] value kind integer um let's call this value value and we'll say value kind here um that's fine and then certain value we're going to look at the value integer of value and what did i call that let me go back to internal value integer okay it's just straight up value cool value value value and that should be enough so i will say pass here and we'll see if this actually works i'll add this also to the suite clean that up a little bit and now we're going to run the compiler compilation buffer all right exited uh abnormally hey alex yeah it's c time it's c time baby uh oh value interior undeclared uh 3 14. yeah this is what happens whenever i'm typing and it doesn't pick things up all right so another error test length suite um expected declaration or statement end of input 337. oh how did i end up deleting the final uh curly brace okay something's wrong here exited abnormally i don't see any errors let me take this test out temporarily okay that's very weird oh undefined reference to value log that is in uh script.c value log i didn't make that eval log okay so value log should be eval log we're gonna use that query or place again value log eval log yes yes yes yes all right that's enough recompile all right so now that's not the the test we had we commented that out so let me go and run this again um segmentation fault interesting so apparently something i did here uh it does not like and it's probably the way that i'm trying to do mem copy so let's jump back into script.c mem copy uh we'll just uh we'll take that out and see what happens okay still segmentation fault so one of these things i'm doing is causing a problem so let's just jump into um gdb really quickly geekshell that'll tell us really quickly where it's happening uh m manifest uh gdb hopefully i have gdp installed gdb build slash run tests all right so we're gonna run that and we oh even the assert is a place where it's having a problem so let's see and testlang uh eval integer so value probably value is does it tell me what what is it uh display value yeah that doesn't look like a correct oh i think i wrote to the memory location that's what happened okay let's um jump back to this um hmm that's weird i'm gonna take that ampersand off that could be the problem so [Music] this value is not a pointer itself i think it's gonna do the wrong thing also but we're gonna figure it out really quick let's let's try to run this we're gonna quit that we're gonna go back to compilation buffer and run it again exit the abnormally it does not like the fact that this is not a memory reference or pointer now let me think a little bit about what this means so i probably need to do this i'm guessing just to make sure it's pulling the pointer of the right thing so let's uh let's run that again all right so another seg fault and all right so let me check what we're doing with our memory locations here one thing i can do is eval log and i'm going to write out the location of this value let me just delete that um all right so copying to and then new value copying integer two let's just write that out release just quickly see what's going on there did i compile it with dash g uh i don't know i need to check the make files because c make generated all that ah eval log okay it needs to be once again a pointer so let me just pull that extra value i might need to debug that a little bit more we're going to make progress on this faster i think all right so eval log it didn't even get to that point why not oh i wonder if this didn't even get hit that's very likely actually um so i'm not even returning anything from this function so let me just return null to do uh we probably should never get here all right so let me write out a log statement for um for this and i'm going to write it out at the location of the value cursor the current location of the value cursor so we know where we're at when dealing with values and um let's see eval ing or let's say eval expression kind and we're going to write this part out exper kind okay so we're evaling expression kind one which is oh it's a list okay that makes sense so so we're at the point now where we need to unwrap that top level list so hmm [Music] maybe what i need to do is do that unwrapping in the helper function for the tests code so the eval function here we're going to unwrap it so we have the result unwrap the first um expression in the result so is it logging it to standard error uh no right now it's everything's being logged to standard out and i'm just getting the output basically i can log to any file or uh file descriptor unwrap the first expression in the l and the results so let me think about this for a second we have a result here expert list well we're assuming that we're getting a result back from that i guess that it's a pretty safe assumption because we are creating that top level list so this returns an expert list so we can we can depend on that uh result uh items zero i'm pretty sure i can just roll with that because we just want the first one let me jump though to the definition to make sure that i'm not doing the wrong thing okay items zero what doesn't this not like oh unused fine whatever so i can pass that directly in and i think i'm gonna have to yeah i'm gonna do an ampersand on that all right let's see if that works go back to the compilation buffer okay so now we got a little bit different result which is good uh alex says if the logs don't flush and hit a seg fault you might not see them yep that is a problem sometimes and i think in my log function i'm flushing so every time i write a log log line i'm flushing it just to make sure okay so um i need to put in a new line in one of those lines inside of script.c let's see let's see where was i yeah this i'm using a mouse i should really be using avi for these kind of point-and-click type of movements all right cool let's run it again just get more clarity okay eval expression kind four and uh in the list of expressions i really need a way to translate those back to the name of the kind that would require some kind of function probably uh go to internal expression kind zero one two three four it's a value kind which is what we expect that's great uh next value requested current kind is 2079 of course so do i need that yeah uh-huh unhandled unhandled value type so at least for the first one i need to also initialize it here value cursor uh current and then for the current one i need to set the kind to zero just so that we don't end up in any problems now huh i don't have to call this right now actually i could probably use the current location and this is sort of a a pattern that's slightly inconsistent between um tokens and the tokenizer and the parser so for now hmm so let's look at the the way we call next functions elsewhere because i want to be somewhat consistent with one of the approaches so here we're calling next after uh setting the value for the tokenizer okay i probably want to switch that to the other approach but it does require changes to the cursor struct we need not only the current we also need the um the starting point all right i'll i'll fix all that later i want to get this going as fast as possible so let's just let's just do this first um what we'll do is follow a strategy like i was doing before and um so we're logging we're copying the memory we want to copy it to so newton instead of new value we're going to say value integer new value equals uh value cursor current and we're going to just cast that basically to the value integer type so this is what i was saying before with how i'm basically just using a memory location and just putting information there i'm not trying to allocate anything specifically i'm just finding a location in memory i'm just dropping the information and i have a way to go back and iterate over that again when i need to so what i'm doing here is i'm using the current cursor location which should be placed at a location where i can put something which the more i think about it the more i think this is the wrong order to do it but we'll we'll get that to that another time and then i am copying the value from the expression that i'm evaluating into this new location in memory and then i'm moving that cursor to the next location so you can see why i'm sort of reticent about this pattern uh prepare the next cursor location okay so now we're returning the value and i think this should actually work now okay so two failed at least we got uh something running expected integer negative 15. did i actually say that let's go to teslang 311 is what it's supposed to be and i think i'm putting this in the wrong order aren't i oh value value integer it wants a value integer oh okay i'm pulling the wrong thing here so in fact it already expects to see a value integer this macro that i defined so it wants a value oh so i'm just passing the the result directly to that okay cool so let's just do this value and 311 and now let's see if that works i think it will work nope wrong all right so what's wrong with this got zero zero is kind of interesting because it's wrong but it's very specifically wrong why is it zero um we have successfully validated that it's an integer um let me write out the thing myself and just make to make sure i'm just going to use a plain old printf uh value is put a new line in and then uh i'll cast that to value integer pointer oops i managed to turn off my audio that was fun let me know if you have any trouble hearing anything in the stream double checking good so value um value i think let's just write it out see what we get value is two that's not right uh let's see i wonder if i'm copying it wrong because two i wanna say 2 is actually 012. that's not even right [Music] so when we copy it all right we're turning it into a value integer we're treating the memory location as if it's a value integer and we're copying the current value into it um i have one thing i haven't done yet is look at the actual value that's being stored here so if i were to do value integer uh old value once again my finger did not type that g equals we will copy this little bit of code here i probably need to put an ampersand in front of that we're going to find out real fast whether this is right or not um and then i will use uh x uh yeah 311 is being converted from using a to i for sure i'm gonna put in the uh old oh wait a second what is that yeah okay old value value and then i'm going to put this as a d so we can actually read the integer value uh copying integer two okay so something is wrong with how i'm grabbing this memory location so let's think about this for a second we're getting uh an extra value oh wait a second hold on this is probably due to that thing i was talking about before let me just go and check this is definitely because of what i did before right here i'm not setting the value for one thing and the reason why is because it caused some very freaky behavior in one of the other tests let's see if we even get that far yeah okay so let's let's just focus on the one test for right now and then we'll turn the rest back on whenever i fix all this other crap the value member is the address it doesn't contain an address so it um we're actually oh let me show you so with this struct uh we have this expert header which is not a pointer it's actually a value i suppose you could say and also value header here also is a value and the reason why is because we want an actual location for this thing to be defined in the memory layout of this struct so i can get the pointer to this location and then write the value data to it it's a little bit weird that's the reason why i left a comment here saying don't trust the size of this field because where i'm actually going to overwrite the field to be a specific value type which i don't know if that's a good practice and c but that's something you can do and it's what i'm doing to make this work so now if i turn off all those other tests run the code um one pass one failed expected integer 311 got zero which is very strange um eval says copying integer 311 so that part is right hey astraz i've basically written a new programming language for uh the project because i was unsure what to do about the uh the scheme implementations okay so uh that's amore says uh maybe unions are good for that um the reason why i can't use unions is because uh the contents of certain subtypes are the the potential values for them like are strings where there's there's no predefined length of what the string could be so you can't use a union for that because unions need to know what the actual sizes of all structs are that could be in the same place because it just allocates the amount of space for the largest struct in the set of unions or the in the the union list so can't really uh use a union for this unfortunately which is why i'm using a more uh complicated approach but it does work okay back to what i was looking at before we did see that it has 311 as the integer next value requested current kind one that's right so it must be to do with how i'm copying that location so i'm i'm pulling that location of the value as a pointer i'm referencing the the pointer the value it's writing out the right value but then whenever it gets down to the test code it does not seem to have the right value let's set it directly let's maybe mem copy is not doing what i'm expecting it to do so how about we do this instead we're going to write it out directly we're going to say new value kind equals value kind integer new value oh wait hold on that's not right it needs to be header i'm not getting as quick of a feedback loop from eglot tagged union with pointer in direction uh well if you can do pointer interaction but yeah pointer interaction requires you have an an address some other location in memory and i'm trying to store all this data sequentially in one strip of of memory so just allocate it all in one line and i don't want to have to allocate pieces of that and put them somewhere else on the heap basically mainly for performance obviously performance is not totally necessary here but as i mentioned earlier in the stream i'm doing this because i kind of want to learn how to do this effectively um all right so header and then value equals old value value let's see if this works all right still it's very strange to be honest something else is wrong for sure so i'm returning the value here trying to think what else this could be so if we look at the internal headers uh expert value is an expression header oh hold on wait a second yeah yeah that's right expression header and then value header value header is really just a kind okay this is very strange to me why these values aren't coming through um expected integer got zero did i do something wrong somewhere else in the test code i wonder value is oh no right here in my face value is 311. so the test code is not validating it correctly so am i see cast well hold on why are we doing that uh okay okay that's why i'm using assertant value i wonder if that's part of what i was having no no i can't be i need to change what i'm doing here i need to change that macro to not uh pull the memory reference of value which is going to break my other test but i'll go fix them alright so we'll do that and then hopefully that should fix this nope can i convert to a pointer type tesling value itself is not a pointer that's why oh okay for all these for sure i guess the cheapest thing i could do is just de-reference this value but that feels really nasty all right i'll do that for now just to make this work in fact let's just undo undo undo then jump back and then dereference okay now it passes so um let me leave a little note for myself here that i shouldn't have to do that to do um i shouldn't have to de-reference this rethink the macro all right so we have our first eval uh working for just a simple value type which is great that means that we at least can eval something that came from uh normal code so for those of you who weren't here before so far i've written a tokenizer and a parser tokenizer takes just the input stream from a file and decides what type of syntax element the input is and then the parser then interprets that into um expressions and in this case it's just a plain value expression for an integer so it's able to return a an integer value back whenever running eval so this is sort of the foundation for doing higher level types of eval so we could try to do the same thing for a string as well if we wanted to just making sure that the different types work correctly so we're going to eval a string and then put a string in there we're going to have to use quotation marks because strings in this language have quotation marks just like you know other languages generally and we're going to say that this is a string kind we're not going to print this out anymore we don't really need to um i'll leave that out of this i think i have an assert string all right i search string and i'm pulling it up manually which is probably the better approach all right start string and usually another thing that it's a little bit inconsistent i'm putting um the expected value first so let's say flux harmonic should be expected value and here i'm going to pull out why is it doing that i don't want to see that um value string stop all right string right i think that's right let's go look at value string really quickly uh yes string and then i need to go back to script.c and add a case uh hold on a second so in this um check i need to have another function that can uh basically copy a value out based on the the kind of value so probably oh actually i wonder if i should try mem copy again let's do that click compile yeah okay still works good so mem copy was not the problem after all um and we can go add this function into the test suite just so that it will run and we'll see that it should fail yeah even to the extent that it uh oh what happens man for some reason it keeps eating the thing at the bottom okay cool so not getting what we expect but that's that's expected because i need to go fix that function so let's go back into script.c um so value header flux script ev uh no fluxscript value copy that's what we're gonna do here and we're gonna pass in a value header value um value cursor as well because we need to be able to move that cursor for value cursor and then we're going to switch on the type of value that it is so what we can do is return flux script value copy and i think it's what expression is a yeah we're gonna copy the expert value exper value i probably need a handle for that okay then i'll take this code actually the whole thing let's do this let's just take all that and put it into this function we also need a switch on the uh value kind and uh let's see case whoops case value kind whoa wrong value value value kind syntax you can see that my brain is starting to lose um clarity as time goes on all right so some of the stuff we can simplify because we don't need to do all the casting anymore new value is value cursor current okay that's fine then old value we're just going to uh cast our old values value so just plain old value so let's just say value that's the only place where it mattered value value so we have a new value [Music] that is bogus code we're gonna put a value here i think all right and then mem copy and we need to change this also we've already pulled that out so value here okay that looks right so um that works for integers what about strings case value uh kind string now this is a little more complicated because we need to um do a little bit more work to calculate the size uh also maybe what i should do here yeah let's let's do this just to simplify the pattern a little bit um i'm going to say value header new value equals null hopefully this is not gonna be a problem but we'll fix it later if it is um return new value and we're gonna leave both these things here and uh let's just put a break there we're not gonna return this we're just to let new value get set so new value is um the location of the current in fact that really applies for everything so we could probably just uh do this all right so let's do that so the way that we copy the memory is the thing that actually matters and then at the end these two things now should not be uh annoyed okay so copy our eval statement copy the mem copy as well uh here we need value string plus um let's see i need to temporarily have a handle on well let's just cast it so value string value length so we need to know the length of string um where else am i doing this so length so it's all plus ones which because we need to copy the null terminator as well alright so value string plus the length of the string so this is necessary because of the way i'm storing this stuff in memory the [Music] specific instance of a value string struct can have a string of an arbitrary length but we know what the length is because we've already parsed it we have the linked information but when we try to handle this object or this destructive memory we have to be aware of its specific size whenever we're doing things like copying it or iterating over it to get to the next item in the vector so right here we should have what we need copying string we're going put uh some little slashes above that all right um value value i wonder if that's actually gonna work probably it's fine no value string probably has string right okay string which will require me to actually cast this which is fine i use string star value okay now i think i'm good this should work let me go back and check my test to make sure i have this here uh eval string is good all right let's run it something it didn't like i'm missing a semicolon in uh script.c at line 570. yes uh yeah i need to finish that flex value copy i think that's good enough another problem too few arguments i think i need to have that value cursor as well and the value cursor is a pointer so i can just pass it directly value cursor uh last problem hopefully value header oh yeah okay fair enough uh 542 so i need to cast that to um value integer value integer star put that inside and uh that should be enough all right so i think that both tests are passing now which is great that means that we have excuse me basic value value evaluation is working um so now we're getting into the good stuff we have an hour left the thing that we need to do next is try to evaluate a function so to evaluate a function we need to have a function to be able to call i'm looking at ashford's comment it might be easier to have uh add more types with the respective uh copy functions and have them take the correct pointer type so you might be right about that yeah uh that's something i can definitely add later there's a lot of cleanup i want to do so i'm i'm very uh interested in hearing uh people's ideas on how to clean up the code a little bit because i'm i'm basically making a straight line for getting this working so obviously i'm not writing the cleanest code just yet okay so what we need is a way to register a function to be called so we're going to go into testlink and we're going to [Music] eval basic call and what we're gonna do is have a function let's just call it uh do you wanna call it add might be the best way to do this right now i mean we'll we'll see what happens okay i have to add add a function called add which is not something that would normally have but honestly it doesn't doesn't hurt to have math functions in this so i could actually do that and i would prefer to use plus sign i don't know if it will parse this but we're going to find out now let's go with add first just just for the sake of our own sanity um so we can't actually evaluate a call yet what we what we have to do in fact let me add this test so we can um see if it fails all right so we're going to do this let's just run the code um man it every time it's deleting that last thing all right segmentation fault that seems like what i expect so now we're gonna go script sc and we need to add a case in eval expression for uh expression kind list because any list that we see that we try to evaluate we're going to treat it as if it's a call expression and the only way you can create a list in this code is to call the list function and then the rest of the parameters get turned into a list value in the end at least that's what i have in mind currently all right so we're not going to do a value copy here directly we're going to try to actually invoke something i'm going to return null for now just because you know we don't know yet what we're going to do for that but to actually call we're going to have to pull off the first thing at the beginning of the list so um first of all grab the symbol at the beginning of the list and we're gonna have to throw an error if there isn't one like if you have a list that has an integer at the beginning we're gonna have to um give an error somehow i haven't gotten to error handling yet but we'll get there at some point so we have a list type all we really need to do is say um expert list star expert items zero and this is going to be an expert header uh let's see call nah maybe i don't know let's see call symbol that needs to be a pointer and uh to do check that as a symbol i'm not going to do that just yet what i'll do is just treat it as if it's a symbol just to start with and we can do that so if call that's what we'll do we'll we'll just panic for now that's that's good enough call symbol kind equals expert kind symbol um else panic panic is great you just say uh i want to bail out and write an error message um so for this one we're going to say call expression has a expression of type of kind d in first position and we can say call symbol kind here okay that's good enough just to bail out if we get something unexpected in this case we can then cast this to expert symbol star call symbol and what i really want to do is grab the name off of the symbol so we're getting the name um let's see care star symbol name equals all right so now we have the name and we could look that up in a symbol table so that requires us to have a couple things first of all we need a symbol table and we also need a way to register a function pointer to a symbol so that sort of indicates if we want to have a uniform way to evaluate a symbol and get a uh excuse me um a value back then probably what we need to do is um have a value type for function pointers so we can go back to flux internal flex internal header now the thing is if we do that we're gonna have to have a pretty vague um signature for that right now it may be that we change that we have a a consistent signature for all function pointers but we may have to do some different types of function registrations later depending on what type of inputs they take thanks ashras um so we'll just go with a simple thing for now and then we'll flush it out later as we always do okay so we're gonna have a type def struct um value function pointer because it's not really a value function because it's not a function being stored as the value it's just a pointer i mean i guess technically speaking i mean we could just call it a value pointer but let's be a little bit more specific for now value header header and then what do we want our binding functions to look like what kind of signature do we want them to have so we could go let's just go right up here so when you call a function in the end you kind of want it to return a value i think value header um so flux script funk add so now if we have this function what parameters does it need to take i think that it probably needs to take uh a list of some sort a list of expressions i mean for now we're just going to have all these functions take that remaining list of expressions and then handle them directly later we can come up with some abstractions so that we don't have to write the bindings manually but for now i think that the best way to do it is to manually write the bindings and have these functions wrap other functions and sort of translate between them effectively so the add function would take some kind of list of expressions it can't take the extra list but what it could do is take a cursor to use to iterate over the list so um yeah that actually sounds like a good idea so expert list cursor list cursor this needs to be a pointer now the important part here is that whenever you iterate over this cursor it should only iterate for the number of parameters remaining in a given expression and that's something i don't think i've implemented yet but we can get to that in a second so the goal here would be to iterate over the remaining expressions evaluate them and do some work so in the end what we would want to do is uh yeah we also need the value cursor too because we need a way to store the result and kind of makes me think i will have to think about a better abstraction for this at some point but we'll like i said we'll go with this for now uh value cursor value cursor okay so um yeah we need a way that's the problem we're gonna have to have a way i think the cursor is going to have to have um extents built into it so that we can configure a cursor to only go so far and then stop even though the memory that surrounds it has a lot more stuff it can go to we just want it to be able to stop so we are basically just you know re-implementing uh iterators here that's great okay so while let's just write out what this code would be um list cursor let me go to the actual list cursor implementation uh index it would be nice let's see unsigned int index if you set it at a specific location and say how many indices it has then the code can handle this and the index can start at zero it's all just a matter of the context where you set it up so um let's go back one step higher for a second and look at how we would invoke that so um at this location we would want to have an expert list cursor uh arg cursor our cursor would then so um yeah i think that's what i want to do okay cool our cursor i think list what just happened okay this song changed okay for a second i thought uh my streaming setup died all right so list equals we're gonna grab oh wait i got a little bit of syntax issue there you know i could just set this up uh directly in fact at the very beginning so here i'm going to say let's paste that in we're going to take this move it up here we're going to say the list is items um arg cursor index is zero and then we're going to iterate it flux script um what is it exper come on now exper something here is very slow i think that eglot is not coping with ccls very well expert what is it parse list next where come on why is it not updating as i'm typing all right i think i'm going to have to pull in lsp mode here because this is going to drive me nuts um that's not the right function so script parse list okay so let's find all the next functions next value next token next expression list next of course all right so go back where we were exper list next all right and then i think we just drop in our cursor here if i'm not mistaken yes it looks like it come on expert list next i also have the cursor in it there which is nice cool okay that's definitely what i want but i also need to take a look at um so how do i indicate that there's no further iteration possible returning null is such a dirty thing to do i really don't want to do that and i can't do that currently uh short mark the signature was shown in the minibuffer yeah it was shown for like a split second i think but i didn't trust it um because things are not working like i expected to with iglot at the moment because i've been working with lsp mode on another computer for the last few days and uh like i've gotten used to what it does and this is kind of freaking me out a little bit all right so list uh cursor init is that used somewhere let me just drop this down here because we'll put it next to the function that uses it okay let's speed things up a little bit so we have list and we have list cursor um cool i could use that but first i need to go and find a way to set up a maximum extent and how do i communicate it back i guess what i have to do is have another function that's able to check it so maybe that's the way to do it um and then maybe panic or something if someone calls it when they're not supposed to i guess that's also a good way to know that uh that someone's doing something wrong in the code so expression list finished p can you do question marks no you definitely can't do question marks and see function names um i guess you could say has next has next okay how about that so someone can call this function i know the names are getting a little bit ridiculous but i'm trying to have sort of like a name spacing approach to the function names so if i go to expressionlist cursor unsigned int uh count count is a little bit vague though length maybe is a little bit more appropriate so the semantics here would be that in some cases we don't know what the length is yet uh oshawa says what kinds you have to find is there a nil void unit type or similar that's a great question that is a great question that could be useful i don't know though if if we would want to actually legitimately return it but i might need to have some way to um have like a nil of some sort all right so for now here's what we're gonna do we're gonna go back here we also need to go to init and what i'm gonna do in init is list cursor uh length equals negative one by default we don't know how long the list is now that's sort of the pr hold on we do know how long the list is right because the list itself if it's an expert list expert list it has a length okay so i already have that so the sub list should have its own link i think i'm updating that correctly so let's just think about that for a second i don't need to add anything new to the cursor the cursor can just use the information from the list itself so long as we do actually keep a pointer to a list and if i go into the hmm interesting okay too many thoughts at the same time so we're gonna do this if uh list cursor index is going to be zero based so if it's equal to the let's see so let's say we have five items link this five the index is currently four we can't yeah we're gonna just check if it's um greater than or equal to the list cursor list dot length yeah dot length no minus one um and i probably should return a number for this can i just return like a care i just want a very short number care is probably good enough i know there's there's like a standard bool or something oshawa says something's akin to air and rust or either in haskell yeah probably i would need to go there but i have to allocate that somewhere well no no no i don't do that because i can return it as a value type and just it will copy from the stack so maybe i could do that that's a good idea let's i'll think about that after we've gotten past the stuff i want to get something done before we um get out of here today i think we're gonna get function registration potentially done or at least working to some degree let's just see if we can hard code it maybe if we have to all right so um in this case well actually i could just return the result of this right i don't know what what value type it expects it to be so list cursor index if list cursor index okay it has to be less than uh list length minus one so i can check this function to see if we can iterate further so uh in this list next function i'm not gonna use it but i will use it here where is it in this funk ad so while um flux script expression list has next in list cursor okay so while there is something left then we can try to i guess that's a problem what do we do with the intermediate values i guess all right so this is add let's just assume that we're adding integers okay and let's say that for now the implementation only accepts two so uh int numbs two and i equals zero yeah this is just not this is very simple implementation okay we're gonna we're gonna polish this as we go so um in fact we could even try to do alex i think you still have some line width if you want to call a longer name function yeah i think you're being sarcastic we only want to grab two so in this case we could use uh four i equals zero let me just get rid of that i knew it dude uh i is less than two i mean why am i even looping in that case well it makes it a little bit cleaner okay so i plus plus and then get rid of that um we're not going to do any checking we're just going to try to straight up implement this as fast as possible so uh nums i equals at first we need to get the value out so we're going to use uh flux script expression list next i really wish i had a better completion going on here uh i think that's still gun says i'm glad he's not using java yet java would be way way worse there is a function called that isn't there flux script expression list next what is your problem list cursor okay so whoa where was i right there yeah whatever okay so list cursor and uh we've we've moved it to the next huh yeah i think we should assume that actually so we're gonna move it to the next and then we're gonna grab the value out so here to do check that the value is actually oh no we need to eval that first in fact huh yes exactly asuraz it would be a lot of fun writing all that nested uh builder code okay so we need to eval okay uh expression header uh result because these are all expressions that are here so we're going to eval every one of them we're going to call flux script eval expression so this is where sort of the uh nested evaluation of the entire tree starts to come into play you know like these functions can cause evaluation of things below them so flux script eval expression we're going to use the list cursor current and then [Music] our value cursor we need the value cursor because we need to know where to store that value in memory okay so when we evaluate this it doesn't give us that it gives us a value header so we get a value out of this and um did i delete that comment to do verify that it's an integer okay so now let's just assume which is a great thing to do in c value integer and i'm missing the g again fantastic value uh value all right cool so that should give us two values from the inputs we're not doing any error checking there's obviously things we need to do to verify that we have the right parameters but that's good enough for a basic add function and now um well actually i don't need to do that i just need a sum right so int sum equals zero and then uh sum plus equals and then honestly we could just have any number of inputs at that point we could just take whatever number of uh integers were passed in so that's a improvement that can be made to do don't limit the number of inputs eval caching to um interesting point that could be useful yeah i haven't thought about that uh ashrae says there may need to be some kind of eve out caching all right so now we've got the sum we can return uh we can't return the sum directly we need to actually get a value that we can use and this really is a place where it becomes obvious to me that first we need to ask for a new value uh slot i guess you could call it honestly it's fine to do i don't know i have to change the shape of the algo a little bit for that okay we're gonna assume to do don't assume that we already know the slot all right so for a value cursor value integer value value the reason why i don't go make some of these fixed fixes that i need to make is because i don't want to like drag things down i'm trying to get function evaluation working before the end of the stream so we're sort of under the gun a little bit we've got 30 minutes left in fact we got less than 30 minutes because i have a meeting at the go-to literally as soon as this remains which is great okay so um we're casting value integer have value headers and output argument i had thought about that good point um value integer yeah i guess you're right you could well okay we'll think about that okay so value equals sum yeah we also have to initialize this which really i need i need some some functions to to make it easy to set these values in new slots for sure um header.kind equals value integer i should not have to be doing this all over the place uh value kind integer so i should be keeping a list of improvements that need to be made and then i will return yeah i should be doing this value integer result equals uh no not value value cursor so that was already a bug value cursor current okay and then result header.kind and then result that value so let's delete uh result value sum okay all right then we can uh return the result okay so now we've written a function that in theory should work uh we haven't written any logs yet but if we have problems we will uh write out some log statements okay so now we have a flux script funk add and for invoking a general thing what we're gonna have to do is um look up symbol make sure it's a function pointer and then invoke it with the with a cursor for the uh expert list okay so that actually makes sense ah let's see that's a good point uh ashford is asking if i'm returning something that's bogus which is actually true i should probably yeah so we're gonna do this uh value header the reason why i can't return result specifically is because result is a struct that technically has a value header as the first uh field in the struct even though they both start at the same memory location you have to be a little bit more specific about which uh face i guess you could call it of that struck that you're referring to okay so symbol table we need another block of memory which is another case where we need all these stupid functions for iterating over the block of memory so i'm definitely going to put together that vector implementation that i had started on and then refactor all this code to use that vector implementation because i i'm tired of writing the same thing over and over again let's see let's cheat cheating's great that's how i made it through college just joking all right so what we'll do is we're just going to look specifically for the add function and we're going to invoke it directly to simulate what it would be like if we did have a uh symbol table since we don't have one yet and it's probably not the right time to implement that i can do that off stream so here we go um if where is where do i have the name of the symbol right there so if a string compare is that right symbol name uh add equals zero which means it's the same then we can call flux this is effectively what we'd be doing with a symbol table anyway flux script funk add where is it flux script funk add and the input is the list cursor and the value cursor okay so uh list cursor value cursor i think one of those is going to need to be oh which one we need the arg cursor for this purpose because what we wanted is for it to be restricted to the arguments to this function or at least the the list that comprises the call expression okay and then the result of that is a value that can be returned directly so let's return whatever this gives us all right symbol name uh symbol name add okay let's see what this does let's go back to the test script make sure that we have something there that should work okay let's try to run it see what happens okay compiler error um 5.99 in scripture c okay uh oh yeah these are not um pointers they are just regular accessors and then this one is uh 605 expression list incompatible types and initializing type expression header call symbol uh why oh okay fine so expression header does that make you happy and then i cannot convert a shouldn't the r cursor start on index one then um yeah we'll get to that you're right oh no actually um this next is actually going to take care of that so it's going to increment the index let's double check that it does that we're actually depending on this index haha this code does not increment the index let me see how do i know they're all breaking okay so let's just say list lust list cursor index plus plus okay increment the index because otherwise we're gonna have a lot of trouble okay let's go back where were we um we're trying to appease the compiler here incompatible type for argument one of uh funk add this needs to be an ampersand and the previous one expression header can i convert to a pointer type what do you want me to give you an uh okay okay fine let's do that come on now value is undeclared in 572. um yeah you're right so result result is what i'm looking for all right segmentation fault our favorite so let's think about why this is happening so we're we're doing some tokenization stuff here interesting that it keeps going gdb uh yeah you're right let's let's actually do that instead of trying to do psychic debugging i should just run gdb when this happens run okay so oh okay interesting so basically flux script token next why can we get a back trace okay cool parse okay interesting we got some issue oh is this the same problem we were having before i had a freaky issue earlier with integer value types somehow overwriting stuff in another location in memory and i don't know what i did i still haven't figured it out yet uh let's see eval script parse parse list parsed list that's the nested list token next line 89 and script.c so at this point segmentation fault while pulling probably this current kind i would think how is the cursor busted at this point it must be that same problem it's overriding memory so let's go look at my um yeah i'll tell you how i'm debugging these things let's go back to the compiler output here so uh all of a sudden we've got so so these these hex values at the very very beginning of the lines are memory addresses of the things being worked on you can see when we get to right here we've asked for a next token and then we've got this next value which is completely out of left field this rom we're getting a segmentation fault because it's reaching for something that's completely out of the scope of the memory block we have so um somehow the token list is being corrupted and i'm not i that's that's where i was where i left off earlier this afternoon whenever i was looking into this somehow setting a value for a value integer magically corrupts the original token i don't know what i did there's some pointer stuff i'm doing that probably caused it so i think we're seeing a manifest manifestation of that right now um okay so ashrae says f sanitized address gives great additional information is that a gdb parameter all right all right so we got let's say about 10 minutes um we could try to fix that problem the alternative well okay cheating doesn't really help if if we just have a function that takes no parameters and we just call it it doesn't really help i mean that would show that we could call something but we don't have a symbol table yet so we're not validating anything other than just the fact that uh string compare works so um it's better to try to fix this bug i think in fact let's go look uh let's see enable special form registration haven't done that yet haven't done that yet basic evaluation of expressions okay we did that i think we sort of did that uh okay there's a compiler edition that needs to be turned on okay great um all right so um let's really quickly try to figure out why this is happening so the value integer code right here somehow this is setting the token kind something i'm doing here is bad inspect the current state within the back trace not sure that will bring something yeah it's it's right here where something is going wrong i mean maybe i can set a break point there with gdb and uh try to inspect things to see how it could be happening the interesting parts are that we have a couple of symbols here sorry we have a symbol and two integers that are being parsed we have add one two we can see that the tokenizer is picking them up and it's putting the information about those tokens in memory at these locations that you're seeing here somehow i think it's always the last integer that gets uh overwritten so next token is three let me go check tesla let me turn off every other test and see if this is not some other weird issue yeah same problem so we finished tokenization um i think the parsing is happening here and we don't even get okay that was part of the problem too we never even get to the final parenthesis that closes out the expression because we've already destroyed the final uh tokens in the list so we never even find the last uh closing paren which is a very strange thing somewhere in all this code i'm doing something really bad and watch this somehow it manifests by doing that like if i if i take this line of code out all of a sudden well okay it's still seg faults so maybe this is a different issue maybe it's not the same thing because the output still seems the same let me turn back on the parser logs really quickly and see maybe if anything else jumps out let me uh uncomment that and then comment that out a lot more output now all right so the lines with parser from the parser uh let's see parser got token5 that's an integer setting integer one okay so we're putting that into an expression setting integer two okay so the parser found the one and the two is putting them into that expression list and then somehow next token is three which makes no sense because it's supposed to be one which is the closing parenthesis so somehow uh [Music] something got overwritten let me think about how this could go somewhere i'm using a pointer to set a locat set something in memory actually what seems to be happening very specifically as the kind of a token is changing so maybe if i can see where kind equals is coming up that's all in tokenizer that all makes sense all right here how about excuse me how about we do this dot kind or no slash dot kind all right cool consult line is easier uh alex gets popcorn out all right so parenheader.kind that's the closed paren that's still token code that's a list kind symbol symbol in this case is an expression symbol and we're not setting it to yeah i don't think it's an old termination shouldn't be no determination i don't know though keyword header.kind okay so keyword here is an expression keyword is coming from the expression list um all right integer header kind everything there looks good and that is setting the value that should be fine and we're not dealing with any strings here so none of that should actually have an effect on it uh that's another thing string token okay we're already back to the beginning again you can watch the state of the token but i'm not sure how to set up within gdb terminal yeah there's a way to set watch variables for sure but oh and you can watch memory locations too can't you oh oh you know that would be cool if i can set a watch on the specific memory location and see when it gets set maybe i can figure out when this happens uh okay five minutes can i figure out how to do this let's see um let's pull up a firefox window how about this uh gdb uh breakpoint memory address can i set a breakpoint on a memory access in gdb come on all right set a breakpoint at address address you can use this to set breakpoints in parts of your program which do not have debugging information or source files okay so um what i would need to do is set a breakpoint before a piece of the code and i guess what is the format that it needs that information in i can set that um damn we're gonna run out of time i'm gonna use a gdb integration in emacs to make this a little bit easier let's actually try that's a code address not a memory address is that right watch points ah watch you're right examining memory [Music] watch okay uh gdb watch points this is sourceware documentation what is that give me the actual gdb um so address an address cast to an appropriate data type watch a four byte region at the specified address that's cool all right all right that's gonna be super powerful for figuring this stuff out if you all right watch memory address or expression okay cool i should be able to just type it in 0x and then whatever i'm writing out my own code because that is writing out the memory location so let's just give it a shot really quickly um gdb the good gdb whatever we're going to run this the same way as we do everything else in fact let me pop back to whoops yeah that folder we're gonna run uh geeks shell pure dash m manifest.scm dash gdb full name uh slash build slash run tests all right and then we're gonna use run here okay so now we've got all the same output i want to go into script.c and at the end of tokenization tokenization uh let's see tokenization start where is the end way way way down here all right right here so um yeah let's see uh good break point set a break coin at the current line uh let's see i think it did it so now if i go into the gdb or good shell i can type run again starting from beginning yes i hit the break point so now if i look at let me drop this down a little bit so the integer maybe it's a closed paren that we're having problems with and the closed parent is this memory location so if i copy that uh was it watch ox cannot watch constant value why not because it needs a width all right let me uh try copying that at the beginning because i think it is an unsigned integer that i'm uh i'm dealing with there hardware watchpoint let's see what that does all right so um how do i do continue just continue where's it run where is this happening i think i must hit the point flux script parse list um token cursor list cursor old value new value oh that's it that's what's happening i think right hold on a second at c20 um [Music] hold on a second whoa but the wrong thing no no right 407 c20 407 c2 why does i have that i don't know why i have that same uh line in there alright so c20 next token is zero close paren and close paren is supposed to be that should be one ah um it's all right i guess it is right oh i'm not sure parse list starting at 407 c20 which doesn't make any damn sense well anyway this is definitely the way to figure this out just setting up a watch point in the memory location i need to be more specific about the location and make sure i'm doing it correctly but once i have that set up i should be able to figure out what part of the code is squashing that or stomping on that piece of memory and changing the value in a way that causes problems so that was that was cool thanks for the suggestion um uh let's see was it ashraz who said that but uh anyway i think we made some progress today um i definitely see a number of places where i need to go and clean up the code a bit and make it a little bit uh easier to maintain and easier to trace things that are going on i mean i've got a ton of logging and stuff in here already but some of these patterns are repeating a lot of places and i don't have functions that um abstract some of that out so i really need to take care of that but as it stands as a result of five days of coding on this i think that we were at a very good position to have something working on tuesday where i've got a lot of these issues figured out i may have already implemented stuff like symbol tables and we can actually start using the language now the question is should i go ahead and implement that stuff or is it going to be interesting enough to watch me build the symbol table stuff on stream should i just clean up the code and then wait until tuesday maybe and finish up this stuff it might be more fun to do it on the stream because i think it was kind of fun today we only had two hours because i was talking for one hour at the beginning about the uh design of the language so i'll think about it i might actually wait and do some of the stuff the next stream because we might get to the point where it all works and then we can start you know converting the old code over and then by thursday we'll be using our own language to power uh everything that's going on uh i think it'll be fun let's do that okay so uh let me just write down the stuff in the notes for next time so next steps um see clean up some of the repeated patterns in the code use gdb watch to figure out why uh token kinds are being overwritten um and then we will get back to the rest of this stuff the next time so yeah i don't know i'm pretty happy with the result of things i'm i'm actually very happy i decided to do this because i think it's going to be more interesting having our own language and building up the tooling around it than it would be to just use something else because we have full control like i said before so that's going to be a lot of fun to me so uh thank you everybody who is here today watching this for three hours and giving really good suggestions and feedback i really appreciate all of you who are actually engaged and in helping out this it means a lot to me i really appreciate it and uh yeah we'll see you on tuesday i'm gonna commit the show notes and the code that we worked on uh probably not immediately because i have stuff going on but uh uh very soon i'll be uh posting that up so you can take a look at it if you want to just keep in mind that it's in a different branch it's in the scripting branch on the flux compose repo so uh thank you all for being here uh and we'll see you well some of you i'll see you probably tomorrow because there'll be a system crappy stream but then the rest of you i'll see you on tuesday so anyway thanks a lot and keep it creative see you next time now i have to find my thanks for watching screen bye do you

In this stream, I'll talk about the scripting language I started writing for flux-compose this weekend! We'll discuss the design and goals for the language and then hack on the interpreter to get us back to where we were with Guile Scheme. We may even create the language REPL live! SUPPORT MY WORK: 👍 https://fluxharmonic.com/how-to-help/ CHECK OUT THE CODE: https://github.com/FluxHarmonic/flux-compose SHOW NOTES: https://fluxharmonic.com/live-streams/2022-01-13/ MY CONFIGURATION: https://config.daviwil.com https://config.daviwil.com/emacs https://config.daviwil.com/systems (Guix) If you'd like to learn more about Emacs and Guix, check out my other YouTube channel System Crafters! https://youtube.com/SystemCrafters JOIN THE COMMUNITY: https://twitter.com/FluxHarmonic