AI-Powered Voice User Interfaces and Deepgram's API at Cloudflare Hackathon #0to1AI

if you're ready any know with cl your hands okay so let I'm I'm ready are you ready never never been more already yesterday we went to cler right yes CL office cler yeah cler uh office uh amazing office um big interesting space yeah uh very Google like I would say very yeah what what when you think about um City startup this is what you probably imagine yeah that they off is look like yes it's uh a spa a creative space uh a lot of interesting things like for example you could get a kombucha from a top right or like a uh how it's called like um lemon water right with all different flavors yeah different chemicals in your water yeah so it was interesting office and there was a hackaton uh it was real time voice and multimodel AI organized by the AI engineering Foundation uh by Sasha uh yeah pretty great experience uh there was 26 30 teams I think uh initially 26 uh submitted the the project but as we went the it increased to 32 or 31 so they were past the time I guess so A lot of people people uh were hacking till very end uh I've seen like some of the submissions were yeah submitted after the the deadline yeah long after the deadline long after deadline we had a chance to finish a bit earlier so maybe we could discuss what we uh what we built um so we've seen like a couple days ago there was this presentation by versel during Google uh Cloud next event yeah about jimy and I about Gemini and GMO uh CEO bcel was uh showing showcasing on this like a generative UI um tool or application where you could like uh pull from different sources uh react components right for example a list of flights or uh like essentially other uh like a think out of from internet you could buy you could imagine that you could buy books for using that and uh so that was the idea uh and when he was presenting that he was using the the regular mouse and keyboard mouse and keyboard regular LM interface and we decided that we could maybe improve a little bit that and we could switch the typing to to voice and why why voice is important I I mean why it's better with voice your opinion I'm not sure if it's better I it was interesting to to do it but I mean for me it probably would be easier to use Mouse anyway but for people with I don't know movement impairment or site impairment they it probably would be easier to use voice and right now similar tools exist but they only allow for very mechanical computer like navigation like you can tell the computer uh click the second button or I don't know click the link that's on the center of the screen but we went uh I think a few steps further because you could say to the computer book a flight at 2:40 p.m. and and it did that so natural language processing uh and natural language communication with the computer and it understood the the commands and uh yeah and it worked I think it went pretty well it was pretty interesting yeah um I think we had some troubles presenting the our like the final presentation was a bit hectic I would say uh we had some problems with the sound and uh so I was wondering if they really catch the the whole idea we we we had um but anyway yeah we we buil that and we got a huge from deepgram actually exactly we are using the API provided by deepgram it's an it's a company that's older than open AI actually as we learned and there was this person from deepgram uh Daman Murphy thanks for helping us yeah help us like a bunch of the but because we are dealing with this problem of uh delay or latency we are using uh like standard API we we had some experience with open API so we used the the same approach of dgram where we record a piece of well voice and then we submitt it to the after it's recorded we submitted to the to the API but it turned out it's it's the Laten is just too annoying the ux was not not perfect it's too much right yeah so he showed us a better way to do it to open a web socket connection and then we could stream the audio uh in real time without almost no delay like the delay was not not noticeable anymore and was processed so fast yeah I was impressed so because at the beginning I always was thinking that degram is just this like a um like a alternative to what open AI is doing uh but they have much more in terms of features yeah they obviously they do speech to text and text to speech yeah that's like core uh core feature but then they have a lot of things around that so for example we when we started hacking on this so we were like reading before the hackaton the night before we were reading preparing for the hack reading dogs and one problem we wanted to solve is this like uh interface inter voice interface where when you start talking and the llm responds you can like interrupt with your voice so like of human right what we are doing right now so I'm saying something you interrupt me and you stop and yeah and I stop usually exactly I stop and I'm just taking into account what you just said right yes uh so we wanted to reproduce that the same um let's say Behavior or the same situation with uh Ai and it's it's pretty complicated to build that right using the browser browser API or just the tools we we had and deepgram provides that out of the box almost almost yeah yeah close they're working on improving the as we learned the DX even even further but it's close to yeah close to out of the box so was pretty interesting they also do um things like sentiment analysis yeah intent analysis intent analysis so summarization exactly what else the like the topic they could exct topic yeah so that's uh that's pretty interesting and we just learned that almost at the end of the hack right all all about those all those features uh but anyway we we are thinking about integrating deepgram uh into one of our projects uh the one that we will be doing during the course one to zero AI there's there's one uh project about voice specifically and I think deep gr fits uh the scenario pretty well yeah I think it will be perfect and we'll be using web suckets for that because it's that that it's just imper uh how do you say it in English that the user experience is just it's hard to compare to to the rest API yeah yeah I'm not sure incomparable incomparable maybe yeah that could be a good word I had another word in mind but it's on the tip of my tongue so let's quickly discuss the the winners yeah the results of yesterday yesterday hackaton uh so there were like fre uh fre spots uh so the first maybe let's stop from the from the bottom the third place it was a podcast maybe let's talk about the prizes first okay yeah let's talk about the prices so the the thir price do you remember what was the thir price $600 yeah or or um what was it mini projector I think yeah exactly a Mini Pro so you could you could pick money or the the thing and the second price was 2,000 right do you remember the uh it was yeah it was the GPU uh Nvidia 490 yeah 490 very expensive very good yeah and and the first price was Apple Vision Pro yeah or $3500 yeah yeah and the winners so the the third place was a person uh it was a project called iPod but it's not the iPod like Apple's iPod but it was um a tool for podcasting yeah I podcast I podcast yeah and it was something that allowed you to be more efficient when consuming podcasts yeah I think it was just I don't really know what it does uh but the presentation was pretty good so yes I guess he he won because of the presentation also the was seemed useful I mean it generated text out of the podcast and then yeah you could probably built on top of that uh the second place was a project called wake me up right yeah but it's why was it called wake me up I don't get the name uh because it was they were showing during the demo they were showing this like a situation where you wake up and you want to be um and they were like showing you can discuss with your your like a the upcoming day with Avatar and in that case it was Taylor Swift oh right and she could like a wake you and tell you something nice to get you energized for the day yeah but did she I mean she was kind of mean on on the demo do you remember that yes she was asking question like are you really out of bed yeah so that that that did not work out let me just check should I go straight straight okay yeah yeah it's very complicated to drive here uh you can enable the auto drive and yeah maybe do that and yeah so Tesla decided to switch the lane and uh yeah so it was like a Psychotherapy kind of thingy by the llm and an interesting thing was that the image the Taylor Swift image or video Yeah was was generated was generated and moving right it like a some like a face of a person that it looked con with you it looked creepy it looks very creepy uh yeah it was interesting and the first prize yeah the first prize was Apple Vision Pro went to person who actually had Apple vision and used Apple Vision Pro to create the project yeah so what allegedly so what they did they um they had this like uh 3D space because she connected to uh the the monitor right using the Apple Vision Pro so we could see what she's seeing uh so she had this like a space 3D space with blocks like cubes and uh will Tesla change the lane okay it does and then she arranged the the cubes with different colors and she asked what what's the image about yeah and and it generated some random image unrelated to the cubes nobody knew what's going on and then she won yeah but but the image was 3D right I mean it was more like an object at first it was flat and then it generated a model out of it so so generation of the model out of the image was cool but I don't understand the relation to the Cubes at the beginning yeah I think the cubes were more like the yeah it was like a for me it looked like it's it's seat for the random number generator okay but it's probably not true I was I was thinking more that the cubes were showing po like a shape of of the maybe of the object that is going to be created um and she just took this like a 3D thing and and throw it uh into the audience it was pretty interesting yeah but it was it was pretty cool this person was I whenever I was watching her she was always wearing Apple Vision yeah at all time all the time all all the time so it was funny in a positive way I mean and uh no no I mean it's like the she was very into into this technology yeah futurist now I'm I'm told to this a true early adopter I would say and yeah that was the the hackaton the winners interesting experience um I think we can stop here yep [Music] [Music] [Music] [Music] [Applause] what we did yeah we are leaving the Meetup from that that happened in the GitHub office yeah this is the GitHub office pretty great office and there was amazing space there was a Meetup called unstructured Data so there were San Francisco there were free talks and we met Lori yeah Lori Vos amazing it's like the my hero figure so to say so yeah it's not a figure of speech so the the fre TS were amazing uh we didn't uh we don't want to stay for the networking because we are heading to another Meetup which is happening close CL by but we'll drive uh about three minutes maybe and it's about uh llm evaluations so I'll see you there see you there [Music]

Join 0to1AI 👉 https://www.0to1ai.com This episode covers the development challenges and solutions of integrating Deepgram's API for efficient voice processing, highlights from the hackathon projects emphasizing accessibility and user interaction, and a review of the winning projects. 0:00 Introduction & Overview of Cloudflare's Office 0:22 Tour of Cloudflare's Creative Office Space 0:54 Hackathon Overview: Real-Time Voice & Multimodal AI 1:44 Developing a Voice-Driven UI 3:10 Benefits of Voice Interfaces for Accessibility 4:03 Challenges and Solutions in Project Presentation 4:50 In-Depth Look at Deepgram's API and Real-Time Processing 6:52 Hackathon Wrap-Up: Features and Future Integrations 8:05 Discussing Hackathon Winners and Their Innovations 13:15 Recap of the Event at GitHub's Office and Upcoming Meetups Links: https://deepgram.com/ Follow us: https://www.linkedin.com/in/zaiste/ https://www.linkedin.com/in/mmiszczyszyn/ #VoiceAI #Hackathon #AIInnovation #TechTalks #deepgram