Beyond CloudEvents: Endpoints, Messages, Schemas – CNCF xRegistry Clemens Vasters | MQ Summit 2025

[music] [music] [applause] I'm very excited to be uh at this conference uh because it's uh the first time that uh so many people from so many competing companies uh in this space are kind of in the same room uh which is uh or or it's been a very very long time since that happened and usually there were lots of lawyers present. Let's put it this way. Um so I'm very happy about this. So I'm you you notice that even though I have the Microsoft.com name here um I didn't put the Microsoft logo on this because this really is a presentation about standards. I've been, as Charlotte said, representing Microsoft in Oasis on AMQP, also member of the MQTT TC over there. Um, I've been doing work in the IoT space in general um with OPCUA. Um, and I've been working on uh some CNCF projects. And so I want to go and especially present to you and give to your consideration a project that we've been working on with a few folks uh including Doug Davis who's also been in the um in this uh uh standardization space for a long time. He's been with IBM for I don't know 24 years. Uh then came to Microsoft for a while and I think he's now mostly baseball coach but still working pretty hard on our project. Um and then Claus Daisy who I want to highlight from SAP uh who's also been uh a partner in this uh in this effort. So um where do we come from? So seven eight years ago maybe we got together in on initiative from Google um initially to go and and define what is an event. uh which seems like an easy question to answer but it took us three years to kind of get to the point of you know defining what an event is and we came out effectively with an answer that has seven attributes seven attributes that define what an event is and then having these attributes bound to protocols and to uh encodings. Cloud events has meanwhile been uh fairly successful. It kind of finds its way into numerous CNCF projects. I see it in my daily work. Customers coming to us and say, "Hey, we standardized on cloud events." And I say, "Good choice." Um, so so it's uh it's gaining foothold and as it is with standards, right? It always takes years to kind of figure out whether something really sticks. But what cloud events is is a great way to so to express an event on the wire and also go and document those events on the wire. What we also had as part of this effort is a schema registry. So we have an schema registry uh API that we put into the cloud events project initially um because we had we have an attribute in the cloud event called data schema and data schema is supposed to point to a schema but you need to have a place to put that. So there's a schema registry and we also didn't like the fact that all the schema registries that were out there kind of had some weird license encumbered uh uh ways of you know not being free. So we said let's go and make a spec and then um and then put that spec out there and the cloud events project was a good place to put this. So then we had a problem as people who are building product. So we have been implementing um cloud events in event grid. Event grid is one of the event service eventing services that we have as a push push service originally um that has been kind of focused on cloud events. And when you have like Azure blob store raise events then this is kind of how this came out like there's a documentation page here are the events and these are the event types and here's some examples how you do this in cloud event schema but you basically have to go by example and so we didn't have a machine readable so we have now a nice a nice spec for how to go and express events but we didn't have a metadata story around it so we're like how can we go and build a metadata story around it so it'd be nice if We had something like where's my thing? Here's my thing where we had a a way to go and define um events where we have you know these are the fabricam smart light bulb events and then I have a few messages and then I have the turned on event and the turned on event is a cloud events event and it has an ID and has a type here says what that means. it has a source where is that where is that being raised from it's required for it to have a time and then the data content type that's being carried is an application JSON even though the schema is actually avo um and then having the schema actually sitting alongside of it so having a registry that is kind of a versioned registry for payloads but also is a registry that contains constraints for either filters or templates for um for events. So we can have a you know metadata foundation for which from which we can go and generate the documentation but then also go and generate more interesting things like clients or kind of drive tooling. That'd be nice if we had that. So that was the initial um motivation for um for embarking on the project at this X registry. Meanwhile, this is a CNCF sandbox project. Um the CNCF has kind of levels in which of of maturity of projects. Sandbox is the level where you start having stickers. [laughter] Um which means that sandbox is the level where the the technical oversight committee in the CNCF has first looked at it and said this is not garbage and then it the next level is then kind of the next maturity levels are then kind of counting on um on adoption. when we built so we we set out and and wanted to say we want to have a registry that covers as you just saw message definitions or event definitions and schemas and then also endpoints which you're also going to see in a moment and then we set up creating three specs for those things after we had realized that all putting it all into one spec would be too large of a document and then we found that we had a lot repetitive patterns across those and then we decide no we are going to factor those things out into a common registry spec and then create a way we can go and and create these these specialized specifications as extensions. So come came about X registry which is the extensible registry because we can now go and take basically any metadata model and plug it and create a universal metadata graph which I'm also going to talk about in a little in a little while. So it's model driven. You you declare effectively to the system to an implementation what is the kind of metadata you want to go and manage and then the implementation can go and uh and create that for you. What we also wanted to wanted to do is um we wanted to create a symmetry between a REST API and a document format because when you go and create a module and your module raises four events, you have to go and put those mod those declarations somewhere, right? You need to check them in with your code and then from there you need to be able to go and publish them. So what we decided is that a representation of a X registry inside of a file with all the with know various model uh instances is a legit registry. Um, and all the tooling should be able to go and feed off that registry. And the API representation of that is basically x numbers of documents stacked on top of each other and served by an API server. And you get kind of a navigation model because loading it all into memory all at once is kind of hard. But we want to make sure that we have that this whole model works both as a document model and as an API model with implementation. And kind of the the split in the middle is that you can go and take the metadata contents of an X registry can shred it out into an S3 store or Azure storage and you can serve that as a static site. So you want to go and have you know full-on server um a a static site server and then single file that should all kind of work the same so that this metadata story works at the at all scales. Um because we are messaging people. We came out of this the cloud events uh space. Our initial focus is on serialization validation schemas, message event definitions and endpoints for asynchronous uh uh protocols effectively HTTP being asterisk when it's used for that. Um so that was the that was the goal. So I already said principle documents API interchangeable. You can go and take the document and import into import into into the registry API and you can also go and export from the X registry API. Keep this all um symmetric version management for resource types is completely built in. So for schemas um it's not meant to be git but the reason why we have version management in there is when you have a messaging system you will have of a schema multiple versions in flight and in flight also means there's there's events which are stored somewhere in in storage which you will go and he rehydrate and run again. So those may be of version one and you're now at version 20, but if you are using a a protocol or sorry an encoding like Protobuff or like Avro, you have to have the original schema to even get at the data. So you need to have these versions of schema concurrently in a system available. So we needed to build a registry that can allows you to go and get at all those versions all at the same time. Because that's true for schema, we made this in a universal model. So you can have you know any resource you want in a versionable way. So the core specification set um that you can go and find at our site. So this is kind of where this lives. So we're at github.comxregistryspec. This is kind of where our stuff lives. um is uh we have a primer. The primer is uh kind of an introduction and kind of gives rationale. You start to realize that I'm doing this talk just mostly to make you read a spec um or a bunch of documents. The core defines the the the various entities and entity models that we have. So we have a concept of groups, we have a concept of resources, we have a concept of versions and then we also have a concept of model and how the model kind of applies to those things. And we try to make these thing we try to make that spec very precise. It is so precise that if you load this into your favorite um AI coding agent, they will im immediately be smart and uh you know put good results out. I I do this all the I basically just load up the specs kind of on the site in my instructions and when I do work with this it basically the the coding agents know this and so we're we're having a lot of there's a lot of AI kind of uh in mind here even though we don't specifically um write for it. Then we have an HTTP REST API which then runs on the core. We've decided to split that off because we um basically reserve the right to do an RPC API and maybe also kind of do an AP an MQP API um uh in the future. And then there's schemas that are built on top of the core. Um so we are generating JSON schema, AVO schema and open API documents for the for the APIs. And then we have a set of events defined. So when the registry changes state then we have a set of events that you can as implement go and raise um so that uh those are also standardized and we have a pageionation spec specifically for um HTTP structurally the there's an entity the API gives you effectively the notion of a group type um then there's a group ID there's a resource type resource ID versions versions and then uh the version number that's kind of generally the structure of it that maps to the document format in the same way. So the document structure and the API uh resource uh path structure are the same so that a relative HTTP path uh uh URI and an Xpointer are exactly the same. um and x pointer meaning JSON pointer um in in when you treat it as JSON. So that's the that's the general model that we have that in how we organize these things. So let's let's talk a bit about know closer to all all of our hearts. I had to go and tell you the abstract story. Um what are the specific things that we have? So we have a schema model, message registry model, endpoint registry model. The best explained this is by me showing you a few examples. You already saw one example here. So I have a uh a message group and the message group is called fabricam lumen. So we also have in um I can show you this also in the uh in the server. Now the IBM people are going to get a kick out of this. Um it's called it's hosted currently as xregistry.sophobub.org. Um so duck uh so Duck Davis is kind of hosting this. This soaphob.org this is that soap is the soap you might be thinking of. [laughter] It's that old. So it's kind of funny. Um so here we have uh um a few endpoint endpoint definitions. message group def me message group definitions. Let's go into those. Um you see that there's some employee events group and I can go and navigate into those and you kind of get the message definitions. So this is kind of the API view and Doug has made a UX could use some help but okay um that kind of layers on top of this. So so here you have kind of these message definitions and of course I have the same thing here kind of as same structure in this model. We have um an envelope section and we have an we have an then envelope metadata section. We currently only define the cloud events envelope. But when you turn on the cloud events envelope, then you kind of light up this envelope metadata section. I'll show you an extreme example which is not cloud events. Um which is uh I have this here I think. Yes. Um here's spark plug B. So spark plug B is an IoT protocol which has several kind of roles and so with this we can go and define the protocol and then the protocol options. So we can go and say this is going to be on this particular topic because we allow for URI templates. There's a will message which is defined kind of in the same set that this refers to and a will topic where this is going to be sent to. You see that this endpoint is now referencing a message group and these are the messages that are associated with the edge node. So these are all still endpoints and now we get to the message group. So these are now the messages that are associated with the edge node which is end birth and end data and end death. And then you have the device messages. So you can go and do this is the most formal declaration of spark plug B that you have yet seen. We can do here's a very simple example of an MQP message also not a cloud event but we say the protocol is MQP protocol options the subject field in the properties has this value. So when you and then it is associated with the following schema URI which is an XSD schema. So you can go and effectively always declare for CFKAN, NATS, for MQTT 311, MQTT5, um for MQP1, um and for what did I forget? HTTP, we have all the the the specific properties, all the the fields that exist and you can go and define them either as filters, right? you you have a declaration like this and you load this into into a dispatcher. The dispatcher now can now say if a message arrives where the subject is is my value then it's obviously the contos of my group.mmyvent message which means that even if no uh schema basically if no schema reference is given I know that I need to load the schema and apply it for validation. So, it's a metadata. Now, you have a metadata story and now you can go and and and format streams. Oops. Hang on. Let me not destroy things on the way. So, schema registry for any schema you like, right? Bring your own. We have predefined a few schemas mostly just in the spec. We said you know JSON schema the the the ident identifier marker needs to be like this and AVO and proto um and XML but if you want to bring ASN1 or whatever other schema you want you can go and and use that for message registry model as I said we have these kind of as predefined protocols and then one is the application neutral envelope cloud events one um and then the endpoint registry model similarly the particular characteristics of MQP endpoints and MQTT endpoints are all kind of predefined um in there. So, and the point is that you can go and create an an a metadata information model and you can go and store it in a well organized way and then you can go and share it out with others and also use consistent tooling. the schema model has um so this is kind of the models that you're creating. This is the the the model for the schema registry. There's a bunch of implied properties that are are not in here that don't need to be in here. But if you want to go and create a new subregistry, you basically create these model files and then you define basically what is my singular name? What's my the plural name is implied by the group's name. But you can also go and do this. there are compatibility statements. You can declare extensibility uh attributes or or disallow them. Um and then you can go and create your own attributes um and uh define them. And this a reference implementation the server loads these models and then kind of enforces the contract that you're declaring here for um these groups. um to create these kind of multi-protocol scenarios we have a very simple mechanism to go and create conditionals. So we have a a a thing called if values. So if you define in the model you define uh I have an attribute protocol and in that protocol there's an if values and if the value is am1 then take these sibling attributes and inject them at this level. So very simple and what we didn't want to do is we didn't want to lean on something like uh JSON schema. Believe me, we tried but we then ended up saying no no no we're going to go and do a super super simple very bespoke uh uh schema model here because that's only here for our tooling and we don't want to get in the hair of any of the other things. Let's go and do something that's very focused and it's actually quite good. So with this so this is the spec set and the spec set is as said in that in that repo there has been a lot of work um done uh there have been you know the core group has been varying between six four six five seven sometimes people we always bring the the result of the work to the greater cloud events working group every Thursday um at 6:00 p.m. European time um uh in the morning at 9:00 a.m. on the e on the Pacific coast and uh um there's a few several other people who are kind of interested in kind of driving this forward. So um and we've been uh also kind of building some tooling around this. the core tooling that's kind of in the in the the the the core set and that's actually the reference implementation is the XR XR server and that's the thing that I just showed you um here that is a containerizable go based implementation of the server um and then SXR which is companion CLI tool um of that server then there's a metadata driven code generator and registry client which is not quite there yet. Um I that's being going to be submitted uh uh very soon. But what that does is the following. So I can go and run based on one of those files that I showed you. So inkjet.xre.json that's an ink definition. I can go and run this. Um, so that's a code generator. Please, if you would. So, Python is always fun. Sometimes it's fast, sometimes it's slow, and you can't really tell why. So, let's go do this. So it now took one of those definitions and now turned this yes I trust the author um and uh emitted um all the data objects that we need and then it also emitted a Kafka producer from this where the user in the end only sends right the oops Well, maybe one less uh where the user sends just the print job started event with the necessary extra parameters, but they no longer worry about creating the cloud event or interacting with fat packaging for for Kafka. It just does it just does does that um in um uh no in the way how the template is is already doing this. So there's a templatize it's a template based generator. There's Ginga templates that are in there. there's a a bunch of extensions for code generation in that project and so right now so the core is kind of done but now the most of the work is basically stabilizing all these templates and since there's a lot to do um that is kind of ongoing work but the the base infrastructure to go and take these definitions and then run them into the template generator that's all there and uh all of these all of the ones that are currently like this is what I just showed you um that's working flawlessly ly and uh every project already includes the uh um the test projects that go with it. So here's a effectively built-in test um that actually runs this against a test container. Then there's an XR registry viewer and the XR registry viewer is an Angular app and this Angular app can be pointed at uh endpoints and so here's an endpoint that is a special X registry and you'll see uh there's net registries and end and endpoint groups and node registries and okay so what's that? Um I'll get to that. Let me just show you. Let me just very quickly show you a video. This is from my talk that I did uh at the recent fabric conference. So we now have a bunch of these these definitions. We have message groups. We have schema groups. Um and uh we're applying them to a in fabric. We have a Microsoft fabric is our data platform. We have a pipeline model which is called event streams. So and on the very left side this custom endpoint source is really an event hub Kafka and uh so now I I can go and associate one of those message groups with it and through this whole pipeline I now have type safe messaging and the effect of that is that if we go and take a look at this so this is inside of the product this is inside of fabric I have a so-called event schema set and the event schema set the nice thing about this now is not only that we have serialization metadata. What's really important for every all of those fields is that they have a description because we have an AI agent in the system. And to feed that AI agent with enough context, we need the descriptions. We need the schemas not only to go and do serialization. We need it because we need the semantics. So you load those things in into it where as you see we're standardizing actually internally on avo schema because JSON schema is terrible. And uh so the effect of that having these definitions is that we can go and now as I said associate those in fabric with these endpoints and now I have this this associated all these definitions associated and now I'm routing in what's called the event house and the event house is our event database um in in the platform. I know. Okay. And so event house ah ah it shouldn't have there anyway. So um so I'm going to tell you what what happens. So in events because we have the metadata we can actually go and co uh generate automatically all the tables that are necessary with the right struct data structures. Okay. So because it's extensible, we can also do a full metadata graph. That's actually what you're seeing here. So I have here um other models loaded and you see Python registries, node registries. So this is the pi package registry as a pass through. So there we have a we have a set of um of uh of um let me go and pick a my advertise. Um so this is a this is one of the packages from pi. We have models that kind of that that that make all the various package registries even as xregistry representations. So we've basically mapped pi, pi, npm, nougat, oci and all those things to uh xregistry models and we have reverse proxies which do those which do that translation live. So this is going towards a universal metadata tree. So if you want to go and have a universal view of all the metadata that's in your system but you want to have a consistent then xregistry is a path towards that. So um so that's that so that's this and status is we are at recent release Canada 2 we want to go and release version one before end of year and um call to action is please please come and join us. um uh participate in the project uh get our code uh file issues um I believe for the broker people this should be something interesting I think because we're solving a problem in the open that everybody has and I would be very pleased if uh we could go and build some greater community around this to kind of make type- safe messaging a thing. Thank you. [applause] >> [music] [music]

✨ This talk was recorded at MQ Summit 2025. If you're curious about our upcoming event, check https://mqsummit.com/ ✨ The CNCF xRegistry project is an offspring of the graduated CNCF CloudEvents project, motivated by the need to formally declare which events can be raised by services and which are available to handle. This session provides an overview of the xRegistry metadata model, its API and the mirroring document format, dives into reference implementations, explains its use in products, and shows you how you can leverage xRegistry to build robust and type-safe event pipelines. Let's keep in touch! Follow us on: 💥 Twitter: / MQSummit 💥 BlueSky: / mqsummit.bsky.social 💥 LinkedIn: / mqsummit