Java Serialization: Spooky Action at a Distance

Two days before the initial release of this video was a big day in the world of Java as 29 years ago on February 19th 1997 Java 1.1 was released. Java 1.1 brought med now fundamental features to the Jawthre platform including Java beans, JBC, remote method invocation, uni code support and more. Another feature that was just below the surface of being included in the Java 1.1 announcement press release, but has had a profound impact on the mental state of the JK engineering team was the addition of the Java serialization mechanism. Outside of applications like a calculator, rare is it for an application to be both stateless and standalone. Eventually, an application will need to either persist to a disk or communicate with an external service. For this to happen requires taking the structure and content of an object and encoding it into a transferable format. A non-trivial task back during this time period. The serialization mechanism was Java's attempt to address the needs of its users and its ease of use led to its widespread adoption which also helped propel Java's own adoption during this time as well. But it's a scary world outside the JVM. And when you start accepting data into your applications without validation, you're opening yourself up to some potentially spooky actions. And that's the subject of this episode of Stack Walker, Java Serialization, its success, its issues, and what we're doing to fix it. If you'd like to learn more about Java serialization, finding your nearest Java meetup to talk with fellow Java developers about serialization alternatives, and more, be sure to check out dev.java. at Java link in the description. And don't forget, Java 1 is just around the corner, March 17th through 19th, and is an opportunity to meet more brilliant Java developers and the JDK team as well. Register now at jav1.com. Link and discount code for $150 off registration in the description. Before covering the issues with serialization, it's worth establishing some context about the problems serialization was attempting to address and the state of an industry at the time serialization was being developed. We will then also do a mid-depth review [clears throat] on how exactly serialization works under the hood. As mentioned in the intro, the serialization mechanism was introduced with the Java 1.1 release in 1997. During this period of the 80s and 90s, persistence was hard work. It required writing a lot of fiddly code to handle the writing and reading of an object's structure and state. To understand the issues facing developers at the time, let's look at a simple example. Here we have the name class with only two immutable string fields, first name and last name. The code for persisting this class to a file is relatively simple and straightforward. Of course, this code isn't covering everything. For example, I'm only capturing the names of the fields and their content and not the encapsulating class info. There's also minimal error handling as well, but let's set that aside for now because when we review the reading process, the complexity of persistence starts to rear its head. [snorts] This is also a bit of a spoiler for later in this episode. Even for a simple two-field class, there's already a lot of fiddliness. Stepping through lines in a file, parsing by delimiter, looking for specific string values. The mechanics of reading and writing to a file start to distract from the business meaningful code. And again, there is minimal error handling or any other validation happening. All would of course add even more complexity. And it [snorts] can't be forgotten applications evolve and grow in capability over time. At some point in the future, the name class might be enhanced to [snorts] add support for middle names, a suffix, a list of nicknames a person goes by, and more. Each of these changes will require making in some cases significant changes to how you write and read this class. You'd also have to consider concerns like backwards compatibility. It was why during this era it was common for an entire role at organizations to be devoted to just handling data persistence. Along with the need for better persistent support, another trend growing in popularity during this era were distributed objects. It was one of many in this case largely unsuccessful attempts to help reduce the complexity of building and maintaining distributed applications. Java's answer for distributed objects was the remote method invocation RMI API also part of Java 1.1 and this also had impacts in the design and implementation of Java serialization where the goal wasn't just to persist the data contents of an object as would become the revealed interest in industry later but to teleport a living instance identity and all it's also why the why not just JSON argument doesn't really make sense as it only covers the encoding aspect of Java serialization. So that's the context surrounding the design of the Java serialization mechanism. Now let's step through the mechanics of how it actually works. To make a class eligible for serialization in Java, I need only implement the serializable interface which has no methods and likely also autogenerate the serial version UID field to resolve the compiler warning and I'm done. And in many cases I need not know more about serialization as it would just work. But when the scale of Java's usage is in the billions with a B, those niche issues of unexpected behavior or security vulnerabilities start adding up to real numbers. The classes that handle the nitty-gritty of serialization and deserialization are the object output stream and object input stream classes respectively. When the my data class is serialized, it will go through the write object method of the object output stream class. From here, things start to get complicated. Through the usage of the reflection API, also a Java 1.1 feature, type checking, moving through an object's hierarchy, and more. The object output stream class determines which fields are eligible to be serialized and depending on if they're type, how they should be serialized. There is also checking on the class if there are some userdefined methods for a man of control on the serialization process. A subject we will return to later. But the right part comparatively isn't the particularly mind-blowing part. For that we need to go to the des serialization process with the object input stream read object which starts off quite innocently with just four lines of code but it gets weird pretty quick. a few method calls down and we arrive at the discreetly named read ordinary object method which will check the stream for the name of the class that is being deserialized. It will then create an object variable and we'll set that object variable to the type of the class being des serialized and then proceed to instantiate an instance of it by calling the first no r constructor within the classes hierarchy. Let me repeat myself. Serialization would often be actively avoiding calling a declared constructor within a class. Once instantiated within the read serial data method, again using the reflection API, the values of the fields within the class are manually assigned. Some cleanup and final checking after this and it returned with a fully living and breathing object that you can work with like any other. And that's the short story of how serialization works following the default path. Again, we'll get there. Ultimately, the downfall of serialization is closely related to what made it so popular in the first place, its ease of use. But that ease of use ultimately relies on a lot of assumptions and reflection magic. Bypassing a lot of the typical behavior we expect in Java, which leads to a number of issues. For this video, I would like to focus on eight issues. There are more, but to go through them all, well, this would be a much longer video. If you would like to learn more, I would recommend watching why we hate serialization and what we might do about it. After watching the entirety of this video, of course, link in the description. I suppose the co-presenters know a thing or two about Java as well. Continuing on, pretty early on as a Java developer, you learned that to create a new instance of an object, you need to call its constructor. And in the example of my data class, because I provide only a single constructor, programmatically, when I create a new instance of my data class, I would need to call that constructor. But as we covered in the previous section, this constructor is bypassed and the fields are set mainly via reflection. Now if this was a dumb constructor naively assigning the pass in arguments to their appropriate field then no harm no foul. However in this constructor we are doing some meaningful work like some validation checks to make sure the pass in values are well valid. By performing these checks at instance creation, it should mean we can assume that an instance of my data class would always be in a valid state and this would mean not having to repeat any such validation in other areas of the application. Serialization by bypassing the constructor could introduce some very confusing bugs into our application which smoothly leads into the next two issues polymorphism and confinement. When reading from the object stream, there are opportunities for inserting malicious or simply incorrect code. In the constructor, I am copying the contents of the past map into the newly created tree map. This would ensure that only the my data class has the ability to modify this map and also guarantees the map implementation I am working with is a tree map. However, with the default behavior or serialization, it's only looking for a compatible type to map. And a hashmap instance would satisfy that, but could lead to unexpected behavior. As for example, a tree map has a consistent order while a hashmap does not. On the more malicious side, the user could send their own special map implementation, evil tree map, that includes malicious code providing them with control and access into our system. The good news is Java serialization provides options for dealing with this. The bad news is well it leads to other issues itself. Serialization provides options for taking over manual control of the serialization and de serialization process. If we wanted to have direct control over the validation of the fields on the incoming serialized stream, we would implement the private void read object object input stream throws IO exception method. Now you might be asking, Billy, why did you just say the entire signature and not just read object? Well, that's because the serialization mechanism is looking for an exact method signature match. And that's the case as well for several other methods a developer could implement for mana control over the serialization and des serialization process including write object serial persistence fields and read object node data. While the writeplace and read replace methods allow you to choose their access modifiers. The problem isn't with the existence of these methods but the opakqueness of their usage. a subject we will return to towards the end of this video. But back to my data class. I implemented the read object method in my data class added the equivalent behavior to match what is occurring in the constructor. At last that leads us to our next issue. Perhaps the eagleeyed of you might have noticed that along with the addition of the read object method. There is also the removal of the final modifiers from the my data and my amount fields which of course makes sense as I am setting those fields outside of the definition in the classes constructor. In order to properly support serialization, we would need to break other good programming practices like sensible uses of final. There is another good programming practice we are somewhat violating and that is repeating yourself. As we take over manic control of the serialization and deserialization process, it adds the additional work of making sure the methods we implement are consistent in behavior with the rest of the class. As I mentioned, I added the appropriate corresponding behavior so that the read object method would perform the same validation checks and defensive copying that the constructor does. But the nature of the read object method and the constructor is such that the code cannot be easily refactored into a single shared method. If at some point in the future we update the my amount field so that it must be greater than zero but less than 100, we would also need to update both the constructor and read object methods. You might also notice that issues up until now have been primarily on the desertilization side. As has been a point of emphasis throughout this video, the read side is where most of the complexity and danger lies. While writing or encoding a class into another format is comparatively simple, but it's not without issues itself. One such issue would be thread safety. Let's assume for reasons an instance of my data class could be referenced by multiple processes simultaneously. To account for this, I could quite reasonably add synchronized to the add to map and remove from map methods. This was insert consistency when updating the map in a multi-threaded environment. However, the serialization process has no knowledge of this and could potentially grab the my data map in the middle of multiple writes occurring leading to an inconsistency. Ultimately, one of the biggest issues with serialization is that it inhibits the evolution of the JDK itself. The need to provide backwards compatibility of serialization, which also means support of these issues just mentioned, can mean trade-offs in the speed of the delivery or capability of new features. But the JDK engineering team isn't just continuing to dig its own grave of serialization and has been actively working on and addressing these issues and finding a replacement. The first step towards addressing some of the serialization issues was through the addition of configurable serialization filters in JDK9 with Jeep 290. This gives users the ability to limit the classes that were eligible for deserialization. This will help prevent some of the polymorphism and confinement issues I mentioned earlier. The next up was records in JK16. If you review the code of read object and write object methods, you will notice there are special conditions for dealing with records. Chief among them is that when des serializing a record class, the records canonical constructor will be called which addresses the extra linguistic consistency undermining final issues. Of course, records can't be used everywhere, so it's not a full replacement, but we are working towards a replacement to serialization, a serialization 2.0, if you will. Back during the issues with serialization section, I mentioned how these methods that allow for more manual control weren't a problem in and of themselves. The issue primarily lies in that they are difficult to use and aren't a published part of the serialization mechanism. And that is ultimately the direction serialization replacement will need to go. Reducing the reliance on magic and requiring a bit more manual work from the user. Defining an intermediate representation between the final form of the class and the encoded form like in this example here. If you'd like to learn more about this, be sure to watch Victor Kling's talk on his efforts in redesigning serialization. Link in the description. Serialization holds a unique role in Java's history, instrumental to Java's rise in popularity, but also a barrier to its continued evolution. But we now have a good idea on where we are heading, even if that destination might still be a ways off. All right, that's it for this episode of Stack Parker. Thank you to the members of the JDK Engineering team who helped with the research of this video. Don't forget to like and subscribe and until next time, happy coding.

Serialization has been a part of the Java Platform since the 1.1 release. While serialization brings with it a lot of utility and was key in Java's early success, it does have some pretty significant issues as well. In this episode of Stack Walker, we review the history of serialization, the concerns it was attempting to address, and how some of those design decisions have resulted in more than a few headaches for JDK engineers and Java developers alike. We will then look at how the JDK engineering team is working towards addressing the issues that serialization presents and ultimately a replacement. Come to *JavaOne 2026* in Redwood City, March 17th-19th, and get $150 off with code J1WEB150: https://www.oracle.com/javaone/ Want to learn more about serialization, other topics, and finding your local Java Users Group (JUG)? Check out: https://dev.java *Additional Reading and Viewing:* Towards Better Serialization: https://openjdk.org/projects/amber/design-notes/towards-better-serialization Why We Hate Java Serialization And What Might Do About It: https://www.youtube.com/watch?v=dOgfWXw9VrI Serialization 2 0: A Marshalling Update!: https://www.youtube.com/watch?v=F89sNgG9dRY https://openjdk.org/jeps/290 https://openjdk.org/jeps/395 0:00 Intro 2:15 History of Serialization 5:30 How Serialization Works 8:23 Serialization's Flaws 9:07 Extra Linguistic 10:15 Polymorphism and Confinement 11:21 Difficult to Use Features 12:39 Undermines final 14:19 Thread Safety 14:54 Inhibits JDK Evolution 15:15 The Way Forward 17:02 Conclusion Attributions: https://commons.wikimedia.org/w/index.php?curid=6372844 https://commons.wikimedia.org/w/index.php?curid=9886955 https://lotr.fandom.com/wiki/Gandalf https://commons.wikimedia.org/w/index.php?curid=49570 Tags: #Java #JDK #javaprogramming