Amplifying Invisible Motion

How much do tall buildings really sway? What about the wobbliest bridge in London? And is it possible to actually see the purring of a cat instead of just hearing it? Also, why is my 3D printer moving like this? These are all questions that we can answer with this tiny global shutter video camera and this weirdly ruggedized laptop. This video is sponsored by Shopify. You can't actually use this camera on its own because it's the laptop that does the capturing of the data from the camera. But really, the important thing is that unlike with DSLR or a phone camera, there's no compression. So, we're not losing any information, which means we can do some really interesting analysis. I'm talking about motion amplification. I made a video about this years ago, but I didn't actually have the camera and the software. So, I was showing other people's footage, but now I actually have the camera and the software for myself. So, instead of looking at factories and things like that, we're going to look at all the weird things that I wanted to look at. One thing I really wanted to know is how much tall buildings actually sway because it's quite hard to get solid information on that. So, we took the camera into London on a fairly windy day to see what we could see. And it turns out that some things are easier to amplify than others. Like look at the lamp post in the foreground of this shot of the BT tower. It looks like it's not swaying at all, but when you amplify the motion, it goes like crazy. But what about the BT tower itself? Well, that's a lot harder. It turns out it sways a lot less than a lampost. We originally set up here to look at the BT tower because I mean, god, look, what an iconic shot. But actually, we needed to get a lot closer. And we needed a sturdy reference building in the foreground. And in fact, we've lined up the shot so that they're almost touching, which should make it easier to see any motion in the tower itself. If you just try to amplify all the motion in a scene, well, you have to choose between not very much amplification or loads of noise. But look, if I pick out this little area, I can get an analysis of all the frequencies of the motions in that area. And look, there's a peak at 0.133 hertz. And if I tell the software to only amplify that frequency, then we're no longer amplifying the noise. And we can really amp up the magnification. This is 250 times amplification. And you can really see the building sway. 0.133 hertz is a little over 7 seconds per swing. So what you're seeing here is sped up. The cool thing is if you input the focal length of the lens and the distance to the subject, the software will tell you how much the thing is moving. So look, we can see that the BT tower is moving about 5 mm in reality at the top there. And here we can see that my 3D printer is moving back and forth just 33 microns. You can characterize the movement in lots of different ways. For example, here's a plot of the movement of this aerial over time in the vertical direction and the horizontal direction. Or you can plot it as a kind of orbit. And look, here's a vector of that motion laid on top of the video. These two are waggling at the same frequency, but they seem to be out of step with each other. In other words, they're out of phase. And I can paint the whole image based on phase. Look, it shows that these are about a quarter of a cycle out of phase with each other. Kind of like this. This bit of footage from RDI shows it quite nicely, too. These are exactly out of phase with each other. This is why RDI does motion amplification in the first place. By the way, it's like putting a million tiny sensors all over your factory so you can monitor the things that shouldn't be moving. And additional information like phase can tell you whether two parts are moving in opposition to each other, which could be important. Later on, we'll find out just how out of phase my cat's left paw is from her right ear when she purr. We didn't have any luck with the shard, by the way. That thing doesn't seem to sway at all. Or at least it didn't while we were there and not in the direction that we would be able to see it from this angle. But we weren't far from the Millennium Bridge, which you might know had a serious wobble problem when it first opened. They fixed it very quickly. But look, it does still have a bit of a wobble. And based on the focal length of the lens and the distance between the camera and the bridge, we know the bridge is only moving about 2 mm. I recently told you about my Shopify store, but I actually have two, Maths Gear and Festival of the Spoken Nerd. And I had to get this thing working between the two where orders from one would generate oneoff discount codes for the other. And this is the thing that I love about Shopify. You can go as nerdy or non- nerdy as you like. And you can set up a really nice store with no technical knowledge whatsoever. But if you do want to tinker, the technical stuff is really easy to get to. For example, I decided to write code that generates salted hashes for the discount thing that I was trying to do. Another example is look, the analytics is really good and it just works. But if I click here, it exposes the SQL query that generated this report. So if I wanted to, I could tweak that for something niche that I want to figure out. One thing that's really stressful about owning a store is when legislation changes and you need to get compliant. for example, GDPR or European digital goods tax. With Shopify, there's always been a solution built in before the deadline. Having that kind of stress just go away is amazing. I don't have time to talk about all the features, but something you might appreciate is integration with YouTube. You can link your store with your channel and get proper product links under your videos. And they do other platforms, too. So, if you're thinking of selling something or you already do and you suspect there might be a better way, go to shopify.com/stevemold to start your trial. Or you can use the QR code and the links in the description as well. I mean, there's so many ways to do it. But how does this motion amplification thing work? Well, here's how I assumed it worked. Computers can do this thing called edge detection. So, oh, look, here's the edge in this image. And in this next frame, clearly it's moved. And if I wanted to amplify that, well, I'll just move it even more. And actually, that's not what's going on here at all. In fact, it can't be what's going on because the motion we're typically amplifying in these shots is subpixel motion. So, here's how it's actually done. For simplicity, let's just think about a 1D line of pixels. And let's convert this into a graph of pixel brightness. So, now we've got this curve. And you might know that any curve can be represented as the sum of sine waves. It's called a furia transform. For example, I worked out that this curve is made up of all these sine waves added together. Each sine wave has one more bump than the one below it. In other words, this is the harmonic series. And for each sine wave, you just have to work out the amplitude and the starting position that leads to your original curve when they're all added together. Now imagine in the next frame of this 1D video everything is shifted to the left. Well now when we do the furer transform we find out that all of the sine waves have shifted to the left as well which makes sense. But when you shift a sine wave you can ask the question how far has it shifted through one cycle of that sine wave? Because look if I shift this sine wave all the way over here well that's a full cycle and the sine wave is back where it started essentially. Whereas this big sine wave here, it's moved the same distance, but it's only moved slightly through its cycle. And this medium-sized one has moved halfway through its cycle, and so on. How much a sine wave has moved through its cycle is called the phase shift. So, what if I wanted to amplify this motion by a factor of 10? Well, all I have to do is look at the phase shift between the two frames for all the different sine waves and then multiply that phase shift by 10. Move the sine waves over that much and then add them all back together to get back into pixel space. And look, the image has now shifted over 10 times as much as it had before. The amazing thing is at no point in the process did the software need to know about objects in the image. It didn't have to do edge detection or anything like that. Like it just works when you do the math. It's pretty cool. Now, what we've shown here is the analysis for when the whole image shifts, but we need to be able to handle different parts of the image moving independently. So what you actually end up doing is kind of several FIA transforms at different scales for each pixel. And it gets even more complicated when you shift to two dimensions cuz you need to think about direction and all this sort of stuff. But anyway, that's the basic principle for how it works. And by the way, that might not be exactly how RDI software does it, but it does seem to be the state-of-the-art for motion amplification in general. Okay, but what if we wanted to amplify the motion of someone humming, for example, because like maybe my throat is [music] wobbling, but it's wobbling at 200 hertz or something. And this camera can go up to maximum frame rate of 100 hertz. So that's why we shift to this high-speed motion amplification camera. Again, if you just try to amplify the whole thing, the motion of the hum will be drowned out by larger motions and noise. But looking at the frequencies present in this little square, well, this is probably my humming frequency. So if we filter by that, look, that's cool, isn't it? But we can make it even better. And this is really cool. So this is a plot of side to side motion in this square. It's really jagged because my throat is moving back and forth 127 times per second, and the camera is taking 500 frames per second. So, the actual movement of my throat is probably a fairly smooth curve that's roughly a sine wave, but the camera is only capturing data a few times per cycle. So, it looks spikier than it is, and it's why the video looks quite jerky. But here's the really cool thing. Because the footage is cyclic, you can borrow intermediate frames from future parts of the video and cram them all into one cycle to make it smoother. That's what this HDR thing means. It doesn't stand for high dynamic range. It stands for high density recording. So, it's not like motion smoothing that you would get in editing software where the intermediate frames are invented. This is actual data from the camera being used to fill in the gaps. Before we get to the cat pair, I just want to show you a few other cool things. Here's me balancing on one leg. And look, I'm trying really hard to stay still, but obviously there's still loads of motion going on. When you fixate on a point with your eyes, you might think that your eyes aren't moving at all, but actually they're always doing these tiny motions called micro cicades about twice a second. And I really wanted to see if I could see this with the motion amplification camera. The challenge is my head is moving all the time. But actually there's this feature where you can ignore transient motion and only focus on the motion of small things. And if you wanted to get data about microscopic movements of a point that is part of something that has large movements, well, we can do that. Look, I'm tracking the pupil. I can strip away the large scale motion of my head and find out that my pupil moves about a fifth of a millimeter with each of these micro circades. This is a crane. Actually, cranes in general look pretty cool under motion amplification. You can often see more than one mode of vibration and sometimes higher harmonics. Okay, but the hardest challenge was the cat. So again, amplifying everything isn't going to help us see the purr because the purr is going to be such a tiny part of the overall motion in this image. And so again, we look at the different frequencies to see what we can filter out. But this time, it's not immediately clear if any of these peaks are the purr. I thought it might be this one at 13 hertz, but it looks like that's just a duty cycle or something on the dimmable LEDs in my living room. There's a peak around one herz. That's probably the heart rate, but actually that's a little slow for a cat. Or it could be breathing, but that's a little fast for a cat. This might be the heartbeat actually around two hertz. And it seems to be sending a shock wave into my arm. But I couldn't figure out how to see the purr. But then I had a genius idea. I recorded audio of the cat purring. And then look, in this software, I can see how many bumps are there per second. And it's pretty much exactly 25 hertz, which would have been really hard to pick out from this graph of frequencies. But when I filter by it, we get this. How cool is that? Just like with the heartbeat and breathing, the pair actually causes her whole body to move. I think because she kind of had her head tilted upwards in a funny way, the movement of her head was kind of amplified, like literally in the real world before being then amplified by the software. So, her head moves quite a lot, but look, even her paws are going. And actually, as promised, her paws and her head are out of phase with each other. And it's around [clears throat] 180°. In other words, they're almost exactly out of phase with each other. So, there you go. If you could see a cat pair, this is what it would look like. RDI Technologies aren't paying me or anything like that, though, they did lend me the cameras. So, a big thank you to them. RDI technologies for all your motion amplification needs. Hope you enjoyed this video. If you did, don't forget to hit subscribe and the algorithm thinks you'll enjoy this video next.

Get your business started today with Shopify: http://shopify.com/stevemould Motion Amplification and Video Magnification are techniques that find subtle changes in a video that are invisible to the naked eye and amplify them so they become visible. They have huge diagnostic potential in industry and medicine. Original Motion Amplification Video: https://youtu.be/rEoc0YoALt0 Here's RDI Technologies: https://rditechnologies.com/ You can buy my books here: https://stevemould.com/books You can support me on Patreon and get access to the exclusive Discord: https://www.patreon.com/stevemould Twitter: http://twitter.com/moulds Instagram: https://www.instagram.com/stevemouldscience/ Facebook: https://www.facebook.com/stevemouldscience/ TikTok: https://www.tiktok.com/stevemould Buy nerdy maths things: http://mathsgear.co.uk