Deciphering 'The Algorithm'

In this video, I'll discuss a few examples to share my observations on how I think that advertising and content auction systems work. But first, I'll give a bit more context. These observations are based on running a few thousand worth of ad campaigns as well as my experiences in creating online content for the better part of the last decade. I mention that because I think there's a lot of overlap between the algorithms that decide what content you see and the advertising auction system that determines which ad gets served. In modern advertising, there is no longer a clear distinction between ads and content. So, you should consider this information accordingly. Now, since the target market of this video is for people who are involved in engineering, I should point out that a lot of supposedly smart tech people are surprisingly illiterate about how advertising works and how important it is to business. It's surprisingly easy to find tech people who will say things like, "Adds just don't work." But if that was true, why would people pay $100 billion per year to Facebook to have them served? Clearly, it seems there are a lot of common myths to dispel. I think there's a lot of people out there who have the idea that the only reason to run an ad is because you want people to click on it and immediately buy something. In reality, it's extremely rare for people to see an ad for something that they've never heard of before, click on it, and then buy something. Most people don't make a purchase on the first impression. They [clears throat] make a purchase on the 17th impression. A lot of technology enthusiasts also have major objections with the privacy issues that come with advertising. I think that these concerns are reasonable and should be respected. But the truth is, many members of the public really just don't care, especially when it comes to tracking minor things like food or entertainment preferences. Another thing that many technology enthusiasts seem to believe about advertising is that it's just some greedy capitalist attempt to make 10% extra profit or something. But that's not the right way to think about it. Using advertising to build a business is literally the difference between using an openloop system and a closed loop system. It can literally make or break the financial viability of an entire business model. If you want to build a rocket ship to Mars, you'll need to use some kind of feedback control system to make sure that you stay on track. Without feedback, the navigation errors will accumulate exponentially and you'll get lost in outer space. And the exact same thing is true in business. Another common thing that many tech people say is, "I don't click on ads at all. I don't even remember the last time I clicked on one." But the myth here is that if you don't click on an ad that that's not valuable to the company at scale, it could be just as valuable as if you did click on the ad. Of course, advertisers do actually want to make sales, but what they want even more than that is to build a list of prospective customers who are likely to buy from them in the future. One way to get that list is to compile a list of everyone who clicked on the ad and then build a demographic profile of those people. You can try to extrapolate that demographic to the entire population. But what you might not think of at first is that you can also use a set theory approach and do the opposite. Build a list of people who don't click on the ad. Then extrapolate that list to the entire population. When you set up the ad campaign, you can exclude that audience instead of including them. And if your sample is representative enough, this should theoretically give you the same thing. If you try advertising a product like a baby stroller to an extremely broad audience, it shouldn't take the algorithm very long to figure out that single men who don't have children probably aren't interested in that product. Whenever you see ads that you're not interested in, that doesn't necessarily mean that the ads aren't working. Having said that, there are other potential explanations for seeing ads that don't make sense for you. For extremely large companies with massive ad budgets, it is also possible that the marketer may have set the targeting options wrong. When this is the case, they can waste a lot of money very quickly. But if they're doing their job right, this loss should be caught quickly and corrected. Another potential explanation is when companies advertise products that have an extremely high conversion value. things like insurance, mortgages, cars, even if you don't click on the ad to buy the car, being bombarded with the company's logo repeatedly will imprint it in your mind. And someday, if you do eventually buy a car from that company, the conversion value could be thousands of dollars, which would justify hundreds of dollars worth of ad impressions over the years. Another possibility is when the marketer is just starting out and doesn't have a lot of data to work with. If you don't have any data to start with, you'll have to build your marketing funnel from scratch. To do this, you could start by targeting a general interest category. Targeting a general interest category probably won't get you many sales, but it will help you build a profile of people who will move to the second phase of your conversion funnel. Then you can build a second custom audience based on the conversions from the second stage of your conversion funnel. You can then repeat this process until you actually get people to buy your product. So, to start discussing a bit about how the auction systems work, let's assume that you're completely naive about running ads and you just want to get started as fast as possible. Something that you're likely to do is just leave all of the ad campaign options on the default. Since you don't know what to bid, you'll just leave it on automatic bid. Later, you come back eagerly to check your ad campaign, and you find that your ad has generated one click for a total cost of $21, which resulted in zero sales. And of course, this leaves you extremely disappointed. So you decide to turn off the automatic bid process. Instead, you put in a fixed bid of $1. But unfortunately, after running this ad for a week, you don't get any views. So you decide to raise your bid to $8. And now you start getting clicks, but none of the clicks result in sales. So you conclude that the traffic must be all fake or consisting of bots. But in my opinion, that's not necessarily true, and the truth may be more complicated. To help explain what's going on, I've created this diagram. The purpose of this diagram is to illustrate the inner workings of a hypothetical auction system that determines which ads or content gets served to a user on the platform. Here you can see four different examples of ads that four different advertisers are running on this platform. And for each advertiser, you can see how much they're bidding to run the ad. And here you can see three overlapping circles that represent the addressable audience that the top three ads would be applicable to. And the outer pinkish area here represents every possible user on the platform. So if you were to run an ad on this platform and not specify any targeting information, your ad could be served to anyone anywhere in this diagram. It could be someone out here in the pinkish area or it could be a user that's found anywhere inside of these overlapping circles. So to start with, let's review the advertising auction system in the case where no targeting is used. Now in this example, the insurance ad, restaurant ad, and indie video game ad are being run by experienced marketers. So the ads that they're running have lots of highquality targeting information. And the targeting information indicates that the probability of a conversion event like a sale or a click is very low for these three ads. So in this case, the lowquality garbage ads are the ones that win in the auction. Now let's look at the auction system for the opposite scenario. In the center of this diagram, you can see the white area that represents users who would be interested in all products. They're interested in insurance, restaurant food, and indie video games. So the probability of a conversion event is much higher for all three of these. And as you can see, the insurance ad is the winner of the auction. This is despite the fact that the probability of a conversion event is lower than it is for the restaurant food ad or the indie video game ad. The reason being, of course, is because they're bidding so much more. Next, let's review the area of the diagram that represents users who are only interested in indie video games. As you can see, these users have no interest in insurance. So, the insurance ad doesn't win the auction, and neither does the restaurant food or lowquality garbage ads. And a similar scenario plays out for the area of the diagram that represents people who are interested in restaurant food but not interested in insurance or indie video games. And predictably, the same thing plays out in the blue area of the diagram that represents users who are only interested in insurance. It's also interesting to review the overlapping regions. In this area of the diagram, the insurance ad and restaurant food ads are competing for attention. And as you can see, once again, the insurance ad wins despite the lower conversion rate. And the same thing is true for the area of competition between the insurance ad and the indie video game. And for the area of competition between the restaurant food and indie video game, the restaurant food wins due to a higher bid. Now, reflecting on these ideas from this hypothetical auction system, I think that we can come up with some better ideas for why the typical experience of people who try out advertising is so poor. Clearly, if you don't have a lot of information to convince the advertising platform that your ad is likely to result in a conversion, they're probably going to rank you very low in their auction system. This means that you're going to have to bid significantly more in order to get chosen in the auction system. The other observation is that most advertising platforms have a large pool of leftover users that almost all advertisers want to exclude from their targeting. Now, you might say, "If some of these users are ad scraping bots or people with glitchy phone screens that click randomly, shouldn't the ad platform just go ahead and automatically exclude them?" The reality is that the advertising platform doesn't necessarily know all this information. They just know that this set of users has a very low conversion rate for many different types of ads. For example, it might be very difficult for the advertising platform to tell the difference between ad scraping bots and iPad kids. But having said that, sometimes iPad kids might ask their parents to buy them things or the iPad might be shared by the parents sometimes. Another significant category that I don't think many people think about in advertising is people with mental disabilities. If someone has a significant mental disability, it's much less likely that that person is in the workforce. And if they're not in the workforce, it's very likely that they spend a lot of time on their computer or phone. And an important question for advertisers is do people with mental disabilities buy things? And of course, the answer to that question depends a lot on the person and the type of disability that they have. So in general, you can't necessarily wholesale exclude this group of people. The vast majority of people are interested in the vast majority of products. And another observation about trying to lowball the bidding process. Most likely, as the quality of our traffic goes up, the cost per click will also go up, not down. And this is why the average cost of advertising for many different products can converge to a very similar value. And it's very rare that you'll find someone who only clicks on restaurant food ads, insurance ads, or indie video game ads. In fact, the size of the areas represented in this diagram that only cover a single interest category are likely exaggerated. This diagram only considers three types of ads. But a real advertising platform has many more than that. And importantly, it's also worth observing that these thin slices are the best possible case scenario for every different type of advertiser. As you can see here, the restaurant food ad could potentially bid much lower than $20 and still win the auction. But this condition is only true in this thin slice of the diagram. And remember what we said before about people who behave like outliers. In general, it can be very difficult to tell the difference between them and people who have inauthentic behavior. And for the subset of people out there who only click on insurance ads and actually buy it, I think that it could be very easy to confuse those users with people who have mental disabilities. In general, the concept of fake clicks doesn't really exist anymore. It's your problem as an advertiser to figure out which clicks are good or not. You need to be tracking the actual amount of money you're making and optimizing for that. There is no such thing as fraud traffic. If someone with so-called inauthentic behavior is willing to become a long-term customer and pay you lots of money over the years, then the ruse is actually helping your business model. There's also all sorts of other unusual human behavior that's difficult to account for. You've likely met a young kid before who's going through a phase where he only wants to talk about a certain video game or a movie. This kind of psychological phenomena is something that even adults go through. And if you're looking at click data from someone who behaves this way, it might be indistinguishable from a bot. It's a common accusation that people make that these advertising platforms exploit people's mental weaknesses. And even though that probably is true, I think that it's actually even worse than that. There are most likely certain kinds of mental weaknesses that the platforms don't even know about and the algorithms simply optimize everything mindlessly. In fact, if you thought about the reverse problem, if the platforms wanted to protect people from their mental weaknesses, this would be extremely difficult to do because most likely you probably wouldn't be able to figure out why someone was buying so much insurance all of a sudden. Modern advertising has become very difficult. In my opinion, advertising was probably much easier for inexperienced marketers back in the early 2000s. But now that the algorithms are better for experienced marketers, they're also worse for inexperienced marketers. Another important observation is that I described this as an auction system. In fact, I think you could compare it to a bid and ask system, like a stock market order book. Back around the 2020 US election cycle, I was actually using Facebook ads at the time. Within hours of the election being over, I noticed a massive decrease in the prevailing bid for ads, which is not surprising given how much money is poured into political advertising. So based on what I've described about how the auction system works and my own observations, you will generally get the traffic that is the highest quality for you but also the lowest quality for everyone else because that's the optimal answer to the matching problem. Another thing that people say is that the algorithms deliberately try to provoke and radicalize us. I can't prove that it's not deliberate, but I will say that polarization and echo chambers are likely a mathematical property of an optimal matching system. Obviously, polarization and radicalization are not great for society. But I think that trying to legislate away a problem like this is likely impossible. Water always flows downhill to find the lowest energy state. And ads or content will always get served to the user who will cost them the least. To illustrate how abstract and general these targeting algorithms can be, I've written this example Python script. Obviously, this is just something that I wrote as a demonstration for this video. But if I can come up with this in a couple of hours, imagine what thousands of engineers working at Facebook or Google can come up with. When it comes to online advertising platforms, there are two major different types of audiences. The first and most simple audience type is a custom audience. A custom audience is basically just a list of specific individual users. It could be based on a list of specific user emails or it could be based on a list of users who performed a specific conversion event such as visiting a web page, clicking on a video or buying a product. And the second type of audience is much less precise but is usually much larger. It's based on predictions in demographics. Different advertising platforms call these audiences different things. On Facebook, they're called lookalike audiences and other platforms may call them something else like predictive audiences. And in general, these platforms give you almost no insight into how these features actually work because if they talked about implementation details, they would likely be exploited. In this Python script, I've attempted to give a simplified model of how I believe that lookalike audiences probably work internally. In the early days of the internet, we used to have these things called website counters. These were counters that you could place on a website that would go up every time the website got a visitor. Back in those days, we didn't have as much bot traffic. So, getting a visitor actually meant something. And this is why the early advertising platforms were only based around getting impressions. And of course, eventually advertisers moved to tracking clicks instead because that was a better indication of legitimate user interest. But eventually that stopped being enough as well. So eventually people started tracking more behavior to make sure that the clicks were authentic. And eventually we ended up with what we have today where the platforms try to track everything. And in this simple Python example, I have a small collection of madeup users. For each user, I have a list of metrics that the platform might have access to. And to simplify everything, all of the metrics are compressed to a value between 0 and 1. So for a metric like age, 0.4 could mean 40 years old or something. The details don't really matter as long as all of the users on the platform are ranked between zero and one. Here I've provided a list of five metrics, but in reality there could be thousands. In general, you could look at all kinds of things like the make and model of the person's phone or what time they get up in the morning. How much time do they spend traveling? As you can see here, the example platform only contains four example users. And in this example, we have two different advertisers who want to run ads on the platform using lookalike audiences. The first advertiser wants to create a lookalike audience based around an interest in technology. And the second advertiser wants to create a lookalike audience based around an interest in mobile gaming. And here you can see a list of waiting parameters that was created for each lookalike audience. These waiting values are determined by analyzing a custom audience that was provided by the advertiser. For example, this could have been determined by looking at a list of customers who actually bought products from that advertiser. Generally speaking, these waiting values will be determined automatically by the advertising platform. The only thing that the advertiser has to do is specify the custom audience to sample. And in general, even though there could be thousands of different parameters here, you don't really need to think about which ones are actually useful. You can just throw all of them into a machine learning algorithm and figure it out automatically. Since all of the lookalike audiences in this example are just a list of waiting values, you can create a scoring function and evaluate every user in the platform according to how closely that user can be associated with the lookalike audience. To calculate this, just take all of the waiting values for every single attribute and multiply it by how much that person has that given attribute. Then just add them all together and now you've got your scoring function. Then to figure out who you should advertise to, simply run a sorting function on all users on the platform and use the overall scoring function is the sorting key with the applicable set of weights for that lookalike audience. Now you can mathematically calculate the best user on the platform to advertise to for any given lookalike audience.

Become A Channel Member: https://www.youtube.com/channel/UCOmCxjmeQrkB5GmCEssbvxg/join Terminal Block Mining Simulation Game: https://github.com/RobertElderSoftware/robert-elder-software-java-modules