What You Need to Know About Analyzing Backtests

[00:00:00] Welcome to Two Quants and a Financial Planner, where we bridge the worlds of investing and financial planning to help investors achieve their long term goals. Join Matt Zeigler, Jack Forehand and me, Justin Carbonneau as we cover a wide range of investing and planning

[00:00:09] topics that impact all of us and discuss how we can apply them in the real world to achieve the best outcomes in our financial lives. Justin Carbonneau and Jack Forehand are principals at Volidia Capital Management. Matt Zeigler is managing director at Sunpoint

[00:00:20] Investments. The opinions expressed in this podcast do not necessarily reflect the opinions of Volidia Capital or Sunpoint Investments. No information on this podcast should be construed as investment advice. Securities discussed in the podcast may be holdings of clients of Volidia Capital or Sunpoint Investments.

[00:00:32] So Matt, today we're going to talk backtesting. Very exciting topic. When we talk about p-hacking and heteroskedasticity and all the fun words. We have a limit on number of syllables that can be used out of our mouths in the

[00:00:45] podcast so and heteroskedasticity is over that. So unfortunately we won't be going there today. But we are going to think more about like how your average person can think about these because a lot of times when you get people like me talking

[00:00:56] about backtests, it's all this fancy stuff and all these words and like all the details behind the scenes of the tests people have run. And like that's all very, very important obviously for the person running the test. But it's

[00:01:07] also like as a user who sees these, as a user on the other side of these, it's important to maybe think a little bit higher level in terms of how can I ask questions because your average person is not going to ask these detailed

[00:01:17] questions of somebody who ran a backtest. It's more like what are the high-level questions I can ask or what are things I can think about that will help me better evaluate this? Everything. There's nothing we can talk about that doesn't have some form of a

[00:01:30] backtest in it. Every word we pick, everything we choose to say, we've got some prior knowledge that we're calling forth in our heads with some parameters around it to say this is our understanding going forward. So even

[00:01:44] if we're not going to get into all the really hard and tricky statistical terms and concepts, which I would just demonstrate immediately how wildly unqualified I am, talk about any of back testing and just understanding some

[00:01:56] of the language and ideas around it remains one of the most important things in finance because I mean honestly think about this. Is there anything that somebody presents to you without some version of a backtest behind it? Like

[00:02:09] do we ever talk about anything where there's not some some backtest in concept? If you think about the idea of just using the past to try to predict the future and if you think of a backtest in that context, we use it all

[00:02:19] the time. I mean it's not, it's just as quants like me. A lot of people look at that, look at the past and try to say what might happen in the future. I mean even in the world of financial planning you do that so like when you

[00:02:28] broaden out the concept, you don't think about it as some like quant test like the idea of how to evaluate a backtest that becomes a much broader thing even if you're not in like the nerdy world that I'm in.

[00:02:38] So let's dive straight into the deep end with this. You wrote a wonderful article. It's up on Valyria from who knows how long ago at this point but this is where we got stuck talking about this thing. In this

[00:02:51] article you let off with your first question of, well I guess explain what you wrote, give us the context then I'll ask you the first question and we'll talk about it. Yeah I just wanted a high level to give people some criteria to

[00:03:02] evaluate these things but I think the first thing you want to talk about is the idea that usually if something's too good to be true it is so let's look at two different things here. Let's look at the performance

[00:03:10] of mutual fund, active mutual funds over time. So let's say in general and I'm gonna get this wrong but it's good enough. Let's say the market produced 10% over time and let's say on a gross basis mutual funds produced about the same, 10% and then net of fees they produced

[00:03:23] 9% or something because they had a 1% fee. So that's the real world of people running investment strategies and trying to outperform the market. Now let's go into the back testing world. What is the average performance in the back testing world? Probably 20 give or

[00:03:35] take like this is the stuff I see. Typically the periods of struggle are much shorter or they don't exist at all. There's not much of a risk return trade off and so you always have to evaluate

[00:03:45] it from that perspective of thinking like you know you can I can go produce you back tests right now. It'll have 20% returns. They're not gonna they're not gonna have 20% returns in the real world. So you almost have to start from this context of like what

[00:03:56] of the best people trying to run active strategies in the real world? What have they produced? What have these backtested produce backtest produce? Like why is there that huge gap here? And that gap is what should make you very skeptical when you

[00:04:08] see any backtest because it's very easy to run these that have great performance but you have to look at it and say why if this is really what this person is capable of doing why is nobody else doing that in the real world? You know that to

[00:04:21] me is the starting point before you start thinking about how to evaluate them. This applies across the board in the financial planning space in the individual client no matter no matter how many commas are after your name. This applies to your life in

[00:04:36] the advice industry because there's way too many people who have way too many amazing backtests and amazing sounding ideas that they will present to you as if they have hard data for it. And if you're not I don't want to say

[00:04:52] trained but if you're not experienced in having a conversation like what we're about to have if you're not up on how to just ask questions about this at a minimum to say is this too good to be true? But then at

[00:05:05] a maximum to start to unpack some of these levels so that you can actually appreciate some of the nuance because I'm gonna just say this bluntly like that 20% backtest might be too good to be true but it also doesn't mean it has

[00:05:17] like no merit to look inside of what's going on there. Is that fair to say? Yeah you want to look at the details I mean the 20% itself may not have much merit but what they did could have merit like it's possible like a strategy

[00:05:30] that's back tested at 20% will actually outperform the market going forward just not by that much. So there might be merit in there but the thing you have to always look at and for people like me who build strategies behind the scenes you become very familiar

[00:05:42] with this which is the idea that I can keep tweaking my strategy all day and getting better results and I could convince myself that those results actually are predictive of the future and that's the hard part about it and that's why you see these 20% back

[00:05:55] tests. These were not like people started we'll talk about economic principles and stuff in a minute but these weren't like people started with these economic principles they ran one test they got this great 20% and they moved on with their lives like this was

[00:06:06] oh well what if I what if I adjust this a little bit and what if I adjust this a little bit like that's how these things play out and before you know it you've made the 20% but you've done things that

[00:06:15] are not predictive of the future and that's like for someone who's behind the scenes making these you realize that there's that temptation to do it and you realize that most of the people that are putting these things out because you see like

[00:06:24] ETFs can't do it specifically but the ETFs that follow an index can do it like the index can have a back test and you've got to be really careful when you go to the indexes separate website where they show that back test you got to

[00:06:35] be really careful about what is in that back test and do I think that actually is going to carry into the real world. So you're saying my two times levered Dave Ramsey mutual fund ETF where you can get to three times if you just

[00:06:48] never pay your taxes you're saying that's not a winning marketable strategy. Probably not. And the other thing that's good to ask is like what is you know what is the actual data in the back test is one thing of people if like let's say they have some

[00:06:59] data set that nobody else has access to and they've run a back test I might have a little more confidence in that but like if that back test is the P E ratio and earnings growth and the current ratio or whatever like everybody

[00:07:09] in the world has access to this data. So like how is it that like this person who's putting this random back test out there has found a way to take this publicly available financial information and you know beat all these computers that are analyzing on a daily basis

[00:07:21] all these smart people that are analyzing on a daily basis. The answer is they haven't what they've done though is they've continued to play with that criteria because you'd be really shocked like if you run a back test and you say all right I

[00:07:32] want P ratio less than 10 and earnings growth greater than 15 percent and whatever else like as you start to play with that. No we know what maybe I only want 14 percent earnings growth you know what if the P is less than eight that's better than 10 like

[00:07:43] it is so easy to go down that and you can make your performance way especially these focused strategies don't return that many stocks like just getting the right stock in there can change everything you know getting GameStop in there you know in 2021

[00:07:55] or whatever can completely change your entire back test. So it's just important to understand like is it likely this person who's showing me this back test has found a way to take this data that's available to everybody else and like use it in a completely different way

[00:08:08] that nobody else has used it in and generate 20 percent returns. It's just very highly unlikely and important to add even in those private data sets because we're seeing more and more and more and more of this especially in private markets seeing a ton of this in venture right

[00:08:23] now and I'm not saying it's I'm not saying it's bad. It's one thing if somebody is making these claims on publicly available data. So if anybody can go out there and find the data to your point they might be mining or

[00:08:36] doing some shady things to get to those numbers are not genuine things to say this is reproducible. However we are seeing some crazy numbers from what people are doing with private data as well. Some of that may be real some of that

[00:08:48] we're going to believe gets arbed away over time but this is why having the language to ask these questions is so important because it shows up everywhere and you have to have some framework in your head. You have to rumble through enough of these conversations be I'm saying

[00:09:05] tortured through enough due diligence meetings but that's just my experience with people far smarter than I who can ask the right questions but you have to be exposed to this stuff to know. So I think let's start with your first question. Let's start with one of the questions

[00:09:20] you said to ask first is does this test does this back test account for never thought you would have started here. Does it account for human emotions. Yeah let me before I answer that I also want to just say you know one of the

[00:09:32] things I've probably been a little bit too negative on back test here from the perspective of I don't think they're complete garbage. I don't think there's we wouldn't be asking these questions if the answer was just throw out the back test and

[00:09:41] never look at it we wouldn't be asking these questions because you could just throw them out at the beginning and not ask any questions. There are things you're still a quant you haven't announced your quant religion just yet. Right. And properly done back test can add value.

[00:09:53] It's not like every back test you know I just you know unfortunately in terms of what you see what the end user sees a lot of the back test are not well done. And it's just a matter of there's questions you can ask where you can

[00:10:04] learn information from back tests. It's just the 20 percent return is not the information you want to learn a lot of times a lot of times it's a lot of other stuff and so this what you just said is a really important thing which is let's say

[00:10:15] let's say this is not an investment strategy but let's say Amazon was an investment strategy investing in Amazon and that was an investment strategy that back tests exceptionally well like if I bought Amazon at its IPO and held it till now in the returns of that are

[00:10:27] better than any back end or what tens of thousands or thousands of a percent whatever it is it's way better than any strategy I could ever back test. But could your average person have held Amazon to achieve those returns. And the answer is almost undoubtedly no.

[00:10:41] There were multiple 90 percent drawdowns across that period. You had to sit there when everybody you know when people thought Amazon was just a book seller when Amazon was going to fall apart like all these problems that you know Amazon will never make money. Amazon is not actually profitable

[00:10:54] like all this stuff. You had a combination of the strategy or the stock is doing horribly and you've got these facts that say you shouldn't be invested in it. You had to sit there and stay through it. It's the same thing with back

[00:11:06] test as you have to look and say as a human being could I you know with my own personal risk tolerance could I have sat through what inside of this back test. And this gets into the details of what we're talking about before is it's not just the

[00:11:18] end return of the back test. It's the ride along the way. And so one of the things you can do to analyze a back test is you can say all right maybe I have this great return at the end but what did it look like in the middle.

[00:11:27] Like was it down 80 percent in a given year. Well can I you know was it underperforming the market by you know 30 percent for five straight years. Like when there's you know could I have endured these things I have to endure to get to the end result.

[00:11:40] And that's what a lot of back testing misses. That's what like those long term charts of the S&P 500 miss is like you know you've got these great long term charts and you're like obviously you should buy stocks that's fantastic. And 2008 is like this little

[00:11:51] move down that like he's going up like you can't. That's not the real world. The real world is you're going to have periods where you are panicked or you think this thing doesn't work anymore. And like if you can identify those periods in the back

[00:12:02] test and if you can say could I have stood through you know could I have sat through that. And it's hard to do that in retrospect but to look at those down periods and to look at where your emotions would have been their highest

[00:12:12] and say could I have actually followed this back test I think is really important to do. So my amended strategy where I just buy the 90 percent dips you're saying that I can do that to improve on your model. Yeah you miss the downtown

[00:12:25] and just buy you know by the dip somebody had DTFD man. Somebody had like the greatest hedge fund strategy ever something which is tongue in cheek might have been Barry Ritholtz years ago who was just like this is what your returns would have been if

[00:12:37] you just bought Amazon every time it dropped 90 percent. So just a funny story and this like one time I was running like a crazy back test on my database and like I was leaking like some data in the future with data in the past.

[00:12:49] I forget what I was doing but somehow I managed to use like the relative strength one year forward is my criteria. So effectively I was just buying stuff that I had no idea was going to go up but actually had gone up and I

[00:13:00] got the results back and it was like 10 million percent it was like this ridiculous thing. Something has gone horribly wrong here. Like obviously I never published it or anything but it took me a long time to dig into it. I'm like this is not right

[00:13:11] and like that's why it was as I was I was using like one year forward like the returns like one year forward that I obviously would have known no way to know so I was just buying the stocks that have gone up the most.

[00:13:20] So you made a strategy that looked forward a year in time said what did the actual best and then made sure you owned it in real time. It was like it was like the time machine strategy you would call it like basically just like take whatever

[00:13:31] ended up doing best and just buy that so you were you were biff with the sports all the neck and back to the future. You achieved that. That's amazing. And that's the thing behind the scenes of these a lot of times is like when you

[00:13:42] are running them like you know you sometimes you look at it you're like something's wrong with this and like you got to dig into the details and you got to figure it out and like some sometimes a lot of those people don't do that and

[00:13:52] they make it to public. You know the back test that just have obvious flaws and we're not going to get into all those reasons here but they make it public. So you know the idea is just to be skeptical of this stuff. But if you want to invest

[00:14:02] in my one year forward relative strength strategy it's a it's a very strong one. It's produced incredible returns. You just have to have a way of knowing how the stock's going to do in the next year in advance. Well make sure you sign me up for that.

[00:14:12] I want to I want to add I think especially when we talk about the human emotion aspect something that everybody who's ever invested a dollar for themselves can relate to. If you're only investing for yourself so you're not investing for your spouse or your kids or

[00:14:27] whatever else you either have a play money account or maybe you're I'm thinking of friends and family who are like in their 20s and they've like either done some YOLO stuff or they've done whatever on their own just because they're curious and investing. You know somebody gifted

[00:14:40] them Disney when they were six years old and they still had it. Whenever the case may be there's a psychological shift when you're doing something just for yourself and you're not managing money for anybody else where you start to think about stuff like this.

[00:14:54] As soon as you add a single other entity or a stated purpose that you're investing for. So yet you add a spouse or you have a kid or you think about I need this to buy a house or whatever else your entire human interaction with those invested dollars

[00:15:12] changes. This is such a basic concept. There's a million behavioral reasons that go into this but it's the idea that no matter what strategy you start out with as soon as a new piece of information enters not just the data set but as soon

[00:15:26] as a new piece of information enters your life you are going to be changed in your approach and understanding of that stuff. Something I see all the time in talking with individual clients is they'll have had an experience from something a year ago or five

[00:15:42] years ago or whatever else or they'll have some new goal in the future and they'll go well I remember when I could have bought Amazon at the IPO or I remember when I could have bought more NVIDIA before the pandemic or I remember when I

[00:15:55] could have could have could have could have that idea of having an old memory on something you didn't do or didn't do enough of and a forward looking projection on something I'm trying to achieve and how it can be connected to that potential future outcome. It's a myth.

[00:16:13] You got to break this part thing apart. You have to recognize not just when new information is coming into your back test, but when new information is coming into your life and your experiential relationship with the data, it's gonna mess with your head over and over again. Inescapable.

[00:16:32] How do you think about that? Not just the new information in the back back test, but the new information into you like your lived experience. Yeah, but what's also what's also interesting is use that word could have like that could be the the word you want to use

[00:16:44] to describe back testing. Back testing is what I could have done like back testing is not like what I'm doing today or what's going to work going forward. Back testing is going to the past and say I could have bought these stocks with these characteristics

[00:16:56] or I could have had this asset allocation or I could have run this system. And this is what might I mean, you could say this is what would have happened in the past, but it doesn't tell you much about what might happen in the future.

[00:17:06] So I think that's a great word you came up with like to describe back test. Like if you think about the word could have every time you think of a back test, you'll probably think about it in a different light than like what actually happened in the past

[00:17:18] because it's a different thing. It's a different thing to have real money on the line and be making real time decisions along the way about what to do. Then going back and saying, all right, this is what could have happened in the past and therefore like

[00:17:30] this is cast in stone. It's the same thing as if I had been making those real time decisions because those are two very different things. Like even even for quants like me, I mean, people think quants are like not subject to emotion. Like there's emotion everywhere with quants.

[00:17:44] I mean, I'm affected by all when I build an investment strategy, I'm affected by all the same things everybody else is. When I decide whether to change that investment strategy, I'm affected by all the same things everybody else is. And in that back test, there would have been

[00:17:57] periods where I'm like, you know what? Maybe I got to change this thing. It's not working. Like I got to tweak this thing. Like that is not reflected in the back test. Like any role of what the actual person running the strategy may or may not have done

[00:18:07] is not reflected in the back test. And I think that's really important to keep in mind. Like this is not some hard, you know, cast in stone thing as to what would have happened in the past. Like there's so many variables, including emotion, that go into

[00:18:19] what might have happened if somebody was actually trying to follow that strategy then. This willingness and ability to take risk idea. This is what I what I frequently use with clients is kind of like a could have, would have, should have test. Or it's basically like,

[00:18:33] could you take this? Meaning like, could you actually bear to take this risk? Would you take this? Meaning like under what circumstances would you actually entertain this option because there's probably a big difference in. I don't know, you might go skydiving when you're 22, but not when you're 65

[00:18:50] or something. I wouldn't be skydiving at any age, just so you know. You want to be both. And then should you? Which also like again, to use the skydiving example, like should you do this thing? It's like, well, if you're 60 with a heart condition,

[00:19:03] maybe don't jump out of planes. Like it's not a should thing, even if you would because you're a risk taker, even if you could because hey, you're standing at the, you know, the airport with your 22 year old friend and they're about to do it.

[00:19:14] By the way, there's just a, just as a side on that, there was a video going around on Twitter recently about like the guy that broke the record for like the highest skydive without a parachute. It was like 25,000 feet or something.

[00:19:25] So it was the one who like jumped from like the Red Bull thing or he like jumps out of like the International Space Station practically. Was it that guy? I mean, it was more like he didn't, it was like he was still at a normal like airplane altitude,

[00:19:35] but he had no parachute and he like had to, he landed on a net. So like the guy's in complete free fall with no parachute the whole time. And he has to like hit this net like on land. Exactly right. And I was just like,

[00:19:46] there is no chance in hell I would ever, I would ever consider doing this. Like, I don't even know how with all the winds and everything you possibly can make it. I mean, you can steer I know to some degree when you're up there.

[00:19:56] This is Matt and Jack getting on their total side. That has nothing to do with what we're talking about. But I was just, I was like, that's an example of like a risk Jack would not be taking. I'm right, I'm right there with you. So asking,

[00:20:07] I guess asking those things to understand the human emotion, the human feeling, the human input to this. If you could, should or would take that risk. And then how is that playing into your understanding of the back tested, the prior information? That's big. So, all right,

[00:20:24] jump to the next one with me here. Does the strategy have an economic reasoning behind it? So this is one we're define this for me. This is one where you and I can go off on a tangent and probably finish the entire podcast here

[00:20:36] because this has actually been called into question a little bit. Recently, you know, we did. So let me say what it is first, and then we can kind of talk about why maybe it's been called into question. So the idea is I want to start

[00:20:46] with some sort of reason as to why I think this might work. So like whatever it is, like if I think a stock, you know, that like a value stock is likely to outperform because, you know, of all the reasons we talked about another podcast,

[00:20:57] the value stocks likely outperform or, you know, stocks with certain fundamental characteristics are likely to do better than other stocks. I want to have that first. So I don't want to start again going back to our data mining thing before. I don't want to start running

[00:21:09] random tests on fundamental data. In theory, we'll get back to this in a second. But like this has been the way we've all been taught to do it, which is you don't want to just start running random tests. You want to have a theory. Here's why I think

[00:21:20] whether it be multi asset class, individual stocks, whatever it is, here's why there's a foundation as to why I think this should be working. And then you want to build your test. And the idea is those are more likely to hold up in the future

[00:21:32] because that economic rationale will still be driving it in the future. So that's the idea. In terms of this now, we can get into the other part of this, which is we had Alejandro Lopez Lira and Andrew Chen on our on excess returns.

[00:21:45] And they looked at this idea. They looked at these strategies. They kind of did a couple of different things. They looked at strategies like factors that had an economic rationale behind them. And they looked at, you know, after they were identified in the data,

[00:21:58] how do they do going forward? And then they looked at ones that had absolutely zero. They just ran a data mining exercise and said, you know, these strategies have no economic, you know, whatever it is like dividing a balance sheet number by a cash flow statement number,

[00:22:09] something that should have just absolutely nothing to do with itself. And they looked at both of them. And what they found is in the future they were either indistinguishable or the ones that had no rationale actually outperformed by a little bit.

[00:22:20] And so that research has kind of put all this, you know, turned all this on its head, whether this economic rationale is important. But in general, like the idea was you want to have some reason you think it's going to work. Then you want to test it.

[00:22:33] You don't want to just start throwing a bunch of numbers around. There's a point in this that I often think of. And again, in communicating with this with clients where we're not just dissecting the math. So not with the engineer clients, but with, you know, the regular humans.

[00:22:49] Well, we're talking about this stuff. There's a real importance in understanding the role that intuition and to a degree, maybe even curiosity plays into this stuff. So. You have intuitions about things that you're thinking about when you're analyzing these sets, when you're and it's as simple

[00:23:06] as when you're committing to or saying I trust my advisor to put me in a growth fund or a value fund. It doesn't matter anything in the world. Those intuitions on is this making sense to me or is something fighting me is important to pay attention to

[00:23:21] because then we can back out to that first step of saying what emotions, what feelings, what prior experiences are maybe influencing this on the human level versus what are my intuition tuitions telling me on a more quantifiable mathematical or whatever level to say,

[00:23:36] like this back test is telling me it is actually pretty smart to eat the broccoli and not just live on chocolate cake alone. That balancing act is really important. Asking if it has economic reasoning, not just going to the pure it makes no sense.

[00:23:54] So I'm going to follow anything and we are going to talk about this little bit more because it's so interesting. But that opens up the opportunity to both respect what your gut says to you. And then also the the I believe it's Cliff as the as this ism,

[00:24:08] the sin a little idea. It's it's also OK to say here are my core beliefs and here are my core things. But I can send a little I can do some things that actually may maybe don't jive with the exact back test

[00:24:21] or what I believe to be true or whatever else. But I'm willing to experiment. Back back scratch the back test scratch that itch a little bit. Get outside of the normal box. So long as I'm not doing it in a way that's detrimental to

[00:24:37] the rest of my potential results. How do you think about the like the sin a little part of maybe it doesn't all have to have a reason, but I I feel like most of it needs to have a reason you understand. Yeah, well, let me give you

[00:24:50] the other argument and then we can kind of debate it. Like so the flip side, the Alejandro Lopez, Lira and Andrew Chen argument is is this to some extent? Like once these things that make sense, they get identified in the academic research. They become public. What happens?

[00:25:05] Everybody or a lot of people start following these things that make sense. They will price up those things that make sense. So your forward return is not as good as what you would have expected in the data. And if you start dividing these random numbers

[00:25:20] or not not rampant, we're talking about stuff that's on a balance sheet or on a cash flow statement or on an income statement. We're not talking about some random data that should have nothing to do with stocks like they started with a data set that in theory

[00:25:31] you would argue like stuff that's on these financial statements should have some sort of impact on stocks in some way or another. Like they weren't just off on some random tangent. But if you divide these other variables that may not make as much sense, though people are not

[00:25:45] chasing that stuff. People are not following that stuff. So you might have a better opportunity. And also we did this thing with Alejandro like when we called the lightning round, which is we gave him some of these random combinations and he was able to come up

[00:25:57] with an economic rationale after the fact. So maybe it's not that they make no sense. I mean, maybe there is some degree of sense in some of these dividing of these things. And you're in like more uncharted territory. You're not you're not using the price to book ratio.

[00:26:10] You're a little bit more uncharted territory or maybe other people don't understand that there's a relationship there and you do a little bit better. So that would be the argument on the other side of this. And their test was done very, very well. Like they did.

[00:26:22] You know, they said, here's what I knew in the past. Like, you know, they only you know, they they looked at it in sample, then they looked at it out of sample. They did all the right things with their testing and they still came up with these results.

[00:26:34] So it has to some degree call into question this economic reasoning, although maybe some of these variables that we don't think make sense. Maybe they actually do make a little more sense than we thought. Or do they? So this is a genuine question. Do they just make sense

[00:26:47] for the moment? Like if I decide stocks go up every time a Nicolas Cage movie comes out or something like that, like I pick the most conspiratorial drunk uncle theory I can come up with and I'm like, this is the basis for this week.

[00:27:01] Is it one of those things that you come up with the crazy idea? It's right once, maybe it's right twice. And then like, is this did they talk about does it depend on all the other machines like training their algorithms on these things?

[00:27:16] So like you just have to think of the crazy thing first and then if you get lucky and it's proven right, then a few other machines will try it at least for a period of time until the incremental differences are away. The one thing you have to think

[00:27:26] about is these were both these were long in sample and out of sample periods. So like they were looking from like basically Fama and French published in the early 90s. So they were looking from the early 90s back and then from the 90s forward.

[00:27:37] So both of those periods have substantial amount of data. So the idea of like some random thing that it's possible some random thing would work that has nothing to do with stock performance, but also they are long periods of time and it's a well done test.

[00:27:50] So you'd think there is some statistical significance in what they're saying. You know, I mean, I couldn't really explain it and I don't have the answer to this. I don't think anybody has the answer to this. It's an interesting thing to think about,

[00:28:00] but you can make an argument that maybe my number two here, you know, I wrote this article a long time ago, like my idea that the strategy as an economic reasoning maybe is not as strong as it was in the past. And, you know, you've seen this

[00:28:11] on the short term side for a long time. Like there's a quote from Robert Mercer at Renaissance, which is like our signals that have worked the best over the longest period of time don't make zero sense. Now they're they're in and out. Like they don't have like a

[00:28:23] like a second holding time, but they do have they have shorter holding times certainly than your long term investor. But they found that like the stuff doesn't make sense to other people. Like they just drive themselves with the data and they found like the stuff

[00:28:34] that makes sense to other people, you know, doesn't necessarily work as well for them. So maybe we're seeing that carry to like the longer term space that we do, that we operate in, that, you know, some of this stuff that we where we feel like

[00:28:44] we need the economic rationale. Like obviously there's a million people out here trading stocks and they're seeing that same economic rationale. Maybe if too many people are following that economic rationale, the economic rationale becomes less important. I don't know the answer to this.

[00:28:57] It's a really interesting to date, but you definitely could argue of my things, my five points I have in this article. Like this is the one that is more most called in the question now relative to when I wrote the article. Well, let's keep calling things

[00:29:08] into question over time. What what fun would life be if we weren't destroying that the things we the very things we held sacred? So let's let's jump to the next one. Does the test cover a very long period of time? You kind of just address

[00:29:24] this in the last one a little bit, but I guess we're kind of addressing everything in all these because it can't help it. But this time period thing is one of those you give as much of the mathematical explanation as you want. But talk me through why

[00:29:37] the time period question so important. Well, I think what's important here is the separation between what a very long period of time actually is and what your average person thinks a very long period of time actually is like Medfavor has done a bunch of work around this

[00:29:48] and like he'll start asking people, you know, well, if you underperform for one year and this is not in backtesting this just in the world of like managers, if you underperform for one year, what are you going to do? You think that tells you a lot about

[00:29:58] what's going on with the manager. How about if you got to perform for three years and like what you get to and what he got to is in terms of real world results, like you're probably looking at 20 plus years before you can fairly evaluate any investment strategy.

[00:30:10] But no human being in the world actually has 20 years where they're going to be like this thing is underperformed for 20 years. I'm going to stick with it because the fair period to evaluate is longer than 20 years. Nobody does that. And so that's the big challenge

[00:30:22] is there's a separation. You know, you want the longest period of time you can have. And, you know, one of the things we've learned recently with what happened with stocks and bonds is if for 40 years, stocks and bonds, you know, are correlated with each or uncorrelated

[00:30:34] with each other and then they become correlated with each other. Well, my 40 year backtest is not as strong as I thought it was. And 40 years is a really long period of time. But like if we go into an inflationary environment and that correlation changes, well, that backtest

[00:30:49] didn't tell me that much. And so the idea is you want really, really long periods of time. And also those long periods of time are probably a lot longer than your average person would think is a fair period to evaluate anything. What about the shortage of?

[00:31:05] There's not a long a lot of long periods of time, but then there's also like a shortage of the measurement periods where we're looking across these. So like there's only so many 30 year rolling returns, for example. There's only so many ways to slice like you start

[00:31:18] and end month. How do you think about that? Yeah, like the end is what you're getting at here, like the end of the sample. Yeah, and even, you know, if you think about like a lot of people who do more advanced testing

[00:31:28] will look at like all the data we have on stock returns and be like, that's not nearly enough data. Like, you know, people who look at especially for some of these advanced machine learning applications, they'll be like you don't really have that much data there.

[00:31:38] Whereas we think we have all all this stuff. And like you said, I mean, if you're looking for a 30 year rolling period, like how many 30 year rolling periods are there? Like I got to go back 30 years before I could even start having rolling periods,

[00:31:49] you know, and then I got to go back. So there aren't that many. And also, when you think about data, like the quality of the data we have today is very different than the quality of data we had 30 years ago and 60 years ago.

[00:31:59] And, you know, what you can do becomes much more limited as you look back because you don't have all the same variables. Like, you know, I'm not going to like earning if I'm running like an earning estimate revision strategy, I'm not going to run that back to 1927

[00:32:11] because, you know, what data do I have on like earnings, estimates revisions from 1927? I don't have any. And so like that's a very important thing to understand, too. It's not just like how much data do I have, but also like the data points I'm looking at,

[00:32:22] like how far can I go back with those individual data points? And then that starts to constrain the period I'm looking at because of that. Of the one of my favorite examples of this, and I have to go back to 1994. And if even 20 years ago,

[00:32:37] you told me when you talk about 1994, the top three things you're going to talk about are, you know, Nas's Illmatic coming out, the movie Pulp Fiction and William Bengen's Determining Withdrawal Rates Using Historical Data Paper and the Journal of Financial Planning. Things obviously related to each other.

[00:32:52] Clearly those three things go together just swimmingly well, but I probably spent more brain energy thinking about these three these three things that I'd care to admit. So Bengen in this, that's the 4% withdrawal strategy paper. Everybody's seen it. We all know it. One of the problems,

[00:33:10] I don't want to say problems. One of the, well, yeah, one of the problems with Bengen's paper is that he's looking at these 30 year periods. Well, when you start in like 1954 and you play it out forward, I mean, we only have like 70 or so

[00:33:22] of these like 30 year periods when you slice the data. So does it cover a long time? Will we go, OK, 30 years average retirement, high success rate, whatever else. But we don't have that many cases. We basically have a sample set that says like, OK,

[00:33:38] we still have like less than 100 potential potential annual year retirees doing a thing in a very particular way. And we go, well, two of them. It didn't work out. And there's a whole bunch of others, you know, like the 1994 forward still in 2024 here.

[00:33:56] Like we don't know the outcome. That's a really important detail to just remind us. Yes, we have these rules. Yes, we have these ideas as we have these concepts of why we think this stuff works against some historical record and analog. But. Do we really think

[00:34:16] that that's representative across all the environments, all the experiences, all the things there is a thing to, by the way, just to point out with these rolling 30 year periods, you have to keep in mind is there's massive overlap in them. So like if I start

[00:34:30] with the rolling 30 year period from 30 years ago to today and then go back one year, well, 29 of the years are essentially the same. So it's like I just have to understand like these are this is not like one independent rolling 30 year period to another independent rolling 30 year period.

[00:34:43] If you want truly independent rolling 30 year periods, then you have like almost nothing. Then you have very little to work with. So it's just important to understand like there is there's a bunch of overlap when you look at stuff like that. The heavy bias inside

[00:34:54] of the data set is pretty remarkable. And yes, we have economic reasons. We have stuff that we tell ourselves this is why this is OK. This is why we believe it to be true. This is why we believe it to make sense. But to your point, this isn't

[00:35:11] these aren't independent data sets. 1993 through seven has a lot of things that rhyme with each other, just like 1982 through 89. I'm arbitrarily picking yours like don't actually read into this. But like these there are whole chunks of time where trends persist, where ideas persist, where investment styles

[00:35:30] and other things persist. And we can come up with these things like the 4% rule. But when we apply these backtest questions to it, we just have to say these aren't perfect. They're a good idea and concept to just help us in the way we think about them.

[00:35:48] But they're not actually going to solve that problem for us. Not the least of which I'm going to jump to your next question. Does the test show periods where the investment strategy struggles? I'm going to keep the banging thing going for just a second.

[00:36:03] Do you remember the two failure periods in the 19 whatever 94 paper? I don't know. Funny enough, so it's 65 and 66, at least according to the note I made to myself a year ago last time I really thought about this hard. But it's no mistake that those two failure periods

[00:36:22] are next to each other. You should always see a struggle in the backtest. How do you think about this? And let's refine this for a minute. Yeah, this is the best thing you can do with backtests is you can learn so much from what's going on

[00:36:34] behind the scenes and not just the end result, like what's going on in the middle. And the first thing you have to say is if the period of struggle does not exist, throw it out. You know, that's you see and you think I'm joking,

[00:36:45] but you do see that a lot. You see like these these strategies that just, you know, there just is not the period of struggle. It just does exceptionally well all the time, no matter what. And like if you see that toss it together. You haven't met

[00:36:57] my advisor, Mr. Bernard Madoff, his drawdowns, they're just they're just not there. You just you don't understand because you haven't met Bernie yet. Let me just introduce you. Yeah, it was exactly. That's a great example. Also, my one year forward relative strength strategy from before

[00:37:12] that did not have any periods of struggle. But it turns out that that was not a legitimate strategy. So it's just if you don't see any periods of struggle, then that's obviously a major problem. But inside of that, like I want to learn when does this work?

[00:37:26] When does it not work? What are the bad periods look like? What are the good periods look like? Like there's so many questions I can ask behind the scenes, particularly on the sides of struggle, because I can understand in what type of market environment

[00:37:38] does this strategy not work? You know, what are the really bad period? What can I learn for them? How can I blend them with the rest of my portfolio so it does well in those periods? Like that's where you learn. You learn behind the scenes, you know, again,

[00:37:50] assuming there are periods of struggle, which there should be. You understand like what's going on in those periods and what can I learn about this strategy in terms of when it does well and when it does poorly? This idea of doing the backtest to actually identify

[00:38:05] the struggle periods I think is a profoundly useful topic. This is the bulk of and it's the counterintuitive bulk. People think like you run the financial planning, future projections and whatever else, just to say like, oh, la la, I'm a millionaire in, you know, a year,

[00:38:23] 10 years or 30 years. Like look how much money I have when I extrapolate all these returns forward. And yes, it feels good. But what we regularly remind people of is this is a made up future. This is a fairy tale. We don't actually really care about the right tale.

[00:38:40] We don't really care all that much about how great things look. We're running those tests, those forward extrapolations to actually look for the trouble periods to go in that period of time when you or you and your spouse are basically in your early 60s.

[00:38:58] You're both going to retire and you'll have to pay out extra for health care because it will be pre-Medicare and you've got this like window of your income is going to go to zero and the tax, which accounts are we going to pull stuff out of

[00:39:11] to make this stuff mean? And you watch where on the forward projection, stuff gets really stressed out. The same is true looking backwards and the same is true of this is why in the forward projections we get to see those periods of time. You will learn more

[00:39:26] from the bad periods than you will likely learn from the good period because finance at the end of the day is a game of survival. That's what we're doing. Do you have any good examples of basically like backtests with horribly bad or thinking like all the

[00:39:41] tail risk strategies, like every tail risk strategy basically is like here is 2000 paper cuts. You will probably not bleed out and then suddenly you will be rewarded. What are some examples that come to mind? That's a good example of how to think about a backtest in the context

[00:39:56] of your overall portfolio because, like you said, I mean, there are some newer tailor strategies that are doing a little better job of managing the bleed. But for the most part, like the tail risk strategy, the long term performance of these things is not good.

[00:40:07] They're going down over time. But like if you put that into your investment strategy and in 2020 you add a small allocation to the tail risk fund and you rebalanced it into your assets that were down, that portfolio or that strategy can add to your performance

[00:40:24] even though it has horrible long term returns. And so that's a really good example of how you would think about like I don't just have to look at this on its own island as a standalone thing. I also have to look at it.

[00:40:33] How does it impact my portfolio? If the tail risk strategy has a negative return but is going up 10,000 percent in March of 2020 and it allows me to buy a bunch more stocks, like that actually can add to my return even though it's on its own

[00:40:48] has a bad return. There is the wacky reality inside of that. And this is, I guess, it's one part of financial planning and another part of portfolio theory point. There are things that can do really well in a in a single punch. And let's use the tail risk

[00:41:04] like 2020 strategy. So you could have something that is just down, down, down and down slowly, not plunging down, but just a gradual erosion of value that is a really sharp uptick as a spike. One of the great values of the tail risk type things

[00:41:20] if you're going to use them and most people shouldn't because it's going to drive them more crazy than it's going to help because those periods can be so well, they're visceral in a number of levels. But on that rebalance, so that five percent allocation

[00:41:36] to the tail risk strategy that goes up a thousand percent or whatever number that affords you to rebalance in such it in a much more aggressive way when the 60 percent stock allocation is off 20 percent in the month's time or whatever it is. The correlation of that

[00:41:53] sharp decline and sharp increase that allows you to do the rebalance can make up for all the misery in the however many preceding years where again, just paper cut, paper cut, paper cut, paper cut. Why do you keep on picking up these pieces of paper without properly

[00:42:09] moisturizing your hands? That's what probably like for people who people who have what they call line item risk. So people who don't just look at their whole portfolio, but they look at the line items like that is tail risk is probably the most difficult behavioral strategy that exists

[00:42:21] because you are losing and you think about it like I mean, you got that huge gain in 2020, but it's been bad since then pretty much. And during a lot of periods where people thought it should be good, like 2022, when you get the big decline in the market,

[00:42:32] like that type of stuff didn't really do very well because we didn't get that big spike in volatility and stuff that we expected to get. So like that's a very those are a great strategy. If you're looking at your overall portfolio, you have a small allocation to them

[00:42:44] and you're not worrying about it. They're a horrific strategy if you start, which most people do. You start picking apart like the line items in your portfolio. And one other thing I want to say about this whole periods of struggle thing that I think is really important is

[00:42:56] the periods of struggle should occur when you think they should occur. And what I mean by that is I'm being sold an investment strategy that's going to do certain things that's like if it's a value strategy, it's looking at certain value characteristics. So let's say like Matt

[00:43:09] is a big believer that the price to book is about to come storming back. And then so I'm ready to sell you my price to book strategy and I give you a back test that basically says over the past 20 years, here's my price to book strategy

[00:43:20] that's just destroyed the market. You should basically run as far as possible from that strategy because the reality is it is not a price to book strategy. Like the strategy, a price, a low price to book strategy I would expect to be doing well

[00:43:33] when value is in favor, when low price to book strategies are doing well. Like that's what I'm buying it for. I'm buying it to perform well in that period. And the problem is if you find the person who's selling you the strategy has done well

[00:43:45] in the past 20 years with that type of strategy, what's it going to do if price to book actually comes back? If you're right about that, it's not going to do what you think it's going to do because it's not it hasn't done in the past

[00:43:55] what you thought it should do. So like asking the question of what am I being sold and when should that do well and when should that do poorly? Because obviously if I'm investing in the strategy, I believe in what I'm being sold. So I need to understand

[00:44:07] that if what I think is going to happen, the reason I'm buying that strategy actually happens, is this going to do well? And like that's where you can really learn is you can look through the details of the annual returns and stuff historically and say,

[00:44:18] did this do well when it was supposed to do well? That question of when does this not work with the follow up question of when this does not work, is there something else that I could be doing as an alternate to this one idea is also really useful.

[00:44:33] So as people try to sell you stuff, you know, price to book, I don't know when if or when price to book ever comes back. We far better off with price to books where you just basically look at how much I spend on Amazon and the correlation between

[00:44:46] Amazon stock price probably is more signal in that than price to book these days. But this idea that you're looking for what's the struggle period? What's one answer to it? And then actually asking what's a second answer to it too? Because you need to give yourself

[00:45:03] the room to start to unpack the struggle idea. So the tail risk thing, like if I have a tail risk hedging strategy, there's line item risk. There's a whole thing. I know that that tail risk hedging strategy is going to give me the pop in value offsetting,

[00:45:17] hopefully a good amount of its gradual erosion and decline from the time I bought it. Never buy a tail risk hedging strategy right after the tail risk has happened. That's another bloody quirk of these things. But you can look at that and you go, OK,

[00:45:31] that allowed me to rebalance or over rebalance in the crisis period if I did this right. But then you should also say, well, what if I just had the same amount or a smaller amount of money in cash or in bonds or something else

[00:45:45] coming up with not only the struggle period, the desired behavior, but then mapping out like what's two or three different ways I can do it is a really useful exercise. And it lays over to planning, too. It's the same thing of saying if like I lose my job,

[00:46:00] but my spouse keep working. It's the same way of saying like I retire in this year and then my spouse takes three more years till they retire or vice versa. You're staggering out these things with a series of what ifs, not a single what if,

[00:46:13] because it's that composite what if that actually helps you understand what works in your brain and what works for you and your plan. Yeah, the tail risk strategy is a great example, too, from that perspective, because let's say I look at the test of the tail risk strategy

[00:46:25] and it doesn't have that bleed. It has whatever produces small positive returns. But then when in the 2020 period, it doesn't go up that much. It does OK. Well, that's something I need to know. Like that's an example of going back through and saying

[00:46:38] is this truly a tail risk strategy? If it's if I'm putting it in my portfolio because I know in that type of extreme scenario it's going to go up a lot. Well, if it doesn't do that, then that's something I can learn from a back test,

[00:46:50] regardless of what the long term return of the back test is and all that other stuff. Like I can learn from that. I can say I'm putting this in my portfolio for this reason. When this reason has happened historically and the reasons are never the same,

[00:47:00] they rhyme, but they're you know, they're never the same. Like what is it done? Like is it going to do what I think it's going to do for the reason I put it in my portfolio is such an important question to ask. All right, let's

[00:47:09] let's dive into your fifth question, which is does the back test assume knowledge that wasn't available until after the fact? The answer is yes, it does. Because they yes, and they're ready. They always do. Like, you know, the thing can you unlearn what you know

[00:47:28] is sort of the bigger and you're better at the deep philosophical questions than me. But like, you know what? I know what happened in the past, Matt, I know I know that stocks and bonds had a great 40 year period. I know value investing has struggled.

[00:47:39] Like, is it is it possible for me to construct a back test and like remove all of that stuff from my mind, like to not even think about it? Because a lot of times people who have the best intentions with these back tests

[00:47:51] and you follow all the proper procedures still fall, you know, prey to this thing. Because I know, I mean, I know what worked. I can't unlearn what worked in like my starting parameters I set up are going to reflect the fact that I know that.

[00:48:03] And this is one of the hardest things to do is like you can't unlearn it. You know it. It's in your mind. Like so maybe computers can do better with this or I don't even know. But it's just something to realize. Like there are these things

[00:48:14] that happen in the market. We do know that they happened in the market. We can't unlearn it. Even guys like me that are building quant strategies. It's part of what we know and we have to do our best to try to manage it. But it's a problem

[00:48:25] we can't completely solve. This. The big one that I always think of when I think of new information or information that's not available after the fact is it's kind of what you said. It's the information that was available in the moment. People frequently have

[00:48:44] what I think of as the indefatigable first failure memory. And that's whatever your experience is experience was when you started investing, when you started putting risk on the line, in this risk for real usually. So not just the first dollar on the stupid thing you bought,

[00:49:03] but sometimes this happens too. But the first time you experience like a real downturn. So whether that was the crash of 87, whether that was the tech bubble bursting, whether it was the run up before the tech bubble bursting and then you get yourself,

[00:49:17] you get it served to you, whether it was the financial crisis in banks, whether Brexit, COVID. Take your pick. That first failure, that first face plan, that first thing is indefatigable in your mind. It will never get tired of reminding you of this crappy thing that happened.

[00:49:37] So when you start to think about stuff out of sample, like I should have known this or I could have protected against this or I'm never going to repeat this thing again. I'm never going to buy banks on some value metric before the financial crisis.

[00:49:52] When you start to think or speak in these absolutes, it's deadly. And when you start to think in like absolutes about the future from something that was a negative experience in the past. It doesn't mean you have to go do the I only buy stocks

[00:50:07] when there's a new Nicolas Cage movie out of some machine learning chaos, but it does mean we have to remember that we're just we're horribly biased creatures. We're going to be taking those new experiences and those new things, restacking them into that deck

[00:50:22] and getting in our own way of thinking about like how that colors our experience. Yeah, maybe the computers help, but I don't know. Yeah, like I was thinking about like an example, a good example to give of this. And like if you think about

[00:50:35] like if I'm testing a strategy, like a value strategy and I'm deciding whether to run it unconstrained from a sector standpoint or at least with loose constraints or tightly constrained and I went through, you know, it's a value strategy that tends to select a lot of financial stocks

[00:50:49] and I went through 2008. Am I going to put tighter constraints on financial stocks? I probably am because I realize what happened in even if it's in the back of my mind, you know, whatever it is, like I realized that like a an unconstrained strategy

[00:51:02] that bought a ton of financial stocks in 2008 got completely obliterated, probably beyond what we even thought it could get obliterated. So is that in the back of my mind? And so now I constrain the strategy more than I would have not having that information.

[00:51:16] The answer is probably, yes, I do do that. And that's the challenge. Like that's not some nefarious person with like their back test that they're, you know, just that they know is wrong that they're trying to just put out in the world to show their best returns.

[00:51:26] That's somebody who's trying their best to do a good job, but has this information in their head that they just can't get out of it. In like indirectly, it leads them to something that maybe is a little different than the test they would have run

[00:51:37] had they not lived through that. How do you think about applying like additional regressions or attribution analysis? So you do a strategy. You think about why it worked. You think about what it is and if you like it or not, what those variables are. But then you actually

[00:51:53] do another regression or you do another test to basically say, like, does something else explain this? And I'm thinking of the version you just said, I'm thinking about when this is like 10 years ago now, but it was in the post financial crisis period

[00:52:07] when there was a lot of money flowing into like what we think of as like the ESG strategies. And it was, holy crap, there's a three year and then a five year and then a 10 year number and they're doing really well. When you did the regression

[00:52:20] analysis against them, though, one of the most common explanations of these of these strategies was, well, they don't have energy stocks. The energy stocks have sucked. And it's kind of like if you just bought S&P and you left out energy, a lot of them,

[00:52:38] you kind of got the same return and they were benefactors of having the tech stocks and having the other stuff. And it wasn't until and this is like, again, like maybe five or seven years ago back where it was like, oh, but they got really heavy in banks

[00:52:51] and some other things like that. Like, well, is that a problem? So how do you think about answering the first question of like finding a strategy and why it works, but then finding other ways to shoot holes in it? How's your brain work around that stuff?

[00:53:04] Well, first of all, Matt, I was told that I was doing good for the world and we get a better return with these ESG strategies. So are you telling me I sold a false bill of goods? You know, I'm not saying you didn't do good.

[00:53:15] And I do think I'm not going to take it to rusty Gwyn levels of like burn corporate fiduciary board standards to the ground. But I am going to take it to the level of just saying when we allocate capital, extra Perth toll type stuff,

[00:53:31] like it has an impact that belongs in the conversation when you're being sold a strategy and the strategy is being sold as like you have an impact, but that's not actually front and center selling the product is front and center. There there is a difference and it matters

[00:53:47] that you ask those questions and understand that difference. So this overlooking the. So this gets to what we talked about last week, Corey Hoffstein's point about unintended bets like I'm intending in my back test to make a bet on a certain thing. But then these other things

[00:54:01] could actually drive my performance and I don't realize it. Like and that can be like you said, it could be at a sector level. It could be at a factor level. So one of the things you want to do is you want to look at

[00:54:09] those things, you know, historically and you want to say like how much were financial stocks driving my performance? You know, how much were other factors outside of the factors I'm looking at driving my performance because not that won't necessarily mean your back test is wrong or it's bad.

[00:54:22] It'll mean, though, if those unintended bets were driving my performance in the past, well, they're going to also drive my performance potentially in the future. They probably are. So I have to think about like do I believe a world in the future where those things are good bets

[00:54:35] is also a good world, not just the bet I was making because it's impossible. You know, these you can sort of do it, but like these pure bets are hard to make. Like you can't make a pure bet on one thing and you're not making any other bets.

[00:54:45] I mean, you can constrain your sector allocations if you want to. You can make them the same as the market. And you know, like my friend Westray, you know, our friend Westray would argue that you don't want to do that on a long only value strategy.

[00:54:55] And you probably do on a long short value strategy. He argued that when he came on the podcast. But so you can limit these things, but you do want to understand everything that's driving my performance and what are the bets I'm making because you can to some degree

[00:55:07] manage that historically and going forward, if you want to. If some of those bets are not the bets you want to take. But it's never a pure thing. Like, you know, you run your value test and it's like you just like this is fantastic. And then you realize,

[00:55:18] you know, you were 70% financials or something during a period where financials were doing really well. And you're like, well, that's not that may be not be as predictive of the future as I thought it was because I was making this other bet

[00:55:27] on financials and I'm not a believer. I want to make this big bet on financials in the future that that was just something that was a side effect of what I did. This no pure bets. This idea of like there's no all in or all out.

[00:55:39] There's always like a composite underneath and then understanding if you're betting and this very much applies to financial planning. Understanding if you're betting on some type of financial stakes or emotional stakes, like in front and then getting your priority stack right around these things. If you're

[00:56:00] wise enough, if you're big enough, if you're honest enough to say I can't control for all these things, I can have a back test to inform my best foot forward and then understand what what do I value in a priority stack of these things?

[00:56:14] You got you got a chance. You got a chance of surviving because I think what this opens up, the point you're kind of making underneath that right there is it opens up you saying I think I have a good idea. I have a good expression of this thing

[00:56:27] and I'm OK with struggling or being wrong about this thing because I know why I put this risk on and I don't want it to fail, but I'm comfortable with it failing because I've sized it right because I didn't take too much risk and because like I know

[00:56:41] where my values are in what I believe in for this thing. Yeah, I think the general the general takeaway from all of this is you just got to be skeptical of these things. You've got to, you know, in investing, like you've got to put your ego aside sometimes.

[00:56:54] You've got to always be asking what could go wrong? You know, what are the unintended bets? What am I not thinking about? Like you always have to ask those things. And if you ask those things about back tests, you typically are going to find some problems because again,

[00:57:06] why would someone make a back test that underperforms the market and looks horrible and put that out into the world? They wouldn't. So that's how you get these better before why back test is always outperform is because you don't know what went on before,

[00:57:18] but you know if what went on before was bad, it doesn't see the light of day. And so you just have to be skeptical about it. But you also can learn like there's things like we've talked about when it does bad, when you know, when it does well,

[00:57:29] like what other things are driving the returns? There's so much stuff you can learn even if you admit that there's probably some flaws in the back test that could at least teach you a little bit about what might happen going forward. I'm forever reminded,

[00:57:41] and I think I've brought this up on other I know this has come up at least on just press record. I can't remember if it's come up on other podcasts. We've talked about this that lives on my desk. My single of the TLC hit No Scrubs.

[00:57:54] Yes, we have. Yeah, so. I love the single, by the way. The single is just a wonderful word. It's an amazing word. I don't know if it's a real word or not, but I just love it. Oh, it is. It's a single mashed together for. Yeah, I know.

[00:58:06] It's a great combination of the two. It helped boost the album sale numbers when those things mattered for charts for a brief period of the mid to late 90s. In the ultimate back test, Chile from TLC, you know, in the statement of scrub is a guy who thinks

[00:58:20] he's fly, also known as a busta, always talking about what he wants and just sits on his broke ass that she's basically defining like her back test of what what a scrub is, the type of behavior a scrub is. And then like how to avoid that thing.

[00:58:36] And what's amazing is like she's saying this is how I avoid this thing and my value set, my priorities and how I'm going to do it. The other reality is the world is full of scrubs and the world is full of people who not only think they could,

[00:58:50] I don't know, cat call a girl on the street or whatever the story is here, but it's like all of people who are going to do like idiotic and terrible things. And even if you write a major charting pop song about the stupidity.

[00:59:05] They're still going to be out there doing that thing. So run on the back test, tell the stories, prove it all that you want. Chile tried to teach us all a lesson, even on defining a well-constructed back test with an explanation of the behavioral realities

[00:59:18] of what makes a guy a scrub. But that doesn't mean scrubs aren't going to exist. It doesn't mean GameStop is not going to work. It doesn't mean Amazon's never going to have another 90 percent drawdown or maybe it's happening right now while we record this podcast. I don't know,

[00:59:31] but it's a messy world. We just have to figure out the stuff that gets us through to the other way. The other side. I guess the question is how predictive is her scrub criteria of determining a scrub in the future? I mean, that would be what

[00:59:42] the quant would have to look at to see if that does carry forward in the future. If this is just another one of those back tests. I'm going to say by the guys doing the roof on the house, a few houses behind me

[00:59:53] in my neighborhood the other week that, yeah, yeah, I think it's very predictive. Much better in her case than in our space. It's very applicable. I heard things doing yard work. It was like, wow, somebody might need to knock on that neighbor's door or get chilly over here

[01:00:07] to lay down some law. So I said to you before this, there's no chance you and I are going to talk about for an hour about backtests. And I consistently underestimate our ability to just ramble on for an hour about any given topic. So we're actually going

[01:00:18] to go over an hour on backtests. So I will take that as a great achievement on our part. This is a great achievement. I think we are contributing to the modern pantheon of just quality content by talking about backtests until you devolve me into making TLC references.

[01:00:35] Well, if you stop with us this long, we greatly appreciate it and we will see you next week. Hi, guys. This is Justin again. Thanks so much for tuning into this episode. You can follow Jack on Twitter at practical quant. You can follow me on Twitter

[01:00:49] at J.J. Carbono and follow Matt on Twitter at cultish creative. If you found this discussion interesting and valuable, please subscribe in either iTunes or on YouTube or leave a review or a comment. Also, if you have any ideas for topics you'd like us

[01:01:04] to cover in the future, please email us at access returns pod at gmail dot com. We would like this to be a listener driven podcast and would appreciate any suggestions. Thank you.