Building Intelligent Alpha Portfolios with ChatGPT | Doug Clinton

[00:00:00] You can see it even just in the investment process as we use the large language models. There's no thought of emotion. They don't really care what the market's doing. They, in some ways, have no sense of what the underlying portfolio that they create is even really doing, which I think in most cases is a superpower.

[00:00:16] A good human analyst probably really knows 20 to 30, maybe 40 stocks if you're really pushing it really well for their own personal investment universe. But these models, they can know thousands. And in fact, they can probably know more even on those 20 to 40 that the human knows.

[00:00:35] The bet that we've been making at Intelligent Alpha is that if we fast forward the clock a decade from now, that the AI-powered asset management industry will be a multi-trillion dollar AUM industry. Welcome to Excess Returns, where we focus on what works over the long term in the markets. Join us as we talk about the strategies and tactics that can help you become a better long-term investor. Jack Forehand is a principal at Validia Capital Management. Justin Carbonneau is a managing director at Life and Liberty Indexes.

[00:01:04] No information on this podcast should be construed as investment advice. Securities discussed in the podcast may be holdings of clients of Validia Capital.

[00:01:34] And he's at the cutting edge of technology. It's good to get Doug's perspective on the intersection of AI, investing, and where things are headed in the future. As always, thank you for listening. Please enjoy this discussion with Intelligent Alpha's Doug Clint. Hi, Doug. Thank you very much for joining us today. Good to be with you guys as always. Always a pleasure to have you back on Excess Returns. We've had you on in the past to talk technology trends and the things that you guys focus on at Deepwater.

[00:01:58] But today we're going to spend, I think, more time focusing on sort of a new venture, although not too new. And we've touched on this to some extent with you in the past. But we're going to talk about how you build and deploy AI-powered investment strategies.

[00:02:14] And I think we want to spend some time sort of getting under the hood and exploring how for the nuances of this, how you've tackled it, some of the complexities of using large language models, things like ChatGPT, in developing a stock selection strategy. You have a new ETF that you launched last year that's powered with these AI strategies.

[00:02:39] You run a number of different indices that you're tracking live at a sample performance data on, which is always important. Obviously, with the ETS, you have an actual track record. You're developing one. So, you know, you're pretty I don't know if there's other people doing this. You may know that better than me, but I think you were early on in terms of trying to really figure out, is there something here with these LLMs and investment strategy? So, again, thank you. And, yeah, this should be good.

[00:03:08] Yeah, I mean, I'm excited to talk to you guys about Intelligent Alpha. And just for quick context background, we launched Intelligent Alpha last year. So we formed the company kind of Q1, Q2 of 24. It came from an experiment that we actually started in the summer of 2023. And so we had this question at Deepwater pretty simply, you know, can we use ChatGPT at that time just to beat the S&P 500?

[00:03:34] And so ever since that initial experiment in the summer of 23, we've been tracking a few dozen now different strategies where we use a combination of different large language models as an investment committee to build stock portfolios. And then, as you mentioned, Justin, we launched our first product for Intelligent Alpha, which is the Livermore ETF. The ticker is LIVR, where that uses our process that we use for all of our strategies.

[00:04:03] Committee of GPT, Gemini, and Claude on the large language model side. They kind of look at and are inspired by, I would say, the world's great investors. They try to think like those investors and then build a portfolio of 60 to 90 stocks that represents what those investors would do in the world, given what the environment is right now. And just remind me again, are those indices being tracked somewhere? Where can I find those? Yeah.

[00:04:31] So as part of forming the company, we also registered Intelligent Alpha with the governing body, the SEC. And so we've had to restrict, because of our registration, some of the numbers and performance that we used to share. So those indexes are now tracked internally. We still track all the same ones. Kind of the TLDR, I can just give you a little bit of a sense there, is there's about three dozen different products we track there.

[00:05:01] The overall performance on a net basis, the majority of those indexes are beating their benchmarks since inception, which is that roughly summer 2023 launch date. The wonderful thing about being a regulated entity is that you've got to pull all that stuff behind the curtain now, you know, unfortunately. That's right. Yeah. You get to hear more.

[00:05:24] So before we get into some of the details on this, what do you think are, what would be both the strengths and weaknesses of AI versus a human professional investor? The biggest strength and the one that I see most obviously when we do our work with the large language models is the lack of emotion. And I think that's something, I mean, you study the great investors. You read everything that Buffett wrote.

[00:05:53] I think so much of it really centers around the emotional traps that human beings fall into. You know, we get excited, euphoric when everybody else is really excited. That chases. We get FOMO. We get scared when stocks go down. And that emotion often causes us to make bad decisions and biased decisions. And so with AI, you can see it even just in the investment process as we use the large language models. There's no thought of emotion. They don't really care what the market's doing.

[00:06:22] They, in some ways, have no sense of what the portfolio, the underlying portfolio that they create is even really doing, which I think in most cases is a superpower in markets. In terms of weakness, I think one of the things, and this may not be a weakness forever depending on how the models evolve, but I think abstract thinking still, I mean, the human advantage still is in creativity, right?

[00:06:46] It's in finding, you know, the stock that's going to 10x for some reason that the models just, for whatever reason, they can't understand today. Maybe in the future, they will be able to understand it. But I think that's the thing where, you know, the models are, you can almost think of them in between, you know, an index, a quantitative approach, and a human. It kind of takes pieces of each of those and I think creates something that's very effective in the middle. But over time, I think it'll be able to do even more of the things that humans can do that we can't right now with AI.

[00:07:17] If I were to go to any one of these LLMs, ChatGPT, Gemini, and give it a prompt, what are the 10 best stocks for 2025, blah, blah, blah. Something where I'm trying to get it to spit out what it thinks the best investable opportunities are in the market. I've tried it. In some cases, it gives, sometimes it gives an answer depending on how you prompt it.

[00:07:45] Sometimes it just gives you, like, here's a framework for how you'd get at it. But just for, in either of those instances, like, what is going on? Can you just explain and maybe, like, talk about the process in which it lists specific ideas, specific names. Like, what is actually happening within the model? And maybe it's different for each of these models, but just pick one. Like, what's actually happening behind the scenes?

[00:08:08] So the way that the transformer architecture works, which is what these models are built on, is that the models are trying to figure out, you know, what is a logical sequence of words that creates a response that makes sense relative to the input from the user? Right? And so how do I, how do I make that sound a little bit simpler? Essentially what the models are doing, and they will do this. This is why we get hallucinations.

[00:08:35] They are going to give you an answer that in some way makes sense. To your point, it might be a framework in some cases. It might be a list of stocks in other cases. What we found is that the more structure you can put around the prompts and inject some appropriate data, useful data, the models are much more intelligent and much more, I would say, deliberate about how they actually go through the stock picking process. Just like a human being.

[00:09:04] You know, if you went to a human being on the street who didn't really know anything specific about stocks, didn't have any, you know, earnings data or revenue data, and you just said, hey, tell me 10 stocks that you think you'd like. What they would probably do is tell you 10 companies that just come to the top of their head. So it might be Home Depot. It might be Google. It might be Meta and Apple. And that's probably what you're going to get for the most part if you just go to an LLM today and just ask it for 10 favorite stocks.

[00:09:31] So would it give you basically the stocks it has the most information on? Like it would give you the NVIDIAs and the Apples, and if you just did a very, very simple prompt, that that's what it would return? That's what we have found. Absolutely. It just kind of goes toward where does it have the most information? If you were going to do a search, like imagine if you did a search on Google for, you know, what are the top 10 stocks? Again, you'd probably get NVIDIA and Google and Apple, all these names that people have written a lot about, people talk a lot about. That's what you would get from the LLMs as well.

[00:10:03] So if you're using one of these to find possible investments, there's obviously, you know, an infinite number of ways that you could kind of approach it. But would you say, do you think, do you have an opinion? And I guess, would it, if, could you give the model maybe a little bit of data and then let it work? Or would you sort of find that, you know, more specific detailed prompts is probably going to give you a better output?

[00:10:32] I love this question because we've actually tested this a lot over the last couple of years. And we found there's actually sort of like an efficient frontier where you want to balance giving the models enough data just as giving a human enough data to sort of understand the important parts of the company. But you don't want to give it so much structure that it's just becoming like a glorified screener, right? Where you tell it literally everything that you want it to look for in a stock.

[00:11:00] And then it just becomes an optimization around a bunch of, you know, quantitative metrics, which can be fine as an approach. But I think you kind of lose the secret sauce and the special thing, the special additive value that the LLMs can bring. And so we try to go somewhere in the middle where I think there is certainly a structure and a specificity to the prompt that is useful, especially so that it understands what kind of portfolio is it building, right? Is it a long, short portfolio? Is it a global portfolio?

[00:11:29] Some of the details around that, how it should think about risk. I think those are all great parameters to help the models understand. And then also a little bit of data too. So, you know, just like a human analyst would have maybe a sheet where you've got revenue and earnings and what are expectations from consensus going into the year. Giving that data to the LLMs too is super useful. Yeah, the more you talk, the more I think about this in the same way I think about a human being. Like if I go to my neighbor right now and say, give me the 10 best stocks to buy, I'm probably going to get a bunch of garbage back.

[00:11:59] Like if I go to one of the world's best investors and say, give me the 10 stocks to buy and give them very specific information, I'm going to probably get a much better list. It's almost like it's the same spectrum you see with these types of things. It's absolutely right. And I mean, that's sort of the mission that we have in Intelligent Alpha is we want to replace the human component of investment management on the analysis side and the portfolio management side. And we think that we can do that.

[00:12:23] And I referenced this a little bit before, but I really do think one of the best ways to think about using large language models as an investor is as sort of an evolution of the quantitative finance movement where obviously these models can understand quantitative information and they can act like a traditional quant model. But what they have that traditional quantitative finance models really don't have is that qualitative understanding.

[00:12:52] It's able to understand, you know, Apple makes cell phones and people like iPhones for whatever reason, right? And advanced auto parts sells auto parts where a quantitative model, a traditional quant model really doesn't care about that, right? They're looking specifically just at metrics. And I think it's that marriage of we can understand metrics. We can overlay a little bit of that unemotional qualitative understanding of what these companies do and what their opportunities might be.

[00:13:18] That's where I think you run into kind of this special outcome where you get a little bit of best of both worlds, quant side and human side. How do you think about the LLNs you're using? I think you mentioned you're using Gemini, you're using GPT, you're using Claude. How do you pick those? Are you just trying to get the biggest players in the space right now? So we started with GPT. That was kind of the first one we tested just because in 2023, they were sort of the obvious leaders. And I would say the leaderboard has gotten much tighter.

[00:13:46] And the way I describe it is at the beginning of 2024, like to me, OpenAI was clear, far and away, top of the leaderboard. And then, you know, Claude, Gemini, Llama, some of the other models were very clearly in second place and below. I think the leaderboard has compressed a lot over the last year. OpenAI to me is still the leader, and they've done some amazing things with advanced reasoning in 03.

[00:14:15] But I think we've seen the models get a lot better just in aggregate. And the reason we use the three that we do is we do some testing where we've looked at each of them individually. We've looked at Croc. We've looked at Llama. We've looked at several of the open source models. And we found that sort of subjectively and objectively, the three that we use, GPT, Gemini, and Claude, tend to give us the best results. So they tend to understand what they're asking, what we are asking them to do.

[00:14:45] And then also the performance of the portfolios that they've generated have been more stable relative to some of the other models. Yeah, to your point on the models in general, it's amazing. I use these things all the time. Now, not for stock picking, but I use them all the time. And it's amazing how fast they're evolving, but also how that leaderboard can move back and forth. Like, I think Claude is doing, like, really exceptional stuff right now. But then I'll go back to GPT, and I'll use it for a specific thing. And I'll say, wow, it's doing way better than it did even, like, three weeks ago. It's just amazing how fast these things are evolving. It's crazy.

[00:15:13] I mean, everybody says it's like, you know, months are happening in weeks and weeks are happening in days. And I think it's true. Like, sometimes that analogy feels like an exaggeration, but it does feel true in the AI world right now. And one thing that I think is really fun, just observationally, so we're always doing testing of models at Intelligent Alpha.

[00:15:33] And one of the more recent phenomena in the model space is this timescale inferencing, which is basically the idea of if you let models, just like human beings, think about the prompt that you give it longer, you tend to get better answers.

[00:16:18] And hopefully better portfolios and hopefully better performance over time as well. How different is the output from the models? Like, I assume, and you can correct me if I'm wrong, I assume you probably prompt them in fairly similar ways. Do they produce similar stocks or are they really, really different? That changes over time, too, actually, which I think is kind of fun to think about why that might be.

[00:16:38] But what used to happen was that GPT and Gemini tended to give fairly similar portfolios, if I rewind the clock like a year ago. And Claude tended to be the more contrarian of the models. I would say now GPT actually seems to be the more contrarian of the three models that we use the most, whereas Gemini and Claude now tend to think a little bit more alike.

[00:17:04] I think some of that has to do, obviously, with a function of what is the underlying training data for each of these models, because that's going to dictate how they think. But that does evolve over time, too, which is fun to think about. And also kind of a challenge we have to consider as we build our portfolios in terms of how much differentiation versus consensus we want in those underlying portfolios. Yeah, I think that'll be an interesting thing to think about for you over time, too, because you will, as you do this for longer, you'll start to develop a track record with each individual model.

[00:17:32] And then you have to think about, you know, if one of these models just ends up being better, you know, do I weight that model more in the future? Or, I mean, I guess Grok and some other models could come in. But, like, thinking about how do you evaluate the performance individually must be an interesting thing to think about. It is. In some ways, you can actually, I think, think about it almost like some of these multi-manager or multi-strat shops think about it, where, obviously, they have, in some cases, hundreds of different PMs. Obviously, each of those different PMs are performing in different ways.

[00:18:02] And those PMs, when they perform really well, get more money to manage. I think that's something we thought a lot about how should we structure that at Intelligent Alpha. And it's definitely on our roadmap in terms of trying to do performance optimization, I think we would call it, relative to model selection and weighting. Yeah, we just did an episode on pod shops recently. And, like, it does, it fits, it's very, like, a very similar thing. I mean, you're not pulling out all the risks like pod shops are. But the idea of, like, flowing more money to the better managers, you know, might work here, too. Absolutely.

[00:18:33] Absolutely. And, I mean, conceptually, I think it's kind of fun to think about, okay, what could be an end state or, like, you know, blow out the idea of using large language models for any type of investing? I mean, you could conceivably build a structure where you have a pod model, where every pod is a different, you know, fine-tuned LLM to think about the world in a specific way. You know, an energy expert, a macro expert, all of them having a long and short book.

[00:19:00] And you could even have an overlay on top of that, which is your sort of portfolio manager or risk management layer, just like a pod shop. So I think there's a lot that can be done there. I'm sure there are firms that are experimenting with that. But, I mean, I think you could imagine a world where anywhere we have human analysts and PMs, they could be replaced in some ways by large language models. I want to ask more about how you think about prompting these, because it's interesting. You're trying to track great investors.

[00:19:27] So, obviously, you have to give them information as to what those great investors did or what they can learn from them. I mean, obviously, it's got its own information as well. And then you've got to think about, what data do I give it? You know, do I give it fundamental data? Do I give it, you know, you have a lot of information probably about technology in general. What do I give it? So, how do you think about that in terms of what you tell it, in terms of how you model great investors, and then what types of data you give it besides what it already has?

[00:19:51] We found that to also be this sort of efficient frontier where you want to give it enough data to help it understand the specifics of the task. Because the models, all of these models already have a very general understanding of any task that you might ask of it. And so, you think about the sort of training data sets. Effectively, at this point, all the major models and even the open source models, I think of them as they've been trained on all of the data that's available on the internet.

[00:20:20] And if you conceptualize that in your head, it's like, all right, well, they know a lot about pretty much anything I might want to ask it. And so, your job in using an LLM effectively is really like, okay, if I'm going in a specific domain, what about that specific domain might those models not know from their training data? And how can I optimize their understanding of whatever that specific domain is? And so, to answer your question, Jack, is when we think about how do we help these models understand some of the world's great investors,

[00:20:48] some of it is pointing to more recent material that may not be in their training data sets because a lot of these models have cutoff dates that might be several months old. So, making sure things are current, that's one big effort. And then also making sure things are specific enough about maybe some of the best writings or the best thoughts or some of the most effective investments in the past that these great investors have made.

[00:21:13] Making sure the model understands, you know, this is how this investor thinks, this is what's made them successful and why they're successful. That we find is also helpful to create the general structure. And then once we go past there, then we kind of get into some of the specific stock-related data. And again, that comes back to probably a lot of the things that quantitative managers use. I would guess that if I think about the amount of data the quantitative manager might use for their strategies, we probably don't use as much data.

[00:21:43] I don't know if I would quantify us as, you know, a tenth or a half or a quarter, but something probably like that. You know, it's a little bit more of a limited data set, but still substantial enough so that the models can understand the underlying businesses. I just want to jump back to the multi-model pod shop sort of thing just for a second.

[00:22:03] And tell me if I'm like, this is a crazy question, but I mean, could you, you know, if you're allowing these models to compete against each other, which that's kind of effectively what it would be. Could you then circle back around and tell a model that it's losing the competition? And so to like improve itself, or is that, is that way too much like then you're getting them to actually think like humans and that's too far out in left field?

[00:22:31] So we've actually thought a lot about this particular question in a slightly different way. We call this introspection for the model, which is think about, you know, why you might pick a certain stock or believe that a certain thing might happen in the world. And then challenge yourself, you know, why might it happen? And in some cases you can actually have the models sort of debate each other is another tactic that we've worked on.

[00:22:57] And so I think the spirit of your question is, could you give models more information or kind of force them to think a little bit more about what might be working, what might not be working in terms of the application of strategy? The answer is yes. And I think that that is one of the ways, just like a human being might adapt their performance, depending on how they're doing, learn from their mistakes. The same thing can be true with these models. Yeah.

[00:23:21] Part of the challenge, I mean, just to get into a little bit of the nerdy nuances here is that these models at the moment don't have, um, uh, simple structured memory. You know, like a human being will remember, especially the painful lessons. Those are the ones you remember the most. Um, the models don't have that, which in some ways is a superpower, but in some ways is a weakness. So you kind of have to almost bring in important memories, at least for now, um, back to the surface for them. You think?

[00:23:51] And do you have specific, you don't have to name them, but do you have specific great investors in mind that you're modeling? Or are you trying to model just great investors in general when you think about your prompting? Uh, both. Uh, I think generally I would say our, our base strategies tend to have sort of a, a growthy value tilt or, or a GARP tilt. Um, I think that's, that may be a reflection of how I like to think about the world for better or worse. There may be some bias built in there. Of course there always is when humans get involved.

[00:24:20] Um, but that's, that's kind of the base philosophy. And then I would say, you know, we use it as inspirations, the names that you would expect. Buffett, Peter Lynch, right? George Soros, Dan Druckenmiller, you know, names, names. I think everybody would probably agree are, are on the Mount Rushmore. That makes sense. And then it goes back to what we talked about before, because depending on who you're telling it, the great investors are, you can have a very different experience. So for instance, if I tell the model, I want you to be Benjamin Graham and I want you to

[00:24:48] buy cheap price to book stocks, it has been a catastrophe in the last five years. Yeah. You know, so I guess you, and we do have to figure out like for the world we live in, who are the great investors I think are worth following. That's true. Yeah. And I think that, that again gets to this idea that we use a lot, which is efficient frontier of how much, um, of a push do you want to make one way or the other, right? Like if you say Benjamin Graham and you really want the models to optimize price to book stocks, models can certainly do that.

[00:25:16] Um, another way to do it is to say, we really like the concepts of value investing broadly. And, um, obviously that theme has been more challenged over the last decade, really at this point, uh, you could ask the model, how would you structure a value oriented portfolio in the current regime where, you know, indexes are sort of dominant, the big companies tend to get bigger, right? You can have the model, I think, think about how it might adapt that strategy without giving

[00:25:45] it the answers to the tests that historically have been, you know, associated with whatever exposure you're looking for. How much of any of this is fundamental database? Like I assume it doesn't have access to CompuStats database or something like that. And you mentioned before, a lot of times they probably don't even know what the price of the stock is once they decide to pick it. So like, is something like NVIDIA's PE ratio, is that playing a role here or is this really completely different things? It does in our process. So we'll pull in, uh, we don't use CompuStat, but we use some other, uh, third party, you

[00:26:13] know, well-known data sources where we do pull in price data, PE data, earnings, uh, you know, revenue, things like that, free cashflow. Um, so that's, that's something that we do think is important for the models to understand as they're picking these stocks. We want to give them that sort of quantitative grounding. So they at least know, okay, how expensive is this stock on some metric? Um, and then pair that with the qualitative piece that we talked about before, where it's

[00:26:40] kind of a, okay, the stock may or may not look expensive on a purely quantitative basis, but qualitatively, what are some of the elements of this company that I think might be good or bad going forward? I want to bring you in on the debate we've been having a lot in the quant space, which is that there's this idea in like factor investing that our factors have to be intuitive. So I want to start with something that makes sense and then I want to test it, but that's kind of getting turned on its head, you know, in, in recent times because we're finding

[00:27:07] a lot of strategies that work and we don't necessarily know why they work. I'm wondering, how do you think about, about that with respect to this? Like how much do you know about why these models are picking these stocks and how much do you care that, you know, the reason versus you just know that they have a good process that they're using? Um, I think it's, it's a little bit of both because we have to understand at least generally how the models are picking, working, functioning so that we can explain to our investors how

[00:27:37] the process works. You know, explain ability, I think in AI, whether you're in the financial services space or in any other space is a huge component of having people trust the process and trust the AI. And so I think the explainability piece is really important. Um, how the AI or what the AI unearths to your point, I think, um, that I think can be a little bit more, I mean, we call it, you know, the part of the magic, um, that's in

[00:28:05] the process where even if you talk to AI engineers, they couldn't give you the full explanation of exactly like, you know, line for line or literally like character for character, why a model might be returning something. And so we can't do that either. Um, but I think there's something within that, that is if we can at least understand and explain the process and then, um, have at least an intuition of why does the model think a certain

[00:28:34] way based on all of our experimentation and work with the model, just like we might have to explain how a human PM thinks. Um, I think that's kind of the perfect balance to the process. Will it tell you why? So if you have XYZ stock in there, can, I mean, can you prompt it and say, you know, give me the biggest reasons why this particular stock you've selected for the portfolio? It does do that. Yes. And as part of our process, we often ask the models to provide a rationale with its stock

[00:29:03] pick because we tend to see you get, you get better answers. Um, and as a general rule, like what I would say with large damage models working with them, the more you, you force them or ask them to explain themselves, to explain their process, the more that the, the machine, the AI actually has to go through a process just like a human being. And so you tend to get better results when you ask for more information or back to the idea of introspection. When you ask for sort of a, a why analysis or how might you

[00:29:33] improve this analysis? It does often improve your results at least on the surface. And then I think over time it should have improved your, your returns as well. How do you think about testing? This is something that I think is very different probably than our quant world. Like in our quant world, we want to test something. We want to have in sample data. We want to have out of sample data. We want to have, you know, results over a really long period of time, but with these, that's not really possible. So on one hand, I look at that and I say, well, that might be a weakness of these. But on the other hand,

[00:30:00] I say they really have all available information available to them. So do I really need to just get rid of this whole backtesting process? Do I need to really think about is backtesting necessary for a model like this? Yeah. Well, um, we have a very specific view on backtesting. I'd say, I think, uh, quants, quants are generally very smart, uh, people, much smarter than me, uh, that understand math at a different level. And what we're doing with these, with these models, I think it's

[00:30:24] something that's a little bit different. Um, where math obviously is playing a role in our process and within the data that we present to the models. But I think a lot of what we're doing is actually trying to create structured creativity for the models, like a world to, uh, to sort of be constrained by, but also, uh, act within and be created within. And so it's a little bit of a different process in my view, as I think about what's the difference between using large language models versus,

[00:30:54] uh, traditional quant models. Uh, but to answer your question specifically about backtesting, I think backtesting using large language models is, is almost useless. And there's a few different reasons for that. Uh, the biggest of which, and the most obvious of which is that most of these models have a, a non-current cutoff date in terms of where their training data set ends. And so I think if I remember, I was just looking at, uh, I think it was Gemini. Uh, it could have

[00:31:23] been Claude, but their training data ended, uh, in August of 2024. And so there's, you know, five months now basically of data about the world that that model doesn't understand unless you make sure that you find a way to inject it into the model. And so if you were going to perform a backtest, any backtest that would go past August of 2024, the model already has the answers.

[00:31:48] And we do see in some of our, our tests, cause we'll still backtest just to look, but you do see in backtest that oftentimes when the model has the answer, it will make sure the portfolio kind of reflects what did actually work, um, in the past. And so I think when we backtest to the extent that we do, we'll only look at the data, uh, up to whatever today is. And the cutoff is whatever the, the training data cutoff is because we feel like at least then the model doesn't already

[00:32:18] have the answer within its broader training data set. Yeah. Cause it's not like you could go to the model and say, no, only what you would have known in March of 2000 and, you know, run the same thing. It doesn't work that way. Right. It doesn't. I mean, you, you can certainly try that in my view. I think that your, uh, your answers would probably still be pretty reflective of, of reality. Cause I don't think the model would fully grasp or embrace, uh, not knowing what was in his training data.

[00:32:47] So I want to dig into a little bit of the portfolio construction stuff that we, uh, quants always focus on behind the scenes. Um, and I want to start with the investment universe. Like how do you think about telling it what the, cause obviously you probably don't want to stock, you know, with a million dollar market cap in there or something. How do you think about telling it what the universe is it can choose from? Sometimes it's easy where if we're going to create, let's just say a, you know, a large cap U S equity strategy that universe is, is pretty well-defined. You can, you know,

[00:33:13] you just do a screen of stocks above, let's just say $15 billion in market cap based in the U S. Um, those are easy where it gets a little more challenging is when we try to do more dynamic portfolios, something like, uh, our Livermore strategy for our ETF. Um, something like, uh, we're about to launch a, an LP structured, uh, product later this month for qualified investors. You know, that has a little bit of a more dynamic universe. And the answer to the question is,

[00:33:39] it sort of depends. You know, we always try to start with what is the broadest universe that makes sense for the strategy that we're trying to implement. And in some cases we'll try to pare down that universe by using the LLMs themselves. And so that might be the LLM saying, here are specific themes or specific sectors that I want to focus on for this particular strategy. And that will help us narrow down that

[00:34:04] universe a little bit. Um, but you know, that's, that's the general process is either it's super easy because the strategy is, is fairly basic and targeted at something where there's a defined universe. Or it's something where, you know, we're starting with a very broad universe and using the LLM and the strategy that we're sort of building with the LLM to help cut it down into something more manageable. I think you mentioned you hold between 50 and a hundred stocks before. How did you think about

[00:34:31] setting that number? It varies depending on the different strategies. So that's the case for our Livermore strategy. Um, our, our LP structured strategy that we have coming up is also in the same general range. Uh, but we have some strategies that hold two to 300 different stocks. And we have some strategies that hold as few as 10 to 12. And, um, most for the most part, we have thought about

[00:34:58] the world as sort of, you know, concentrated portfolio exposures or very broad portfolio exposures for the broad portfolio exposures. Those are the kinds of strategies where we're just benchmarking against, you know, the S and P 500. We're trying to create just a comp that we think would be like an AI powered version of large cap U S equity exposure. Um, so that's easy. You want something that might have two or 300 stocks in that scenario for some of our more concentrated strategies. We are

[00:35:27] trying to be in that 10, maybe 30, maybe up to 50 kind of holdings range where we want the AI to be more convicted in the names that it's trying to pick because we're trying to generate more excess return. Uh, we're not trying to just create something that has relatively low tracking error and maybe a little bit of excess of return relative to a benchmark. The concentration is a reflection of,

[00:35:50] uh, us going for more upside in that strategy. Is the AI giving you position sizes too, in terms of how big it wants each position to be? It does. Yes. And we, we set parameters around those two, depending again on the strategy, um, for things, you know, in the ETF world, obviously there are specific rules that we have to comply to in terms of, uh, sizing of various positions. For example,

[00:36:16] we can't have a single holding over 25%, um, as one example, but we'll have specific rules. Generally we'll tell the models that they shouldn't have position sizes over five or 10%, depending on the size of, uh, of holdings in the strategy. But we do ask the models to weight their picks based on their conviction. And how about sector concentration? Like I would think these probably tend to pick a good number of technology stocks. Do you think that's something that should be restricted over

[00:36:44] time to a certain percentage in each sector? Or do you think you should just let the model roam as it wants to? We've experimented some with that. And so we do have some strategies where specifically we will tell the models to think about sector exposure. Um, in some cases, we might even exclude certain sectors, uh, again, just depending on what the strategy is. Um, but I think that's something as we think about kind of our roadmap and into the future and, and thinking about different customer

[00:37:13] sets we can serve that sector exposure, I think is a question where we, we often hear it more from an institutional client relative to a retail client, uh, where the institution is looking more specifically, I think for, you know, orthogonal alpha. They want to see things that are, uh, neutral in terms of sectors, factors, beta. Um, and that requires a little bit more work. Obviously in our end, we're building some products around that. Um, but for our more general strategies, I would say

[00:37:42] we usually let the model sort of roam and go where they think is best. How, how do you think about how often you rebalance and also whether you limit turnover? Um, do you, do you want to have a certain, do you, if the models want to change every single stock every month or something, would you put some limits in there to say like, that's not a great idea? Yeah. It depends on the strategy. So generally for most of those three dozen strategies that we track, most of them are quarterly kind of portfolio reviews

[00:38:06] is how we view it for some of the more aggressive strategies that we run like this LP structured fund. I mentioned that when we look at more every two to four weeks, sometimes every six weeks, just depending on, you know, how the portfolio is performing, what's the market environment. We're kind of constantly having the LLMs actually think about the portfolio, even though we're not changing stocks in it. Um, so it depends on the strategy. And then in terms of the turnover, we've used a few different approaches to that. I would say

[00:38:34] generally we sort of let the models have free reign because that's the purpose in our mind is just have this be a reflection of what AI thinks are the best stocks for whatever specific strategy we're working on. In some cases, if we're doing something that is very long-term oriented, as an example, we might only allow the AI to switch out one or two holdings, uh, every review period or something like that. So it's easy to put limitations on it. And that's usually how we implement it.

[00:39:02] I almost wonder if this could be used as like a direct indexing type of framework or strategy. Like you, if you were able to, you know, hold something like the S&T 500 and then give the model, like the rules around when to tax loss harvest and what type of replacement stocks, I mean, you might be able to sort of get like an automated direct indexing strategy. I know you're running active strategies. I'm just making a comment here as I'm thinking about like the investment

[00:39:30] process side of it. It's just an interesting question. I think. I think it's, it's absolutely, uh, possible. And we've, uh, we've actually looked into that world a little bit. Um, something that we'd be, we'd be open to. I think that again, some of the broad strategies that we track where we're trying to specifically create broad, you know, us equity exposure again, we'll just use the S and P as an example, because it's easy. Um, I think we could implement our strategy and to your point, add on some intelligence about

[00:40:00] tax loss harvesting and hopefully try to create not just a little bit of tax alpha, but also some return from better stock selections as well. Is there any part of the process that a human steps in? So for instance, if the models came up with a stock that had accounting issues or something, what does a human review these? Or is there any other part of the process where you think a human adds value to what the models are doing? So we do have a human oversight layer, which basically happens at the end of the process.

[00:40:28] So a human obviously is sort of, um, at the top of the process, helping figure out, uh, the investment universe and what the strategy intention is, things like that. And then once the models sort of ingest all the data, perform all of the analysis that we're asking it to do, and we have a, uh, you know, a base portfolio, a human looks at that to make sure that there's no, you know, hallucinations or stocks that are in there that shouldn't be,

[00:40:54] you know, if there's a Chinese equity in a U S equity portfolio, obviously we'll take something like that out because the models can still make some mistakes. And I think that for now, at least that human oversight layer is an important part going back to the idea of explainability where the most explainable thing is always that, well, a human is overseeing this still, you know, it's AI power. We're giving all of the decision-making power essentially to the AI to build the portfolio, but we are still taking that final step to make

[00:41:22] sure that nothing went awry where the portfolio is completely, you know, irrelevant for whatever strategy or exposure we're trying to create. And by the way, we've actually never had any egregious, um, issues with hundreds at this point of different portfolio reviews. The worst we usually get is like, you know, a small cap stock will somehow find its way into a large cap strategy. That's the most frequent problem we run into. We've gone over some of these already, but I was thinking before

[00:41:51] we did this, I was like, probably the best thing I could do if we're going to talk about the pros and cons of Claude as a stock picker is let's ask Claude itself, uh, who could your pros and cons. So I did that. Um, and the first one I came up with was its ability to process and synthesize vast amounts of information quickly, which I think would probably be one of the biggest advantages over a human, because I think no matter how much and no matter how smart I am as a person, there's no way I could process as much data as these things are processing.

[00:42:17] A hundred percent. I mean, you think about it this way. I think a good human analyst probably really knows 20 to 30, maybe 40 stocks. If you're really pushing it really well for their own personal investment universe, but these models, they, they can know thousands. And in fact, they can probably know more even on those 20 to 40 that the human knows they probably know more, you know, if you just objectively would test the models on it. Now that brings us back to kind of the

[00:42:46] creativity of the human. That's still where the advantage is. But I think a hundred percent, if you think about what is one of the biggest strengths aside from lack of emotion for these models, it's the scalability of knowledge and the breadth of the knowledge that they bring to the investment universe. And the other one I came up with, that I thought was interesting was the idea we talked about before that they can work 24 seven. But my question around that is like, do you have them working fairly regularly? I mean, is it, is it just around when you're going to rebalance that you're having it look at the portfolio?

[00:43:13] Or is this something where on an ongoing basis, you have them analyzing things pretty regularly? David We do have an ongoing process, uh, 24 seven. Um, and I actually think there's, there's actually something important in there too. Um, where Ben Affleck, funny, funny to bring him up in an AI conversation. He had this incredible quote about the difference between sort of art and AI. And he said, um, AI, you know, is doing the work,

[00:43:39] you know, but knowing when to stop that's art. And, uh, I'm paraphrasing. I think I probably butchered the quote. He said it way better than I did, but the point of the quote, I think is really beautiful, which is you could have these models work on figuring out a stock, thinking about a stock around the clock persistently forever. And it would probably be a function of

[00:44:03] the stock or the, the LLM just giving you roughly the same answer in a slightly different way over and over and over again. So it kind of becomes pointless. This idea of sort of knowing when to stop, I think is still part of how does the human element, right? Whether it's an engineer or somebody who's overseeing the technology still, how's the human think about what's the right time to have these models think a little bit more? Is there new data? Is there something where the model

[00:44:31] can have a new opinion that's worthwhile? Or is it just, you know, if you said a human analyst and said, tell me what you think about Apple every minute on the minute for the next 24 hours, it's not going to change that much. And so I think you can think about the models the same way. And I just want to do one week this before I hand it back to Justin, because I thought this one was interesting. It said, can't adapt to unprecedented market conditions or paradigm shifts. And I'm wondering about that. Like say, you know, China invaded Taiwan or some sort of massive

[00:44:57] change happens in the world. Like, how do you think about these, these models ability to continue to select stocks in a world like that, where something changes really dramatically? Justin Donald I think the conventional wisdom, most people who are at least curious about using AI to invest would agree with that statement. You know, they would say, you know, okay, it's great. Models work when the market's going up, everybody kind of looks smart. But when the market has volatility,

[00:45:25] the models will fail for, for one reason or another. They might point to the quant quake, you know, 15 years ago now, I guess, seems, seems forever ago. Something like that, you know, where we have some evidence where these models can have issues at times. I would probably take somewhat the other side of that bet, just because there's different ways that you can have these models think about the world. And I'll give two specific examples. I mean, one way that you can

[00:45:54] kind of guarantee that the models don't make an egregious mistake is make them think a little bit like indexes, which is you only have them review their portfolios on a set schedule. And so you're not going to make some crazy mistake if the market blows up any more than the S&P 500 would. Right? Now, you wouldn't take advantage perhaps of an opportunity, but if the question is, you know,

[00:46:19] could these models blow up? I think that's one approach, a very simplistic approach to the problem. I think the better approach and the thing that we've been trying to build at Intelligent Alpha is these models are capable now with tools like perplexity. GPT is now building in real-time search and web index to understand the world today, you know, at the moment. Grok doing the same with Twitter data. I think these models now more than ever are actually more capable of

[00:46:47] understanding kind of the status of the world currently. And so I actually do think they can be either now or in the near future capable of adapting to these volatile market conditions, just like a human could. And I think because they don't have that emotional bias, they may prove to be better than the human would be in that case. What do you think this means for the future of stock

[00:47:12] ticking? I was reading one of your articles from earlier in the year and you were kind of talking about intelligent alpha and, you know, moving quickly and why big firms, you know, will be slow to get into this. It's messy. It's complicated. They obviously have their current product shelves that they may not want to cannibalize, but, you know, just in, you know, when you guys think about the future and

[00:47:40] AI driven investment strategies, you know, could there be a future where there's like hundreds of AI models just competing against each other or is that not really likely going to happen? What do you think? I think there could be just like, there are hundreds, even, even thousands of different funds that are run by humans that have different flavors, different strategies, different focuses. I think you will find in 10 years that the same thing happens with AI. So there will be funds that

[00:48:09] focus on emerging technology. There'll be funds that focus on, you know, global equities and everything in between. And I really do think that's the future. I think where human analysts and PMs will have a challenge competing is if they're just doing some, you know, vanilla, plain vanilla strategy, AI will probably do that better. If they find a strategy that is really differentiated, unique,

[00:48:36] it requires some level of creativity or ingenuity well beyond, you know, just looking at simple PE ratios and saying, this is a quality company. I think that's where human managers are going to find themselves being able to add value in the world of AI. So it will get more competitive in many ways. I think AI can bring down the costs obviously to implement those, you know, plain vanilla type strategies. And hopefully,

[00:49:03] and this is my optimism, you know, people always worry about humans losing jobs to AI. I actually think it just creates a stronger competitive environment for the best humans to stand out even more. And so in that future world where creativity is at a premium, I think you'll see some really creative human investors doing things that maybe we've never seen done before because the structure of the market will really push them to think outside the box because most of the other stuff is being done by AI.

[00:49:30] It's interesting to your point. I was listening to an interview, I believe it was with Gavin Baker recently, and he was saying that exact same thing. He was saying that he thinks human assisted, like humans with the assistance of AI is going to be a great, great opportunity for great fund managers to get much better than they are now. I think he's 100% right. And it's, you know, again, something we've experimented with at Intelligent Alpha 2. I mean, our vision is really to try to sort of replace the human

[00:49:54] component of investing. But we've also experimented with the augmentation potential for, can you make a human manager better if you give them some of these tools? You know, maybe free the human manager from thinking about specific things, going super deep on certain stocks and just thinking more broadly about portfolio construction and your end customer, you know, what do they want? How do they trust you? Things like that. That might be how the world shapes as well.

[00:50:22] Well, you know, the big problem is we're running out of ticker symbols for ETS. So the more strategies that come on, the more, the more shortage of tickers there's going to be. But you got a good one, actually. So you kind of locked in on LIVR, right? LIVR. That's correct, right? LIVR, yep. Yeah, cool. You got lucky. Yep. Okay. So just in closing here, let's kind of come out of the investment sort of space for a man.

[00:50:49] That was great, Doug. Thank you very much. This is evolving so quickly. So it's going to be just important, I think, for people that are in the business and whether you're a quant or a stock picker to sort of understand this and how it's being deployed. And you guys are sort of on the front edge of it. So, you know, thanks for sharing all this with us. I want to ask you sort of outside of

[00:51:12] using AI for stock selection. What are the other things that you're most excited about with this technology outside of investing? At the end of last year, so in December, OpenAI put out their O3 model, some of the benchmarks that they achieved that were, I thought, outstanding.

[00:51:38] And really, I don't think the market has priced in this concept of, you know, how close, and you could get into a real philosophical debate of how close are we to AGI? What does AGI even mean? But the TLDR to me, just for anybody who's not familiar, the O3 model is OpenAI, sort of top of the line AI model at this point, on a test where basically the model is asked to solve some puzzles

[00:52:05] that it's never seen before. It scored, I think the highest score it had was like a 91, if there was no constraints on how much compute it could use, how long it could think essentially. And the human benchmark is 100%. And so we're 9 percentage points, you know, that model was 9 percentage points from a human being. I think that that is incredibly powerful, because one of the criticisms of these AI models is they'll

[00:52:32] never be able to think like a human, right? They can't reason like a human being. They can't create de novo sort of intelligence or creativity, learn something new if it's not in their training data. And I think that that achievement with O3 suggests that maybe that's not true. And the reason I find that really exciting is even before O3, go back to GPT 01, right? Or even the model before that, 4.0.

[00:53:00] Like AI as it is today is already powerful enough to do so many things. Like think about how effective chat GPT is anytime you ask it a question. I mean, it's amazing. And so that already has the power to transform a ton of businesses. So to me, there's like all this latent potential, even if AI doesn't get any better than it is today, to make the world a much more efficient place, to superpower human

[00:53:26] beings, right? To help them think faster on their feet, to deliver better customer service. I think that's one of the most obvious use cases, better sales outreach, stuff like that. But that's not, that's not where we're going to stop. You know, my excitement about where these models are going as we head into 2025 and thinking about what might 2026 look like, you know, we might get pretty close to AGI. And I think that's great for humanity. I think that's great for applications like investing,

[00:53:55] because then the models will start to be able to think a little bit more creativity like human beings. And we were talking about before kind of what was the human advantage? I think it is that creative, abstract thinking in the future and maybe even the near future, these models might have that power as well. Do you think it's a foregone conclusion with open AI transitioning to like kind of going from nonprofit to profit that, that the path for them is a publicly traded company? I mean, is that a done deal,

[00:54:24] you think? It seems the most obvious and likely. And also just, you think about their capital needs too. If, if I remember the last thing I read, I think they burned several billion dollars last year. Obviously they're spending a lot to build their leading models. And I think that at some point, there's probably some limit to how much money the private markets and primarily, I mean,

[00:54:50] the hyperscalers like Microsoft really are going to be able to invest directly in open AI, you know, and not own the underlying technology. Obviously Microsoft has some rights in their agreement, but if we get to AGI, as we just talked about, I believe the way that contract works is then, you know, the, the exclusivity that Microsoft has is no longer. So I think eventually that probably means that there's, there's a path, but opening out sort of needs to go public to keep raising more

[00:55:20] capital to keep developing the technology. How far do you think we are from the personal assistant? Cause that seems like it's one of the killer apps for this whole thing is I can just talk to my phone and it can book travel. It knows everything about me. Like I know with my calendar, it can basically do anything. It can be like what a personal assistant would be, but just a digital one on my phone. Do you think that's way far out or do you think that's close? I would put the over under at probably two years. I think we'll start to see some early

[00:55:49] implementations of that this year. Um, this year 2025, and this is probably consensus at this point, but still worth stating, I think is, is really going to be the year of the agent, the AI agent. And so the year of the agent is a step Jack toward what you're talking about, where an agent, if you say, you know, book me a flight to Vancouver from Philadelphia, you know, an agent can understand that query. It understands that it needs to find your frequent flyer number, your credit card number,

[00:56:16] you know, find the right times for your schedule, all those things where today, if you just went to chat GPT and said that, right, you're not going to get a useful answer. And so that agentic AI world, as we start to see what are those products implementations, I think there'll be sort of one off in many cases this year, maybe it'll just be a flight booking service. Um, in 2026, I think you might get to see some of that stuff come together and build into something that looks more like that

[00:56:45] assistant that we would all envision. So one last standard closing question, Doug, and it's, it's a new one for us, um, that we're asking all of our guests, which is what is the one thing you believe about investing that the majority of your peers would disagree with you on? The bet that we've been making at intelligent alpha is that if we fast forward to clock a decade

[00:57:07] from now that the AI powered asset management industry will be a multi-trillion dollar AUM industry. So just like we've seen this boom in, in ETFs and indexing of all past 20, 30 years, I think you will see a similar boom in AI powered investing. Some of that will look active. Some of that might look a little more passive. Um, some of that might look even a little more exotic, but I think

[00:57:33] we're here chatting in, uh, 2035, I think Cliff asked this, just did a prediction piece, 2035. My prediction would be that we'll see a few trillion dollars being managed by AI. Good stuff, Doug. Thank you very much. We really appreciate it. Thank you guys. This is Justin again. Thanks so much for tuning into this episode of excess returns. You can follow Jack on Twitter at practical quant and follow me on Twitter at JJ Carbonell.

[00:58:00] If you found this discussion interesting and valuable, please subscribe in either iTunes or on YouTube or leave a review or a comment. We appreciate it.