Dr. Ernest Chan’s Journey: The Promise and Limits of AI and Machine Learning for Investing

Dr. Ernest Chan is a noted quantitative hedge fund manager and quant finance author. He has applied his expertise in machine learning in positions with IBM T.J. Watson Research Center’s Human Language Technologies group, Morgan Stanley’s Data Mining and Artificial Intelligence Group, and Credit Suisse’s Horizon Trading Group. He is Founder and Chief Scientist of Predictnow.ai, where his API specs can be found, and founder and non-executive chair of QTS Capital Management.

We asked Dr. Chan about his work in machine learning and AI.

Tell us a little about your career.

I went into machine learning because I was too dumb to be in theoretical physics, for which I got my Ph.D. (when one has some leading figures in the Oppenheimer movie as current or ex-professors in one’s department, one can be forgiven for feeling that way.)

AI/ML offered the attraction of scientific research without the burden of having to first learn 400 years of prior knowledge. When I joined IBM’s language modeling group, it was doing some of the cutting-edge research in applying AI research to human languages, stuff that OpenAI and others are doing some 30 years later. A couple of researchers who co-founded our group decided to decamp for a hedge fund just before I arrived, and that piqued my interest in applying AI to investing. (They went on to become co-CEOs of that hedge fund, Renaissance Technologies.) So, after a few very productive years at IBM, I left to join Morgan Stanley’s fledging AI group. Since then, I have always worked at the interface between AI and investing, including at the hedge fund QTS Capital Management and at the AI-driven fintech firm Predictnow.ai, both of which I founded.

What excites you about machine learning and AI?

As I said, AI/ML is like scientific research: you often learn something new and unexpected. It is very exciting to find an AI algorithm that actually improves on things people have been working on for decades, such as portfolio optimization.

What are your current goals?

After the success of the first version of our AI-driven portfolio optimization technology called CPO (“Conditional Portfolio Optimization”), we are working on version 2.0, which leverages some of the advances in deep learning that everyone keeps hearing about these days. It is a joy to work with a team of highly motivated and experienced data scientists and machine learners in creating this product. We are also looking for use cases outside of finance.

What is the focus of Predictnow.ai, including the business model, and how do you deliver value?

After numerous successful proofs of concept, we are now partnering with one of the largest US banks to roll out a fund based on CPO. It is very exciting to see AI helping retail investors avoid risks to their retirement income, which, to me, is one of the most beneficial and humane applications of AI. I am ambivalent about AI taking away creative powers from humans. But to help reduce risks to them and their investments? I am all for it.

You’ve talked about the power and limitations of machine learning in trading, specifically saying that it helps improve strategy but isn’t yet a viable means of actually managing hedge funds or creating black-box strategies.

Can you give us examples of what machine learning does well, what it doesn’t, and why?

Machine learning is great at finding correlations but not so great at finding causal relationships. In other fields, such as medicine, correlations often lead to causal relations.  Not so in finance. The market is too noisy, too non-stationary, and subject to arbitrage activities, such that machine learning has often found correlations that don’t persist. Hence using it to directly generate enduring alpha has been very hard.  However, using machine learning to find and manage risks and recommend optimal capital allocations is easier. For example, anyone can tell you that when oil prices are high, it’s likely that an oil industry ETF will do well. But it’s not so easy to devise a systematic trading strategy that can profitably trade such an ETF with precise timing using machine learning. Financial machine learning tends to excel at things that do not require high predictive accuracy or exquisite timing. 

Do you think that at some point, machine learning will become smart enough to manage funds without much human intervention?

For sure. At Predictnow, we are exploring how to use generative AI to conduct discretionary trading. Our work is showing promising results that I will describe in my next book, co-authored with a GenAI expert.

You’ve said you apply machine learning to manage risk for your hedge fund and that it learns from your track record. You’ve also talked about using it for tail hedges. Can you tell us about those uses?

Managing risk via machine learning vs. using a tail hedge strategy as developed by QTS Capital are quite different processes. The former uses machine learning to learn from the historical daily returns of an investment strategy, combines that with big data (which can be viewed as an expanded set of factors familiar to investment professionals), develops a machine learning model, and uses it to compute the probability of loss for that strategy. We call this “Corrective AI”. Tail hedge, on the other hand, is often implemented as a trend-following strategy on an index, or to simply go long on index options. Our trend-following strategy, called Tail Reaper, only trades S&P futures intraday without holding any overnight positions. Just holding positions for a short time during the trading day can already generate great performance in crisis years such as 2020 or 2022 while not constantly bleeding options premiums if one were to long index options.

Has anything changed in the 18 months since ChatGPT and other generative AI engines became widely available?

Yes. Turning news into sentiment scores used to be very expensive and challenging. With GenAI, we’ve found that to be quite easy.

The Predictnow.ai website highlights these topics: “Applying Corrective AI to Daily Seasonal Forex Trading” and “Conditional Portfolio Optimization (CPO): Using machine learning to adapt capital allocations to market regimes.” Can you tell us something about each?

In the first white paper, we described how Corrective AI can be used to improve an FX strategy’s Sharpe ratio by skipping those days with a high probability of losses as computed by our Machine Learning program. In the second white paper, we described how CPO can improve the Sharpe ratio of portfolios, whether they are portfolios of ETFs or trading strategies, simply by using adaptive capital allocations based on the market environment.

What kinds of research are you currently working on?

Mainly on version 2.0 of CPO, which leverages some insights gained from the deep learning and reinforcement learning literature.

What do you anticipate will be the next stage of machine learning, and are there limitations to what it will ultimately do, or do you expect the technology to keep growing indefinitely?

In terms of alpha generation, there is a limit to how much AI can help. When intelligent robots are trading against intelligent robots, the market simply gains in efficiency. However, in terms of risk management, capital allocation, and other use cases outside of finance, there is no limit. Those are not zero-sum games — the pie just keeps getting bigger, or the risks keep getting smaller. Obviously, generative AI will play a part, but so will more general applications of deep reinforcement and, in general, supervised learning. At Predictnow.ai, we’re excited to be part of this remarkable endeavor.

Go Source