Many applications of bayes rule

Applying Machine Learning and Science to Trading Decisions

TL;DR:

Treat investments like hypotheses to be tested not as risk adjusted returns to be maximized.

The adage: ‘cut losers and let winners ride’ needs revision (to put it kindly), and if you use stop-loss-levels, you should consider using stop-loss-curves.

This revisits the content from an ‘oldie-but-goodie‘ talk I gave in 2016. I was still breaking out of my Hedge Fund / Portfolio Manager comfort zone and was starting to help the Boston Machine Learning community and the Boston Asset Management community learn from each other’s relative strengths.

No man is better than a machine… and no machine is better than a man with a machine.”

Paul Tudor Jones – 2016 (Forbes & WSJ )

This was an exciting set of topics for me because I began formalizing how the various insights of machine learning could lead to enormous process efficiencies for investment managers. I began formalizing what insights active management bring to the table and that they are fundamentally different than their silicone-based forecasting counterparts. Some investors like Paul Tudor Jones stated: ” No man is better than a machine… and no machine is better than a man with a machine.” This has the right sentiment but it isn’t very specific as to why there are these synergies.

An Avocado Anecdote

The reality of machine learning or any type of statistical analysis is that it is backwards looking. The engineers or investors building the quantitative models inevitably have to rely on historical relationships to tease out patterns and most of the time this works extremely well, especially when forecasting highly specific things. For example, data could answer how La Nina could inflate avocado prices and impact Calavo’s quarterly revenue numbers, but it couldn’t answer the question about how it would affect the margins and profitability of Calavo that quarter (1Q2017). Using weekly avocado prices we could see the slowing of sales and the 40% spike in prices. Use quarterly balance sheets to estimate company margins and changes in input costs, the confidence bands on the forecasted margins for the company remained wide.

We concluded in our report that “Supply, on the other hand, is having problems keeping up. An unexpected cold snap this winter led to a worldwide collapse in avocado production. California avocado farms suffered the worst, dropping in production volume by almost 50% year/year according to data released by the Hass Avocado Board. Mexico followed close behind with a 20% year/year decline in production quantity. Price elasticities suggest that, even with the demand spike, avocado revenues in the US could be down around 18%.”

The Guacamole Situation

There are limitations with this type of analysis. We only had limited histories of company data to review and it was rare for avocado prices to spike this much. To top it off, the spike occurred just before Cinco de Mayo! The quantitative models don’t understand how inelastic the demand for guacamole is at that time of year and it may not have a sense for how Calavo could pass along the prices to consumers and distributors.

  • If you believed the historical demand patterns and elasticity for avocados would continue, then the earnings-per-share (EPS) for Calavo $CVGW would have disappointed and likely the stock would have declined.
  • If you believed that due to the holiday or perhaps due to the emerging love of avocado toast among my peers that demand for avocados and that ‘this time is different’, then EPS for avocados would surprise to the upside and the stock should rise.

Ultimately we decided to go with the models and assume a more constant elasticity in the demand for avocados. Our revenue forecasts were dead on. Our EPS forecast… the one that primarily drives stock moves on earnings day, was too low. Demand was strong for avocados, EPS exceeded expectations and the stock went up. C’est la vie

This shows the essential conflict between extrapolating historical relationships and overreacting to perceived changing market conditions. Although in this case, sticking directly with the data and underweighting this special situation led to a worse forecast, this is not always the case. Human’s make errors and this wikipedia article on Cognitive Biases is very long.

Striking the balance between using data and algorithms to identify emerging trends and give you confidence in the persistence of known relationships, while relying on humans to shift the types of inputs and to question whether ‘this time is different’ seemed to be the key way to integrate human insight with a machine’s probabilities. These days I simply state that “humans give you possibilities, Machines give you probabilities”

Humans give you possibilities, machines give you probabilities

Sean Kruzel – August 2016

Machine Learning-Based Trading Decisions Or:

How I Learned to Stop Worrying and Love when Models Break Down

This talk first covers why using ‘Machine Learning in Finance is Hard‘ and then walks investors through three ways to apply Machine Learning and Science to:

  1. Evaluate Portfolio Managers
  2. Enter and Exit Trades
  3. Shift Factors and learn from your models’ failures

The following sections will briefly cover these topics and I’m sure we’ll elaborate on them more in the coming months. Please leave us a note in the comments below or on social media to let us know which one’s you’d like to hear more about.

Investor Decisions: How to evaluate portfolio managers

One of the key attributes of a modern portfolio manager is there ability to high risk-adjusted returns. Whether that is measured by a Sharpe, Sortino or, if you’re fancy, an Omega Ratio, the idea is that the larger the risks required to earn a reward, the more the reward should be penalized. Although this sounds great theoretically, these metrics are often difficult to use in practice, especially for strategies with limited track records. These reasons stem from the fact that metrics like the Sharpe Ratio are similar to a ‘t-distribution’ and have ‘fat-tails’. Meaning, using these measures alone, it is very hard to separate ‘Lucky’ from ‘Good’.

What is Lucky vs Good?

Some common target benchmarks for active managers stem are closely related to the average investment horizons of their investors. As a rule of thumb, I typically expect a good active manager at an Asset Manager to have a Sharpe Ratio (or Information Ratio if you’re being precise) of > 1.0 and a manager at a Hedge Fund to have a Sharpe Ratio > 2.0. The reason for this is that a manager with a Sharpe Ratio > 1.0 is expected to make money 96% of the time after 3 years. A Sharpe Ratio > 2.0 is expected to make money 98% of the time every year. If you work for a hedge fund and you are paid based on a ‘eat-what-you-kill’ model you’d like to be at least 98% sure you’ll be paid in a given year. For asset management firms with more patience and more balanced salary to bonus ratios, having solid returns on a three-year track record is the gold star you’re trying to achieve.

Understanding Lucky vs Good: Is my Sharpe > 2.0?

It is easy enough to calculate your Sharpe ratio for a given timeseries but the real question your investors want to know is whether you are more likely to have a 2.0 Sharpe or a 0.0 Sharpe. If it’s the latter, then they should withdraw their money and you should go back to the drawing board.

It can take almost a year to effectively seperate lucky vs good just by looking at returns (and assuming the same volatility)

It is also true when looking at the distribution of Sharpe Ratios over time. In fact most strategies with high realized Sharpe ratios in the first 3 months are actually just lucky!

Classical Approaches to Evaluating a Portfolio Manager

Excluding other important factors like negligence, style drift, fees, etc… Let’s look at three metrics currently employed to track a PM’s performance:

  1. How much money have they lost in 1 day?
  2. How much money have they lost overall?
  3. What is my faith in their ability to continue to be profitable?

When you stop focusing on the performance measures, which as we showed above can be misleading, and instead focus on investor problem:

An investor in hedge funds wants to invest in as many 2.0 Sharpe strategies as possible and as few 0.0 Sharpe strategies as possible.

Therefore an investor can use the Baye’s Rule and trailing performance data to update their probabilities for each investment.

Below is an illustration of this exercise. NOT INVESTMENT RECOMMENDATION. Use this as food for thought for your own research and not specific guidance for or against any investment.

The above distributions show an investor who believed there was a 10% chance the PM had a 2.0 Sharpe ratio strategy (see Prior Odds Ratio = 1/10) and they wanted to be at least 5% sure that they are invested in a 2.0 Sharpe strategy (Odds Ratio = 1/20). If after 3 months (using only three data points) the Sharpe ratio was < -0.73 they should shut down the strategy (see disclaimer above). Likewise, the same could be said that after 1 year, they should exit the strategy if it only has a 0.66 Sharpe ratio.

Note this is contrary to the received wisdom of investment managers that like to ‘cut losers early’ and ‘let winners ride’. In reality you should ‘give losers some time’ and ‘cut winners if they haven’t won enough’

Trade Entry and Exit

Another type of investment decision that this bayesian approach can help improve is how to set ‘stop loss levels’. A stop loss is a rule-of-thumb that investors use as a signal when they should get out of a particular trade. Typically these rules fall into the following three categories: Fixed Loss, Daily Loss, Cumulative Loss over time.

If you followed the previous section, you might guess that we say that these three methods are inferior to a rule that focused on testing the probability of being right vs the probability of being wrong in a given trade.

To use this approach you need to specify your prior beliefs and your confidence in your strategy vs another strategy. I typically either use what the market is pricing or what my risk models spit out as a generic alternative forecast to test my strategies against.

For those in the finance community, you aren’t typically that familiar with precision and recall curves when applied to trading. We can think of them as showing the relative tradeoff between type 1 and type 2 errors.

  • Type 1 error: your trading idea is invalid but you haven’t hit your stop loss level yet.
  • Type 2 error: your idea was right but you hit your stop loss.

The latter example is often the most painful for traders as everyone hates having the right idea but not making money from it. Most people don’t even realize they make type 1 errors: who likes to admit to themselves that they were wrong but got lucky and made money on a trade.

We go on in the talk to show that by using the Bayes Ratio we can achieve higher Precision and Recall and minimize type 1 and type 2 errors better than other stop-loss strategies.

We show that a better stop loss level comes from looking at trailing Sharpe Ratios over time and deriving a curve based on your prior beliefs and your hull hypothesis. This maximized your probability of staying in a trade when you are correct and getting you out more quickly when you’re wrong. Here’s an example from the talk (using an Abenomics USDJPY example)

Let us know if you’d like us to release a calculator to quickly derive these stop-loss-curves.

Factors as Investment Heuristics

When your models break you learn something

The end of this talk brings back together a framework for how humans and machines can work together in an investment framework. The key insight here is that all quantitative models based only historical relationships will inevitably fail because the structure of markets evolves.

Just think about how many quadrillions of interactions that occur within the real economy and markets that affect the pricing of the S&P500. How much confidence could possibly have in your simple model forecasting the SPY that just uses 1 variable: the P/E ratio?

Strong Opinions, Weakly Held

Paul Saffo

The goal therefore as a modern investor is to obtain thousands of high conviction empirical models that we call ‘Strong Opinions’ and produce a framework to adapt to their breaking down: We call this framework Breakpoints and have been using it since 2016 to forecast economic data, trends and correlations within global macro markets. https://get.breakpoint.report/

Thank you

This has been our longest post to date and I’m excited you made it down here. There’s a lot more we can do to expand on these ideas and make the more actionable. Please leave a note in the comments below or message me on Twitter @seankruzel.

Sean Kruzel

Based in Boston, Sean is a portfolio manager, MIT grad, and founder of Portformer. When he’s not investing or programming, he’s probably sharpening his skis for the next icy, New England snowstorm. Check out his full bio here: https://www.portformer.com/our-team

3 comments

  • Workflows & implementable insights — to elevate each step of the advisor/client experience

    Use our head-to-head Replacement Scores™ to quickly create client proposals, discover funds, improve model portfolios, and monitor investment plans