Musings on (worthless) Past Performance

[DPH]

I was working on a blog post today and whipped together a few tables and charts I thought might be of interest to some.

The general theme was to prove the worthlessness of most past performance statistics for CTAs. Primary target was the Sharpe Ratio, and frankly this tends to cover them all anyway.. Sharpe Ratio Covers Them All

I used a VERY simple strategy of buying the top 5 CTAs in 2007 as measured by their Sharpe Ratio.

I then compared that to my results five years later at the end of 2011 to see how I did. (below)..

[I am not able to embed the interactive tables and graphs here (I tried), you can see them, and download the data on my blog at:]

http://managedfutures.ws/2012/01/29/is_ ... worthless/

In a nutshell, it failed miserably. Performance fell apart, the out of sample Sharpe Ratios fell an average of 70%. Returns fell by roughly 60% and maximum drawdowns increased by roughly 30%.

I had hoped that maybe at least their peer group rankings remained somewhat constant (I.E. everybody fell apart equally) but nope, their peer group rankings went out the window too!

Conclusion........ Sharpe Ratio (and related measures) are reporters of yesterdays news only. Their value as tools to help predict or forecast future results is near zero!

[stopsareforwimps]

The general theme was to prove the worthlessness of most past performance statistics for CTAs ... the out of sample Sharpe Ratios fell an average of 70%. Returns fell by roughly 60% and maximum drawdowns increased by roughly 30%.

Would it be possible to compute a survivorship-bias-corrected prediction of future performance eg using Bayes's theorem, reverse engineering the survivorship bias from the results you report?

I have a second question: if the past performance of your back tested methodology for picking CTAs is good, does this mean it will continue to be good?

[DPH]

There is no question that survivorship bias is a MAJOR issue. As far as I am concerned it is the industry’s “dirty little secretâ€

[rajivm]

I don't know about any ratios..But Past performace normally is an indication of future good performance..Look at Soros, Jim Roger or other long time good performers..They STILL are doing well..
If Sluggo has written insightful articles in past .. I would bet he would write more in future so I better off reading his posts ..
I think in life in general Past performance does predict to a large extent what can happen in future..

[rabidric]

past performance tells you how well they played the most recent iteration(s) of The Game.

The Game can always change it's rules faster than you can adjust them with a backward looking method.

But a Game can maintain it's current rules for longer than you can bear if you try to prempt a rule change too often.

as ever, there are games within games. within games. wrapped up in a game.

At the end of the day you either gotta be very selective about where and how you get involved if you want to do better than the average. Or, you just keep moderate stake in the game through thick and thin, and weather the long droughts, so you are still in play when the good times roll and everybody else in washed out on the sidelines. Option 2 is intellectually/mechanically easier, but psychologically harder.

[DPH]

Raj,

I am afraid you’re “in the trapâ€

[Chris67]

i think , fwiw, its best just to understand that there will probably never be a CTA or fund that has an MAR of > 1 if it lasts 20 years and therefore allocating assets on the basis of teh last few years performance where a new CTA has great stats or an MAR of 3 is a ridiculo9us exercise
I wrote before about how 4 months after I launched my MAR was about 20
Well now its about 0.01 after 15 months
Thers nothing wrong with that but investors should work out what teh long term average is for a strategy and probably sell the MARS of 2 and 3 and buy the MARs of much much lower
Some things never change and I believe investors however will keep making teh same mistakes
Then you get Rentech of course - but that needs some serious thinking about

As for the sharpe ratio - its just nionsense and always has been
for a start nobody can agree on or even define the RFROR

[Moto moto]

DPH - you reminded me of a time that GS showed us a system in the late 1990s that was going to help the risk management team adequately minimise/"eliminate" risk to our long volatility equity option trading books from 8 different traders covering approximately 50 stocks
(paraphasing)
"we took your books, combined them, did in depth risk analysis and through our great system worked out that the best hedge for your combined portfolio is to be short 1.2 million AAA stock"

needless to say the risk guys were impressed and bought/leased the system at a small fortune because the big guys used it

.....even after we told them that AAA stock only traded less than 500,000 shares a day, was a single mine gold stock (that had rumours of predators and ended up getting taken over) and that as long volatility option players we actually wanted dislocated markets and high volatility to help hedge their other internal firm portfolios.....

in the end they scrapped it and went for that other great backward looking - but in its defense - best guess - measure of risk Var.

I guess in the end all you can do is understand that these are not perfect measures, know their weaknesses and work out when to and when not to apply them.....or just talk them up and sell, sell, sell. (note; not short, short, short)

[Chuck B]

Past performance can be an outstanding predictor of future success! That success is however in the marketing of said fund/adviser to the latest performance chasing idiots. Suck in the new money in the largest possible quantities in the shortest amount of time possible.

Of course we are just entering a period of mutual fund advertising of 3yr returns given the 1st quarter of 2009 market action. Marketing managers always look forward to these periods in time as your marketing window opens wide to suck in the unwary.

[rabidric]

chuck B is near the mark imho.

regularly re-calibrate yourself to "the idiots"(a changing population, as they die out to be replaced by new idiots who think they have a better take on things) , and then fade their game.... ***AFTER THEY HAVE TIPPED THEIR HAND***

I don't know if you can see which bit i have tried to highlight there

It is amazing, eerie and fun to watch the orderflow then turn your way and shovel money into your pockets.

Playing the Game means , playing the players.

whether the game is trading price, investing in funds, or a game of "spoof" down the pub.

[LeviF]

Here is an exercise I have for you. I will send you all the historical data of all the CTAs including those who blew out and are no longer in today’s survivorship bias results. I will remove their names so you have no way to peak into the future to know how they would have performed. THEN tell me who the 5 or so programs you would invest in are. Then we will hold those managers for the next 5, 10 or 20 years and see how you did.

I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.

[babelproofreader]

I recently read this blog post which has a nice graphical summary of this problem. Given the possibility of this sort of situation actually occurring, why would anyone trust any historical performance record as a good indication of future returns, whatever metric one chooses? Not to mention that you might be entrusting your money to the next MF Global, LTCM... etc.

[Chris67]

chuck b / Rabidric - yes I agree entirely and its name is
WINTON

[marriot]

I have yet to find a Cta that started with low gains and ended last years in a great way.
Is depressing the evidence that even after ten years of trading you can find sharply drops.
The only consistent point for survivors seem to be that first years are the best.
It will be nice a test like:" if the first year Ror is at least 100 %, jump in and exit after one of to years"
But this has to be done on all the universe, archived program too.

[DPH]

LeviF wrote:

I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.

My question is why only LTTF and not the others? Is it possible you are using hindsight bias because you know today, with the benefit of hindsight that LTTF did better than those other approaches?

Did you know as much about LTTF 10 or 20 years ago as you do today? If not, then what makes you think you would have made this choice back then?

[trackstar]

Here is an exercise I have for you. I will send you all the historical data of all the CTAs including those who blew out and are no longer in today’s survivorship bias results. I will remove their names so you have no way to peak into the future to know how they would have performed. THEN tell me who the 5 or so programs you would invest in are. Then we will hold those managers for the next 5, 10 or 20 years and see how you did.

I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.

my feelings exactly

[LeviF]

I have no idea know how the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, and arb guys did over that time period. LTTF is the only method I believe has a chance over the long run. Influenced by hindsight? Probably, but what is the alternative?

[rhc]

Dean thanks for sharing your thoughts in your blog (as linked above)

I’m wondering if I am reading the tables and graphs correctly since the 2007-2011 values differ from table to graph

For Emory Partners;

The table ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = -0.02
The graph ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = 0.15

For Shinnecock;

The table ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.33
The graph ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.47

For Blenheim;

The table ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.32
The graph ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.69

[rajivm]

I have no idea know how the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, and arb guys did over that time period. LTTF is the only method I believe has a chance over the long run. Influenced by hindsight? Probably, but what is the alternative?

Frankly LeviF,
I feel LTTF wasted a large part of my life in my early search for systems to trade...
My thoughts are unless someone has 20 million plus...How can one possibly even think of making a living out of Trading as LTTF...

Returns are low ....and drawdowns are Long streching more than 1 year often..

[DPH]

Dean thanks for sharing your thoughts in your blog (as linked above)

I’m wondering if I am reading the tables and graphs correctly since the 2007-2011 values differ from table to graph

For Emory Partners;

The table ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = -0.02
The graph ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = 0.15

For Shinnecock;

The table ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.33
The graph ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.47

For Blenheim;

The table ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.32
The graph ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.69

Whoops! Thank you for pointing that out! I had about 20 spreadsheets open on my desk and I accidentally used the one thru 2010 instead of 2011. Fortunately BarclayHedge clearly notates your data on every export. I have included some screen shots because it also shows the basic filters I employed in the data. See below:

2002-2006 data:




2007-2010 data:



2007-2011 data:



The correlation results using the data through 2011 (instead of 2010) was even more confirming. The correlation coefficient on the Sharpe Ratios went from roughly (-.07) to (-.14).