Testing Instrument Selection

[shakyamuni]

Previously, I have made oversimplifying assumptions with respect to my instrument selection algorithms in backtesting in order to keep the process as simple as possible.

Maybe you guys can share your thoughts and make some suggestions of other ideas and avenues that I might want to investigate.

[stancramer]

You describe a method that works in input-space: create a Frankenstein's monster of spliced-together pieces of price series.

Another methoud would operate in output-space: splice together pieces of equity curves.

Start by assuming one "trading system" and N instruments. It'll be easy to generalize to K trading systems at the end.

1. Run the trading system on each of the N instruments, generating and saving N different equity curves
2. Run the "instrument selection" code and for each bar (day, week, ??) of the test, record which instrument or instruments were selected.
3. Use the list of selections in step 2, to carve out pieces of the appropriate equity curve(s) from step 1. Splice them together and add them up.
4. Finito

[ksberg]

Shakyamuni,

I'm doing work in automatic porfolio selection as well. I'm not so concerned about having a system substitute markets real-time as I am in developing an objective measure for market selection. This motivation may lead to different approaches and results, but hopefully we share something in common.

What leads me down this path is noting how some systems perform better on certain markets. For instance, I've had a few favorite markets for a while, and generally refrain from trading markets where Turtle doesn't function as well. One of the mid-ranked markets was British Pound (BP): great enough to trade, but not to make it into my top 10. Counter to my experience, a friend reported that FOREX GBP/USD results were great, but that BP was not. Wait, that's the same underlying market (!). Well ... apparently not.

BTW: Even though that example was a big motivation for understanding the system-to-market relationship, I have not yet characterized the key differences between BP and GBP/USD. I suspect it has to do with liquidity and reduced volatility. But, from a systems perspective I still need better understanding of how FOREX interest differentials, carrying costs, and actual pip spread/slippage effect performance to make a fair comparison.

Here are my thoughts on generally approaches. The first is to develop a measure on the market itself. The second is to run the system on a single market and examine performance statistics. A third is to look at the equity curve(s) from a given system:market pairing.

Wells Wilder developed a method for selecting a market based on its characteristics for potential opportunity, and called it the Commodity Selection Index (CSI). It boils down to a application of his Directional Movement Indicator work (DMI), and the big-point-value. This kind of approach may have merit for short-term switching between markets, but I found no value for longer term market selection. I've convinced myself that this makes sense because of indicator lag: the longer, slower signal to switch occurs too late and loses too much of the move. I've tried some other market characterization methods that show some promise, but don't have anything solid to report.

The brute-force method of running the system on a given market and examining performance results appears to be workable. I used simply used a combination score of CAGR and MAR on single-market test runs. For example, testing single markets from 1983-1993 yeilded JO, JY, BP, ED, SF, KC, SB, LH, C, GC, CD, SI, CT, and PB as top choices (in descending order). Shifting 10 years forward, the same test produced ED, PA, JY, CL, GC, SF, CT, KC, LH, S, W, BP, CD, and SI. I was encouraged to find mostly the same markets in each 10 year period; only 4 out of 14 changed, or 30%. Better yet, the combined portfolio of top performers continued to perform well in the out-of-sample time frame 1993-2003. [for some unknown reason, I didn't include Bonds].

What's interesting to note is there is a fair representation of different sectors (Currencies, Softs, Interests, Meats, Grains, Metals, Energy). I suspected there might be some "bunching" of correlated markets using this technique, but it didn't happen. BTW: this test used a battery of longer term trend systems including Turtle, Thirteen, and PGO. One unanimous finding: SP tested poorly on all systems and time frames. I suspect other index products might be similar.

I'm also midstream to investigating mutliple market combinations. I'll use walk-forward periods with overlap, then a smaller out-of-sample test. For example, 10 year periods 1980-1990, 1981-1991, 1982-1992, etc. plus something like a 2 year out of sample validation. The intent here is to establish the optimal portfolio combos, and to use this as a starting point for market character analysis. I also want to understand nonstationarity of the optimal set: i.e. how does the drift occur?

The third area is using the equity curve, and includes techniques like mentioned by stancramer. My thoughts here are to borrow a page from Ralph Vince; literally, pages 157-170 of Portfolio Management Formulas, "Transporting Portfolio Concepts to Futures". This section describes a technique for balancing allocation between markets. With some modifications it can be used to help select markets. The procedure would be as follows:

• Cut the equity curve into holding period returns (HPR) of a given length (daily, weekly, monthly, whatever). Re-combine HPR segments with either zero or an equal distributed fraction, depending on number held in the portfolio.
  Do this for all possible combinations or markets (constrain if desired)
  Look for optimally weighted result. The maximum represents the ideal portfolio at that particular point.
  Extract markets based on ideal segment.
  [optional] Modify via "thread leveling" so that markets don't swap in-and-out.
  [optional] Level by extending and averaging several segments on each iteration
  [optional] Look for optimal extended segment

The formula for each sample segment becomes

Code: Select all

PortfolioHPR[k] = Sum(0*HPR[i]) + Sum((1/M)*HPR[j])
HPR[i]'s are the incremental equity curve returns for markets not included
HPR[j]'s are the incremental equity curve returns for the M markets that get included
PortfolioHPR[k] represents one possible combination of markets on a given segment

An example segment might be

Code: Select all

PortfolioHPR = 0*HPR#1 + 0*HPR#2 + 0.25*HPR#3 + 0.25*HPR#4 + 0.25*HPR#5 + 0.25*HPR#6

The nice thing about working on the equity curve is that it can be implemented real-time fairly easily. It is necessary to record new holding period returns, and then reference the arrays verses mucking with a system. The result of a selection would toggle a market:system pair on or off.

Anyway, a few ideas and three separate approaches. Feedback?

Cheers,

Kevin