Number of optimized variables and ranking of system goodness
Number of optimized variables and ranking of system goodness
I am wondering about an index or indicator, which takes into account the number of variables a system contains. Should a systems measure of performance be adjusted to compensate for the number of optimized variables required to achieve the back tested result? Based on the belief that a higher number of curve fit variables is inferior or less likely to replicate in the uncertainty of the future. Thoughts???
Re: Number of optimized variables and ranking of system good
How might you go about testing this belief, to determine whether it is true?RedRock wrote:Based on the belief that a higher number of curve fit variables is inferior or less likely to replicate in the uncertainty of the future.
If one is contemplating using Blox Builder to test this belief, an obvious question is: does this belief make correct predictions for the ten trading systems that come pre-supplied with the software? These systems have a nice variety of #variables which control their actions:
- ATR Breakout: 4
- Bollinger Breakout: 3
- Bollinger Counter Trend: 5.5 (counting a true/false variable as 1/2)
- Donchian: 8
- Dual Moving Average: 4.5
- MACD system: 4.5
- RSI Trend Catcher: 5.5
- Stochastic system: 12.5
- Triple Moving Average: 6.5
- Turtle: 10.5
Sluggo,
Thanks for pointing out a superb starting point question.
A quote by c.f. from another current post sparked the thought initally.
"That's perhaps the best rationale for having very simple systems. We can have a much stronger basis for their continuing to work because the amount of truth required for their continued success is very low. In other words, we only have to be right about one or two things to be successful." CF
viewtopic.php?t=3326&highlight=
Statistically defining the validity of the supposition is now on my 'list'...
Thanks for pointing out a superb starting point question.
A quote by c.f. from another current post sparked the thought initally.
"That's perhaps the best rationale for having very simple systems. We can have a much stronger basis for their continuing to work because the amount of truth required for their continued success is very low. In other words, we only have to be right about one or two things to be successful." CF
viewtopic.php?t=3326&highlight=
Statistically defining the validity of the supposition is now on my 'list'...
RR, nice thread. I taught from Mark Fisher to test only those situation where tarders are " caught" , so they rush and cover if they were short or vice verse. Sometimes such moves last for 2 weeks, sometimes for even months... as I see it, there are traders who have some biased opinions (ego is usually involved) and they keep on Shorting the market thinking fundamentals are in their favor and so on, at some point of time they might have profits on their P/L statements, but its only temporary, so they have lotsa shorts and even increase them, them some other guys come in with technical bias or some new fundamentals (the first ones dont know about) and start buying like crazy, first group keeps on shorting, their profits get erased in a week or so, they're still biased it'll go lower. then "intelligent" public joins and pushes it higher, first grooup sits in losses and ego doesnt let them cover (or those are hedgers who dont care cos they have their cash crop going higher so that offsets losses). Good example is Soybean market, my Chicago broker kept on saying how bearish fundamentals are and that kinda bull ****, same was with Wheat, he had 4k loss per contract in Wheat trade. Mark said that MACD and RSI are inferior indicators, check out the NYMEX symposium
http://www.clicklive.com/NYMEX/symposium_2003/
there were 3 days intensive classes about it
http://www.clicklive.com/NYMEX/symposium_2003/
there were 3 days intensive classes about it
Simply put, using the number of indicators a system uses as a measure of system robustness. I think it could give a rough indication but that there'd be a point where lines get blurred. The system's concept would best tell whether or not it's robust, it's easy to see if it's obviously not robust but the rest seems to be a grey area that can't be neatly boxed by order of robustness.
Also, sometimes things like portfolio diversification become the factor if a system's backtest result can be achieved in actual trading.
Just my thoughts....
Also, sometimes things like portfolio diversification become the factor if a system's backtest result can be achieved in actual trading.
Just my thoughts....
robustness - just my idea
Just for a giigle I made up Blair's Law. It goes like this:
Since almost everything increase at a square.
The robustness of a tool decreases at the square of the number of its parameters.
So a tool with one parameter(p) = (p)^2 = 1
The holy grail has no parameters and thus = 0.
So a tool with 3 params is more than twice as bad as a tool with two. Of course none of this is proven it is just my way of thinking.
For the record I mostly use 2 but I am not counting stops or such because I see them as part of the money management part as opposed to the tool which just generates a market status like long or short.
Since almost everything increase at a square.
The robustness of a tool decreases at the square of the number of its parameters.
So a tool with one parameter(p) = (p)^2 = 1
The holy grail has no parameters and thus = 0.
So a tool with 3 params is more than twice as bad as a tool with two. Of course none of this is proven it is just my way of thinking.
For the record I mostly use 2 but I am not counting stops or such because I see them as part of the money management part as opposed to the tool which just generates a market status like long or short.
-
jaym
Hi,
Just as something can be overfit it can also be underfit. Both equally as dangerous. "Everything should be as simple as possible but no simpler." The number of tweakable or tweaked parameters are only one small part of the this question.
I think the best advice I saw in this post was sluggo's. You need to test the testing method like you test your "system".
Jay
Just as something can be overfit it can also be underfit. Both equally as dangerous. "Everything should be as simple as possible but no simpler." The number of tweakable or tweaked parameters are only one small part of the this question.
I think the best advice I saw in this post was sluggo's. You need to test the testing method like you test your "system".
Jay
I have a stock trading system with a large amount of variables and it works REALLY well. Anyone else have a system that's not so simple, yet works well over time? From the posts I've read, most don't stray too far from the donchian, ATR breakout, moving average x-over type of systems. Anyone else go crazy with Trading Blox?
D
D
BARLI WROTE
I was wondering how do you identify those situations? I assume that it could be documented by being on the floor and watching the buzz and forlorn faces while monitoring how much ground a market moved in such and such a time frame. But for someone who has not been on the floor and is just looking at data from a screen, how would you identify those periods where traders were "caught"?
Is it safe to assume that a signal would be generated by the ferocity with which a market moved in such a short time, and then key in on looking for an extension of that move as more and more traders are figuring out that they are caught?
In other words:
IF KEY REVERSAL AND/or ATR < OR > STDEV OF LAST 10 BARS BY 3.5% THEN ALERT "UH-OH "
How would you know traders are "caught" and it was not just another valid break out? Are they one and the same?
I appreciate your insights. Forgive me if I ask something proprietary, you just sparked my curiosity when I perused your post. Thanks
BARLI,taught from Mark Fisher to test only those situation where traders are " caught"
I was wondering how do you identify those situations? I assume that it could be documented by being on the floor and watching the buzz and forlorn faces while monitoring how much ground a market moved in such and such a time frame. But for someone who has not been on the floor and is just looking at data from a screen, how would you identify those periods where traders were "caught"?
Is it safe to assume that a signal would be generated by the ferocity with which a market moved in such a short time, and then key in on looking for an extension of that move as more and more traders are figuring out that they are caught?
In other words:
IF KEY REVERSAL AND/or ATR < OR > STDEV OF LAST 10 BARS BY 3.5% THEN ALERT "UH-OH "
How would you know traders are "caught" and it was not just another valid break out? Are they one and the same?
I appreciate your insights. Forgive me if I ask something proprietary, you just sparked my curiosity when I perused your post. Thanks
hi GENX, its a proprietary study of Mark Fisher, I will bring you a simple example of that. Mark explains about it also in details in his book Logical Trader. He has a system called ACD, its based on this system. ACD uses opening range as a corner stone.
Here's the example of being "caught" in the March 2007 Wheat market:
Usually, when the situation like that happens, Open for the day is also the Low of the day. Same applies to getting "caught" when traders were Long and had to exit their longs at the Open next day.
Here's the example of being "caught" in the March 2007 Wheat market:
Usually, when the situation like that happens, Open for the day is also the Low of the day. Same applies to getting "caught" when traders were Long and had to exit their longs at the Open next day.
I bought this one cos I read Paul Jones' review:
http://www.mbfcc.com/book/paul_tudor_jones.shtml
Paul and Mark been friends for a very long time, Mark taught traders his systems at Tudor. If you watched that speach of Jones at NYMEX 2003 Symposium, you heard that Paul came to speak only because Mark asked him to, that was kinda funny. Paul didn't reveal anything of his stuff of course. Another book I can suggest in this field is of Tom Demark called: New Market Timing Techniques: Innovative Studies in Market Rhythm & Price Exhaustion
http://www.mbfcc.com/book/paul_tudor_jones.shtml
Paul and Mark been friends for a very long time, Mark taught traders his systems at Tudor. If you watched that speach of Jones at NYMEX 2003 Symposium, you heard that Paul came to speak only because Mark asked him to, that was kinda funny. Paul didn't reveal anything of his stuff of course. Another book I can suggest in this field is of Tom Demark called: New Market Timing Techniques: Innovative Studies in Market Rhythm & Price Exhaustion
Excellent! Thanks a lot 
I just took a quick look at the idea. I have a tool I tested recently based on a similar idea but I was just using time and/or range and not factoring in volatility (that was on weekly bars), so I was having trouble pegging the size of the opening range. It won't be too difficult to test I imagine. I shall order the book(s) and try to bang up some tests over the next few weeks.
I just took a quick look at the idea. I have a tool I tested recently based on a similar idea but I was just using time and/or range and not factoring in volatility (that was on weekly bars), so I was having trouble pegging the size of the opening range. It won't be too difficult to test I imagine. I shall order the book(s) and try to bang up some tests over the next few weeks.
Well since I wanted to start with a weekly one I used the last (n) minutes of Friday or the first (n) minutes of Monday.
The results showed a success rate greater than would be acceptable if that period was not significant. then I tried reading back (Friday) or forward (Monday) if the range was too small. Then I tried the same if it was too big. Then I decided that I was missing something because those results were not better. So I shelved it for a while and tidied up my code so I can just feed ticks at a time rather than minutes at a time.
Over the last few weeks I have been using my new version to update the tests using my existing tool, just a moving average based thing. Not much exciting or different but just to reassure myself.
Thanks to you and this website I will have another go at it. Using ATR or some measure of volatility sounds like a real good idea. Anyway better people thanme have it already sorted so if I can get something close then I will be happy with that.
Do you think it goes ok on weekly ?
The results showed a success rate greater than would be acceptable if that period was not significant. then I tried reading back (Friday) or forward (Monday) if the range was too small. Then I tried the same if it was too big. Then I decided that I was missing something because those results were not better. So I shelved it for a while and tidied up my code so I can just feed ticks at a time rather than minutes at a time.
Over the last few weeks I have been using my new version to update the tests using my existing tool, just a moving average based thing. Not much exciting or different but just to reassure myself.
Thanks to you and this website I will have another go at it. Using ATR or some measure of volatility sounds like a real good idea. Anyway better people thanme have it already sorted so if I can get something close then I will be happy with that.
Do you think it goes ok on weekly ?