Hi Jimmy-
I’ll take a stab at it also… isn’t it great to engage with c.f. & read his posts? This is almost like having season tickets to the Chicago Bulls when MJ played, or watching Karl Malone & John Stockon play together. We are privileged.
How do I avoid over-optimizing and/or curve fitting a system? Is this step similar to conducting sensitivity analysis where you leave all components constant and vary one component at a time to see the effects of the change? Then, once you determine the "best" inputs for each component, you move onto step 3?
What I do is something along those lines. I always start with the concept, let’s call it a market map. Then my job is to break apart that map in as many ways as I can and test out each part to see if it makes more then just theoretical sense.
The best way to deal with problems is to resolve problems. By this bottom up approach, I believe it removes a lot of issues that might cause problems later on… the main issue I believe we have as system designers have to aviod is
lack of Robustness. This can be seen in Over-Optimizing or simply short changing our tests with limited data. Don’t think of it as a “system,” but think of it as parts to the whole. It is not a welfare system, every part must contribute to the greater good.
The best way around the over optimizing is throwing a literal ton of data at the system. Across all markets for as far as you can go. Another thing I personally like to see is activity of trades. I want things to occur early and often. And a nice calmness to the results when looking at neighboring variables. Hint: You just don’t want the neighbors to behave, you want the community to behave!
I also test out everything on its own merits, there are some creative ways to do this and they are easily conceived. Say for example an exit. I like to test this not just during an open trade, but I’ll turn it around and make it an entry and run it as such. If your system is only in the market 20% of the time and then you code and run an exit on it, you are missing the chance to run it on the other 80% of the data. Among other things,
exits are to prevent further losses, rotate that statement a bit, squint your eyes, and you will see that it is the inverted definition of the entry as in the
entries are to encourage further gains. So flip the code and make it into an entry, what are the strengths? What are the weaknesses?
When I find an exit that actually makes money when applied as an entry across the universe of data. I will gladly employ that as a component to the system. Do this to each part of the system. But keep the standards high.
Once you have components that are sturdy, then bring them together and then begin to buff out their brilliance. By looking at it in this way, you shouldn’t have any problems in the future in regards to lack of confidence in the system and losses that you didn’t see coming.
Or the one I love… the system stopped working! Yea, it stopped working the moment you stopped testing it.
The object isn’t to just build a system, but to build something that will work & last.
You need to know how the pieces fit together and what the peices are. once you know that, it is rather easy.
Hope it helps.