Incorporating real-world parameters into simulated(paper) trading
"I think you're on the wrong track.  Getting more and more samples from the real world does not make your backtest more accurate, it just confirms that your strategy can withstand one particular sample path of a stochastic process. The reason why you find it simple to incorporate fees, commissions, taxes, etc. is because they're a static and constant process -- well they might change over time but most definitely uncorrelated to the markets. Modelling overnight returns or the top levels of the order book the next day is serious work.  First you have to select a suitable model (that's mostly theoretical work but experience can help a lot).  Then, in order to do it data-driven, you'd have to plough through thousands of days of sample data on a set of thousands of instruments to get a ""feeling"" (aka significant model parameters).   Apropos data mining, I think Excel might be the wrong tool for the job.  Level-2 data (even just the first 10 levels) is a massive blob.  For example, the NYSE OpenBook historical data weighs in at a massive 15 TB compressed (uncompressed 74 TB) for the last 10 years, and costs USD 200k.  Anyway, as for other factors to take into account: So how to account for all this in a backtest?  Personally, I would put in some penalty terms (as % on a return basis) for every factor you want to consider, don't hardcode them.  You can then run a stress test by exploring these parameters (i.e. assign some values in the range of 0 to whatever fits).  Explore them individually (only set one penalty term at a time) to get a feeling how the strategy might react to stress from that factor.  Then you can run the backtest with typical (or observed) combinations of penalty factors and slowly stress them altogether. Edit Just to avoid confusion about terminology.  A backtest in the strict sense (had I implemented this strategy X years ago, what would have happened?) won't benefit from any modelling simply because the real-world ""does the sampling"" for us. However, to evaluate a strategy's robustness you should account for the additional factors and run some stress tests.  If the strategy performs well in the real-world or no-stress scenario but produces losses once a tiny slippage occurs every now and again, you could conclude that the strategy is very fragile.  The key is to explore the maximum stress the strategy can handle (by whatever measure); if a lot you can call the strategy robust. The latter is what I personally call a backtest; the first procedure would go by the name ""extension towards the past"" or so. Some lightweight literature:" You said the decision will be made by EOD. If you've made the decision prior to the market close, I'd execute on the closing price.  If you are trading stocks with any decent volume, I'd not worry about the liquidity.  If your strategy's profits are so small that your gains are significantly impacted by say, the bid/ask spread (a penny or less for liquid stocks) I'd rethink the approach. You'll find the difference between the market open and prior night close is far greater than the normal bid/ask.
I traded futures for a brief period in school using the BrokersXpress platform (now part of OptionsXpress, which is in turn now part of Charles Schwab). They had a virtual trading platform, and apparently still do, and it was excellent. Since my main account was enabled for futures, this carried over to the virtual account, so I could trade a whole range of futures, options, stocks, etc. I spoke with OptionsXpress, and you don't need to fund your acount to use the virtual trading platform. However, they will cancel your account after an arbitrary period of time if you don't log in every few days. According to their customer service, there is no inactivity fee on your main account if you don't fund it and make no trades. I also used Stock-Trak for a class and despite finding the occasional bug or website performance issue, it provided a good experience. I received a discount because I used it through an educational institution, and customer service was quite good (probably for the same reason), but I don't know if those same benefits would apply to an individual signing up for it.  I signed up for top10traders about seven years ago when I was in secondary school, and it's completely free. Unfortunately, you get what you pay for, and the interface was poorly designed and slow. Furthermore, at that time, there were no restrictions that limited the number of shares you could buy to the number of outstanding shares, so you could buy as many as you could afford, even if you exceeded the number that physically existed. While this isn't an issue for large companies, it meant you could earn a killing trading highly illiquid pink sheet stocks because you could purchase billions of shares of companies with only a few thousand shares actually outstanding. I don't know if these issues have been corrected or not, but at the time, I and several other users took advantage of these oversights to rack up hundreds of trillions of dollars in a matter of days, so if you want a realistic simulation, this isn't it.  Investopedia also has a stock simulator that I've heard positive things about, although I haven't used it personally. Hey, sole proprietorships called (don't those comprise roughly 50% of all businesses?) they want to know what corporate tax is.  Hell, most of them want to know what payroll tax is.  They just know it's not fun paying both halves of it.  From my perspective over on the incorporated side: Oh HEY, I'm incorporated as an S-Corp or and LLC -remember those?- and they're going to suck out five percent MORE of my GROSS.  I'll fax you my cash flows statement.  It's going to look like a severed artery.  For those lucky enough to be joining us from the C-Corp world, enjoy trying to retain your key employees without seeing your payroll costs go through the roof.  If you have all your employees by the balls because they don't have the skills to easily transfer [if you think you do, hint: you don't] then I hope you have the stomach to watch them all falling further and further behind and into debt.   I don't. Sounds like you are a candidate for stock trading simulators.  Or just pick stocks and use Yahoo! or Google finance tools to track and see how you do. I wouldn't suggest you put real money into it.  You need to learn about research and timing and a bunch of other topics you can learn about here. I personally just stick to life cycle funds that are managed products that offer me a cruise control setting for investing. I don't know the term sole trader but usually the trade off is a corporation allows for shield from liability (i.e. creditors) more flexibility with respect to personal tax management, but higher operating costs (incorporation, financial statements, etc).  I always incorporate. * In the 70's, 80's and early 90's there were pinstriped brokers who took orders over the phone from people who wanted to buy and sell. They had a huge competitive advantage over the rest of the market due to the lack of transparency in the market's order book. Therefore you got screwed every time you wanted to trade, ie the markets were less efficient because transaction cost was high. Transaction cost is = bid-ask spread + how much you get screwed by the market insiders.  * In the 90's and early 00's there were automated trading systems that allowed people to conduct trades directly with computers, aka Algorithmic Trading. The markets were more efficient, because spreads became tighter as more people were able to enter the market on this platform (e.g. [Lightspeed](http://lightspeed.com)). The ability for market insiders to screw the general market was lessened because the exclusive access to the market's order book was eroded. Of course some Algorithmic Trading operations had a huge competitive advantage because they had great systems and great people. However it wasn't talked about because those who new about it were making a killing and keeping their mouths shut.   * Then in the mid to late 00's there was co-located algorithmic trading on very fast systems, aka HFT, a natural evolution of Algorithmic Trading. Now market insiders (= people with enough resources to field co-located machines and the the engineer/traders to manage them) expanded their  competitive advantage by discovering the market's order book (as they are able to see orders in a fraction of a second and then act on those orders). However to retract this natural efficiency in the markets you would need to create some kind of set of rules to even out the playing field. How can that be done?   ** Option 1) Transaction tax would just make the markets less efficient by increasing the cost of buying and selling. A generally bad thing because it discourages traders (to put money into stocks), which is of course how the capital markets are supported.   ** Option 2) Create rules to ensure everyone sees the same information at the same time and then permit anyone to use whatever technology they want to act on that information, so that the most efficient trading operations win.    ** Option 3) Create some artificial environment where no-one is allowed to have an advantage:  ensure everyone sees the same information at the same time, ensure everyone has the same technology, and ensure that the people who manage the systems have the exact same experience and intelligence etc...      Of course #2 is how it works, and it is the meritocratic basis which underpins Capitalism. I don't see why people have a problem with it. "If you want to make a profit from long term trading (whatever ""long term"" means for you), the best strategy is to let the good performers in your portfolio run, and cull the bad ones.  Of course that strategy is hard to follow, unless you have the perfect foresight to know exactly how long your best performing investments will continue to outperform the market, but markets don't always follow the assumption that perfect information is available to all participants, and hence ""momentum"" has a real-world effect on prices, whether or not some theorists have chosen to ignore it. But a fixed strategy of ""daily rebalancing"" does exactly the opposite of the above - it continuously reduces the holdings of good performers and increases the holdings of bad.  If this type of rebalancing is done more frequently than the constituents of benchmark index are adjusted, it is very likely to underperform the index in the long term. Other issues in a ""real world"" market are the impact of increased dealing costs on smaller parcels of securities, and the buy/sell spreads incurred in the daily rebalancing trades. If the market is up and down 1% on alternate days with no long tern trend, quite likely the fund will be repeatedly buying and selling small parcels of the same stocks to do its daily balancing." Algorithmic trading essentially banks on the fact that a price will fluctuate in tiny amounts over short periods of time, meaning the volatility is high in that given time frame. As the time frame increases the efficiency of algorithmic trading decreases and proper investment strategies such as due diligence, stock screening, and technical analysis become the more efficient methods.  Algorithms become less effective as the time frame increases due to the smoothing effect of volatility over time. Writing an algorithm that could predict future long-term prices would be an impossible feat because as the time frame is scaled up there are far less price fluctuations and trends (volatility smooths out) and so there is little to no benchmark for the formulas.  An algorithm simply wouldn't make sense for a long-term position. A computer can't predict, say, the next quarter, an ousted CEO, a buyout, or anything else that could effect the price of the security, never mind the psychology behind it all. Vice versa, researching a company's fundamentals just to bank on a 0.25% daily swing would not be efficient. Tax advantages or not, it is the most efficient methods that are preferred for a given time-scale of trading. nice work! really enjoyed looking through your website. do you see any possible application of Machine Learning (specifically tensorflow) to this? I was thinking about building a trading bot that uses data from various APIs as a strategy just as an experiment but I'm wondering what your insights are.