Data-driven individual stock approach

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
psteinx
Posts: 2659
Joined: Tue Mar 13, 2007 2:24 pm

Data-driven individual stock approach

Post by psteinx » Tue Sep 19, 2017 10:42 am

So, from time to time I've considered something like this:

* Obtain access(1) to large set of high quality data on stock fundamentals (key data from income statement, balance sheet, etc), for large number of US stocks(2) going back many years(3).

* Do my own deep analysis, data mining, etc. I have good C/C++ skills, and experience with databases, statistical programs, etc.

* Hopefully come up with, and implement, some nice strategies, with positive risk-adjusted expected value, relative to the market.

(1) Perhaps by subscription
(2) International would be nice too, but good, comparable data might be tougher to get
(3) Ideally back to 1926-ish, but realistically might have to settle for less. Depending on the approach, 20 years of data would be a rough minimum, more would be better.

====

OK, so first reaction of lots of you readers - Boo!!!!! Individual stocks = BAD. Trying to beat the market = BAD. Blah Blah. I understand skepticism. I'm not looking to offer a deep defense of the proposal here, and I hope the focus is a bit more on the how than the why or the pros and cons. That said, this approach is, I think, broadly similar in concept to what DFA has been doing for years (though they relied on academics for some of the initial research), and the broader smart beta approach (in its many flavors), more recently. There are at least three advantages that I see for a self-rolled approach versus the fund packages offered by other providers:

1) I can make the underlying algorithms as simple or as complex as I choose, with focus on effectiveness. The big boys want effectiveness, but they also want something they can explain to clients - something that's marketable

2) Scale issues - I can have less market impact and work with less liquid names

3) Tax control - I can optimize loss/gains realization and other tax issues generally, and with regard to my own tax situation

====

OK, so a quant approach. I'm mainly thinking intermediate to longer term - hoping to find that companies with favorable ratios of price to X, Y, and Z on the IS/BS have favorable returns, or that internal relationships on these statements suggest favorable/unfavorable performance. I think most of this stuff, realistically, is going to play out on a scale of, say, 12-60+ months. I'd be open to shorter timescales, but then, IMO, the focus shifts away from the IS/BS fundamentals and more to outside issues and/or price movements that suggest shorter term earnings surprises and/or technical/momentum returns.

Data/implementation. I'm aware of lots of potential data sources, and with searching, could probably find many more. I realize that some subscriptions for high quality data are quite expensive (~5 figures). I'm amenable to that. But of course cheap (or free), high quality data sources would be great. I believe there are also services that sort of roll data and implementation together. The one I recall looking at had data, and an interface to perform backtests. I can't quite remember the business model - pay them/subscribe to use it for implementation? The problem with a sort of "walled garden" approach is that there would likely be limited flexibility to analyze the data and implement backtest portfolios in the way(s) that I'd really want.

====

Anyways, while I realize that much more passive, hands-off approaches predominate here, I'm wondering if others have taken approaches broadly similar to what I've outlined, and what thoughts/experiences you might be willing to share.

rbaldini
Posts: 756
Joined: Mon Mar 23, 2015 3:20 pm

Re: Data-driven individual stock approach

Post by rbaldini » Tue Sep 19, 2017 10:51 am

As a (I think) similarly skilled data scientist, I'd wonder if its worth the time. Could you spend those hours coding up something else that might advance your career and earning potential? Or working harder on your day job?

If you do a good job, it's totally possible that you could beat the market on average - at least minus trading costs (which I don't know anything about). And again, maybe not minus the opportunity cost of what you could be doing with all your time.

But... are you good enough? Can your neural networks beat the ones that the best investing firms are using now (I don't actually know if they do this, but someone's gotta be doing it)? It's not a new idea, after all. Kaggle (the place for data science competitions) held a data competition a year or two on exactly this topic. The results are now public knowledge. Can you beat these guys? Maybe, but I'm not so confident. https://www.kaggle.com/c/the-winton-sto ... -challenge

jebmke
Posts: 7005
Joined: Thu Apr 05, 2007 2:44 pm

Re: Data-driven individual stock approach

Post by jebmke » Tue Sep 19, 2017 10:57 am

My wife was a portfolio manager managing primarily small and mid-cap value portfolios. It was a full time job. She was paid well for it so you would have to impute some income for your labor and subtract that from your investment returns to measure the success of the strategy. In her case the cost of managing the portfolio was spread over hundreds of millions of dollars. Scale is important.
When you discover that you are riding a dead horse, the best strategy is to dismount.

gips
Posts: 278
Joined: Mon May 13, 2013 5:42 pm

Re: Data-driven individual stock approach

Post by gips » Tue Sep 19, 2017 11:01 am

I've worked on wall st for 40 years, all in an around IT (I have an ms in cs). If it's a hobby, or you want to improve your IT skillset, great idea. But I don't think you're going to make money.

Imagine this scenario: Ivy league educated portfolio/hedge managers paired with a team of talented (mit/caltech/top Russian and Chinese graduates) IT professionals. A budget that pays for every piece of data and software platform(s) they want. Very low transaction costs. Knowledgeable traders and conplex algos to execute trades. On a risk-adjusted/luck adjusted basis, 90% (or more) don't beat the market long-term. Now repeat that scenario 10k times across the world and you have a sense of your competition.

alex_686
Posts: 2584
Joined: Mon Feb 09, 2015 2:39 pm

Re: Data-driven individual stock approach

Post by alex_686 » Tue Sep 19, 2017 11:09 am

I do believe that active investing in individual stocks can work. It take a fair amount of time and effort to do so. Now, let me shoot down much of what your are purposing.

I am going to focus on fundamental factors back from 1926. You are going to face many secular changes since then. Accounting methods have changed multiple times since then, so you are going to need to account for those. Plus some more subtle ones. The tax code has changed so companies have manipulated their accounting to maximize their returns. Same economic returns, different accounting returns. Companies have changed in how they work. A good example is book value. This was once an important measure, now less so.

Here is a thing to chew on, "pro forma" reporting can at times lead to better predictive results, other times less so. Stock based compensation, writing off good will, are cases in point.

I think you have 2 choices.

The first is to go down the fundamental path. Be like Warren Buffet and start reading annual reports. Stock screeners can help you limit the number of stocks to review to a manageable level but it will require subjective judgement to pick the winners. This works well for long holding periods. Say more than 3 years. I would advocate this method.

The second would be statistical arbitrage or technical trading. This sounds more like what you are trying to do. This would be holdings for less than 12 months. Actually, probably for a much shorter period than that. I might point you towards the Chartered Market Technician program. This not not a Bogglehead approach. Whatever magic sauce you create will work for a long period of time and then it won't. Long Term Capital Management is a extreme case of this.

livesoft
Posts: 57191
Joined: Thu Mar 01, 2007 8:00 pm

Re: Data-driven individual stock approach

Post by livesoft » Tue Sep 19, 2017 11:17 am

I have written software that I use to help me. I use ETFs and not individual stocks. Have some fun! And no one will believe you, but so what if you are laughing all the way to the bank!

Read some books, too, such as "The Quants"
This signature message sponsored by sscritic: Learn to fish.

gips
Posts: 278
Joined: Mon May 13, 2013 5:42 pm

Re: Data-driven individual stock approach

Post by gips » Tue Sep 19, 2017 11:22 am

livesoft, I know you're a student of the market. if no one believes you, why don't you post your trades before you make them? I'll volunteer to setup a google doc that timestamps your entries.
Last edited by gips on Tue Sep 19, 2017 11:24 am, edited 1 time in total.

psteinx
Posts: 2659
Joined: Tue Mar 13, 2007 2:24 pm

Re: Data-driven individual stock approach

Post by psteinx » Tue Sep 19, 2017 11:22 am

alex_686 wrote:
Tue Sep 19, 2017 11:09 am
I am going to focus on fundamental factors back from 1926. You are going to face many secular changes since then. Accounting methods have changed multiple times since then, so you are going to need to account for those. Plus some more subtle ones.
In fact I generally agree with this, and would perhaps emphasize it. I think some of those who advocate DFA-ish value tilting or slice and dice ignore the extent to which book value, in particular, has become a less meaningful marker as corporate value and profit-potential has been driven more by intellectual property versus factories and machinery.

I'm not thinking of a dumb, brute force, "this worked in 1928, therefore it should work in 2018" approach. I think a problem of academic approaches is that the publishing environment is not conducive to a nuanced approach. A financial academic often structures his/her papers in the form of 1-3 independent variables influencing 1-2 dependent variables in a fixed way. But markets evolve, and if I'm not trying to satisfy a reviewing committee I think I've got quite a bit more flexibility.

bigred77
Posts: 1933
Joined: Sat Jun 11, 2011 4:53 pm

Re: Data-driven individual stock approach

Post by bigred77 » Tue Sep 19, 2017 11:29 am

gips wrote:
Tue Sep 19, 2017 11:01 am
I've worked on wall st for 40 years, all in an around IT (I have an ms in cs). If it's a hobby, or you want to improve your IT skillset, great idea. But I don't think you're going to make money.

Imagine this scenario: Ivy league educated portfolio/hedge managers paired with a team of talented (mit/caltech/top Russian and Chinese graduates) IT professionals. A budget that pays for every piece of data and software platform(s) they want. Very low transaction costs. Knowledgeable traders and conplex algos to execute trades. On a risk-adjusted/luck adjusted basis, 90% (or more) don't beat the market long-term. Now repeat that scenario 10k times across the world and you have a sense of your competition.
+1

This is what the professionals do and they can't make it work. Why would you be able to?

Maverick3320
Posts: 357
Joined: Tue May 12, 2015 2:59 pm

Re: Data-driven individual stock approach

Post by Maverick3320 » Tue Sep 19, 2017 11:30 am

gips wrote:
Tue Sep 19, 2017 11:01 am
I've worked on wall st for 40 years, all in an around IT (I have an ms in cs). If it's a hobby, or you want to improve your IT skillset, great idea. But I don't think you're going to make money.

Imagine this scenario: Ivy league educated portfolio/hedge managers paired with a team of talented (mit/caltech/top Russian and Chinese graduates) IT professionals. A budget that pays for every piece of data and software platform(s) they want. Very low transaction costs. Knowledgeable traders and conplex algos to execute trades. On a risk-adjusted/luck adjusted basis, 90% (or more) don't beat the market long-term. Now repeat that scenario 10k times across the world and you have a sense of your competition.
This.

livesoft
Posts: 57191
Joined: Thu Mar 01, 2007 8:00 pm

Re: Data-driven individual stock approach

Post by livesoft » Tue Sep 19, 2017 11:32 am

gips wrote:
Tue Sep 19, 2017 11:22 am
livesoft, I know you're a student of the market. if no one believes you, why don't you post your trades before you make them? I'll volunteer to setup a google doc that timestamps your entries.
I do post my trades before I make them at a place where I cannot control nor fake the timestamps. I've been doing this for years.
This signature message sponsored by sscritic: Learn to fish.

alex_686
Posts: 2584
Joined: Mon Feb 09, 2015 2:39 pm

Re: Data-driven individual stock approach

Post by alex_686 » Tue Sep 19, 2017 12:36 pm

psteinx wrote:
Tue Sep 19, 2017 11:22 am
I'm not thinking of a dumb, brute force, "this worked in 1928, therefore it should work in 2018" approach.
So what are you thinking? The problem that I have with a data driven approach is that the stock market has many stable regimes that are not. There is a signal but it slowly weakens. There is a shock where the market rapidly moves from one regime to another. I think that there is wisdom to be gained from a historical review of the data but that it requires a subjective approach - but I doubt that is your view. What are you hoping to gleam?

User avatar
bottlecap
Posts: 5048
Joined: Tue Mar 06, 2007 11:21 pm
Location: Tennessee

Re: Data-driven individual stock approach

Post by bottlecap » Tue Sep 19, 2017 12:45 pm

You are not plowing any new ground here. Why not post your question on a stock picking website? At least you won't be reinventing the wheel.

JT

aristotelian
Posts: 3143
Joined: Wed Jan 11, 2017 8:05 pm

Re: Data-driven individual stock approach

Post by aristotelian » Tue Sep 19, 2017 1:10 pm

I would forget about the data stuff. No matter what you are going to want to build a diversified portfolio, and at that point you are best off just replicating the market. I do think it would be possible, if you had the time and sufficient funds, to do a low cost version of something like an equal-weight S&P that would give you lower costs than a S&P mutual fund but with the added benefit of tax loss and gain harvesting that might increase your after-tax returns. I think tax efficiency, not beating the market, is where you are better off focusing your energy.

NHRATA01
Posts: 439
Joined: Sat Oct 22, 2011 1:57 pm
Location: New York City area

Re: Data-driven individual stock approach

Post by NHRATA01 » Tue Sep 19, 2017 1:26 pm

Reading the OP, I am reminded of one of my favorite engineering memes on the subject of investing.

Image

psteinx
Posts: 2659
Joined: Tue Mar 13, 2007 2:24 pm

Re: Data-driven individual stock approach

Post by psteinx » Tue Sep 19, 2017 1:53 pm

alex_686 wrote:
Tue Sep 19, 2017 12:36 pm
psteinx wrote:
Tue Sep 19, 2017 11:22 am
I'm not thinking of a dumb, brute force, "this worked in 1928, therefore it should work in 2018" approach.
So what are you thinking? The problem that I have with a data driven approach is that the stock market has many stable regimes that are not. There is a signal but it slowly weakens. There is a shock where the market rapidly moves from one regime to another. I think that there is wisdom to be gained from a historical review of the data but that it requires a subjective approach - but I doubt that is your view. What are you hoping to gleam?
Well, it's hard to say exactly what path(s) I might go down (deeply) until I started exploring things. That said, I might model things with some degree of decay in signals/signal strength.

gips
Posts: 278
Joined: Mon May 13, 2013 5:42 pm

Re: Data-driven individual stock approach

Post by gips » Tue Sep 19, 2017 3:01 pm

livesoft wrote:
Tue Sep 19, 2017 11:32 am
gips wrote:
Tue Sep 19, 2017 11:22 am
livesoft, I know you're a student of the market. if no one believes you, why don't you post your trades before you make them? I'll volunteer to setup a google doc that timestamps your entries.
I do post my trades before I make them at a place where I cannot control nor fake the timestamps. I've been doing this for years.
well that's very cool! what is the site?

acanthurus
Posts: 270
Joined: Sun Aug 04, 2013 8:02 am

Re: Data-driven individual stock approach

Post by acanthurus » Tue Sep 19, 2017 3:48 pm

Removed
Last edited by acanthurus on Tue Oct 31, 2017 6:31 pm, edited 1 time in total.

psteinx
Posts: 2659
Joined: Tue Mar 13, 2007 2:24 pm

Re: Data-driven individual stock approach

Post by psteinx » Tue Sep 19, 2017 3:59 pm

acanthurus wrote:
Tue Sep 19, 2017 3:48 pm
Have you identified any free tools for scraping EDGAR or datasets resulting from that?
I haven't really done a deep dive into the possible datasets, including the free ones. Years ago I looked into CRSP/Compustat some (it was very pricey then). Bloomberg is an option. GFD is an option. Maybe even Morningstar on the low end. And I think there are multiple sites that kind of roll things together - provide data access and tools for backtesting and the like. The issue with the latter is flexibility. The issue in general (including with the likes of Bloomberg, GFD, etc) is quality, depth, breadth and cost.

I'm not averse to simpler to use solutions, but I'm at least open to something that requires more work on my end, in exchange for improvements in the other constraints, particularly flexibility. And anyways, right now this is more of a thought balloon by me. I realize there are a lot of reasons why doing it effectively would be hard.

snarlyjack
Posts: 496
Joined: Fri Aug 28, 2015 12:44 pm
Location: Montana

Re: Data-driven individual stock approach

Post by snarlyjack » Tue Sep 19, 2017 4:00 pm

Psteinx,

Their was a company of genius that had the same idea.
Their average IQ was 160.
I think the name of the company was "Long Term Management"?
Highly leveraged.
Long story short...they lost it all & then some. I think Allen Greenspan (Fed)
got involved & a bail out was involved. (Talk of jail time)...

I believe what Jack Bogle said..."No one knows anything".

livesoft
Posts: 57191
Joined: Thu Mar 01, 2007 8:00 pm

Re: Data-driven individual stock approach

Post by livesoft » Tue Sep 19, 2017 4:04 pm

I don't think numbers will help in this regard. My niche is learning about behavioral finance errors and traps. Bogleheads.org is a great place to get the pulse of investors making common "peace of mind" and "sleep well at night" mental accounting mistakes. Psychology and emotions of investors allows one to take advantage of them. I can't say how to capture that in a data-driven approach yet.
This signature message sponsored by sscritic: Learn to fish.

acanthurus
Posts: 270
Joined: Sun Aug 04, 2013 8:02 am

Re: Data-driven individual stock approach

Post by acanthurus » Tue Sep 19, 2017 4:06 pm

Removed
Last edited by acanthurus on Tue Oct 31, 2017 6:31 pm, edited 1 time in total.

User avatar
patrick013
Posts: 1878
Joined: Mon Jul 13, 2015 7:49 pm

Re: Data-driven individual stock approach

Post by patrick013 » Tue Sep 19, 2017 4:09 pm

I look at data, mostly PE and Debt/Equity ratio, but first look at
what if anything is happening that will give this company a
brighter future. More revenue, profits, and at a current low
PE.

Bought a steel mill and the next day it took out a billion dollar
loan after an acquisition. Decided to stay the course and it
eventually started running up good price gains.

Missed NVidia.

Bought a computer services company and sold for 100% gain in
a complete buyout/takeover.

Used to see opportunities buying companies like Boeing during a
labor strike. Most often business as usual after the strike.

So stay diversified and look for that event with $$$ on the end of it.
age in bonds, buy-and-hold, 10 year business cycle

acanthurus
Posts: 270
Joined: Sun Aug 04, 2013 8:02 am

Re: Data-driven individual stock approach

Post by acanthurus » Tue Sep 19, 2017 4:10 pm

Removed
Last edited by acanthurus on Tue Oct 31, 2017 6:31 pm, edited 1 time in total.

foo.c
Posts: 9
Joined: Sat Jul 15, 2017 4:55 pm

Re: Data-driven individual stock approach

Post by foo.c » Tue Sep 19, 2017 4:29 pm

Check out Quantopian.

I started there and then rolled my own test and simulation harness for algorithm development. For data I am using CSV files from Yahoo, so I'm rather limited by that, but there is price data for the ETFs and funds I use going to back to their inception.

My personal goal is more to limit the downside risk, so I am not really competing with the professionals or trying to get rich quick. I have written a couple that back-test and simulate really well, but I haven't put them into real world use yet.

incognito_man
Posts: 35
Joined: Fri Sep 08, 2017 11:00 pm
Location: Madison, WI

Re: Data-driven individual stock approach

Post by incognito_man » Tue Sep 19, 2017 6:12 pm

livesoft wrote:
Tue Sep 19, 2017 11:32 am
gips wrote:
Tue Sep 19, 2017 11:22 am
livesoft, I know you're a student of the market. if no one believes you, why don't you post your trades before you make them? I'll volunteer to setup a google doc that timestamps your entries.
I do post my trades before I make them at a place where I cannot control nor fake the timestamps. I've been doing this for years.
what's the subscription fee? :P

livesoft
Posts: 57191
Joined: Thu Mar 01, 2007 8:00 pm

Re: Data-driven individual stock approach

Post by livesoft » Tue Sep 19, 2017 6:19 pm

incognito_man wrote:
Tue Sep 19, 2017 6:12 pm
what's the subscription fee? :P
Are you saying that you would pay me money to watch me lose money right and left? Amazing!
This signature message sponsored by sscritic: Learn to fish.

User avatar
triceratop
Moderator
Posts: 3744
Joined: Tue Aug 04, 2015 8:20 pm
Location: la la land

Re: Data-driven individual stock approach

Post by triceratop » Tue Sep 19, 2017 6:25 pm

gips wrote:
Tue Sep 19, 2017 3:01 pm
livesoft wrote:
Tue Sep 19, 2017 11:32 am
gips wrote:
Tue Sep 19, 2017 11:22 am
livesoft, I know you're a student of the market. if no one believes you, why don't you post your trades before you make them? I'll volunteer to setup a google doc that timestamps your entries.
I do post my trades before I make them at a place where I cannot control nor fake the timestamps. I've been doing this for years.
well that's very cool! what is the site?
I believe the implication is that the site is his brokerage's ;)

Always read livesoft with an assumption of a bit of droll humor.
"To play the stock market is to play musical chairs under the chord progression of a bid-ask spread."

DenisD
Posts: 17
Joined: Sun Apr 19, 2009 9:16 pm

Re: Data-driven individual stock approach

Post by DenisD » Tue Sep 19, 2017 8:08 pm

psteinx wrote:
Tue Sep 19, 2017 10:42 am
I believe there are also services that sort of roll data and implementation together. The one I recall looking at had data, and an interface to perform backtests. I can't quite remember the business model - pay them/subscribe to use it for implementation? The problem with a sort of "walled garden" approach is that there would likely be limited flexibility to analyze the data and implement backtest portfolios in the way(s) that I'd really want.
Maybe the one you were looking at was https://www.portfolio123.com/

Compustat data back to 1999 for the US and Canada. Europe promised one of these days. Very powerful backtesting and portfolio management. But steep learning curve. Low to moderate cost, depending on how you want to use it. Free trial.

Post Reply