Anyone use Python to analyze their finances?

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
JD2775
Posts: 220
Joined: Thu Jul 09, 2015 10:47 pm

Anyone use Python to analyze their finances?

Post by JD2775 » Sun May 07, 2017 10:22 pm

I am in the midst of a 4-day "Python for Data Science" bootcamp where we are learning how to do data analysis through python scripts. It is very interesting and practical and makes me wonder if anyone has put this to use for finances....

I would love to be able write a script that would:

1. Log into misc finance websites, grab pieces of data (balance.. etc)
2. Take that data and compile it into a single document (spreadsheet or whatever)
3. Analyze the data and/or make predictive calculations

Has anyone done something similar using Python? If so, what have you done and how have you done it?

Just curious, I am sure there is a lot I can do with this

soda_bandit
Posts: 4
Joined: Tue Dec 30, 2014 3:26 pm

Re: Anyone use Python to analyze their finances?

Post by soda_bandit » Sun May 07, 2017 10:37 pm

I've used python a fair bit for running financial simulations and projections (#3 in your list) and for data cleaning and compiling (#2 in your list). I know enough to know writing the code to log into websites and grab data is possible, but it'd be a lot more work than either of the other two points so, thus far, I mostly export data out of existing aggregation websites like Mint, and have my own python scripts to parse the outputs.

Single biggest thing I've enjoyed about working with code instead of excel is how much easier it is to model uncertainly. For my own net worth projects I have a 50% confidence interval and a 90% confidence interval going forward (based on the shiller historical monthly stock market dataset).

JD2775
Posts: 220
Joined: Thu Jul 09, 2015 10:47 pm

Re: Anyone use Python to analyze their finances?

Post by JD2775 » Sun May 07, 2017 10:55 pm

soda_bandit wrote:I've used python a fair bit for running financial simulations and projections (#3 in your list) and for data cleaning and compiling (#2 in your list). I know enough to know writing the code to log into websites and grab data is possible, but it'd be a lot more work than either of the other two points so, thus far, I mostly export data out of existing aggregation websites like Mint, and have my own python scripts to parse the outputs.

Single biggest thing I've enjoyed about working with code instead of excel is how much easier it is to model uncertainly. For my own net worth projects I have a 50% confidence interval and a 90% confidence interval going forward (based on the shiller historical monthly stock market dataset).
Good stuff. I use SQL at work all the time but that's the extent of my "programming" knowledge. I am looking forward to putting Python to some good use...

techrover
Posts: 98
Joined: Tue Jun 12, 2012 2:23 pm

Re: Anyone use Python to analyze their finances?

Post by techrover » Sun May 07, 2017 11:43 pm

I have done OO for a long time in professional life - java/c++. Last year I dabbled with python for a financial analytics project and was very impressed by the numerous utilities/frameworks available - few notable ones: scipy/numpy, matplotlib.
However I have yet to do full fledged financial analysis as I had to abandon the project due to time constraints. If one of you can share some pointers to a working project, it would be good.

AlohaJoe
Posts: 3562
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Anyone use Python to analyze their finances?

Post by AlohaJoe » Mon May 08, 2017 12:07 am

I use Python a lot. I'm not sure it is really needed to "analyse" finances, though :) A pretty simple spreadsheet and occasional use of online tools like cfiresim are good enough for 99.9% of people, I reckon.

That said...

Pandas is a great library for data analysis: http://pandas.pydata.org/pandas-docs/stable/index.html
I use it extensively in my own analysis.

It has a related project, pandas-datareader https://pandas-datareader.readthedocs.io/en/latest/ that provides simple & easy access to Yahoo Finance, Google Finance, FRED, Fama/French, the World Bank, OECD, the TSP fund data, Oanda historical currency exchange rates, and a few other things. The bit on "caching queries" -- which actually uses the requests_cache library -- is also useful.

I've mostly used seaborn https://seaborn.pydata.org/ for visualising things. Though I'm thinking about trying out plotly https://plot.ly/python/ since it offers interactivity, which seems like a near-mandatory thing nowadays.

happenstance
Posts: 45
Joined: Sun Jul 26, 2015 11:24 am
Location: NYC

Re: Anyone use Python to analyze their finances?

Post by happenstance » Mon May 08, 2017 12:28 am

Yes, I use Python extensively for my financial understanding. I keep my data in Google Sheets and then connect it to a Python data platform for running analysis. More detailed, I have:

1) A Google Spreadsheet that contains every transaction, with one fund per tab. I just manually enter my transactions whenever I make a trade, and I add rows for dividends and capital gains when I receive my monthly account statements. This sheet also has a rollup tab to compute cost basis, number of outstanding shares, dividends, etc. for all my funds.
2) Another Google Spreadsheet where I track my overall allocation, rebalancing bands, and monthly investment calculator. This imports the transactions rollup sheet to get current positions and cost basis.
3) Jupyter/Pandas notebooks that use Google APIs to connect to my spreadsheets. Jupyter is a web-based interface to something that resembles Mathematica/Matlab. Pandas is a powerful Python data processing library that was originally created at AQR, so it's well-suited for handling timeseries financial data. These notebooks allow me to import my entire transaction history into a single "data frame", so that I can look at various things:
  • Rate of return on a monthly basis. With full transaction data, I can calculate this using the modified Dietz method, which is important for me since I'm in the accumulation stage and my inflows do cause other calculation methods to get biased.
  • Rate of return of my portfolio against any benchmark I can import. Pandas can import CSV feeds easily, so it's trivial to compare against any arbitrary benchmark returns.
  • Fama-French factor regressions against my precise portfolio, over time. Using the pandas.datareader module, you can import French's datasets, and then use the scikit Python module to run a linear regression. I tilt, so this is a rather significant tool for my portfolio management needs.
  • Compound annual growth rate over any time window.
  • Investment cash flows on a monthly basis.
  • In my transactions notebook, I also record NAV when I buy ETFs. This allows me to calculate how much I'm losing in trading costs against NAV.
What I really like about the Jupyter notebook setup is that I can slice any one of those above analyses arbitrarily by fund or time window, since it's all ultimately tied back to my transactions spreadsheet data. And I can output the results in a variety of charts or as tables.

The initial setup to get all my transactions organized was a bit of work, though I'm lucky that I only started investing a few years ago. Learning Pandas and the world of other PyData projects (Seaborn, PyPlot, Bokeh) has been fun. But I am a software engineer by trade, so this does come rather easily for me.

FamilyGuy87
Posts: 24
Joined: Tue Feb 18, 2014 2:26 pm

Re: Anyone use Python to analyze their finances?

Post by FamilyGuy87 » Mon May 08, 2017 8:31 am

I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.

When I combine both of these I can create very cool tools. I can keep track of all my transactions and display the transactions in very cool ways such as the sankey method as seen below.
Image

It also automatically downloads all the investment price data using pandas datareader and calculates Betas and Alphas for my portfolio. You can see the image below. I also used to calculate the efficient frontier and display that information also. But I did not find that as useful for personal investing. Since I keep track of my transactions and portfolio prices, I can calculate my IRR and other items. I have a dashboard page that shows all that information in detail.
Image

Finally, I can create different goals and create projections. The one below if for college savings.
Image

I think python is a really easy tool to be able to accomplish all three items in the OP.

FamilyGuy

User avatar
DartThrower
Posts: 812
Joined: Wed Mar 11, 2009 4:10 pm
Location: Philadelphia

Re: Anyone use Python to analyze their finances?

Post by DartThrower » Mon May 08, 2017 10:04 am

I've also been looking into Python, but for me it's been about building my career skill set. I don't know if I'll commit to learning Python in depth or not, but I just thought I'd share a link comparing Python to R. It seems that Python is used much more in general, but R is probably better for stat analysis tools. I have no horse in the race, so I will probably end up learning some of both. (My main language is SAS currently). Hope you find this useful:

https://www.datacamp.com/community/tuto ... gs.=DoSG9Y
A Boglehead can stay the course longer than the market can stay irrational.

Yeti
Posts: 115
Joined: Wed Oct 01, 2008 8:58 am

Re: Anyone use Python to analyze their finances?

Post by Yeti » Mon May 08, 2017 10:06 am

Interesting. I have been going through Codecademy and 'Learn Python the Hard Way' just to get familiar with the language over the past few weeks. I don't really have any strong background in writing code, mainly VBA to facilitate spreadsheets.

Are there any books that you would recommend on Python that you have found to be invaluable? Any other resources that you'd recommend?

Thanks!
Yeti

User avatar
JupiterJones
Posts: 2673
Joined: Tue Aug 24, 2010 3:25 pm
Location: Nashville, TN

Re: Anyone use Python to analyze their finances?

Post by JupiterJones » Mon May 08, 2017 10:12 am

I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).

Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
Stay on target...

User avatar
aj76er
Posts: 566
Joined: Tue Dec 01, 2015 11:34 pm
Location: Portland, OR

Re: Anyone use Python to analyze their finances?

Post by aj76er » Mon May 08, 2017 10:33 am

So far, google sheets does everything I need. I use one set of sheets to track expenses and another to track portfolio holdings (for rebalancing).
I make heavy use of the GOOGLEFINANCE() function.
I do wish it was a bit more mobile friendly, however. But I think that's just the nature of spreadsheets.
"Buy-and-hold, long-term, all-market-index strategies, implemented at rock-bottom cost, are the surest of all routes to the accumulation of wealth" - John C. Bogle

freebeer
Posts: 1983
Joined: Wed May 02, 2007 8:30 am
Location: Seattle area USA

Re: Anyone use Python to analyze their finances?

Post by freebeer » Mon May 08, 2017 10:57 am

JupiterJones wrote:I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).

Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
+1 to this. I see no big reason to use Python for personal finances, vs. any other general-purpose programming language, and there would be advantages to using a math/statistics-specialized language like R. But the real point is that doing custom software development for personal finances seems like major overkill (even for someone who is a professional or hobbyist programmer and I can't imagine it making sense for someone who is not).

If you want sexier charts than Excel / Google Sheets can do, maybe consider Tableau, see e.g. https://www.interworks.com/blog/skenned ... -too-often

FamilyGuy87
Posts: 24
Joined: Tue Feb 18, 2014 2:26 pm

Re: Anyone use Python to analyze their finances?

Post by FamilyGuy87 » Tue May 09, 2017 11:02 am

freebeer wrote:
JupiterJones wrote:I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).

Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
+1 to this. I see no big reason to use Python for personal finances, vs. any other general-purpose programming language, and there would be advantages to using a math/statistics-specialized language like R. But the real point is that doing custom software development for personal finances seems like major overkill (even for someone who is a professional or hobbyist programmer and I can't imagine it making sense for someone who is not).

If you want sexier charts than Excel / Google Sheets can do, maybe consider Tableau, see e.g. https://www.interworks.com/blog/skenned ... -too-often

I agree, you can do majority of the analysis using either excel or google sheets. I have used both prior to my python tool. The only time I found Excel/Google to be insufficient is when I wanted to integrate my personal budget with my investing and keep track of all my information in one place. It is doable with excel; however, it is cumbersome the more complex you make your tool.

So i guess if you are looking for something simple that gets the job done I would go with excel/google sheets. If you are looking for something complex I would lean towards python/R.

FamilyGuy

farnsy
Posts: 82
Joined: Tue Dec 15, 2015 8:49 pm

Re: Anyone use Python to analyze their finances?

Post by farnsy » Tue May 09, 2017 11:25 am

I use python for some general purpose programming, but when it comes to finance and statistics, including optimizing my own portfolio, I use R.

Both are very flexible and powerful. Python's a nicer language overall, but R is designed for these types of problems and it's the language that's used by people who solve them.

CppCoder
Posts: 809
Joined: Sat Jan 23, 2016 9:16 pm

Re: Anyone use Python to analyze their finances?

Post by CppCoder » Tue May 09, 2017 12:41 pm

Yeti wrote:Interesting. I have been going through Codecademy and 'Learn Python the Hard Way' just to get familiar with the language over the past few weeks. I don't really have any strong background in writing code, mainly VBA to facilitate spreadsheets.

Are there any books that you would recommend on Python that you have found to be invaluable? Any other resources that you'd recommend?

Thanks!
Yeti
My first Python book (15ish years ago) was Learning Python by Mark Lutz, which I liked. I think I read either the first or second edition, which was a much shorter book than the current version. I think the edition I read was in the 350 - 400 pages range. Now, the book is over 1600 pages. In fact, it used to be that you read the Learning Python O'Reilly book for an intro and the big Python O'Reilly book for details. Now, they're both 1600 pages, so I don't know the difference anymore. In general, I like O'Reilly books. My one complaint is that they often seem to embody the mantra of "Why say in one paragraph what you could instead say in five." That said, that style does lend itself to easier reading for beginners.

As to the OP's question, I do not use a programming language to handle any of my investing. I find that everything I need can be done in a spreadsheet, and I say that as someone who manufactures programming projects to do for fun. To those who don't get it, I equate it to other people doing crossword puzzles. That said, I have some fairly extensive experience with HTML screen scraping (necessary if the website doesn't provide a useful API to gather information) from several hobby projects (my favorite is my rental car price watching application). Anyway, unless you are doing this simply for fun, it's quite laborious and probably not worth your time. You have to manually examine the HTML and essentially write routines to parse/interact with it. My experience is in interacting with the DOM tree using webkit in a mixture of C++ and javascript. The biggest pain is that after your "done," your application will require updating forever to keep up with every little change the vendor makes to their HTML code. If you go that route, good luck.

Beliavsky
Posts: 688
Joined: Sun Jun 29, 2014 10:21 am

Re: Anyone use Python to analyze their finances?

Post by Beliavsky » Wed May 10, 2017 6:44 pm

AlohaJoe wrote:I use Python a lot. I'm not sure it is really needed to "analyse" finances, though :) A pretty simple spreadsheet and occasional use of online tools like cfiresim are good enough for 99.9% of people, I reckon.

That said...

Pandas is a great library for data analysis: http://pandas.pydata.org/pandas-docs/stable/index.html
I use it extensively in my own analysis.

It has a related project, pandas-datareader https://pandas-datareader.readthedocs.io/en/latest/ that provides simple & easy access to Yahoo Finance, Google Finance, FRED, Fama/French, the World Bank, OECD, the TSP fund data, Oanda historical currency exchange rates, and a few other things. The bit on "caching queries" -- which actually uses the requests_cache library -- is also useful.

I've mostly used seaborn https://seaborn.pydata.org/ for visualising things. Though I'm thinking about trying out plotly https://plot.ly/python/ since it offers interactivity, which seems like a near-mandatory thing nowadays.
To this good list of resources I would add Statsmodels for econometric models to predict volatility and returns (volatility is somewhat predictable based on GARCH models, but returns predictability depends on the asset and is more controversial) and the book "Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython". The low-cost brokerage firm Interactive Brokers has introduced a Python API for automated trading -- they have long had APIs for C++, Java, and C#.

Someone interested in Python for finance should also consider R, which was discussed in a recent thread.
Last edited by Beliavsky on Wed May 10, 2017 9:57 pm, edited 1 time in total.

User avatar
LadyGeek
Site Admin
Posts: 46769
Joined: Sat Dec 20, 2008 5:34 pm
Location: Philadelphia
Contact:

Re: Anyone use Python to analyze their finances?

Post by LadyGeek » Wed May 10, 2017 8:57 pm

If anyone would like to share their code, I can add it to the wiki: Using open source software for portfolio analysis

If there's a lot, upload the script to Google drive and share the link.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.

Gadget
Posts: 153
Joined: Fri Mar 17, 2017 1:38 pm

Re: Anyone use Python to analyze their finances?

Post by Gadget » Wed May 10, 2017 9:10 pm

I'm a software developer and am surprised at the extent some of you are tracking your finances with various scripts. To me, the Boggleheads wiki google sheets spreadsheet is good enough for most everything. If I want to track anything fancier, I can just use Mint or Personal Capital. But maybe I just don't like coding at home. Feels too much like work.

happenstance
Posts: 45
Joined: Sun Jul 26, 2015 11:24 am
Location: NYC

Re: Anyone use Python to analyze their finances?

Post by happenstance » Thu May 11, 2017 10:26 am

Gadget wrote:I'm a software developer and am surprised at the extent some of you are tracking your finances with various scripts. To me, the Boggleheads wiki google sheets spreadsheet is good enough for most everything. If I want to track anything fancier, I can just use Mint or Personal Capital. But maybe I just don't like coding at home. Feels too much like work.
Indeed, most of what I do in Jupyter/Pandas could be done in a spreadsheet, and it is what I used to do. But I think it's a lot easier to write a few short lines of Python where my data can have clear variable names, rather than several nested SUMIF()s and COUNTIF()s of anonymous ranges in a cramped formula bar. I also think it's a lot easier to create custom graphs using the various plotting libraries available, rather than being limited by specific spreadsheet template graphs. For me, this is more fun than work, but I definitely understand the not coding at home feeling (except for small automation tasks like this, I seldom do personal coding projects).

Regarding R, it is great for statistics, but I personally find that the language has a lot of unusual semantics that don't match most general purpose programming languages. With Numpy and Pandas, I can't think of anything that you can do in R that you couldn't do in Python. So if you're new to programming, I'd recommend a more general-purpose language like Python with data crunching libraries, where knowledge of Python could then be used to automate and program other tasks, rather than an a specialized language like R or Matlab.

McGilicutty
Posts: 144
Joined: Tue Dec 13, 2016 5:24 pm

Re: Anyone use Python to analyze their finances?

Post by McGilicutty » Thu May 11, 2017 10:50 am

I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.

User avatar
SeeMoe
Posts: 985
Joined: Sat Jul 18, 2015 11:30 am
Location: Near Philly..

Re: Anyone use Python to analyze their finances?

Post by SeeMoe » Thu May 11, 2017 1:11 pm

McGilicutty wrote:I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.
Q. Do you trust this stuff, or is it just another form of " crystal ball" gimmickry? I can see using the Bloomberg , in depth, company profile programs for educated guesswork, but a subscription is about $25k per month! According to CBS.

SEEMoe.. :annoyed
"By gnawing through a dike, even a Rat can destroy a nation ." {Edmund Burke}

GLState
Posts: 133
Joined: Wed Feb 15, 2017 10:38 am

Re: Anyone use Python to analyze their finances?

Post by GLState » Thu May 11, 2017 1:30 pm

For the OPs intended use, I would lean to using Excel for its easy input and short learning curve. I use R to explore financial data beyond my portfolio. With R or Python, we can do the work of the Portfolio Visualizer web site, Simba's spreadsheet or Morningstar's Growth of $10,000 charts, with the funds/stocks and weights of our choosing. If we want to compare the many Lazy Portfolios, Larry Portfolio, or the Permanent Portfolio with the Three Fund, Four Fund, or Eight Fund portfolios, we can in a few lines of code. I like to be able to translate something that I read into something I can see and manipulate.

IMO, Python requires a better programmer than R. R has over 6000 packages than contain functions for just about whatever we want to do. With Python, we may have to translate a formula into code ourselves. In the R packages, others have done much of the coding for us.

McGilicutty
Posts: 144
Joined: Tue Dec 13, 2016 5:24 pm

Re: Anyone use Python to analyze their finances?

Post by McGilicutty » Thu May 11, 2017 6:40 pm

SeeMoe wrote:
McGilicutty wrote:I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.
Q. Do you trust this stuff, or is it just another form of " crystal ball" gimmickry? I can see using the Bloomberg , in depth, company profile programs for educated guesswork, but a subscription is about $25k per month! According to CBS.

SEEMoe.. :annoyed
From the checks I've done the data usually matches up pretty closely to Google Finance's and Morningstar's. In any event, my program is just an initial screener. It doesn't execute trades. Before I did any trades I would do some more in-depth research.

Like I said, I've only been running the program for a couple of days and it's really just an experiment so far.

RoboFan
Posts: 33
Joined: Thu Apr 27, 2017 7:17 am

Re: Anyone use Python to analyze their finances?

Post by RoboFan » Fri May 12, 2017 7:34 am

Would some of the individuals writing high-level code be willing to share their motivation for doing this and comment any obstacles they may have come across? Specific questions are:

1. What do see as the main deficiencies in the free portfolio analysis tools offered by analysis provided by Personal Capital, SigFig, Morningstar, and Vanguards own Portfolio Analysis tool. I purposefully left Mint off this list as I see thins too being more geered to analyzing how much you spend (it was pretty quick to tell me I spend an insane amount in coffee).

2. How much of the source data you obtain is available using API's? Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis? Can yo access any of your out accounts via APIs.

For anyone one unfamiliar with APIs (Application Program Interface) these are internal calls that are used by different cloud based programs to talk to each other. Sometimes they are given out for free; sometimes you need to pay for them. In the finance world they allow data aggregators like Yodlee to access data from whatever account you are willing to provide the password too - but they are used by travel apps to merge flight information, with hotel accommodation and car rentals. Any data in the cloud. If you have ever downloaded data from one program to your computer and then uploaded the same data to a different program you have performed an unnessecary time consuming step that can be avoided by an API. They also allow specific security measures to be put in place, controlling what data can be downloaded or transferred.

I'm interested in the 2 questions above as it seems there are a lot of concerns about different Portfolio Analysis tools providing different answers. Different answers can be due to errors in coding, correct coding but different assumptions or definitions, or differences in source data.

McGilicutty
Posts: 144
Joined: Tue Dec 13, 2016 5:24 pm

Re: Anyone use Python to analyze their finances?

Post by McGilicutty » Fri May 12, 2017 8:19 am

RoboFan wrote:Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis?
Below is a code snippet for getting some data from Yahoo! I use the re and urlib.request libraries. Basically, for each ticker, the code downloads the associated HTML and then does some regular expression matching to get the data I want. I don't know if this is the best way to get the data, but it's the way I do it [Edit: When I copied and pasted my code it removed the indentations. I'll see if I can figure out how to get it back.]:

import urllib.request
import re

p2 = re.compile('MARKET_CAP.*?><.*?>(.*?)</span>')
p3 = re.compile('reactid="36">(.*?)<')
p4 = re.compile('reactid="37">(.*?)<')
p5 = re.compile('reactid="41">(.*?)<')

listStrMarketCaps = []
listFltDayPrices = []
listStrDayChanges = []
listFltAHPrices = []

intCount = 1
for strGoodTicker in listStrGoodTickers:
text = ""
try:
local_filename, headers = urllib.request.urlretrieve('https://finance.yahoo.com/quote/' + strGoodTicker + '/?p=' + strGoodTicker)
html = open(local_filename, encoding="utf-8")
text = html.read()
except:
print("An error occurred at line 59")

matchMarketCap = p2.search(text)
if (matchMarketCap != None):
listStrMarketCaps.append(matchMarketCap[1])
print(str(intCount) + ' Market Cap = ' + matchMarketCap[1])
else:
listStrMarketCaps.append("0.0")
print(str(intCount) + 'Market Cap = 0.0')

listStrDayPrice = p3.findall(text)
fltDayPrice = 0.0
try:
strDayPrice = listStrDayPrice[2]
fltDayPrice = float(strDayPrice)
except:
print("Error getting day price")

listFltDayPrices.append(fltDayPrice)

listStrDayChange = p4.findall(text)
strDayChange = ""
try:
strDayChange = listStrDayChange[1]
except:
print("Error getting day price change")

listStrDayChanges.append(strDayChange)

fltAHPrice = fltDayPrice
if blnAH:
listStrAHPrice = p5.findall(text)
try:
strAHPrice = listStrAHPrice[2]
fltAHPrice = float(strAHPrice)
except:
print("Error getting AH price")

listFltAHPrices.append(fltAHPrice)

print("Day price = " + str(fltDayPrice))
print("Day change = " + strDayChange)
print("AH Price = " + str(fltAHPrice))
print("")

intCount += 1

GLState
Posts: 133
Joined: Wed Feb 15, 2017 10:38 am

Re: Anyone use Python to analyze their finances?

Post by GLState » Fri May 12, 2017 11:11 am

With these two lines of code in R:
library(tidyquant)
data <- tq_get("IBM", get = 'key.stats')

I can get all of the following metrics for IBM:

Code: Select all

[1] "Ask"                                        "Ask.Size"                                  
 [3] "Average.Daily.Volume"                       "Bid"                                       
 [5] "Bid.Size"                                   "Book.Value"                                
 [7] "Change"                                     "Change.From.200.day.Moving.Average"        
 [9] "Change.From.50.day.Moving.Average"          "Change.From.52.week.High"                  
[11] "Change.From.52.week.Low"                    "Change.in.Percent"                         
[13] "Currency"                                   "Days.High"                                 
[15] "Days.Low"                                   "Days.Range"                                
[17] "Dividend.Pay.Date"                          "Dividend.per.Share"                        
[19] "Dividend.Yield"                             "EBITDA"                                    
[21] "EPS"                                        "EPS.Estimate.Current.Year"                 
[23] "EPS.Estimate.Next.Quarter"                  "EPS.Estimate.Next.Year"                    
[25] "Ex.Dividend.Date"                           "Float.Shares"                              
[27] "High.52.week"                               "Last.Trade.Date"                           
[29] "Last.Trade.Price.Only"                      "Last.Trade.Size"                           
[31] "Last.Trade.With.Time"                       "Low.52.week"                               
[33] "Market.Capitalization"                      "Moving.Average.200.day"                    
[35] "Moving.Average.50.day"                      "Name"                                      
[37] "Open"                                       "PE.Ratio"                                  
[39] "PEG.Ratio"                                  "Percent.Change.From.200.day.Moving.Average"
[41] "Percent.Change.From.50.day.Moving.Average"  "Percent.Change.From.52.week.High"          
[43] "Percent.Change.From.52.week.Low"            "Previous.Close"                            
[45] "Price.to.Book"                              "Price.to.EPS.Estimate.Current.Year"        
[47] "Price.to.EPS.Estimate.Next.Year"            "Price.to.Sales"                            
[49] "Range.52.week"                              "Revenue"                                   
[51] "Shares.Outstanding"                         "Short.Ratio"                               
[53] "Stock.Exchange"                             "Target.Price.1.yr."                        
[55] "Volume"                                    

User avatar
LadyGeek
Site Admin
Posts: 46769
Joined: Sat Dec 20, 2008 5:34 pm
Location: Philadelphia
Contact:

Re: Anyone use Python to analyze their finances?

Post by LadyGeek » Fri May 12, 2017 2:51 pm

McGilicutty wrote:[Edit: When I copied and pasted my code it removed the indentations. I'll see if I can figure out how to get it back.]:
Use the "Code" post edit button to surround your code, which is shown in the previous post. You can edit your post to fix the content, then bump the thread to let everyone know it was changed.

I should also mention that you don't need to pay for MS Excel. LibreOffice Calc is a free and open source alternative that works just fine. There are some compatibility differences on charts and possibly macros, but it shouldn't stop you from using the software.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.

User avatar
dual
Posts: 505
Joined: Mon Feb 26, 2007 7:02 pm

Re: Anyone use Python to analyze their finances?

Post by dual » Fri May 12, 2017 7:46 pm

Many here have mentioned that they use spreadsheets for their calculations. IMHO the use of a spreadsheet for anything more complex than adding a column of numbers is a serious mistake. The reason for this is that for anything more complex than that it is almost impossible to:
(a) validate the calculations to gain assurance that you are actually calculating what you want
(b) modify or extend the calculation to other cases.

I am not the only one that will not use a spreadsheet for serious computations. See this article by statistician Patrick Burns:
http://www.burns-stat.com/documents/tut ... addiction/

I would like to see more use of a high level language like Python in the computation software posted on Bogleheads. I am a Matlab man myself but I can also swing with Python or R. I will draw the line at Java or C++ but they are still better than Excel.

AlohaJoe
Posts: 3562
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Anyone use Python to analyze their finances?

Post by AlohaJoe » Fri May 12, 2017 8:50 pm

GLState wrote:With these two lines of code in R:
library(tidyquant)
data <- tq_get("IBM", get = 'key.stats')

I can get all of the following metrics for IBM
Sure you can do the same thing in 2 lines of python. Most normal people don't do what the previous poster did when fetching Yahoo quote data :) There are libraries for most things under the sun and generally people use those libraries...just like you use a library in your R example.

Code: Select all

import pandas_datareader.data as web
f = web.DataReader("F", 'yahoo')

AlohaJoe
Posts: 3562
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Anyone use Python to analyze their finances?

Post by AlohaJoe » Fri May 12, 2017 9:12 pm

RoboFan wrote:Would some of the individuals writing high-level code be willing to share their motivation for doing this and comment any obstacles they may have come across?
I wouldn't say that I do "portfolio analysis" since that's not what I'm interested in.

Here are some things that I've written in Python that would be either hard or impossible to do in Excel.

See if using CAPE10 to market time rebalancing makes a difference when using monthly data

Build a bond fund simulator that allows me to simulate historical performance of bonds

Extend that to do the same for Japanese government bonds, to see how bonds outside the US have performed

Compare 18 different withdrawal strategies in retirement

Compare 8 different "harvesting" strategies during retirement

Compare the relative importance of early "bad returns" and "high inflation" on retirement

Calculate safe withdrawal rates in Japan for various portfolio allocations

Compare various income smoothing strategies in retirement

Calculate how many months of gains are wiped out by various market corrections

What kinds of things are easier to do in Python?

1. I can run lag-1 autoregressions to simulate bond yields being low right and slowly drifting higher over the next several years...instead of assuming historical averages
2. I can calculate certainty-equivalent withdrawals, Gompertz mortality, and Sustainable Spending Rates -- all of which have slightly complicated math
3. I can calculate Milevsky's "risk of ruin" using the Gamma distribution CDF.
4. I can run 50 different Monte Carlo simulations -- varying lots of parameters -- in a simple for-loop instead of needing continually modify a spreadsheet (or have 50 tabs).

User avatar
LadyGeek
Site Admin
Posts: 46769
Joined: Sat Dec 20, 2008 5:34 pm
Location: Philadelphia
Contact:

Re: Anyone use Python to analyze their finances?

Post by LadyGeek » Fri May 12, 2017 9:30 pm

This thread is now in the wiki: Using open source software for portfolio analysis

I have added AlohaJoe's github links to the Python section.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.

dave_k
Posts: 219
Joined: Sat Dec 26, 2015 8:25 pm

Re: Anyone use Python to analyze their finances?

Post by dave_k » Fri May 12, 2017 9:45 pm

I haven't used Python specifically, but I have written custom code in various languages over the years to analyze financial data, including C++ and JavaScript. I have also used Excel quite a bit, which is more capable than many people realize, but is slow and cumbersome for running simulations.

User avatar
masterofinvesting
Posts: 80
Joined: Sat Jun 22, 2013 9:40 am

Re: Anyone use Python to analyze their finances?

Post by masterofinvesting » Fri May 12, 2017 10:32 pm

I have a related question. A lot of us have years (or decades) of financial data in Quicken, but we still rely on Quicken's reporting to extract and visualize and data that we need. Wouldn't it be nicer to have a scripting interface for Quicken? (just like Word or Excel - both of which can be scripted via VB macros or Python/Perl). Even a read only interface to Quicken's database would be quite useful. This way we can do some deeper and highly customized analysis than what Quicken reporting provides - of course, this isn't for everyone - probably for those (like me) who find this kind of analysis fun.

Couple of recent books I've read on intermediate Python that I found incredibly enlightening:
  • Fluent Python: Clear, Concise, and Effective Programming - Luciano Ramalho
  • Python Cookbook: Recipes for Mastering Python 3 - Brian K. Jones and David M. Beazley

McGilicutty
Posts: 144
Joined: Tue Dec 13, 2016 5:24 pm

Re: Anyone use Python to analyze their finances?

Post by McGilicutty » Sat May 13, 2017 6:42 am

McGilicutty wrote:
RoboFan wrote:Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis?
Below is a code snippet for getting some data from Yahoo! I use the re and urlib.request libraries. Basically, for each ticker, the code downloads the associated HTML and then does some regular expression matching to get the data I want. I don't know if this is the best way to get the data, but it's the way I do it.

Code: Select all

import urllib.request
import re

p2 = re.compile('MARKET_CAP.*?><.*?>(.*?)</span>')
p3 = re.compile('reactid="36">(.*?)<')
p4 = re.compile('reactid="37">(.*?)<')
p5 = re.compile('reactid="41">(.*?)<')

listStrMarketCaps = []
listFltDayPrices = []
listStrDayChanges = []
listFltAHPrices = []

intCount = 1
for strGoodTicker in listStrGoodTickers:
text = ""
    try:
        local_filename, headers = urllib.request.urlretrieve('https://finance.yahoo.com/quote/' + strGoodTicker + '/?p=' + strGoodTicker)
        html = open(local_filename, encoding="utf-8")
        text = html.read()
    except:
        print("An error occurred at line 59")

    matchMarketCap = p2.search(text)
    if (matchMarketCap != None):
        listStrMarketCaps.append(matchMarketCap[1])
        print(str(intCount) + '  Market Cap = ' + matchMarketCap[1])
    else:
        listStrMarketCaps.append("0.0")
        print(str(intCount) + 'Market Cap = 0.0')

    listStrDayPrice = p3.findall(text)
    fltDayPrice = 0.0
    try:
        strDayPrice = listStrDayPrice[2]
        fltDayPrice = float(strDayPrice)
    except:
        print("Error getting day price")

    listFltDayPrices.append(fltDayPrice)  

    listStrDayChange = p4.findall(text)
    strDayChange = ""
    try:
        strDayChange = listStrDayChange[1]
    except:
        print("Error getting day price change")

    listStrDayChanges.append(strDayChange)

    fltAHPrice = fltDayPrice
    if blnAH:
        listStrAHPrice = p5.findall(text)
        try:
            strAHPrice = listStrAHPrice[2]
            fltAHPrice = float(strAHPrice)
        except:
            print("Error getting AH price")

    listFltAHPrices.append(fltAHPrice)

    print("Day price = " + str(fltDayPrice))
    print("Day change = " + strDayChange)
    print("AH Price = " + str(fltAHPrice))
    print("")
        
    intCount += 1

Spedward
Posts: 140
Joined: Mon Jun 13, 2011 4:48 pm

Re: Anyone use Python to analyze their finances?

Post by Spedward » Sat May 13, 2017 8:41 am

masterofinvesting wrote:I have a related question. A lot of us have years (or decades) of financial data in Quicken, but we still rely on Quicken's reporting to extract and visualize and data that we need. Wouldn't it be nicer to have a scripting interface for Quicken? (just like Word or Excel - both of which can be scripted via VB macros or Python/Perl). Even a read only interface to Quicken's database would be quite useful. This way we can do some deeper and highly customized analysis than what Quicken reporting provides - of course, this isn't for everyone - probably for those (like me) who find this kind of analysis fun.

Couple of recent books I've read on intermediate Python that I found incredibly enlightening:
  • Fluent Python: Clear, Concise, and Effective Programming - Luciano Ramalho
  • Python Cookbook: Recipes for Mastering Python 3 - Brian K. Jones and David M. Beazley
I use Alteryx to do this. It works wonders and really takes reporting and data analytics based on info from quick to the next level.

msimon
Posts: 53
Joined: Tue Feb 19, 2008 11:57 am

Re: Anyone use Python to analyze their finances?

Post by msimon » Sat May 13, 2017 8:26 pm

I do not use Python for my finances yet (use it for a few other things like data validation etc).

I use a collection of data files, C programs, Unix shell scripts, Awk scripts and Excel spreadsheets to do the following:

Parse OFX transaction data from Vanguard. Download monthly closing price data from Yahoo.

Keep track of basis lots for all my mutual funds.

Produce monthly portfolio printouts with asset value / capital gain information.

Keep track of overall portfolio composition, monthly asset class / net worth totals over time in both nominal and real terms.

Compute internal rate of return of portfolio and each asset class.

Compute return / volatility data for each of my mutual funds.

Lately I have been looking into OFX file parsers - there is a github project for a python implementation that looks good. If I had the energy and time, today I might implement most of this with python, a SQL database and Excel. Most of what I have was coded over 20 years ago.

I suspect many of the more technically minded folks have needs (desires) that are so far from what the average person wants that most commercial financial portfolio tools / websites will simply not suffice.

RoboFan
Posts: 33
Joined: Thu Apr 27, 2017 7:17 am

Re: Anyone use Python to analyze their finances?

Post by RoboFan » Sun May 14, 2017 7:48 am

Lots of great answers to my question about why people are writing code (with example code!). I think the last poster captured the answers nicely - individuals want to do things well outside what the average investor want to do.

What I did not pick up was whether anyone is using API's per se. There is another thread "Vanguard versus interactive brokers: API?" that address some of what I was getting at:

viewtopic.php?t=174129

But the thread is a bit old and I'm not sure everyone on it recognized what an API is. Some posters did not believe Vanguard has API's and expressed security concerns suggesting they would be unhappy about it if they found outs such a thing exists (thing being a piece of code that allows another program to access their account information). To be clear - API's are everywhere. The fact that apps such as Personal Capital and SigFig can, if I provide a password, access my Vanguard account suggests Vanguard provides API support. This is not a bad thing. From a security perspective Vanguard is still responsible for your account, and they control what access they give via an API. This is different from read only - an API can limit what get reads, when it get read, whatever, the owner of the API get to specify what it does.

What I got out of the thread referenced above was that Vanguard does not provide API support to individual users but TDAmerica does. Not sure if this is true today. What I'm curious about is whether any of the "tech savvy" posters on this thread have tried to access their own Vanguard data via software. I see examples of accessessing Yahoo and other "public" data but no examples of accessing their own "secure" data.

Anyone tried to do this?

AlohaJoe
Posts: 3562
Joined: Mon Nov 26, 2007 2:00 pm
Location: Saigon, Vietnam

Re: Anyone use Python to analyze their finances?

Post by AlohaJoe » Sun May 14, 2017 7:56 am

Vanguard, along with most banks, do not provide any sort of API. Personal Capital and the like don't use an API. They screenscrape. Or go to a known URL to download a CSV or OFX. I haven't checked but they probably use Yodlee and outsource the screenscraping to them, it seems like everyone does.

Nova1967
Posts: 156
Joined: Fri May 27, 2016 3:22 pm

Re: Anyone use Python to analyze their finances?

Post by Nova1967 » Sun May 14, 2017 3:09 pm

I can't see what use Python would be to track a portfolio when a simple Excel spreadsheet would do.
Sounds to me like more of a hobby.

Tommy
Posts: 126
Joined: Mon Mar 07, 2016 11:47 pm

Re: Anyone use Python to analyze their finances?

Post by Tommy » Sun May 14, 2017 7:32 pm

Aren't those things contradict each other? One is that past performance not guarantee future results and in the the same time complex portfolio analysis trying to make predictions. If you cannot relay on the past for some reason no point trying to predict because prediction based on the past data analysis.

RoboFan
Posts: 33
Joined: Thu Apr 27, 2017 7:17 am

Re: Anyone use Python to analyze their finances?

Post by RoboFan » Mon May 15, 2017 6:37 am

AlohaJoe wrote:Vanguard, along with most banks, do not provide any sort of API. Personal Capital and the like don't use an API. They screenscrape. Or go to a known URL to download a CSV or OFX. I haven't checked but they probably use Yodlee and outsource the screenscraping to them, it seems like everyone does.
Screenscraping, as I understand it, involves pulling data from the display screen once the account is accessed and the data are displayed. However, there have been reports on other threads that SigFig can access user information from Schwab despite the user have implemented 2-factor authentication. The SigFig thread - dating back to 2014 - never really resolved how SigFig managed to get around the 2-fact authentication. I just assumed they did so via an API.

For readers who are not following this so well, and it is confusing, an API allows one program to access a limited set of information and commands available from another program. Two-factor authentications should prevent any person or program from obtaining access to any data or commands until you enter a code sent to you via a second device you have with you (typically cell phone).

I'm thinking this issue might be worth a separate new thread.

Post Reply