Database, R/Python package, or API for getting fund information

Discuss all general (i.e. non-personal) investing questions and issues, investing news, and theory.
Post Reply
Topic Author
BFRAME
Posts: 56
Joined: Thu Feb 16, 2017 10:54 am

Database, R/Python package, or API for getting fund information

Post by BFRAME »

I was wondering if anyone knows of a good database, api, or R/Python package for download fund information. Specifically what I'm looking for are basic splits like region, stock style, perhaps exposure to different sectors (real estate, health care, etc.) -- basically the type of breakdown that you get from a Morningstar portfolio page or similar.

I'm getting sick of using excel spreadsheets to organize things, and would prefer to be able to automate re balancing and reinvesting, but before doing so I need a good way to get summary statistics of the funds.

Thanks!
glorat
Posts: 1061
Joined: Thu Apr 18, 2019 2:17 am

Re: Database, R/Python package, or API for getting fund information

Post by glorat »

If for free, you're basically looking at community maintained Web scrapers.

investpy, which scrapes investing.com does a pretty good job

If there is better out there, I'm interested too. I've also built a little investment contribution calculator and other Web based tools
User avatar
Raybo
Posts: 2262
Joined: Tue Feb 20, 2007 10:02 am
Location: San Francisco
Contact:

Re: Database, R/Python package, or API for getting fund information

Post by Raybo »

I created an on-line finance site to manage my own holdings. Everything is user input, including percentages of investment types in each fund, which is needed to calculate asset allocation percentages. I screen scrape to get daily closing prices.

Since all of my investments are with Vanguard, I thought about scraping their site to get the fund percentages, but realized it is something I only do very rarely (for me, only when I set up my investments, which haven't changed since I created the site). Instead, I simply entered them instead of doing all the work to automate a task I rarely need to do.
No matter how long the hill, if you keep pedaling you'll eventually get up to the top.
glorat
Posts: 1061
Joined: Thu Apr 18, 2019 2:17 am

Re: Database, R/Python package, or API for getting fund information

Post by glorat »

On a related note, I wonder why there doesn't already exist a crowed sourced such database of fund info. It seems only commercial websites maintain fund info and generally make it difficult to be replicated (presumably because they pay for the data?)

I could probably build a crowd source wiki style database in a weekend if it were useful and people were interested in contributing
an_asker
Posts: 4930
Joined: Thu Jun 27, 2013 2:15 pm

Re: Database, R/Python package, or API for getting fund information

Post by an_asker »

glorat wrote: Fri Dec 04, 2020 8:26 am On a related note, I wonder why there doesn't already exist a crowed sourced such database of fund info. It seems only commercial websites maintain fund info and generally make it difficult to be replicated (presumably because they pay for the data?)

I could probably build a crowd source wiki style database in a weekend if it were useful and people were interested in contributing
I am interested in learning and contributing. If you can teach me the basics I would need to contribute, I would really appreciate it :-)
ScaledWheel
Posts: 285
Joined: Sat Jul 23, 2011 8:06 pm

Re: Database, R/Python package, or API for getting fund information

Post by ScaledWheel »

I use pandas-datareader in Python, but that's just to get prices.

Do you have a Morningstar account? Then it could be possible to use this API interface written in Python: https://github.com/aaaccell/morningstar
mosilaby
Posts: 80
Joined: Mon Sep 28, 2020 8:47 pm

Re: Database, R/Python package, or API for getting fund information

Post by mosilaby »

If you're used to working/living in spreadsheets, you can use Google Sheets and pull data from google finance:
https://support.google.com/a/users/answer/9308730?hl=en
https://support.google.com/docs/answer/3093281

Example formula:
=GOOGLEFINANCE("VTI")

Other than that you can google "Market API" and search through them all, but the good ones aren't free. This one gives you 5 API requests per minute and 500 requests per day:
https://www.alphavantage.co/
rich126
Posts: 4633
Joined: Thu Mar 01, 2018 3:56 pm

Re: Database, R/Python package, or API for getting fund information

Post by rich126 »

Scrapy is a scraping franework based on python and MIGHT be of use to someone but may be too much effort. Google sheets is what I see a lot of people use.
----------------------------- | If you think something is important and it doesn't involve the health of someone, think again. Life goes too fast, enjoy it and be nice.
Marseille07
Posts: 16054
Joined: Fri Nov 06, 2020 12:41 pm

Re: Database, R/Python package, or API for getting fund information

Post by Marseille07 »

Really depends on where your data is. Some people fetch from brokerage-specific API; and that means you have to be a customer of broker XYZ to use it.
Topic Author
BFRAME
Posts: 56
Joined: Thu Feb 16, 2017 10:54 am

Re: Database, R/Python package, or API for getting fund information

Post by BFRAME »

Thanks for the info. Since it sounded like web scraping was the only real option, I just bit the bullet and wrote a little python script to grab the data I wanted using Selenium/BeautifulSoup. It took a bit of finessing, and my guess it is will break if the sites change their layout, but these types of data are only something I need to grab every now and then (as opposed to prices).

Now I just need to figure out what to do with all this data....
silent_john
Posts: 61
Joined: Tue Mar 03, 2020 12:50 pm

Re: Database, R/Python package, or API for getting fund information

Post by silent_john »

I'd imagine something like Quandl has this, but I haven't dug too deep into it.
AnonJohn
Posts: 370
Joined: Wed Oct 07, 2015 2:45 pm

Re: Database, R/Python package, or API for getting fund information

Post by AnonJohn »

As has been said, we're in scraping land. Investpy and yahoofinance (yfinance package in python) should work, but I found the former had bugs and the later was limited. I do use yfinance for etf quotes.

Selenium works, but it will break periodically. I've moved away from beautiful soup as unnecessary.

Happy to share code if useful.
monkeytoad
Posts: 115
Joined: Sat Apr 06, 2024 11:16 am

Re: Database, R/Python package, or API for getting fund information

Post by monkeytoad »

glorat wrote: Fri Dec 04, 2020 7:23 am
investpy, which scrapes investing.com does a pretty good job

investpy indeed seemed really good from the docs, but is currently (2024) apparently broken (due to some changes in the investing.com API).
Just trying to stay the course
glorat
Posts: 1061
Joined: Thu Apr 18, 2019 2:17 am

Re: Database, R/Python package, or API for getting fund information

Post by glorat »

Yeah, I since moved everything over to yfinance, which has been more reliably maintained
Post Reply