Amazon Glacier cloud stoarge? Any experience?

Questions on how we spend our money and our time - consumer goods and services, home and vehicle, leisure and recreational activities
Post Reply
Topic Author
rgs92
Posts: 2503
Joined: Mon Mar 02, 2009 8:00 pm

Amazon Glacier cloud stoarge? Any experience?

Post by rgs92 » Thu Mar 26, 2020 5:32 pm

This seems to be a low cost cloud storage service. Any comments would be appreciated.

This is the description I found:

Amazon Glacier is a low-cost cloud storage service for data with longer retrieval times offered by Amazon Web Services (AWS). A developer uses a cold data cloud service such as Amazon Glacier to move infrequently accessed data to archival storage to save money on storage costs.

wootwoot
Posts: 453
Joined: Tue Jan 27, 2009 7:37 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by wootwoot » Thu Mar 26, 2020 5:46 pm

This is mostly used by businesses for data backups that need to be accessed very infrequently but stored long term. When you need your data it takes hours for Amazon to retrieve it but they charge you a very low rate for the storage. What is your use case?

Topic Author
rgs92
Posts: 2503
Joined: Mon Mar 02, 2009 8:00 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by rgs92 » Thu Mar 26, 2020 5:59 pm

I just want a place to back up my data in the cloud (my photography,music,documents) for safekeeping. It's for archiving things, not for quick back and forth like I can on Google Drive.
I also want something where I could retrieve a fairly deep directory structure without difficultly, which Google Drive cannot seem to manage without errors.

arf30
Posts: 646
Joined: Sat Dec 28, 2013 11:55 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by arf30 » Thu Mar 26, 2020 6:02 pm

Glacier is for offline backups with hours long retrieval time, you probably want S3 One Zone-IA (infrequently accessed).

Elysium
Posts: 2410
Joined: Mon Apr 02, 2007 6:22 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by Elysium » Thu Mar 26, 2020 6:05 pm

Glacier is the cheapest cost of cloud storage, it is also the slowest to retrieve. It takes 12-24 hours for retrieval of data. Typically this is intended for large business data that you really do not need, but need to archive for a number of years for compliance purposes until it reaches end of life. Most of the time, this data is never retrieved, it just goes into the storage and forgotten unless there is a need for retrieval for audit.

For personal use you can get AWS cloud storage for free up to a limit with faster access levels, and even paid cloud storage for personal usage is not very expensive. For what you are trying to use you should try the S3-Infrequent as someone else suggested above.

HawkeyePierce
Posts: 1037
Joined: Tue Mar 05, 2019 10:29 pm
Location: Colorado

Re: Amazon Glacier cloud stoarge? Any experience?

Post by HawkeyePierce » Thu Mar 26, 2020 6:21 pm

You definitely don't want S3. You want a consumer backup service like Backblaze.

My team at work has a few petabytes of data on S3. It's excellent for that. It would be confusing and painful for consumers.

asgard_fleet
Posts: 1
Joined: Thu Mar 26, 2020 6:14 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by asgard_fleet » Thu Mar 26, 2020 6:27 pm

As others have mentioned, Amazon Glacier is not the correct solution for the use case you mentioned. The other aspect that you have to keep in mind, that I have not seen mentioned yet, is that retrieving data from the Cloud is where the costs really start to add up. In general, it's cheap to add the data but expensive to download it.

If you absolutely want to stick with a solution from a major cloud platform, Google Cloud Storage is often a more cost effective option (Glacier pricing with Amazon S3 functionality). You can also look at other smaller companies like Wasabi who usually are significantly cheaper.

Personally, I would stick with solutions like Dropbox or Google Drive since they are more consumer focused. Although you did mention issue with file structure depth.

Another option is to purchase a Network Attached Storage (NAS) device for your house. Think of these as externally accessible hard drives that you can access on your home network. Companies like Synology have great products that are really easy to set up and consume.

jerseysure
Posts: 1
Joined: Thu Mar 26, 2020 6:09 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by jerseysure » Thu Mar 26, 2020 6:28 pm

Longtime lurker.. registered just so I could chime in here! I have a home server that I use for shared storage. I back up all my critical files nightly to AWS S3 using the Glaicer Deep Archive storage class. I use a tool called rclone on Linux to do only differential backups. As another user mentioned, any Glacier storage class is going to be slow and expensive to recover the data, but it's as a last resort it's a great option. Of course you could use other forms of redundancy as well.

At $0.00099 per GB/month, it cost me 16 cents to store 163 GB in February. You can't beat that.

Happy to answer any questions.

Edit: after reading your 2nd post this doesn't sound like a great fit but hopefully someone else finds it useful.

SpaethCo
Posts: 202
Joined: Thu Jan 14, 2016 12:58 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by SpaethCo » Thu Mar 26, 2020 7:07 pm

I do offsite backup from NAS using restic and have settled on Backblaze B2: https://www.backblaze.com/b2/cloud-storage-pricing.html

$0.005 per GB per month. (first 10GB free)
$0.01 per GB downloaded, but you get 1 GB free every day.

This is enough download to be able to do a "restic check" on a regular basis to make sure indexes are intact, and I can splurge a few cents to do a sample pull of archive data occasionally to check backup integrity.

Backups are only viable if you can test them. Be aware that retrieving data from Glacier can be have some surprise costs.

Topic Author
rgs92
Posts: 2503
Joined: Mon Mar 02, 2009 8:00 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by rgs92 » Thu Mar 26, 2020 7:26 pm

Thank you for all this. Just to be clear, I do want the data stored in the cloud, not on a physical server in my house since I want an offsite backup.

User avatar
oneleaf
Posts: 2441
Joined: Mon Feb 19, 2007 5:48 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by oneleaf » Thu Mar 26, 2020 7:34 pm

These cloud services are very confusing for non-developers... Especially AWS. Not just general usage, but even understanding billing is tough.

My personal experience is that Google Cloud Platform is easier to use, but it's still very much geared towards developers. But it's quite easy to navigate between their normal cloud storage, Nearline, and Coldline storages. You are still generally using the command line, but both usage and billing is easier to understand.

If you are willing to get down and dirty and use a commandline and actually have an interest in learning AWS or GCP, I'd say:
- AWS Glacier is confusing and extremely difficult to use for non-developers.
- Google Cloud Platform Nearline/Coldline is significantly easier, but still a pain
- A service that uses those services but gives a better user experience is probably what you want. Like Arq? Or other backup services geared towards consumers like Backblaze, Carbonite, Crashplan, etc.

I'm a user of Google Cloud Platform. I don't trust proprietary services that do too much for me, so I prefer to use command line options similar to Linux RSync, and GCP Cloud was the closest I could find for a balance between being very DIY but not too hard to use.

Topic Author
rgs92
Posts: 2503
Joined: Mon Mar 02, 2009 8:00 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by rgs92 » Thu Mar 26, 2020 8:58 pm

Thanks for all that. Yep, command lines are fine. I did Unix/C/C++ development for many years.

ARoseByAnyOtherName
Posts: 808
Joined: Wed Apr 26, 2017 12:03 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by ARoseByAnyOtherName » Thu Mar 26, 2020 9:12 pm

rgs92 wrote:
Thu Mar 26, 2020 7:26 pm
Thank you for all this. Just to be clear, I do want the data stored in the cloud, not on a physical server in my house since I want an offsite backup.
How much data do you have?

You almost certainly want Backblaze, not Glacier.

spae
Posts: 124
Joined: Tue Oct 08, 2019 9:29 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by spae » Thu Mar 26, 2020 9:14 pm

If you're a developer and this is for personal use, consider tarsnap. Tarsnap's interface is developer friendly but not suitable for normal consumers.

Topic Author
rgs92
Posts: 2503
Joined: Mon Mar 02, 2009 8:00 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by rgs92 » Thu Mar 26, 2020 10:04 pm

I could use about 2 or 3 Terabytes, that's all. Thanks for the new suggestions; is tarsnap named after tar in Unix?

onourway
Posts: 2321
Joined: Thu Dec 08, 2016 3:39 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by onourway » Fri Mar 27, 2020 5:10 am

Backblaze is available for as little as $4.58/month, and is a real, automated backup system, not simply replicated cold storage. And it’s a one-time setup and forget about it. This is only a hair more than even Glacier’s rock-bottom pricing when talking about 3TB ($2.97/month @ .00099/GB), and with Amazon, your bill will grow over time if you keep adding more data or want to continuously back up changes to the files.

lazydavid
Posts: 2820
Joined: Wed Apr 06, 2016 1:37 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by lazydavid » Fri Mar 27, 2020 6:55 am

As others have said, Glacier is designed for data that is never updated and rarely accessed. I currently have 67 TB of database backups in it, which costs me $288/mo. Uploading is simple. Downloading is painful, slow, and expensive. Last month, for the first time in a long time, I had to download about a TB of data. It took about 3+ days over a 100Mbps fiber connection, failed the first two times (transaction timeout), so took another three days to download, then another day to reassemble the parts it had downloaded. All told it took me almost 10 days and cost $203 to recover 1.1TB.

Mine is an ideal use case for Glacier, and I'm actively looking for an alternative.

revert
Posts: 67
Joined: Fri Jun 02, 2017 9:35 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by revert » Fri Mar 27, 2020 7:17 am

tarsnap is excellent, but it's going to be very expensive for 3tb of backup.

Glacier's per-gb storage price is tempting, but the small print is _very_ important. It can be quite expensive to do a restore, and it can take a very long time. 12hrs sounds okay on paper, but if you're in a situation where you actually need a full backup that may take too long. If you go with Glacier, do spend time with the AWS calculator to compute the cost of a restore and think carefully about how quickly you would really need data if all your onsite, etc. backups were inaccessible. Generally I would only use Glacier for data that I had to save for compliance reasons, but had a very high chance of never needing and definitely wouldn't need to access within a week or two timeframe.

Previously I had my backups on S3 with an auto rotation to Glacier (I think they've since made that workflow easier to setup). That was a reasonable compromise, though it was still difficult to understand the dollar/time cost if I needed to restore, so I never felt wholly comfortable with it.

I've since moved to Backblaze's B2, which is their blob storage service, _not_ their consumer backup service. For 3tb per month, the storage cost is around $15 compared to S3/Glacier's $12, and data transfer costs are significantly cheaper (like 9x). It's also a normal cloud storage style API, so no complexities like Glacier's tiered retrieval levels.

B2's API is very similar to S3, so it has wide client support, including rclone (CLI), Arq (excellent GUI), Duplicati (so-so, but free GUI), and duplicacy (sort of unique backup system aimed at extreme deduplication).

Edit: Two other things I thought of:
1. Purely on cost, Backblaze's consumer offering is going to be the best offering if you don't mind using their client and are backing up 3tb from a single machine.
2. Most of the 3rd party clients (rclone and Arq being my two particular favorites) support all the cloud storage backends. So another way of going about this is trying out some clients and picking one you like, then comparing the prices of the backends it supports. This is essentially the route I took - I found a client I liked and trusted, one that wrote data in a format I knew I could retrieve via raw API access if the client itself failed, and over the years I've fairly seamlessly changed the location where it writes data, often writing to two backends at once (onsite drives plus Glacier, onsite drivers plus Glacier and Backblaze during the transition, and now onsite drives plus Backblaze).

ARoseByAnyOtherName
Posts: 808
Joined: Wed Apr 26, 2017 12:03 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by ARoseByAnyOtherName » Fri Mar 27, 2020 12:53 pm

rgs92 wrote:
Thu Mar 26, 2020 10:04 pm
I could use about 2 or 3 Terabytes, that's all. Thanks for the new suggestions; is tarsnap named after tar in Unix?
Use Backblaze.

It’s cheap, unlimited backup, and you can provide your own encryption key so they cannot decrypt your data stored on their servers.

Saving$
Posts: 1876
Joined: Sat Nov 05, 2011 8:33 pm

Re: Amazon Glacier cloud stoarge? Any experience?

Post by Saving$ » Fri Mar 27, 2020 4:02 pm

ARoseByAnyOtherName wrote:
Fri Mar 27, 2020 12:53 pm
rgs92 wrote:
Thu Mar 26, 2020 10:04 pm
I could use about 2 or 3 Terabytes, that's all. Thanks for the new suggestions; is tarsnap named after tar in Unix?
Use Backblaze.

It’s cheap, unlimited backup, and you can provide your own encryption key so they cannot decrypt your data stored on their servers.
PLEASE elaborate on this encryption with Backblaze - how does this work? If you would be kind enough to explain it in non-tech language, that would be great. I am looking for a secure backup that is NOT at my house, so that my data can be restored if the laptop in the house is stolen, there is a fire, etc. However, the data includes sensitive docs, such as tax returns, bank statements, scanned medical results, etc. I'd love to get away from the every other month backup to a hard drive that I then store in another location. It's a particular concern now since I can't get to my off site stored hard drive..
If this Backblaze solution is incremental and automatic and will backup all my data as it is updated, that would be terrific - there would not be a 2 month lag in backups that I currently have. Seems more reliable. I don't care about restore time - the few files I need I could wait 24 hours for and the rest I could wait a month for if necessary.

ARoseByAnyOtherName
Posts: 808
Joined: Wed Apr 26, 2017 12:03 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by ARoseByAnyOtherName » Fri Mar 27, 2020 7:07 pm

Saving$ wrote:
Fri Mar 27, 2020 4:02 pm
ARoseByAnyOtherName wrote:
Fri Mar 27, 2020 12:53 pm
rgs92 wrote:
Thu Mar 26, 2020 10:04 pm
I could use about 2 or 3 Terabytes, that's all. Thanks for the new suggestions; is tarsnap named after tar in Unix?
Use Backblaze.

It’s cheap, unlimited backup, and you can provide your own encryption key so they cannot decrypt your data stored on their servers.
PLEASE elaborate on this encryption with Backblaze - how does this work? If you would be kind enough to explain it in non-tech language, that would be great.
Here’s a good overview of how it works:
https://www.backblaze.com/backup-encryption.html

From that page:
You have the option with Backblaze to add an additional layer of privacy via a user-selected passphrase. This passphrase will be used to encrypt your private key. This passphrase is your responsibility to remember and safeguard. This is important: if you forget or lose this passphrase there is no way that anyone, including Backblaze, can decrypt, and thus restore, your data. When you choose to add your own passphrase there is no “forgot passphrase” mechanism as Backblaze does not know your passphrase.
Note that what I wrote in my previous post is technically incorrect. The additional passphrase you provide encrypts the private key, it itself is not the actual encryption key. End result is still the same though.

teCh0010
Posts: 50
Joined: Mon Oct 31, 2011 11:20 am

Re: Amazon Glacier cloud stoarge? Any experience?

Post by teCh0010 » Sat Mar 28, 2020 9:41 am

I’m using crash plan for small business to backup my home server. 10 bucks a month, unlimited backup. I have confidence in them, I have done a restore. I have about 3TB out there.

I was a consumer client at $55 a year you got converted to small business when they shut down consumer.

Post Reply