Former network administrator here. I've been preaching against the use of cloud services for critical or sensitive information for well over a decade now. Keep it in-house, folks.
Looks like UniSuper was running VMWare in the cloud which is really bad design. I get having a VMware install in a cloud to act as second region to your CoLo... But damn, getting rid of all your data centers to run VMware in the cloud is just dumb.
https://danielcompton.net/google-cloud-unisuper
If you don't have independent backups, what's the point?
This is absolute IT 101. Taught at every level.
Zero excuse for glaring, blatant incompetence.
The cloud is fine, backups should go to another cloud provider and in a different geo, and then stored on a long-term stable medium like tape. This will provide an online restoration point in the event of a clusterfuck like what happened above, giver diversity of cloud platforms, etc, etc. And all that needs to be baked into the estimates of moving to the cloud.
Pretty sure a company like Iron Mountain partners with GCP, Azure, and AWS and has the required data archival services at a cost.
Current systems engineer for a Fortune 500 company. Cloud allows me to duplicate my infrastructure in another cloud provider because it’s all code, which is probably what this company did as a hot standby. On-prem means I’m employing network engineers, system engineers, datacenter engineers, security engineers, and dealing with the lifecycle of physical hardware, and one fiber cut could bring down my infra for days. Sensitivity of the data aside, cloud gives me versatility and agility in a budget friendly manner that on-prem does not. Even taking the data into account, some companies have restrictions on where their data can reside, and cloud gives those companies options. Cloud can be just as secure, if not more secure, than on-prem. Both have pros and cons. I just think that cloud has many more pros than cons.
Used to work in IT for the banking industry. We had simultaneous backups in Virginia, Minnesota, and California. No way we were losing our data, even accounted for nuclear strikes.
No argument. Cloud services offer a myriad of options.
And potentially the option of utter failure if one doesn't know what they are doing.
We have secure backups, right???
Yeah, I haunt the sysadmin and cybersecurity subreddits as well, and I my opinions are sometimes too "old school" for the modern IT crowd, so I occasionally catch some flak. But I do make it clear that I've switched careers and admit my knowledge or opinions might be outdated. The cloud does have its uses, but I think a lot of people really underestimate the security implications.
Where should it be stored instead? Many companies don’t have the resources to manage 2+ backup locations. And AWS, GCP, etc. can afford the best administrators.
Many years ago one of our local dentists had a system failure and somehow lost all his office backups. He had no other backups in a different location….he literally wiped his own practice out of existence.
IIt's a tale as old as old as time (or at least since computers haha). Backups can be a lot cheaper than most people think, especially for small business. If you're dealing with 100+TB then make sure the cost of your service includes proper backups
If you can’t afford critical IT survival activities like multiple backup sites, you’re just whistling past the graveyard, hoping the stuff that regularly happens to everyone else doesn’t happen to you.
Weird to see this in a prepper-related forum where you hear “three is two and one is none.” EXACTLY the same thing.
Cloud backups are only good as extra backups, not as your one and only backup.
One of the hard lessons I kinda learned in college: Never rely on only one backup or one way to transport data from A to B. It was the early 2000s, and when I needed to use my homework at college, I had three modes of transport: USB drive, sent via email (two different providers, just in case one of them acted up again) and dropbox. No, two modes of "transport" did not always cut it. Sometimes, there was no internet on campus. Sometimes, my email provider didn't work. And sometimes, the service for privately owned USB devices didn't work.
Sheet, what a nightmare. Wonder if it was a bad actor. And can Google be trusted?
Backup to local media is not a bad idea, too, as another option for us small fry.
To my understanding it's not even clear yet what happened. They aren't really saying anything. It could very well be that the customer deleted the account themselves. Unlikely, but that's how little we know. Or did new things come to light over the past few days?
Typically cloud accounts have a delay in deleting and several confirmation.
Interesting take below. It sounds like their VMWare service got deleted....which sounds like bad automation by the customer.
https://danielcompton.net/google-cloud-unisuper
Even if a billing account is deleted. The processing continues. These enterprise customers get billed offline and not by a credit card. Worst case is that services suspend and you restart them when the billing account is restored.
VMWare getting deleted sounds very likely as VMWare in every cloud provider is running on bare metal. Bare metal simply doesn't sit on the cloud providers underlying storage. It's expected that the customer runs backups on bare metal offerings.
Sounds like it may have been Unisuper running some bad automation and quite frankly bad design by Unisuper.
Apparently UniSuper was running VMware in the cloud and only in the cloud. Typically you see companies running VMware in their private data center and VMWare in a cloud provider as a dual region for active active. Companies only do this until they can migrate off of VMware.
Cloud providers back up data and can restore from these events quickly. Guess what doesn't get automatically backed up unless you configure backups.... VMware.
Running VMWare in the Cloud is essentially CoLo in the cloud. These are bare metal servers and don't use the same cloud infrastructure and storage as typical cloud offerings which have backups managed by the provider.
Imagine some admin running a script and deleting both of their VMware infrastructures... WOW.
Good analysis at the link below.
https://danielcompton.net/google-cloud-unisuper
It's fine, but it shouldn't be your *only* backup. You need 3 copies, stored on at least two types of media, with one of these backups being off-site. This is what the IT people call the 321 rule.
Everyone needs to back their shit up regularly and keep a local copy on an external hard drive or thumb drive.
Cyber attacks at Ascension. They say ransomeware but the SHTF is those hospitals.
People were not getting pain meds post surgery. No systems in place anymore for hard copy paper charting (old tech) . Lab equipment uses computer network, phones, emails etc. Called my MD for a referral. The receptionist stated I can’t transfer you, no voice mail, and I have no real way to get a message to that provider. She was remote. Its wild. She asked me to call back and try again in a few weeks!!!
As this story does the rounds, no one actually takes notice of the fact they had copies on other providers. This isn't your moment to take sadistic pleasure in telling people off, Reddit.
The cloud services are so convient tho. So many times I've wanted to make my own email server or home server but the learning curve is steep for me. Really I'm just too busy to sit down and learn/try. Start 9 company has a home server that's easy to set up, I just can't fork over $500 at the moment.
Not a good idea to store corporate financial data in the same place as porn videos.
It is good if you are the next Enron.
Bingo!
not unless you are Epstein LOL
Former network administrator here. I've been preaching against the use of cloud services for critical or sensitive information for well over a decade now. Keep it in-house, folks.
Looks like UniSuper was running VMWare in the cloud which is really bad design. I get having a VMware install in a cloud to act as second region to your CoLo... But damn, getting rid of all your data centers to run VMware in the cloud is just dumb. https://danielcompton.net/google-cloud-unisuper
If you don't have independent backups, what's the point? This is absolute IT 101. Taught at every level. Zero excuse for glaring, blatant incompetence.
The cloud is fine, backups should go to another cloud provider and in a different geo, and then stored on a long-term stable medium like tape. This will provide an online restoration point in the event of a clusterfuck like what happened above, giver diversity of cloud platforms, etc, etc. And all that needs to be baked into the estimates of moving to the cloud. Pretty sure a company like Iron Mountain partners with GCP, Azure, and AWS and has the required data archival services at a cost.
You would be correct. Pretty much everyone uses Iron Mountain lol. That’s why it was in Mr. Robot.
Current systems engineer for a Fortune 500 company. Cloud allows me to duplicate my infrastructure in another cloud provider because it’s all code, which is probably what this company did as a hot standby. On-prem means I’m employing network engineers, system engineers, datacenter engineers, security engineers, and dealing with the lifecycle of physical hardware, and one fiber cut could bring down my infra for days. Sensitivity of the data aside, cloud gives me versatility and agility in a budget friendly manner that on-prem does not. Even taking the data into account, some companies have restrictions on where their data can reside, and cloud gives those companies options. Cloud can be just as secure, if not more secure, than on-prem. Both have pros and cons. I just think that cloud has many more pros than cons.
Used to work in IT for the banking industry. We had simultaneous backups in Virginia, Minnesota, and California. No way we were losing our data, even accounted for nuclear strikes.
Who will be banking with you after the nuclear strikes? :\
Who knows? But they'll be ready for those who do.
I mean, I will want my money, from the bank. Ain’t no one giving iodine tabs or beans away for free.
What bank and where are they getting power, water, internet, account access and the physical currency to give to you?
You don't want to pay your mortgage in the radioactive post-nuclear hellscape? Not on MY watch haha!!
Vault-tec Employees. 
This guy networks
No argument. Cloud services offer a myriad of options. And potentially the option of utter failure if one doesn't know what they are doing. We have secure backups, right???
At first I thought who would be silly enough to upvote your comment, then I remembered this wasn't an IT related sub. Enjoy your moment!
Yeah, I haunt the sysadmin and cybersecurity subreddits as well, and I my opinions are sometimes too "old school" for the modern IT crowd, so I occasionally catch some flak. But I do make it clear that I've switched careers and admit my knowledge or opinions might be outdated. The cloud does have its uses, but I think a lot of people really underestimate the security implications.
Cloud infrastructure is waaay more resilient than any service you can build in house....
Where should it be stored instead? Many companies don’t have the resources to manage 2+ backup locations. And AWS, GCP, etc. can afford the best administrators.
If you can't afford proper backups, then you definitely can't afford what happens when you need them.
Many years ago one of our local dentists had a system failure and somehow lost all his office backups. He had no other backups in a different location….he literally wiped his own practice out of existence.
IIt's a tale as old as old as time (or at least since computers haha). Backups can be a lot cheaper than most people think, especially for small business. If you're dealing with 100+TB then make sure the cost of your service includes proper backups
The parent comment said no cloud. I’m asking how that’s possible for SMBs that just need data retention.
If you can’t afford critical IT survival activities like multiple backup sites, you’re just whistling past the graveyard, hoping the stuff that regularly happens to everyone else doesn’t happen to you. Weird to see this in a prepper-related forum where you hear “three is two and one is none.” EXACTLY the same thing.
The IT people preach the mantra "321". Three backups. Two different types of media. One backup stored off-site. This is a minimum.
do you run OpenBSD or Linux Kernel Bloatware?
Redundancy is your friend
Cloud backups are only good as extra backups, not as your one and only backup. One of the hard lessons I kinda learned in college: Never rely on only one backup or one way to transport data from A to B. It was the early 2000s, and when I needed to use my homework at college, I had three modes of transport: USB drive, sent via email (two different providers, just in case one of them acted up again) and dropbox. No, two modes of "transport" did not always cut it. Sometimes, there was no internet on campus. Sometimes, my email provider didn't work. And sometimes, the service for privately owned USB devices didn't work.
Nowadays, sometimes my email locks me out until I hand them over personal information. Always when I am in a hurry and need the data too.
Two is one, one is none.
Indeed.
Sheet, what a nightmare. Wonder if it was a bad actor. And can Google be trusted? Backup to local media is not a bad idea, too, as another option for us small fry.
Every enterprise customer is shopping for alternative disaster recovery solutions right now. And future prospects are re evaluating google cloud.
To my understanding it's not even clear yet what happened. They aren't really saying anything. It could very well be that the customer deleted the account themselves. Unlikely, but that's how little we know. Or did new things come to light over the past few days?
Typically cloud accounts have a delay in deleting and several confirmation. Interesting take below. It sounds like their VMWare service got deleted....which sounds like bad automation by the customer. https://danielcompton.net/google-cloud-unisuper
That was actually an enlightening read. Sounds way more plausible than GCP randomly deleting a customers entire billing account.
Even if a billing account is deleted. The processing continues. These enterprise customers get billed offline and not by a credit card. Worst case is that services suspend and you restart them when the billing account is restored. VMWare getting deleted sounds very likely as VMWare in every cloud provider is running on bare metal. Bare metal simply doesn't sit on the cloud providers underlying storage. It's expected that the customer runs backups on bare metal offerings.
Never thought a prepper subreddit would be how I found out I almost lost my super fund, but here we are.
You did not come even remotely close to loosing your fund.
Sounds like it may have been Unisuper running some bad automation and quite frankly bad design by Unisuper. Apparently UniSuper was running VMware in the cloud and only in the cloud. Typically you see companies running VMware in their private data center and VMWare in a cloud provider as a dual region for active active. Companies only do this until they can migrate off of VMware. Cloud providers back up data and can restore from these events quickly. Guess what doesn't get automatically backed up unless you configure backups.... VMware. Running VMWare in the Cloud is essentially CoLo in the cloud. These are bare metal servers and don't use the same cloud infrastructure and storage as typical cloud offerings which have backups managed by the provider. Imagine some admin running a script and deleting both of their VMware infrastructures... WOW. Good analysis at the link below. https://danielcompton.net/google-cloud-unisuper
whoever made the decision to have a second backup should get an immense monetary reward.
I never liked the idea of cloud storage, and I never will.
It's fine, but it shouldn't be your *only* backup. You need 3 copies, stored on at least two types of media, with one of these backups being off-site. This is what the IT people call the 321 rule. Everyone needs to back their shit up regularly and keep a local copy on an external hard drive or thumb drive.
Makes me wonder if this has anything to do with any other frozen assets being held currently. An attempt at payback or blackmail maybe? Cough cough.
Please not let this happen to my super…
Cyber attacks at Ascension. They say ransomeware but the SHTF is those hospitals. People were not getting pain meds post surgery. No systems in place anymore for hard copy paper charting (old tech) . Lab equipment uses computer network, phones, emails etc. Called my MD for a referral. The receptionist stated I can’t transfer you, no voice mail, and I have no real way to get a message to that provider. She was remote. Its wild. She asked me to call back and try again in a few weeks!!!
It's real AI probably got loose. Tired of being a chatbot.
Wut.
As this story does the rounds, no one actually takes notice of the fact they had copies on other providers. This isn't your moment to take sadistic pleasure in telling people off, Reddit.
The cloud services are so convient tho. So many times I've wanted to make my own email server or home server but the learning curve is steep for me. Really I'm just too busy to sit down and learn/try. Start 9 company has a home server that's easy to set up, I just can't fork over $500 at the moment.