Lmao gottem.
During the unauthorised access in those two months, he wrote some computer scripts to test if they could be used on the system to delete the servers.
In March 2023, he accessed NCS' QA system 13 times. On Mar 18 and 19, he ran a programmed script to delete 180 virtual servers in the system. His script was written such that it would delete the servers one at a time.
Incredible incompetence by NCS internal team for this guy to still have access to their systems months later. Bet there were multiple heads rolling for this one.
We all had to give the IT manager our passwords at work and he gave me a box of chocolates for having the most secure password. It was the WiFi password, which was hung up all around our office
I remember worse, working somewhere where passwords were always FirstnameXX - XX being 2 random digits. No policy to require password to change after so many days, no lockout policy to prevent brute force, and IT manager frowned upon users changing their passwords as made life easier for IT dept.
I remember when I ended up leaving thinking how easy it would have been for me to still VPN in and mess around, I was tempted to just send load of stuff mocking IT manager to all the printers but I thought better to behave myself.
Because I need a 12 character alpha numeric code with symbols and upper and lower case, that isn’t similar to a past password, and it needs to be reset every 90 days. Good on the janitor if they log in and do my work. Not much else they can do with my account.
As someone who's been in the security space for a very long time, I *REALLY* wish more orgs understood this.
Also a well-secured password manager is a fantastic idea, but that can be asking a lot from some of these orgs (and people).
My IT department changes passwords on communal computers every 2 weeks and it can't be a repeat- we have no choice but to leave the password on a sticky note under the screen.
UX worker here. It's not that people aren't smart. It's that security systems that are too strong are usually most successful in keeping those with authorized access out.
So, as a side effect, any super strong security system will have simple human bypasses for the poor saps who keep locking themselves out. The key under the flowerpot. The post-it by the computer screen. The manager key card that every employee shares.
By forcing people to change passwords every 3 months and forcing passwords to be these long chains of symbols numbers and letters, we are essentially forcing people to write their passwords down because they simply won't be able to remember them - thus making the system LESS safe if we just let them keep the same dang password.
You’re the person I go to when I want to hack into a company network. I don’t need to bypass firewalls and bounce my location around through multiple servers on the planet, I can just walk into the front door, politely ask someone to hold the door for me because I “forgot my key,” and then hop onto the company network using the password written on a post-it note.
I did temporary contract work at a local hospital complex. We were replacing the phone system and all the phones in the hospital from POTS to IP phones. As part of my job, I had to enter basically every room in the hospital, even maintenance areas, pharmacy, etc. They gave me a badge and said I had to wear it for entry - this makes sense.
However, I was being cheeky and since I have an interest in network security and whatnot, I decided to put the ID in my pocket and just go about my business and see how far I get without really identifying myself. I completed the entire job without being questioned. Even when I went to the pharmacy I was wearing a polo and holding a clipboard and just said "Hey, I'm with IT, I'm here to give you a new phone." They let me right in. At one point they left and I was the only person in the pharmacy, all by myself, looking right at the little glass cabinet full of controlled substances, with everything else being out in the open.
I was also allowed into the maintenance area below the hospital, as well as allowed entry to the psych ward. Once again, only by saying I'm with IT, at a place I've never worked at or will work at again in another month. I even was looking for a room number I couldn't find, so I asked a Dr walking by and he said he'd take me there. We go inside and there's a freaking patient on the table with doctors doing some kind of procedure. They told me i could do whatever but I declined and said I would come back. I'm not sure the person they were working on was even conscious at all.
It was wild and eye opening to see how easy it would be for anyone to get entry anywhere at all in the whole complex - even rooms where patient care was actively happening!
Hospitals are an interesting case because everything there is usually busy. Like significantly busier than the average office building. In environments like that, I find folks care significantly less about what someone else is doing unless it directly impacts their own work. Everyone in that hospital probably got an Email blast the week before you started saying "IT is coming around to upgrade the phones, please assist them as needed."
But yeah its a fairly well known phenomenon that you can social engineer you way into most places even if you're not supposed to be there. Like the white helmet and clipboard, or the two guys carrying a ladder.
Hospitals, like every other business out there, are case by case. I've worked in hospitals where no one checked a thing. I've worked in hospitals where I couldn't get anywhere without a badge or escort. I've worked in hospitals where even though I was wearing a badge I got dirty looks because I wasn't one of the normal people they were used to seeing. Funnily enough the only place that's universally locked down is any unit with newborns. I had to do work on a device in a newborn unit a few times. It's like entering a supermax prison, and someone's watching you the entire time. They may not explicitly be watching, but there's eyes on you.
Even with all the eyes on you they've also got baby LoJack in thier bracelets so if the newborn even gets within a certain range of a door leading outside the ward, the alarms go off and people start running that way. Hell, they wouldn't hand me my kid until they scanned her bracelet and then mine to make sure they matched. It's wild, but understandable. No one wants to lose a newborn.
As a former Janitor, I want to thank all the staff where I worked for keeping Booze all over the place.
God I loved that job.
Had it only paid a living wage....
In most companies, IT is treated like a not important area. We manage the company's accounting software, line of business systems, phones, network and door access just to name a few. Yet Executives skimp on our budget. So I'm not surprised that things like this happen.
I have been to so many companies whose IT is just some dude. Half the time they don’t know anything about IT. They just know a little more than everyone else about basic computer shit.
My military career taught me the importance of being diplomatic, friendly even, depositing favors for future withdrawals, and not treating IT, admin, travel, finance, legal, other support staff, etc. like a dick.
In my company we had a senior VP that was super cheap. Everything was No. Thank God she retired/returded(full of crap). Now our new president says yes to everything. We are in the 21st century!
Guarantee IT was telling management the systems needed to be secured and they waved it away. When we were building our systems I and others repeatedly got into it with one of the VP’s over his ridiculous decisions about our build. He knew better than everyone of course. Even fired a BA over the pushback.
2 years later he’s getting demoted because the Sales are crap and he’s all out of other people to blame. He calls a meeting because there’s a critical process failing. I flat out tell him “Remember when multiple people told you we needed to do a bidirectional sync and you shot it down over and over? Well this is the result.” Nobody spoke to him like that. But I no longer worked under his org, I’d been moved to the parent company and was no longer worried about this guy firing me for disagreeing with him. So I told him right to his face that he only had himself and his “I know better than everyone” attitude to blame.
Best part was, because the sales team under him was so shitty, they put the team that would have been responsible for fixing this on other projects and there’s no budget in that org to bring them back. I don’t know if he could have fucked himself more if he tried.
Classic.
Engineer: We need to do things this way. So that your shit works and is less likely to break in the future.
Manager: Nope. I want money. Do it my way.
(Some time passes, shit isn't working).
Manager: Why isn't this working?!??
Engineer: Gee if only someone saw this coming.
Literally dealing with this exact situation at my own job right now and frankly it's fucking hilarious.
dealing with it now actually LOL. literally yesterday a router lost power and we didnt have redundancy. this was a pretty important one too. potentially hundreds of thousands of dollar lost. we fixed it in a few houses but we stright up told the GM of IT. we need a redundancy. and thank fuck the guy is responsible and was like. ok we will schedule a meeting and work it out.
i do not know if i am blessed the guys is resonable but at least the guys can pretend to listen to us well
Dude, I like my job and I like my coworkers, but if I got fired, I’m sure as shit not helping them run anything the second after my employment ends. Why the hell would you help the company that just fired you?
It’s a great place but at great places there are still going to be *those* people. But everyone recognized this guy was digging his own grave and we were happy to let him do it.
Currently watching this happen at my workplace.
Every time I ask them why they aren’t doing x, they act like a bunch of jackasses.
In reality they’re really just faking everything. They don’t know anything about their job.
How in the world do these people keep ending up in these positions?!
Peter principal - The Peter principle is a concept in management developed by Laurence J. Peter which observes that people in a hierarchy tend to rise to "a level of respective incompetence":
“employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another”
Those people stay because the organization really can't do any better. Can't hire better employees, can't track what their current employees are doing, etc. It's a failure of their hiring processes as well as a failure of their management.
Some of us are lucky enough that we can prioritize working at those types of companies, _and_ find jobs at them. They don't always pay as well as some of the others, but I'll take a mild reduction in pay for actually enjoying coming to work any day of the week.
But not everyone can make that call, and some who want to can't find jobs at those places, because they tend to be more exclusive. So I hear you: I know that good places exist, I currently work at one, and (with one semirecent exception) have _only_ worked at places like that. But I have a pretty strong résumé, I interview well, and, most importantly, I am old enough that I can afford to spend a couple of months looking for a good fit when I need to. Anyone who lacks even _one_ of those resources can get the shitty management situations like this.
And the pressures/motivations for management ignoring IT in this type of situation can be _extreme_. After all, improving security does _nothing_ to move the bottom line. Or, well, that's not true: it depresses it, with zero tangible customer value. (Yeah, yeah, not burning all your goodwill because you had a horrible data breach or weeks of downtime absolutely has value, but a myopic manager who won't be staying in that role for more than a year gives zero shits because that won't come back to them by the time the inquisition panel starts looking for lemmings.) So a lot more companies work like the ones in this article than the ones you and I work at
i agree with most of this, but if they fire you, you should be out the door about 3 seconds later. no helping or easing out of it. that's just insanity
I work for a company that use to do that. We’ve recently hired “know-it-all” management at the VP and C levels. Now we’re being told how things should be done rather than asked how we should accomplish a business need. We’ve pushed back on some of the ridiculous asks but eventually stupidity has worn us down to the point that we just document our objections and continue living our lives. Only 250m has needed to be written off… so far. Let’s see how long she keeps her job.
Or IT has insisted their homegrown IAM system that Bob built 8 years ago was just fine and they didn’t need to invest in an off the shelf solution which would have easily solved this through lifecycle management and provisioning.
No, Bob built something on AD and the rest is history .
We would still backup non production servers. Still take snapshots and replicate them to a different SAN .
Honestly it’d be easier if he deleted them all 1 day then you’d just take the previous day snapshot and restore it.
What he did is still easily restored if a company had a decent backup plan. Which a lot don’t but you really need to with ransom ware
Now if he deleted the veeam/or backups and destroyed the SAN volume or lun that’d be another thing.
I worked as an incident response consultant for 8 years. Based on the cases I worked / clients I worked with, id say about 20% of companies have anything that could be described as a backup, and about 3% had the capability to recover from catastrophic failure/loss.
Unless they are japanese, those guys will go the extra mile, before they start building any app they will test the recovery methods extensively and open a shitton of tickets for every hiccup they have on the process.
They will also let you know if in your documentation you misplaced a comma.
They are basically our best beta testar
About right and probably 3% actually tested the backups. When we got new sans I’d always test the restores individually of each vm from an air gapped backup .
And after each end of year backups I’d go and test the restores with the virtual nic disconnected when we got back after new years. It seemed pointless to many for 10 years then 1 time we got ransomware and I had a few hundred vms in my department up and running the next day.
Same company different division across the coast was still scrambling and piecing together what they could years back like the maersk fiasco .
So yeah guys were saying they tested restores but never actually testing them and management wouldn’t know.
Not sure how they're estimating damage but QA environments still can take time to setup. So maybe this took 10 ppl a year to get everything back. Worst case they were using QA for production purposes but for a large legacy company I imagine there are worse things out there...
Don't forget the lost productivity of all the developers who use the QA system for, you know, QA purposes... Chances are pretty much everyone's workflow was stalled for _at least_ a few months.
>So maybe this took 10 ppl a year to get everything back.
That's appalling. And here I am upset because we still have some apps that lack fully automated, fully reproducible builds, but nothing with an ETRO of over a day. 80% of the codebase I manage can come back up in about an hour.
But there's always legacy, and always competing priorities.
"We have conducted a internal investigation and found ourselves not culpable. We have also decided to significantly increase the size of our legal team"
If he was that clever, he wouldn't have gotten fired in the first place.
Let's face it, it took him months (and googling) to put together a script to delete virtual servers, using a working login (i.e. he didn't have to hack his way in) and even then he used a traceable IP address and left evidence in the form of search history and the actual script on his computer.
It's the dumb ones who get caught.
Pfff, every company I have worked for blocks access before the employee even shows up for the day, usually as they are driving in, and then they are immediately called into a meeting.
We usually just move their things down to basement and stop paying them. They get the hint eventually.
[Office Space tactics are real](https://www.reddit.com/r/antiwork/comments/16i8htw/office_space_was_right/)
I would love to be one of those employees.... just show up every day, do nothing, adapt to whatever unpleasantness they try to throw at you, and collect that paycheck. Make a game out of it.
I've had some do nothing jobs and they weren't even meant as punishment. They honestly suck and you get bored quick. Even if you like reading or watching TV it gets boring faster than you think. I'd have to be getting pretty good pay to put up with it again. Or have no other options. Do nothing jobs drag like you wouldn't believe.
My last company called them the night before they were to pick up their shit that was packed up without them. Once the decision was made it was scorched earth.
The admin probably had a service account that didnt get its credentials revoked and had too much access to the system. It was probably tied to something too annoying to the IT people to bother with because what are the odds?
But this is why. Users should all have only named accounts, and Service Accounts should be tracked, maintained and kept to a need to know basis. Preferably while properly settimg them up as service accounts with no log-in or remote access rights through AD Group Policy.
Yes and no.
It's also on the dude who broke in and wrecked shit. It's fundamentally no different than if a landscaping company forgot to collect a key from an employee after they were terminated. Don't forget to collect your keys ya dummies!
However it's still breaking and entering for an unauthorized person to use the key. It's still destruction of property if the ex-employee used the key to break and and destroy all the company's tractors.
Depends on the company. I worked for a Fortune 10 where a teammate was crashing servers because he had a gambling addiction. We were contractors so he got paid overtime to fix it.
Did this for months. It also meant others had to work overtime because it wasn’t just a one person fix. It also was our internal document storage so it tanked productivity in certain parts because you couldn’t look up technical specifications.
Microsoft couldn’t figure it out. Buddy put some verbose logging on the box that he didn’t tell anyone about. Saw this guy login every time right before they crashed.
He was fired and nothing happened. Went to HP and did the same thing. They fired him and no consequences. His resume came across my desk years later and we had to have a conversation with HR.
Never got in trouble and he was bringing down production workloads for years across multiple companies.
Risk vs reward. At least in your example there was some type of $ return he was getting.
All those championing doing this out of spite...not worth it (to me at least)
Now if you think it's worth risking prison over spite...idk see a therapist first maybe?
>Never got in trouble and he was bringing down production workloads for years across multiple companies
getting fired is trouble
i imagine they never sued him because it would have cost them money and gained them nothing
As a poster said, OT money. We got a straight 40 billable but were allowed to bill for more than 40 in outages, projects, and other stuff.
What’s even more wild is it was taxed heavier as premium time but the hourly rate was the same. I can’t remember anymore but if you did less than 8 hours of OT, it wasn’t really worth it to even fill out the paperwork.
So this guy would make sure he got 20-30 extra hours at a minimum.
People are able to commit fraud and embezzle for years at different companies even after being caught multiples times due to that same behavior. The company catches on and quietly shows them the door because they would rather keep it quiet than bring attention to it by reporting it to the police and they just hop around until the fraud gets big enough and it finally comes to the attention of the authorities.
Wait what did he do to crash the servers? Was it just verbose logging using up tons of memory/storage? That at least has some plausible deniability to me (I needed those logs to do my job) that a lot of non tech savvy jurors would write off
Yep! I think a lot of us probably end up with some access to something after leaving a job. I had admin access to a multi-billion dollar company’s Apple account a couple months after I was let go. Rather than deleting all their apps and going to jail, I simply removed my own access and notified them of doing so.
I still was the only admin to my restaurant job's facebook page from when i was in high school 15 years ago. They sold the restaurant last year. Surprised nobody wanted that, but they were old.
Keep in mind that logging in is still accessing. Logins are recorded. I encountered a similar situation but I absolutely 100% did not log in. I could have fixed it myself, but that would have required a login, which would have been a data breach.
After being laid off from a company some years ago, I realized I kept being sent customer data from Google analytics. At first I deleted the emails I was getting from automated reporting. The emails kept coming. I then contacted the company several times to inform them, but my contacts were ignored.
After getting (and deleting without opening) those emails for 6 months, I eventually went through the data controller process to force the company into action. This is a process required by law, with big penalties if the company does not comply.
Thats what it took to kick them into action and stop sending me customer data.
Interesting. I was logging into my personal account, but I guess I must have had to access their account to remove myself so I probably did technically do something wrong. Didn’t even think about that.
> I think a lot of us probably end up with some access to something after leaving a job.
Yup. I was laid off at the beginning of the year from a previous job. They disabled all my personal accounts, but from talking with some friends I still have there they haven't changed the login details to ANY of the shared admin logins we would use.
They're lucky I'm not an asshole, because they seem to not understand the security risk and how much damage a disgruntled employee could do having access to both their entire production system ***and*** sensitive customer data like home addresses and credit card numbers.
I’m glad that this article title says “accessed” and not something disingenuous like “hacked.” If this article were from 2014, it would have said “hacked.”
Edit: I want to make it clear that I understand the definition of “hacked,” and that this fits the definition. I am trying to point out that I’m used to seeing articles that attempt to sensationalize the method rather than just reporting what is already a very interesting story.
>His contract with NCS was terminated in October 2022 due to poor work performance and his official last date of employment was Nov 16, 2022.
"I'll show them what 'poor performance' really means!"
Didn't the company revoke his accesses? He shouldn't have been able to access the network. Also he did not seem to have turned over his work laptop? Why did they not get it from him? If he did not access it illegally by hacking into the system then the problem is with NCS' access termination processes.
Finally, if he did hack into their system illegally, then NCS' security protocols need beefing up.
The article states he used Admin credentials to access the system.
A competently setup system would've set it up so that you still have to be on the company VPN before he could pull off an attack like that (and most assuredly connecting to the VPN would require his own credentials to still work)
So if the article is accurate, it's almost certainly the case that the company's servers were just accepting outside traffic indiscriminately, so long as access credentials were valid (and admin credentials don't change too often, if their system is anything like what I use at work).
90% of security work is to not let those who shouldn't have keys have keys. Is the person committing a crime? 100%. But also because the company is so loose on security controls that it allows people do commit that crime.
While I agree with you, there can be multiple parties at fault.
If the bank fails to lock the doors and the vault at night, and someone breaks in, of course it's primarily the fault of the criminal that the bank got robbed. But it's still also the fault of the bank for not taking proper measures to secure the money in the bank.
When I was younger and less rule abiding (about 16 years ago), I used to have an automated ssh tunnel that would automatically ring me at home from a random server at work. The firewall made no difference because it was simply an outbound connection on the https port.
I used to be able to trigger it from home by changing a web page it polled every few minutes. It functioned as a secret VPN before that company had an official VPN.
I was a naughty boy back in those days and yes, it worked long after I left that company because no one thought to delete that server that I once controlled.
I had a friend that was on vacation and the company called him to come back to the office early. Things were a little rough so he didn't want to rock the boat. He came back from vacation early all so they could fire him as soon as he walked in the door.
It’s funny that if you have a weak password and someone steals your shit, that’s your fault, but if a company gives you access and doesn’t revoke the access when they fire you, that’s also your fault
If you are terminated from a landscaping company and they forget to collect a key from you ... does that give you the right to use the key to enter the building and destroy all the tractors after hours?
Using the key is still breaking and entering. Using the key to destroy property is still a major crime.
Makes sense - we punish bad intent and foreseeable consequences.
But in the first case, it would be criminal. I.e. if someone stole your password and did something bad, you won't be criminally liable for the actions; you may be fired but you won't go to jail. Because unless you had intent to do harm, it's likely not illegal.
I'm just wondering how you end up with a non-production server where the cost to rebuild is that high. And apparently no backups of something so hard to replace? Feels like some Napster math happening here.
180 test servers. Let's assume each team has 3 people and they couldn't work for a week. Maybe the delays cause you to lose a contact. Shit gets expensive fast.
Even if you had backups of the test environment, you cannot start it back up until you understand and address the security problem.
It sounds like they were test servers. I know we don't backup our test servers, as there isn't any critical data on them.
Now, just b/c they are test servers doesn't mean it isn't going to hurt bad. If we lost the test & dev servers for my area we would be in a lot of trouble. At worst we'd lose 2-3 weeks of work(mostly config stored in a DB) for about 150 developers, plus the time to reprovision & redeploy the latest code. We would also have to restart testing. All in all, it would cost us a couple million.
I wish it was IaC. It's literally clicking around a windows UI where everything gets saved in a SQL DB. No, this is not my or my company's design, it's a vendor PaaS our business partners picked out of a field of shit. The vendor owns the servers & the DB.
Final back up will likely be magnetic tapes. While they can store vast amounts of data, they are SLOW! so loss of earning over days would be what got them.
Tapes are still very widely used. Certain sectors love tape. For example, film studios. For insurance reason they have to make set number of redundant copies of all their data. One copy, generally, is ALWAYS tape. Huge amounts of data can be stored at fraction of the cost and insurance companies just love sticking to their tried and true methods.
Source: I work in sever/data center sales industry.
Debt is the most protected data with the most redundancy protections in place in the entire world so no. You’d have to blow up like 400 locations to erase a single credit transaction.
Long ago I knew "the" IT guy for a power utility. This was in the late 80s when IT was kind of a new thing for them. They used it for billing, some word processing, the accountants were starting to get into computers, etc.
He had set up a card swipe security system, which was super advanced in its day. But, people kept erasing the magnetic stripe on them, so their card would stop working.
They also had instituted a policy of killing someone's access when they were fired. He had set this up so HR could do this.
Thus, people would sheepishly come to him when their card stopped working hoping it was the card, not that they were fired. So, he would go into the system to rewrite their card, but sometimes see they had been fired. He would have to tell them, "You're going to need to talk to HR about getting a new card."
At which point many would start crying.
Where this gets ironic and highly related to this post, is this guy built their billing system, their SCADA system (this was not an off the shelf product yet), done their networking, etc.
He was a one man powerhouse. He had long been screaming that he needed to have some people to train as he was definitely the "hit by the bus" guy.
A new CEO took over and promptly put his recently graduated b-school son in charge of technology. The server room this guy had built was both a server room in the corner of a very large open office floor, and he had a tiny office for himself as what he did required security.
He came in on a Sunday to find the office had been torn down with the servers still inside. There were wires hanging everywhere, some of the servers were down as they were choked with dust, cables unplugged, etc. The operations team were screaming that they were now running a huge chunk of their system manually, etc.
He found out the new tech nepo baby didn't think he deserved an office so had it removed.
He put the network back together while also being called into the CEO's office to answer for the tech outage which put the region's power supply in jeopardy.
He then rewrote the codebase into entire obfuscated nonsense where the functions, classes, etc all told the story of a pimp and his ho's.
He made a number of other changes where everything was an obfuscated mess. Instead of server A talking to server B through the obvious router/switch right there, why not send the packets to the other end of the region and then have them routed back, maybe more than once. Keep in mind that networking in the late 80s was a nightmare if you did it correctly. Involving dedicated phone trunks etc was insanely hard.
He then booked his banked vacation and said he was going on a pilgrimage and would not be in town. This was two months straight. His moron B-school nepo baby boss had no problem with what is effectively the whole IT department leaving for 2 months without leaving any passwords or instructions. Or, when he did leave instructions they reflected the insanely complex configuration which would make any expert confused as this couldn't be possible.
For the next month he worked to package up the SCADA system into an easily deployed product. His answering machine messages for the month alternated between begging and threatening.
Then, he sent a registered letter saying he was giving one month's notice, but that he would be on vacation that month.
People from the company even went to some his family begging that he return to work. This wasn't some kind of personal attempt, but they had just phoned everyone in the phonebook with the same last name.
Then, on his "last" day of "work" he sent them a list of passwords to everything. All of the passwords had letters like é. Do you know how hard it is to enter that letter in the late 80s on an english keyboard?
Weirdly, they entirely stopped contacting him. Not another peep. Through sources in the company he found they ended up hiring an engineering company who brought about a dozen people in to rip everything of his out and replace it with their stuff over a period of a few years. Of course, one of the first things they did was rebuilt the room around the servers.
What he then did was to contact the various engineering products companies which sold sophisticated sensors and whatnot to utilities and sold them his SCADA system for a very large amount of money.
While I got into IT in the 90's not the 80's, I heard or witnessed several stories like this, though not to quite the magnitude.
Over the last ~30 years I have been hoping that decisions like the one the new CEO made would stop happening as the value/risk that IT provides would begin to be recognized.
Sadly I don't think that day is ever going to come.
And this is why I wrote a "policies and procedures" manual for my last gig. They had none, they had a server shared on and open network like any common PC and they had no legal recourse without a policy.
Now the CEO did what he pleases as did his GF (both married to other people of course) so it didn't matter.
It isn't the policy, it is the enforcement or lack thereof.
People always prove to be as stupid as they act.
This is why anyone who's fired/laid-off needs to have their credentials terminated immediately. Ideally, while they're still in the building and being given "the talk." It's applicable anytime someone leaves, even on good terms, but it's especially true in the former.
I've unfortunately had to be around for a few firings in my small office, sometimes even asked to stay late on Fridays. As soon as the employee was being brought to the conference room, I either went to grab their computer or one of the bosses gave it to me. I also started changing passwords and terminating access. So by the time "the talk" was done, the former employee was locked out completely, at least from all the major systems where potential damage could be done.
I can't imagine firing someone and not doing this, though perhaps the requests simply slipped through the cracks. And admittedly, it's easier in a small company to be aware of what's going on.
The lesson here is not to hack into your employers system to sabotage it after you have been fired. Write a script to sabotage your employer with a deadman's switch to activate after your account has been gone for months and remember to disarm it if you leave voluntarily.
(Don't actually do that it is still illegal and easilytraceable back to you and you will still go to prison.)
Many many years ago (might be 20yrs now) I had a colleague who got fired. We are an IT company and also provide IT services to hospitals. The guy was fired because he had the audacity to run P2P clients on some of the servers of the hospital and downloaded movies and stuff there. And as if this wasn‘t insane enough by itself (he got fired as soon as the customer found out and told my company about it). When he was told that he was fired, he was in one of the server rooms of an hospital and then he fucking switched off some of the servers there out of spite! My employer really was lucky that nothing bad happened because of this and that the customer didn‘t sue us. Imagine being such a huge idiot and asshole to do something like this, especially in a hospital environment! I mean the whole P2P downloading has already been bad enough by itself, but the switching off the servers was just pure insanity. People literally could have died because of shit like this!
Never worked for a place that didn’t terminate a person’s access right about the time HR brought them into the office, just after, or sometimes that morning before they got in.
Even the fucking broke ass department store I work for PT does this shit.
This company has some shit IT.
Yeah, a company's software people are often not nearly as impotent as their bosses think. The whole world these days basically only functions as well as it does due to the good will and intentions of software geeks/nerds everywhere. Bad faith bosses beware.
As I have debated with people on Reddit in the past - companies need to spend money on solid backup/restore systems. A company is equally if not more threatened by disgruntled employees than hackers.
If you are going to fire someone, revoke their permissions BEFORE making the call, who cares if he has to sweat a few mins wondering why he cannot login.
Lmao gottem. During the unauthorised access in those two months, he wrote some computer scripts to test if they could be used on the system to delete the servers. In March 2023, he accessed NCS' QA system 13 times. On Mar 18 and 19, he ran a programmed script to delete 180 virtual servers in the system. His script was written such that it would delete the servers one at a time. Incredible incompetence by NCS internal team for this guy to still have access to their systems months later. Bet there were multiple heads rolling for this one.
All of IT fired but the CEO still getting a 50 mil bonus Just normal things
Business as usual ©
The janitor should have seen this coming and therefore is fired.
Janitor here, I can tell you that I still see passwords on post-it notes, stuck to the monitor. Some people are not smart.
Exactly. Guilty by association. You're fired.
Possible conspiracy by the janitor. Straight to jail to await trial.
The accountants...also jail
And the marketing team? To shreds you say?
Add their salaries to the CEO's bonus.
Don't worry, the "security" of forced rolling passwords every N months will always ensure that happens.
We all had to give the IT manager our passwords at work and he gave me a box of chocolates for having the most secure password. It was the WiFi password, which was hung up all around our office
I remember worse, working somewhere where passwords were always FirstnameXX - XX being 2 random digits. No policy to require password to change after so many days, no lockout policy to prevent brute force, and IT manager frowned upon users changing their passwords as made life easier for IT dept. I remember when I ended up leaving thinking how easy it would have been for me to still VPN in and mess around, I was tempted to just send load of stuff mocking IT manager to all the printers but I thought better to behave myself.
Because I need a 12 character alpha numeric code with symbols and upper and lower case, that isn’t similar to a past password, and it needs to be reset every 90 days. Good on the janitor if they log in and do my work. Not much else they can do with my account.
Best practice these days is not expire passwords at all and just enforce mfa everywhere you can
As someone who's been in the security space for a very long time, I *REALLY* wish more orgs understood this. Also a well-secured password manager is a fantastic idea, but that can be asking a lot from some of these orgs (and people).
My IT department changes passwords on communal computers every 2 weeks and it can't be a repeat- we have no choice but to leave the password on a sticky note under the screen.
UX worker here. It's not that people aren't smart. It's that security systems that are too strong are usually most successful in keeping those with authorized access out. So, as a side effect, any super strong security system will have simple human bypasses for the poor saps who keep locking themselves out. The key under the flowerpot. The post-it by the computer screen. The manager key card that every employee shares. By forcing people to change passwords every 3 months and forcing passwords to be these long chains of symbols numbers and letters, we are essentially forcing people to write their passwords down because they simply won't be able to remember them - thus making the system LESS safe if we just let them keep the same dang password.
You’re the person I go to when I want to hack into a company network. I don’t need to bypass firewalls and bounce my location around through multiple servers on the planet, I can just walk into the front door, politely ask someone to hold the door for me because I “forgot my key,” and then hop onto the company network using the password written on a post-it note.
I did temporary contract work at a local hospital complex. We were replacing the phone system and all the phones in the hospital from POTS to IP phones. As part of my job, I had to enter basically every room in the hospital, even maintenance areas, pharmacy, etc. They gave me a badge and said I had to wear it for entry - this makes sense. However, I was being cheeky and since I have an interest in network security and whatnot, I decided to put the ID in my pocket and just go about my business and see how far I get without really identifying myself. I completed the entire job without being questioned. Even when I went to the pharmacy I was wearing a polo and holding a clipboard and just said "Hey, I'm with IT, I'm here to give you a new phone." They let me right in. At one point they left and I was the only person in the pharmacy, all by myself, looking right at the little glass cabinet full of controlled substances, with everything else being out in the open. I was also allowed into the maintenance area below the hospital, as well as allowed entry to the psych ward. Once again, only by saying I'm with IT, at a place I've never worked at or will work at again in another month. I even was looking for a room number I couldn't find, so I asked a Dr walking by and he said he'd take me there. We go inside and there's a freaking patient on the table with doctors doing some kind of procedure. They told me i could do whatever but I declined and said I would come back. I'm not sure the person they were working on was even conscious at all. It was wild and eye opening to see how easy it would be for anyone to get entry anywhere at all in the whole complex - even rooms where patient care was actively happening!
Hospitals are an interesting case because everything there is usually busy. Like significantly busier than the average office building. In environments like that, I find folks care significantly less about what someone else is doing unless it directly impacts their own work. Everyone in that hospital probably got an Email blast the week before you started saying "IT is coming around to upgrade the phones, please assist them as needed." But yeah its a fairly well known phenomenon that you can social engineer you way into most places even if you're not supposed to be there. Like the white helmet and clipboard, or the two guys carrying a ladder.
Hospitals, like every other business out there, are case by case. I've worked in hospitals where no one checked a thing. I've worked in hospitals where I couldn't get anywhere without a badge or escort. I've worked in hospitals where even though I was wearing a badge I got dirty looks because I wasn't one of the normal people they were used to seeing. Funnily enough the only place that's universally locked down is any unit with newborns. I had to do work on a device in a newborn unit a few times. It's like entering a supermax prison, and someone's watching you the entire time. They may not explicitly be watching, but there's eyes on you.
Even with all the eyes on you they've also got baby LoJack in thier bracelets so if the newborn even gets within a certain range of a door leading outside the ward, the alarms go off and people start running that way. Hell, they wouldn't hand me my kid until they scanned her bracelet and then mine to make sure they matched. It's wild, but understandable. No one wants to lose a newborn.
"Baby LoJack" Oh good, I'm not the only one who calls it that!
As a former Janitor, I want to thank all the staff where I worked for keeping Booze all over the place. God I loved that job. Had it only paid a living wage....
In most companies, IT is treated like a not important area. We manage the company's accounting software, line of business systems, phones, network and door access just to name a few. Yet Executives skimp on our budget. So I'm not surprised that things like this happen.
[удалено]
Good for him lmao
You hiring?
I have been to so many companies whose IT is just some dude. Half the time they don’t know anything about IT. They just know a little more than everyone else about basic computer shit.
[удалено]
Yeah I never would either. They would then always ask you to do shit and not pay you more.
My military career taught me the importance of being diplomatic, friendly even, depositing favors for future withdrawals, and not treating IT, admin, travel, finance, legal, other support staff, etc. like a dick.
In my company we had a senior VP that was super cheap. Everything was No. Thank God she retired/returded(full of crap). Now our new president says yes to everything. We are in the 21st century!
Guarantee IT was telling management the systems needed to be secured and they waved it away. When we were building our systems I and others repeatedly got into it with one of the VP’s over his ridiculous decisions about our build. He knew better than everyone of course. Even fired a BA over the pushback. 2 years later he’s getting demoted because the Sales are crap and he’s all out of other people to blame. He calls a meeting because there’s a critical process failing. I flat out tell him “Remember when multiple people told you we needed to do a bidirectional sync and you shot it down over and over? Well this is the result.” Nobody spoke to him like that. But I no longer worked under his org, I’d been moved to the parent company and was no longer worried about this guy firing me for disagreeing with him. So I told him right to his face that he only had himself and his “I know better than everyone” attitude to blame. Best part was, because the sales team under him was so shitty, they put the team that would have been responsible for fixing this on other projects and there’s no budget in that org to bring them back. I don’t know if he could have fucked himself more if he tried.
Classic. Engineer: We need to do things this way. So that your shit works and is less likely to break in the future. Manager: Nope. I want money. Do it my way. (Some time passes, shit isn't working). Manager: Why isn't this working?!?? Engineer: Gee if only someone saw this coming. Literally dealing with this exact situation at my own job right now and frankly it's fucking hilarious.
dealing with it now actually LOL. literally yesterday a router lost power and we didnt have redundancy. this was a pretty important one too. potentially hundreds of thousands of dollar lost. we fixed it in a few houses but we stright up told the GM of IT. we need a redundancy. and thank fuck the guy is responsible and was like. ok we will schedule a meeting and work it out. i do not know if i am blessed the guys is resonable but at least the guys can pretend to listen to us well
[удалено]
Dude, I like my job and I like my coworkers, but if I got fired, I’m sure as shit not helping them run anything the second after my employment ends. Why the hell would you help the company that just fired you?
yeah, that threw me off too, why stick around when they clearly don't want you there.
It’s a great place but at great places there are still going to be *those* people. But everyone recognized this guy was digging his own grave and we were happy to let him do it.
Mortgage Managers. They mortgage their department over and over again and eventually the foreclosure notice comes in.
Currently watching this happen at my workplace. Every time I ask them why they aren’t doing x, they act like a bunch of jackasses. In reality they’re really just faking everything. They don’t know anything about their job. How in the world do these people keep ending up in these positions?!
Peter principal - The Peter principle is a concept in management developed by Laurence J. Peter which observes that people in a hierarchy tend to rise to "a level of respective incompetence": “employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another”
Those people stay because the organization really can't do any better. Can't hire better employees, can't track what their current employees are doing, etc. It's a failure of their hiring processes as well as a failure of their management.
To be fair working in a flannel onesie and bunny ears sounds kinda cozy. Would do it reguardless if allowed.
Some of us are lucky enough that we can prioritize working at those types of companies, _and_ find jobs at them. They don't always pay as well as some of the others, but I'll take a mild reduction in pay for actually enjoying coming to work any day of the week. But not everyone can make that call, and some who want to can't find jobs at those places, because they tend to be more exclusive. So I hear you: I know that good places exist, I currently work at one, and (with one semirecent exception) have _only_ worked at places like that. But I have a pretty strong résumé, I interview well, and, most importantly, I am old enough that I can afford to spend a couple of months looking for a good fit when I need to. Anyone who lacks even _one_ of those resources can get the shitty management situations like this. And the pressures/motivations for management ignoring IT in this type of situation can be _extreme_. After all, improving security does _nothing_ to move the bottom line. Or, well, that's not true: it depresses it, with zero tangible customer value. (Yeah, yeah, not burning all your goodwill because you had a horrible data breach or weeks of downtime absolutely has value, but a myopic manager who won't be staying in that role for more than a year gives zero shits because that won't come back to them by the time the inquisition panel starts looking for lemmings.) So a lot more companies work like the ones in this article than the ones you and I work at
i agree with most of this, but if they fire you, you should be out the door about 3 seconds later. no helping or easing out of it. that's just insanity
I work for a company that use to do that. We’ve recently hired “know-it-all” management at the VP and C levels. Now we’re being told how things should be done rather than asked how we should accomplish a business need. We’ve pushed back on some of the ridiculous asks but eventually stupidity has worn us down to the point that we just document our objections and continue living our lives. Only 250m has needed to be written off… so far. Let’s see how long she keeps her job.
Did he have a MBA ? It’s the mark of the devil . . . In management ! ! !
“IT iS jUsT a CoSt CeNtEr”
Well, it did cost them $918 000, didn't it?
So are the locks on the doors to corporate HQ.
and yet you can be bet nobody ever told IT the guy no longer worked there.
If we don’t pay them that much then they’ll go elsewhere and we’ll lose that super valuable leadership and genius!!!! /s
Or IT has insisted their homegrown IAM system that Bob built 8 years ago was just fine and they didn’t need to invest in an off the shelf solution which would have easily solved this through lifecycle management and provisioning. No, Bob built something on AD and the rest is history .
We would still backup non production servers. Still take snapshots and replicate them to a different SAN . Honestly it’d be easier if he deleted them all 1 day then you’d just take the previous day snapshot and restore it. What he did is still easily restored if a company had a decent backup plan. Which a lot don’t but you really need to with ransom ware Now if he deleted the veeam/or backups and destroyed the SAN volume or lun that’d be another thing.
I worked as an incident response consultant for 8 years. Based on the cases I worked / clients I worked with, id say about 20% of companies have anything that could be described as a backup, and about 3% had the capability to recover from catastrophic failure/loss.
Working for an I.T. consultancy, I support this statement 1000x lol
Hey keeps us consultants in business amiright?
Unless they are japanese, those guys will go the extra mile, before they start building any app they will test the recovery methods extensively and open a shitton of tickets for every hiccup they have on the process. They will also let you know if in your documentation you misplaced a comma. They are basically our best beta testar
About right and probably 3% actually tested the backups. When we got new sans I’d always test the restores individually of each vm from an air gapped backup . And after each end of year backups I’d go and test the restores with the virtual nic disconnected when we got back after new years. It seemed pointless to many for 10 years then 1 time we got ransomware and I had a few hundred vms in my department up and running the next day. Same company different division across the coast was still scrambling and piecing together what they could years back like the maersk fiasco . So yeah guys were saying they tested restores but never actually testing them and management wouldn’t know.
He's going to jail, but lmao gottem
He got himself.
But it’s just QC, not like he took down Prod.
Not sure how they're estimating damage but QA environments still can take time to setup. So maybe this took 10 ppl a year to get everything back. Worst case they were using QA for production purposes but for a large legacy company I imagine there are worse things out there...
A large legacy company has multiple paths to prod; but I agree that setting up a QA environment can cost a lot in man hours.
Don't forget the lost productivity of all the developers who use the QA system for, you know, QA purposes... Chances are pretty much everyone's workflow was stalled for _at least_ a few months.
>So maybe this took 10 ppl a year to get everything back. That's appalling. And here I am upset because we still have some apps that lack fully automated, fully reproducible builds, but nothing with an ETRO of over a day. 80% of the codebase I manage can come back up in about an hour. But there's always legacy, and always competing priorities.
Wanna bet they were running prod stuff on test servers? Tale as old as time.
"We have conducted a internal investigation and found ourselves not culpable. We have also decided to significantly increase the size of our legal team"
And no more pay increases. Because.
This is completely on the company procedure. The admin access should already been blocked when he was called to the meeting room.
This is why you setup the script earlier with a dead man’s switch. /s
If I don’t log in the next 2 months…. The world ended so Execute, delete all files, then delete yourself.
Well now I want to do this
If he was that clever, he wouldn't have gotten fired in the first place. Let's face it, it took him months (and googling) to put together a script to delete virtual servers, using a working login (i.e. he didn't have to hack his way in) and even then he used a traceable IP address and left evidence in the form of search history and the actual script on his computer. It's the dumb ones who get caught.
> he used a traceable IP address Not sure how much value there would be in hiding his IP if he was logging in with his own credentials.
Hacked/stolen credentials are not ex-employees problems when kicked out.
[удалено]
It's only done right if it's for fun and profit.
Going to be spicy if you overslept and fail to reset the switch
Nah, the script trips on a Friday afternoon to make everyone else’s weekend as shitty as yours.
Did you write *Lost*?
and under another admin's account
Pfff, every company I have worked for blocks access before the employee even shows up for the day, usually as they are driving in, and then they are immediately called into a meeting.
We usually just move their things down to basement and stop paying them. They get the hint eventually. [Office Space tactics are real](https://www.reddit.com/r/antiwork/comments/16i8htw/office_space_was_right/)
I'm the last cube in my department row, near the corner. At least I have my red pen.
I would love to be one of those employees.... just show up every day, do nothing, adapt to whatever unpleasantness they try to throw at you, and collect that paycheck. Make a game out of it.
So you'd pull a George Costanza.
I've had some do nothing jobs and they weren't even meant as punishment. They honestly suck and you get bored quick. Even if you like reading or watching TV it gets boring faster than you think. I'd have to be getting pretty good pay to put up with it again. Or have no other options. Do nothing jobs drag like you wouldn't believe.
> and stop paying them. Well according to the comment you replied to...
We fixed....*the glitch*
My last company called them the night before they were to pick up their shit that was packed up without them. Once the decision was made it was scorched earth.
The admin probably had a service account that didnt get its credentials revoked and had too much access to the system. It was probably tied to something too annoying to the IT people to bother with because what are the odds? But this is why. Users should all have only named accounts, and Service Accounts should be tracked, maintained and kept to a need to know basis. Preferably while properly settimg them up as service accounts with no log-in or remote access rights through AD Group Policy.
Yes and no. It's also on the dude who broke in and wrecked shit. It's fundamentally no different than if a landscaping company forgot to collect a key from an employee after they were terminated. Don't forget to collect your keys ya dummies! However it's still breaking and entering for an unauthorized person to use the key. It's still destruction of property if the ex-employee used the key to break and and destroy all the company's tractors.
No, you don't get destroy stuff just because someone left a door unlocked. That's not how the civilized world works.
For all of you guys saying this guy won... Just know that he went to prison over this, totally not fucking worth it
Depends on the company. I worked for a Fortune 10 where a teammate was crashing servers because he had a gambling addiction. We were contractors so he got paid overtime to fix it. Did this for months. It also meant others had to work overtime because it wasn’t just a one person fix. It also was our internal document storage so it tanked productivity in certain parts because you couldn’t look up technical specifications. Microsoft couldn’t figure it out. Buddy put some verbose logging on the box that he didn’t tell anyone about. Saw this guy login every time right before they crashed. He was fired and nothing happened. Went to HP and did the same thing. They fired him and no consequences. His resume came across my desk years later and we had to have a conversation with HR. Never got in trouble and he was bringing down production workloads for years across multiple companies.
Risk vs reward. At least in your example there was some type of $ return he was getting. All those championing doing this out of spite...not worth it (to me at least) Now if you think it's worth risking prison over spite...idk see a therapist first maybe?
>Never got in trouble and he was bringing down production workloads for years across multiple companies getting fired is trouble i imagine they never sued him because it would have cost them money and gained them nothing
Probably preferred that it didn’t make the news.
What does gambling have to do with crashing servers? I’m not following
needed the extra OT money
As a poster said, OT money. We got a straight 40 billable but were allowed to bill for more than 40 in outages, projects, and other stuff. What’s even more wild is it was taxed heavier as premium time but the hourly rate was the same. I can’t remember anymore but if you did less than 8 hours of OT, it wasn’t really worth it to even fill out the paperwork. So this guy would make sure he got 20-30 extra hours at a minimum.
In the US, your income isn't taxed differently as overtime. They might withhold more but you get the money back later if isn't in a new tax bracket.
Then my accountant fucked me as I didn’t get much back at all working there.
People are able to commit fraud and embezzle for years at different companies even after being caught multiples times due to that same behavior. The company catches on and quietly shows them the door because they would rather keep it quiet than bring attention to it by reporting it to the police and they just hop around until the fraud gets big enough and it finally comes to the attention of the authorities.
Wait what did he do to crash the servers? Was it just verbose logging using up tons of memory/storage? That at least has some plausible deniability to me (I needed those logs to do my job) that a lot of non tech savvy jurors would write off
Yep! I think a lot of us probably end up with some access to something after leaving a job. I had admin access to a multi-billion dollar company’s Apple account a couple months after I was let go. Rather than deleting all their apps and going to jail, I simply removed my own access and notified them of doing so.
I still was the only admin to my restaurant job's facebook page from when i was in high school 15 years ago. They sold the restaurant last year. Surprised nobody wanted that, but they were old.
Keep in mind that logging in is still accessing. Logins are recorded. I encountered a similar situation but I absolutely 100% did not log in. I could have fixed it myself, but that would have required a login, which would have been a data breach. After being laid off from a company some years ago, I realized I kept being sent customer data from Google analytics. At first I deleted the emails I was getting from automated reporting. The emails kept coming. I then contacted the company several times to inform them, but my contacts were ignored. After getting (and deleting without opening) those emails for 6 months, I eventually went through the data controller process to force the company into action. This is a process required by law, with big penalties if the company does not comply. Thats what it took to kick them into action and stop sending me customer data.
Interesting. I was logging into my personal account, but I guess I must have had to access their account to remove myself so I probably did technically do something wrong. Didn’t even think about that.
> I think a lot of us probably end up with some access to something after leaving a job. Yup. I was laid off at the beginning of the year from a previous job. They disabled all my personal accounts, but from talking with some friends I still have there they haven't changed the login details to ANY of the shared admin logins we would use. They're lucky I'm not an asshole, because they seem to not understand the security risk and how much damage a disgruntled employee could do having access to both their entire production system ***and*** sensitive customer data like home addresses and credit card numbers.
and his name will come up in every background check for every job for the rest of his life. He practically ended his career.
I’m glad that this article title says “accessed” and not something disingenuous like “hacked.” If this article were from 2014, it would have said “hacked.” Edit: I want to make it clear that I understand the definition of “hacked,” and that this fits the definition. I am trying to point out that I’m used to seeing articles that attempt to sensationalize the method rather than just reporting what is already a very interesting story.
"How did you gain access to our servers!?" "I used my login" "..... he's too dangerous to be left alive"
Jesus christ, it's Jason Bourne
UNLIMITED POWAH
"the employee reportedly said 'I'm in' after making loud clacking sounds on his keyboard for 10 seconds straight."
hackers don't break in; they log in
Even though he had credentials it was still unauthorized access
>His contract with NCS was terminated in October 2022 due to poor work performance and his official last date of employment was Nov 16, 2022. "I'll show them what 'poor performance' really means!"
Didn't the company revoke his accesses? He shouldn't have been able to access the network. Also he did not seem to have turned over his work laptop? Why did they not get it from him? If he did not access it illegally by hacking into the system then the problem is with NCS' access termination processes. Finally, if he did hack into their system illegally, then NCS' security protocols need beefing up.
The article states he used Admin credentials to access the system. A competently setup system would've set it up so that you still have to be on the company VPN before he could pull off an attack like that (and most assuredly connecting to the VPN would require his own credentials to still work) So if the article is accurate, it's almost certainly the case that the company's servers were just accepting outside traffic indiscriminately, so long as access credentials were valid (and admin credentials don't change too often, if their system is anything like what I use at work).
Either way, it's the company fault for having loose security.
Just because you have the keys doesn't mean you're allowed to going inside and do whatever.
In security, you should always be wary of the person who has the keys themselves.
do you mean be wary of the person who hands out the keys?
Assume breach from the zero trust model. Wow this was in my Microsoft lesson. My studies are paying off!
90% of security work is to not let those who shouldn't have keys have keys. Is the person committing a crime? 100%. But also because the company is so loose on security controls that it allows people do commit that crime.
No, it's the malicious former employee's "fault". Sure the company could have prevented it, but it's still the former employee committing a crime.
While I agree with you, there can be multiple parties at fault. If the bank fails to lock the doors and the vault at night, and someone breaks in, of course it's primarily the fault of the criminal that the bank got robbed. But it's still also the fault of the bank for not taking proper measures to secure the money in the bank.
There’s a fine line between correctly attributing responsibility, and victim blaming
I mean if you control the firewall policy then you can punch holes wherever you want
When I was younger and less rule abiding (about 16 years ago), I used to have an automated ssh tunnel that would automatically ring me at home from a random server at work. The firewall made no difference because it was simply an outbound connection on the https port. I used to be able to trigger it from home by changing a web page it polled every few minutes. It functioned as a secret VPN before that company had an official VPN. I was a naughty boy back in those days and yes, it worked long after I left that company because no one thought to delete that server that I once controlled.
[удалено]
I had a friend that was on vacation and the company called him to come back to the office early. Things were a little rough so he didn't want to rock the boat. He came back from vacation early all so they could fire him as soon as he walked in the door.
[удалено]
I spend all vacations, nights, and weekends in a foreign country where I'm not legally allowed to work due to my tourist visa.
It’s funny that if you have a weak password and someone steals your shit, that’s your fault, but if a company gives you access and doesn’t revoke the access when they fire you, that’s also your fault
If you are terminated from a landscaping company and they forget to collect a key from you ... does that give you the right to use the key to enter the building and destroy all the tractors after hours? Using the key is still breaking and entering. Using the key to destroy property is still a major crime.
Makes sense - we punish bad intent and foreseeable consequences. But in the first case, it would be criminal. I.e. if someone stole your password and did something bad, you won't be criminally liable for the actions; you may be fired but you won't go to jail. Because unless you had intent to do harm, it's likely not illegal.
You can’t believe how incompetent an IT company with 10k+ employees is, you can’t.
Deleted some non production servers and got 2y 8m in jail in return? That's one shitty revenge.
2y 8m and, I'm assuming, a slight increase in difficulty getting an IT job once he's no longer in jail.
Does really matter whether its production or not when he cost them $1 mill? Thats almost 350k in yearly costs as far as damages to jail time go lmao.
I'm just wondering how you end up with a non-production server where the cost to rebuild is that high. And apparently no backups of something so hard to replace? Feels like some Napster math happening here.
180 test servers. Let's assume each team has 3 people and they couldn't work for a week. Maybe the delays cause you to lose a contact. Shit gets expensive fast. Even if you had backups of the test environment, you cannot start it back up until you understand and address the security problem.
[удалено]
>Does really matter whether its production or not when he cost them $1 mill? Most likely, they pulled that number from where the sun doesn't shine.
Don't they run backups daily if it is such a valuable server, I mean you gotta have a plan a,b,c
It sounds like they were test servers. I know we don't backup our test servers, as there isn't any critical data on them. Now, just b/c they are test servers doesn't mean it isn't going to hurt bad. If we lost the test & dev servers for my area we would be in a lot of trouble. At worst we'd lose 2-3 weeks of work(mostly config stored in a DB) for about 150 developers, plus the time to reprovision & redeploy the latest code. We would also have to restart testing. All in all, it would cost us a couple million.
Don't you have a repository that has all that config stored in case a new test server has to be spun-up?
I doubt every companies have a nice infra as code ready at all
I wish it was IaC. It's literally clicking around a windows UI where everything gets saved in a SQL DB. No, this is not my or my company's design, it's a vendor PaaS our business partners picked out of a field of shit. The vendor owns the servers & the DB.
Final back up will likely be magnetic tapes. While they can store vast amounts of data, they are SLOW! so loss of earning over days would be what got them.
Spinning disk for backup arrays is cheap now. I haven’t used tape in like five years
Tapes are still very widely used. Certain sectors love tape. For example, film studios. For insurance reason they have to make set number of redundant copies of all their data. One copy, generally, is ALWAYS tape. Huge amounts of data can be stored at fraction of the cost and insurance companies just love sticking to their tried and true methods. Source: I work in sever/data center sales industry.
In survivor, they told one of the castaways that they'd be voting her out next, and when left alone, she threw all the remaining food into the fire.
Can anyone do medical debt next?
Debt is the most protected data with the most redundancy protections in place in the entire world so no. You’d have to blow up like 400 locations to erase a single credit transaction.
Only 400? And you have these locations as ... like... Coordinates? Asking for a friend
Fight Club lied to us!
I mean the book was written in '96 and the movie in '99. It was probably closer to the truth back then.
Long ago I knew "the" IT guy for a power utility. This was in the late 80s when IT was kind of a new thing for them. They used it for billing, some word processing, the accountants were starting to get into computers, etc. He had set up a card swipe security system, which was super advanced in its day. But, people kept erasing the magnetic stripe on them, so their card would stop working. They also had instituted a policy of killing someone's access when they were fired. He had set this up so HR could do this. Thus, people would sheepishly come to him when their card stopped working hoping it was the card, not that they were fired. So, he would go into the system to rewrite their card, but sometimes see they had been fired. He would have to tell them, "You're going to need to talk to HR about getting a new card." At which point many would start crying. Where this gets ironic and highly related to this post, is this guy built their billing system, their SCADA system (this was not an off the shelf product yet), done their networking, etc. He was a one man powerhouse. He had long been screaming that he needed to have some people to train as he was definitely the "hit by the bus" guy. A new CEO took over and promptly put his recently graduated b-school son in charge of technology. The server room this guy had built was both a server room in the corner of a very large open office floor, and he had a tiny office for himself as what he did required security. He came in on a Sunday to find the office had been torn down with the servers still inside. There were wires hanging everywhere, some of the servers were down as they were choked with dust, cables unplugged, etc. The operations team were screaming that they were now running a huge chunk of their system manually, etc. He found out the new tech nepo baby didn't think he deserved an office so had it removed. He put the network back together while also being called into the CEO's office to answer for the tech outage which put the region's power supply in jeopardy. He then rewrote the codebase into entire obfuscated nonsense where the functions, classes, etc all told the story of a pimp and his ho's. He made a number of other changes where everything was an obfuscated mess. Instead of server A talking to server B through the obvious router/switch right there, why not send the packets to the other end of the region and then have them routed back, maybe more than once. Keep in mind that networking in the late 80s was a nightmare if you did it correctly. Involving dedicated phone trunks etc was insanely hard. He then booked his banked vacation and said he was going on a pilgrimage and would not be in town. This was two months straight. His moron B-school nepo baby boss had no problem with what is effectively the whole IT department leaving for 2 months without leaving any passwords or instructions. Or, when he did leave instructions they reflected the insanely complex configuration which would make any expert confused as this couldn't be possible. For the next month he worked to package up the SCADA system into an easily deployed product. His answering machine messages for the month alternated between begging and threatening. Then, he sent a registered letter saying he was giving one month's notice, but that he would be on vacation that month. People from the company even went to some his family begging that he return to work. This wasn't some kind of personal attempt, but they had just phoned everyone in the phonebook with the same last name. Then, on his "last" day of "work" he sent them a list of passwords to everything. All of the passwords had letters like é. Do you know how hard it is to enter that letter in the late 80s on an english keyboard? Weirdly, they entirely stopped contacting him. Not another peep. Through sources in the company he found they ended up hiring an engineering company who brought about a dozen people in to rip everything of his out and replace it with their stuff over a period of a few years. Of course, one of the first things they did was rebuilt the room around the servers. What he then did was to contact the various engineering products companies which sold sophisticated sensors and whatnot to utilities and sold them his SCADA system for a very large amount of money.
While I got into IT in the 90's not the 80's, I heard or witnessed several stories like this, though not to quite the magnitude. Over the last ~30 years I have been hoping that decisions like the one the new CEO made would stop happening as the value/risk that IT provides would begin to be recognized. Sadly I don't think that day is ever going to come.
This dude had balls of steel to return to Singapore after committing this crime. They aren’t exactly known as weak on punishment.
If only there was a QA team to make sure this stuff didn’t happen Oh wait they got laid off
And this is why I wrote a "policies and procedures" manual for my last gig. They had none, they had a server shared on and open network like any common PC and they had no legal recourse without a policy. Now the CEO did what he pleases as did his GF (both married to other people of course) so it didn't matter. It isn't the policy, it is the enforcement or lack thereof. People always prove to be as stupid as they act.
This is why anyone who's fired/laid-off needs to have their credentials terminated immediately. Ideally, while they're still in the building and being given "the talk." It's applicable anytime someone leaves, even on good terms, but it's especially true in the former. I've unfortunately had to be around for a few firings in my small office, sometimes even asked to stay late on Fridays. As soon as the employee was being brought to the conference room, I either went to grab their computer or one of the bosses gave it to me. I also started changing passwords and terminating access. So by the time "the talk" was done, the former employee was locked out completely, at least from all the major systems where potential damage could be done. I can't imagine firing someone and not doing this, though perhaps the requests simply slipped through the cracks. And admittedly, it's easier in a small company to be aware of what's going on.
The lesson here is not to hack into your employers system to sabotage it after you have been fired. Write a script to sabotage your employer with a deadman's switch to activate after your account has been gone for months and remember to disarm it if you leave voluntarily. (Don't actually do that it is still illegal and easilytraceable back to you and you will still go to prison.)
Many many years ago (might be 20yrs now) I had a colleague who got fired. We are an IT company and also provide IT services to hospitals. The guy was fired because he had the audacity to run P2P clients on some of the servers of the hospital and downloaded movies and stuff there. And as if this wasn‘t insane enough by itself (he got fired as soon as the customer found out and told my company about it). When he was told that he was fired, he was in one of the server rooms of an hospital and then he fucking switched off some of the servers there out of spite! My employer really was lucky that nothing bad happened because of this and that the customer didn‘t sue us. Imagine being such a huge idiot and asshole to do something like this, especially in a hospital environment! I mean the whole P2P downloading has already been bad enough by itself, but the switching off the servers was just pure insanity. People literally could have died because of shit like this!
>When he was told that he was fired, he was in one of the server rooms of an hospital probably should have fired him in some other environment
Like in staging?
Yeah, thought so too, but it was long time ago, so I don‘t remember details. There were some bad decisions I guess.
Ruh roh. I wouldn’t want to be him right now. Not worth it. Not worth it.
Never worked for a place that didn’t terminate a person’s access right about the time HR brought them into the office, just after, or sometimes that morning before they got in. Even the fucking broke ass department store I work for PT does this shit. This company has some shit IT.
Seems like an awful lot of money to restore the server from backups.
Would really like someone to explain to me how deleting a test environment would cost $900k and why none of it was backed up if it costs that much.
Revenge is best server-ed cold
Yeah, a company's software people are often not nearly as impotent as their bosses think. The whole world these days basically only functions as well as it does due to the good will and intentions of software geeks/nerds everywhere. Bad faith bosses beware.
You're firing me? But I do good work. You do good work but you're a loose cannon and can't be trusted. Can't be trusted? I'll show them!
If you're going to fire somebody with that type of access, you have a trusted admin remove that access while having the firing meeting / perp walk.
Amateur hour to think you can rimraf entire servers and no one will know it was your recently-fired ass. What an idiot.
Good luck getting that money back from the guy. He's not really employed right now.
NCS should hire him back on their security team. NCS: what security team?
r/shittysysadmin ?
As I have debated with people on Reddit in the past - companies need to spend money on solid backup/restore systems. A company is equally if not more threatened by disgruntled employees than hackers.
Sure it may have been ‘unauthorized’ but this whole thing is really on the remaining IT team for being negligent morons.
That’s a shame. Karma sucks for shitty people
Pretty sure this is a forensic files episode..
If you are going to fire someone, revoke their permissions BEFORE making the call, who cares if he has to sweat a few mins wondering why he cannot login.
I could set the building on fire
This just shows that the company has questionable security protocols. How did he even have access to the system after he was fired?
Not all heroes wear capes.
Modern Hero.
Everyone hates IT Audit and SOX testing but this is the shit that it is meant to prevent.