July 28, 2021
Think the ransomware outbreak won’t affect you, that it’s someone else’s problem? After reading this story about an IBM i store’s recent experience with cybercriminals, it might make you think twice about your approach to security.
Greg is the IT manager of a mid-sized distribution company located in the South. Jungle Computing complies with her request to keep her last name and company name out of this article. But Greg was determined to share his story with the wider IBM i community, in the hope that it would inspire them to take the ransomware threat seriously and improve their approach to security.
The ransomware attack began on May 15, 2021, a Saturday. When employees arrived at work on Monday, May 18, they discovered that they did not have access to Windows PCs and servers. Greg started to search the network and noticed files with strange extensions. He also came across a “readme” file in a strange location.
“This is where we found the ransom note,” said Greg Jungle Computing. However, no one in the company actually reviewed the ransom note or read its demands. “We didn’t download the browser they said to download. We haven’t looked at the ransom. We weren’t paying it, period.
Like many midsize IBM i stores, the retailer relies on a mix of different systems, most of which worked in-house before the attack. In April, the company had just taken delivery of a new IBM Power9 server, which runs its IBM i-based ERP applications. It also operated its own Microsoft Exchange server, an Active Directory server, two additional Windows servers for shipping software, and a separate Windows server for processing AS2 transactions.
The company has subscribed to several backup and security services, including endpoint protection from Webroot, firewalls from WatchGuard Technologies, online backup of IBM i server from Carbonite, and backup of Windows systems through a foreign company.
All of the company’s servers and PCs were compromised in the attack except for the IBM i server and AS2 server, which was offline due to a failed fan.
According to a forensic investigator hired by the company, the attackers exploited a security hole in Exchange Server to send a malicious attachment from a legitimate email account, Greg said. When an employee opened the malicious attachment, which looked like a legitimate business, it allowed cybercriminals to enter the corporate network. The strain of ransomware that was used is called Conti, Greg said.
It was not an automated ransomware attack. Once inside the network, the attackers created an Exchange administrator account on the Exchange server, according to Greg. “Then they moved to Active Directory,” he said. “They entered this server. They deleted the server backup files that we had on the file server. Someone, or something, placed the ransomware on a Group Policy in Active Directory, which allowed the encryption routine to deploy to all endpoints.
Webroot endpoint security software has blocked the spread of some, but not all, malware. It turned out that there had been an update, and as part of the update, the distributor was not informed of the new “portal” required to activate the software. With the help of the forensic investigator and Webroot software, which Greg called “very, very good,” the company began its recovery on Tuesday, just a day after discovering the attack.
The first step in the recovery was to find the offending DLLs and EXE files and block any IP addresses that the cybercriminals were using to communicate. The company started a recovery, which took a while because the disaster recovery company it uses for Windows was overseas and all communication was via email. “You get what you pay for,” said Greg. (The company is consolidating its online backup for Windows and IBM i with Carbonite.)
During all of this, the IBM i server was not affected, even though it was on the network and cybercriminals had full access to it. “I guess they just didn’t see it, or they saw it and just didn’t know what it was,” Greg said.
The company was very lucky because the company’s security on the IBM i server was relatively low. “For the most part, our object authorities are wide open,” said Greg. “Public has * USE or * ALL because the legacy app we’re running doesn’t use the adopted authority or something like that.” We were at password level 0.
Since the company’s core business is IBM i, it was able to resume operations and ship the product on Thursday. “A lot of people familiar with these types of attacks have told me that we have recovered much faster than most,” said Greg. “We were lucky.”
However, the company’s luck would soon be put to the test. Thursday night, while Greg and another IT employee were staying late to monitor systems, they suffered another attack. Greg had just left for dinner, which he and the other employee ate at the site before returning to work. When they left their consoles to eat, the cybercriminals came back and encrypted the systems again.
“When that happened, we went to the computers and pulled cables from the servers, network cables, Internet connections. We even dropped the Wi-Fi in the building, ”said Greg. “On a forensic level, we found out after the fact that the term they used was ‘brick our server’. “
They packed the server up and sent it in for analysis, and there was nothing they could do to get it back, Greg said. “They couldn’t see anything after this second attack,” he said. “So at this point we put the IBM i out of the way and basically wiped out every server, wiped out every workstation, and reinstalled. “
While the company has been able to resume shipment of the product thanks to the “security-by-obscurity” provided by the IBM i server, it will take another six days to bring Windows systems into service. But HE would never look alike again.
“We’ve completely re-evaluated, completely reorganized all of our security,” said Greg.
The company has adopted a new security appliance called Coretex XDR from Palo Alto Networks, which adds behavioral analytics to the security mix. It outsourced the management of firewalls to a dedicated company. He turned off Exchange Server and switched to Office 365. He reduced all users’ Windows credentials to an absolute minimum. It is also in the process of implementing multi-factor authentication.
“I assumed our Windows environment was relatively secure,” he said, “and I was wrong.”
But the redesign doesn’t stop at Windows. Greg is also reassessing the security of the IBM i server. He took advantage of a free safety assessment, and the results weren’t horrible, but there are a few things to clean up. To begin with, Greg went from password level 0 to password level 3, which is required to use complex 12-digit passwords. The company has adopted TLS encryption for ACS and RDi sessions, even for developers and administrators running the software in-house.
The company is also in the process of adopting point-of-exit software from one of the well-known IBM i security vendors. While the menu-based navigation system of its old ERP system minimizes the risk of damage, Greg said, there’s still a possibility that criminals or scoundrels can wreak havoc. The ability to access or delete just about anything on the system through Execute SQL Scripts prompted Greg to tighten up security. With an exit point for SQL and ODBC in place, this should eliminate this threat, he said.
In April, as part of the Power9 upgrade, the company also started using a new LTO-7 tape drive, which it had not used before. With weekly backups to the tape drive on Sunday evenings, Greg can rest easy knowing that at least his business data is safe.
“I have been assured by our business partner IBM where we back up our data that there is no way to delete any of our backups as it is not a software capability,” said Greg. . “In other words, you couldn’t go into IBM i and delete a backup set, or something like that. It would actually have to be in the data center.
All in all, it has been a great learning experience for Greg and his team. Ironically, Greg was already taking the initiative to beef up IBM i’s security before the events, but it turns out he needed to make investments across the board.
After recovering not from one ransomware attack, but from two, within a week, the experience left a lasting impression. He reported the attack to the FBI, who took the report. “The field office that handles this is being overtaken, so we spoke to another field office,” said Greg. “It’s bigger than what the news suggests.”
While the company has recovered its systems with only a small amount of data loss, it has had a lasting impact. “From a personal standpoint, I lost a lot of sleep,” said Greg. “I had nightmares about it. It’s like someone has broken into your house, tore everything up and you’re not sure they’re still there.
Greg credits the business owner for being very proactive when it comes to safety. He spoke with all employees about ransomware identification and the expected time for IT to train on important tasks, such as reimaging Windows machines, reinstalling ACS, and adding people to the domain. .
With the owner’s support, Greg pledged to share his story in the hopes that it will inspire others to take action.
“People have to know how to understand gravity here,” he said. “The advice I would give is to take it seriously and act now, not after it has happened. I highly recommend that you review your overall security policy, have a conversation with your endpoint protection, review your firewall, review your administrator accounts, implement multi-factor authentication.
“If we can help someone because of what we’ve been through to avoid this, that’s what we want to do.”
IBM thwarts ransomware in FlashSystem with new data protection
Ransomware attacks strike closer to home
Ransomware outbreak reaches epic proportions and IBM i stores are taking note
Christmas for ransomware: How COVID-19 is fueling cybercrime
IBM i ransomware threat assessment