Solar Winds hack Dec 2020

Hardware and technical stuff
1

This is an unprecedented attack.

I can only imagine CTO’s, CIOs and CEO’s all over the world asking:

  1. Is SolarWinds running on any of our servers?
  2. How did they compromise the update and could our updates be compromised in similar fashion

Imagine you are Intel, Microsoft, or Google, whose software is routinely updated to nearly everyone’s PC’s.

Reading through a few resources: (The fireeye is far far better)

So, according to Fireeye:

SolarWinds.Orion.Core.BusinessLayer.dll is signed by SolarWinds, using the certificate with serial number 0f:e9:73:75:20:22:a6:06:ad:f2:a3:6e:34:5d:c0:ed. The file was signed on March 24, 2020.

This means that all companies that downloaded the update since then are at risk. My take is that while this likely compromised tens of thousands of companies (at a minimum), the hackers likely just set up an admin account for later activity if and when they want to do something nefarious.

However, for the targets they were looking for and were compromised, I’m guessing that they are so far compromised that their entire infrastructure will need to be rebuilt from scratch. Having admin access for months would allow so many backdoors and backup-backdoors created, there is no way of cleansing a system without just rebuilding it completely.

Looking at SolarWind and the update that was compromised, I’m having a hard time coming up with a scenario that doesn’t include an internal person putting this in place - I try in the next paragraph to outline how that might have worked. This is because software updates are notoriously idiosyncratic processes. It’s a build process - you check out code, do the build and then it’s signed but I’ve never seen two groups let alone two companies ever use the same process. Saying that SolarWind’s update was compromised means that someone likely slipped code into their build.

Reaching really far into trying to figure out how this could have been done without an internal person - someone internal decided to use a 3rd party dll as part of their build and that 3rd party dll was the infected portion. This happens all the time b/c people want to save time and finding a tool that helps you is enough incentive to use these 3rd party dll’s.

It will be interesting in the coming weeks to see all the companies that are compromised. We will likely only see a fraction of those b/c admitting you were a target is a problem unto itself.

2

Here is a good rundown how they gain access to the networks with almost anyone’s userid spoofing the multi-factor authentication server:

3

Nation state hacking, where they have the money, resources, and time to do targeted attacks is frightening. I sort of dotted-line to our IS Security folks and this came up this week.

This doesn’t have to be internal. What the Russians can do is pretty scary shit. Most “hackers” just go for the quick hit on something. Nation states can set a trap years in the setting that is almost impossible to detect.

It also didn’t help that I think some SolarWInds account had an easily guessed password.

4
5

Yeah, I read those but what kind of access?

I highly doubt it had access to a server and being able to modify source, compile and sign with THEIR cert is not happening on an update server.

An update server is likely something that they give out ftp access to other companies on occasion so they can download something, hence the easy password.

If you can find any other claim besides this Vinoth Kumar guy, and how serious this was, I’m all ears, but from what I know and how teams release code, this password was not in play.

6

So, like, do I need to be worried about identity theft or my bank account or anything from this.

7

Good question. Were your websites utilizing solarwinds software? That’s probably not known by the general public.

If they were, they could easily impersonate you & drain your accounts - but the good thing is you are but one of a million targets and they only have so many people - from what I gather, the access is highly manual meaning that it took considerable effort to pretend to be one person to gain complete unfettered access.

However, because it’s manual, the odds that any of us are personally going to be attacked is remote.

But - if they gained access to something like LastPass? Time to reset all passwords. Every single password. Are password managers like LastPass saying anything yet? I use LastPass and I haven’t heard a peep from them.

So should we assume the worst?

This is 2020 after all.

8

Well this is terrifying.

9

I am still being recommended and served ads for SolarWinds software for network monitoring. Maybe it’s not the best time to be making the big push for new users.

10

Here is fascinating cloudfare data on the hack - tracking the subdomains it generated.

Looks like a cumulative 5000 subdomains - not sure if this correlates to # of targets that it compromised, but it might be a good guess.

11

My company is one of the 18,000 SW customers that were hit by this. The guy who manages SW says the malware wasn’t “activated.” Not sure how he knows this. I know we don’t use the SW agents, which would limit the infected payload to handful of SW servers only. But I don’t know if the infected file is part of the agent.

12

So a company that specializes in reverse engineering code has published a report that shows how new source code was injected into the build system as far back as Oct 2019 as a proof of concept before they added the real code.

The amount of detail the hackers went to to camouflage this code injection is incredible.

And whoever the employee is who committed this change is going to be in for some serious interrogation.

13

Interlude time!

For the non-techies here, let me attempt to give you an understanding to how builds go.

Disclaimer: I haven’t done a build in years, but managed large teams that released 10-15 packages a quarter and so my high level attempt may be fraught with peril, but hopefully it gets the point across.

To start with there is the source code control system - this is where all the source code is stored. It would be a mistake to think of it as one large file. It comprises likely hundreds if not thousands of files and stores versions of each of these files from the very first time that file was introduced to every single change made to that file since then.

A competent team will have rules set up in the source code system to check-out / check-in changes to these files. Anyone can check out a file - make changes and test it - before they merge it with all the other changes for QA testing. A good team would not check in a piece of code without it being associated with either a new requirement or a bug or tech debt which is a fancy word for improvements that a developer wants to do.

(The before mentioned SolarWinds code change - how did it get introduced? - someone had to associate it with some sort of work item or were they so lax they didn’t require that??)

At some point, someone will make a determination that changes are locked in the release candidate build is done. A build gathers all the files that make up the software - files that haven’t changed, files that did change, 3rd party dll’s that are sw snippets the developers utilize to save time, and language files that take the source code and add the magic sauce. It is then compiled into many different files - you can have web apps / services that run in the background / database engine calls / file services / mail services / etc and all these new files are created that together make a new release.

(so how did this new code get added to be a new / changed file? - a Developer has to “check-in” the code to the source repository. I have never seen a source code control system that allowed an unattributed file to be checked in. So my comment above - the userid of the developer who checked this in is going to get grilled. Were they compromised and someone impersonated them? How did they get the credentials of the developer and access to the internal systems?? )

Testing commences - QA starts to bang on it and bugs are reported and fixed and this goes on for a period of time until such time that the bug count is very low with no serious bugs being found. fyi - It is common to release software with low / medium bugs that are understood.

(so the hackers were pretty svelte - to introduce new code that didn’t have any bugs that popped up during QA. That is quite the feat because it is very common for developers to step on each other’s code inadvertently and resolve it during QA)

Once it passes QA, the build happens AGAIN but this time it is digitally signed with the company’s certificate (cert). This cert is a way of authenticating the file so that when people download it and apply the fix, they know they didn’t grab a bad copy or someone else tried to impersonate their file.

The files are then pushed to an update server where companies come and get the files.

These companies normally go through their own testing process - they have an isolated system where they apply these changes and test themselves - because no two environments are alike - HW / OS / other software running - before they then decide to apply the changes to their production environement.

(FireEye said that the trojan attack waited a period of time - 2-3 weeks before activating - again showing a good knowledge of how people test before releasing. While some companies may take months to release - they would have installed the SW multiple times in that time period to “reset” that timer).

So this shows how sophisticated this attack was - from the source code, making it through QA undetected, then making it downstream through the “supply chain” to the companies who use the software.

This is going to be a doozy. I’m betting companies right now are scrambling to understand if they’ve been compromised and how far they were. With the privileges that this hack gives, it could almost be undetectable. I feel sorry for the IT people at these companies. I bet a lot of them will be working on Christmas day and through New Years.

14

I really hope they were already thinking that. At least the CTOs and CIOs. It’s far from the first time an attack like this has happened, it’s just the most effective/dramatic.

15

FireEye published a pretty detailed breakdown of the fingerprints this attack vector leaves behind. My guess is your SW guys looked over that in detail and didn’t find any of these fingerprints.

16

You are also at less risk if your organization doesn’t use the Orion platform as part of it’s SolarWinds deployment. Orion ties together monitoring and control for your entire network stack, which is incredibly useful in large organizations but also presented a juicy target for these attackers as gaining access via Orion basically lets them run rampant across an organization’s entire network.

Several IT consultancy and support organizations have published good information on combing through your network to see if Sunburst is present. I like this short guide from LIFARS that includes Powershell commands and the hash info direct from FireEye.

I have a good friend who is the IT Admin for a mid-sized company that uses SolarWinds, and he and his people are going nuts this week. Luckily they don’t use the Orion platform, so their exposure is probably limited, but I’m sure there will still be lots of questions from management and lingering concerns for a long time to come. While I am very glad this week that my small organization doesn’t use SolarWinds, it still makes me wonder how many other hacks like this could be out there right now, unnoticed, in software that phones home for updates regularly.

The fact that this has been active on government agency networks for months… we are well and truly fucked. The United States government should assume that Russia knows pretty much everything that is accessible via their compromised networks at this point.

Also, I give this less than a week before the Trump conspiracy clowns attempt to claim this hack allowed Russia to throw the election to Biden or some similar nonsense.

17

Just setting aside the technical details for a little comment, but I mean at this point governments seem better going off pen and paper only for sensitive information.

18

That was a great read thanks!

19

Ah, good, we don’t use Orion.

20