Information though is tied to its use. Modern operations of any size or scale rely on digital information, not analog sheets of paper. Much of what is sensitive isn’t the old-school “Burn After Reading” one-sheet of compromising information or agent reports, but consists of massive amounts of digital data, analysis of that data, and ancillary supporting information. Often, the sensitive info is going from one machine to another, and those machines are rarely located anywhere near each other in meatspace. Having at one point in my life had to courier top secret information internationally, by hand, I can tell you it sucks donkey balls to do so.
don’t count on it. FireEye was hacked and their Red Team penetration tools stolen. If you use ANY of the products from ANY of the companies hacked with this, you are at risk.
EG. do you run VMWare?
Either an inside job, or they had some dev credentials and access to source control, or they just plain compromised source control.
There’s stuff you can do to protect yourself - making all commits to releasable branches go through a webform, require two approvals with 2FA required of approvers, if you have on-premise source control restrict access to the internal network, but none of this really protects against an inside job.
And if it’s an inside job at the nation state level - I can’t imagine how you defend yourself against this kind of thing. Software companies emply hundreds or thousands of developers all over the world. But developers aren’t security vetted at all. And developers from nation states that maybe aren’t so friendly have no particular reason not to be loyal to their own nation.
I think by no means did it have to be an inside job. With recent hacks, you compromise one user’s credentials on the network and you can own the domain server. From there you learn the dev process until you can use your permissions to implement the desired penetration.
A number of federal systems I log into every day from the Department of Education went down late this morning or early afternoon. I am wondering if this is related. Maybe the are pulling plugs to try to assess the damage.
LastPass (and LogMeIn) issued a statement that they were not impacted
Sure. I said that an inside job was only one of the possibilities. What I was really saying is that an inside job is much harder to guard against.
Thanks for this breakdown, Tman.
“Yikes, this is not what I am used to from this site. Seems odd to correlate the events just because they happened near the same time. SolarWinds being the vector to get access to internal systems so it being used to then touch a VMware product flaw that requires auth discounts the hundreds of other ways that hackers every day get internal access. Maybe you know something we don’t but the article reads as speculation versus reporting”
Does seem a bit ass about face. The VMware flaw requires internal network access (presuming mgmt interfaces are not presented to the internet), so an attack like Solarwinds would come before exploitation of a VMware flaw, not the other way around.
Once a network is compromised, there are potentially hundreds of vectors open to an attacker to move sideways. VMware one does look bad though, since it seems to allow potential compromise of some core identity and trust services.
Actually, they (SolarWinds) apparently left their update credentials on git. I believe the password was solawrinds123, and they had been up there since March. I think.
Right, but ftp access to the update server wouldn’t explain how the modified dll was created (the ReversingLab article explains what was done in more depth). Of course if they used solarwinds123 in other places that might explain how the attackers got into a position to compromise the source control server or a dev account with access to it.
As most of us know, a git server can serve up any kind of documents or files. It would be a scandal of epic proportions if this was indeed the build server then SolarWinds will be done as a company. No one will ever trust them again. I saw the tweet by Vinoth Kumar and he pasted the reply by SolarWinds - who said it was removed from being public access, but while Vinoth purported this was a build server, we haven’t had confirmation. I’m sure at some point, someone will do a debrief, but whether it’s public or not where we can find out…doubtful but you never know.
As an IT professional for several decades, it would just boggle my mind that anyone is that stupid - particularly a company with 300,000 customers.
I think this is getting a bit confused.
There was a public github, presumably intended to be public-readable, that including credentials for the update ftp server, which allowed at least some write access. I don’t see any claims from Vinoth beyond that.
From the register article:
“Kumar is not saying alleged exposed server credentials played a role in the compromise of SolarWinds’ Orion platform, though he acknowledges that’s a possibility. If anything, it’s an indicator SolarWinds’ security prowess.”
And I’d agree, ftp upload itself isn’t very enticing, especially to an organisation like the attacker here that wanted to fly under the radar for a long period. Trying that password, and similar insecure passwords, out in a bunch of other more interesting attack vectors though? Extremely enticing.
Well, read the tweet thread he put out there where he includes the email discussion. Inside that he says “Via this any hacker could upload malicious exe and update it with release Solar Winds product” which I don’t agree with - you need the cert to sign it after all, but in any regards, clear as mud.
Beyond the hack itself and the data the Russians exfiltrated, there’s a secondary impact-- the US government and many companies used it for monitoring, and now that they turned it off their systems and services will be unmonitored. That will have huge knock-on effects with uptime. Without monitoring you can’t fix issues proactively.
Each individual outage could be relatively minimal, perhaps 2000 employees at the IRS would be unable to work for 4 hours on some Wednesday or whatever, but it’s industry and government wide and that stuff stacks up quick.
The problem here is one of monoculture, like the famous bananas. The US government has such stringent infosec requirements that it’s overwhelmingly difficult to get on their approved products list in the first place, which leads to using a single product everywhere. When that product goes bad they’re stuffed. Their focus on security actually led to the exact opposite, and they would have been better off approving more vendors through a less stringent process for a more heterogenous environment.
I’m actually going through certification on the DoD/DLA APL for a product now myself, so I know of what I speak!
They would be better off approving a vendor that was actually, you know, secure. Rather than just well connected and willing to jump through process hoops.
It’s not that it doesn’t make sense to have stringent requirements, the problem is they are the wrong requirements, enforced in the wrong way.
Nobody can really protect against a nation-state attacker. Any vendor can be compromised in this manner. The only real defense is a heterogeneous environment with multiple vendors.
I’m not sure this is true? Guidance (not from Solarwinds but from FireEye, Microsoft, and others) I’ve seen includes updating to a newer version of the solarwinds orion libraries without the backdoor code, so it doesn’t seem like security professionals are encouraging disuse of solar winds.
This was my first thought when I saw it, because that’s my responsibility. We dodged this bullet thankfully, but I will be stepping up my conversion off Solarwinds for the few teams that actually use one of their products. I was already doing this because of standardization, capabilities, and just plain preferences, but it will be an even easier sell now.
Very glad we don’t use Orion, but we do use a similar product from a different vendor and I have been grilling them for awhile now on shit like this. This is the stuff nightmares are made of.
It’s a good point but I can’t see seriously endorsing it. I mean we run this way currently and are very much trying to get away from it.