The IT Nerd

CircleCI, a company that develops testing and deployment tools for software engineers, has shared details about how hackers broke into its systems last month and compromised customer data. CircleCI chief technology officer Rob Zuber said hackers gained access to its networks after infecting an employee’s laptop with malware. And here’s what happened next:

On December 29, 2022, we were alerted to suspicious GitHub OAuth activity by one of our customers. This notification kicked off a deeper review by CircleCI’s security team with GitHub.

On December 30, 2022, we learned that this customer’s GitHub OAuth token had been compromised by an unauthorized third party. Although that customer was able to quickly resolve the issue, out of an abundance of caution, on December 31, 2022, we proactively initiated the process of rotating all GitHub OAuth tokens on behalf of our customers. Despite working with GitHub to increase API rate limits, the rotation process took time. While it was not clear at this point whether other customers were impacted, we continued to expand the scope of our analysis.

By January 4, 2023, our internal investigation had determined the scope of the intrusion by the unauthorized third party and the entry path of the attack. To date, we have learned that an unauthorized third party leveraged malware deployed to a CircleCI engineer’s laptop in order to steal a valid, 2FA-backed SSO session. This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software. Our investigation indicates that the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.

Because the targeted employee had privileges to generate production access tokens as part of the employee’s regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys. We have reason to believe that the unauthorized third party engaged in reconnaissance activity on December 19, 2022. On December 22, 2022, exfiltration occurred, and that is our last record of unauthorized activity in our production systems. Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.

Clearly the threat actors knew who to target to get what they wanted. That’s scary. The company has put out a security alert that has been consistently updated since this incident happened. I’d spend some time reading this if you are using CircleCI products. An example of this is that Datadog’s RPM GPG signing keys and its passphrases were exposed during this breach. Anyone who uses their products, and any vendor who uses those products are potentially at risk.

Kevin Bocek, VP of Security Strategy and Threat Intelligence at Venafi had this to say: 

“Another day, another software supply chain attack. It’s clear that this type of threat isn’t going away. Targeting a developer tool and delivery platform, like CircleCI, was clearly intended to fly under the radar and slip into other development environments. In this case, they were able to gain access to Datadog’s environment meaning that its RPM GPG signing machine identities were exposed. Fortunately, Datadog has responded quickly to rotate the impacted identities and it doesn’t appear that they’ve been abused. But if an attacker had seized this opportunity, then it would have given them a very powerful weapon – potentially allowing them to spread across Datadog’s customer networks by enabling them to sign and send malware while appearing completely trusted. This could have had serious repercussions.

“This incident demonstrates the growing risk of attacks targeted at developers, machine identities and modern development pipelines. When combined with the speed of modern development, widespread use of automation and use of the cloud, an attacker with access to powerful machine identities can create ripples fast which are extremely hard to protect against or remediate. In a machine-driven world, having a control plane to manage the lifecycle of your machine identities is essential. As this incident shows, you can be doing all the right things and still find yourself exposed. All businesses – whether they be a software publisher, or a consumer of software – need to be able to automate controls that say who and what can and can’t be trusted, and to have the agility to respond to change.”

This isn’t a trivial hack and should not be treated as such. If you’re reliant on CircleCI products, you should be ensuring that you are not exposed. And you should double check with your vendors that they have done their due diligence as well.