Business
Codecov ditches Bash Uploader for a NodeJS executable
Software testing and code coverage company, Codecov has now introduced a cross-platform uploader meant to replace its former Bash Uploader.
This new uploader is available as a static binary executable currently supporting the Windows, Linux, and macOS operating systems.
The announcement follows the recent Codecov supply-chain incident that lasted two months, in which attackers had altered the Codecov Bash Uploader to collect sensitive credentials from customer CI/CD environments.
New NodeJS uploader to replace Codecov Bash Uploader
This week, Codecov launched a beta release of its all-new uploader capable of running on the Windows, Linux, Alpine Linux, and macOS operating systems.
This uploader, written in NodeJS would replace the Bash Uploader that the company previously had in place for use by customers.
“For the last 8 months, Codecov has been developing a new uploader that does not rely on the bash script that we currently provide to our customers.”
“We initiated this project because, as usage of Codecov has grown and our development velocity has increased, the Bash Uploader has become increasingly complex to properly maintain,” said Codecov CTO Eli Hooten.
Hooten cited multiple reasons for taking this step, including Bash scripts being difficult to maintain, extend, distribute, and test, as complexity increases.
Further, the “curl | bash” style commands previously used by the customers to upload data to Bash Uploader drew scrutiny after the recent supply-chain attack in which the Bash Uploader had been compromised.
“To combat this incident from a product perspective we initially provided better documentation on how to verify the Codecov Bash Uploader until our new Uploader was complete, but our ultimate long-term goal has always been to replace the Bash Uploader altogether,” continued Hooten.
Additionally, Codecov states their new uploader comes with new features, added benefits, and improved security.
The uploader is offered as a natively compiled binary produced from the open-source NodeJS code that the community, customers, and anyone can audit and contribute to.
In a blog post, the company also explained that a compiled binary “makes it more difficult for code to be modified by a middle man,” and offers enhanced security compared to the former Bash Uploader.
Moreover, the new uploader combines separate language-specific uploaders into a single executable.
However, some observers have raised concerns with regards to the new uploader.
Software developer Maximilian Hils highlighted that the new uploader relied on 579 dependencies, and was 43 MB in size:
Hils further expressed surprise at some of the statements touted by Codecov, that a compiled binary was harder to modify for a man-in-the-middle, and that the new Uploader provided “a more secure, verifiable distribution when compared to the Bash Uploader.”
Another developer Josh Pitts told BleepingComputer that binary patching via Man-in-the-Middle (MitM) has been a possibility for a long time, pointing to his earlier discovery.
Additionally, the developer said, that the new uploader binaries were not signed by Codecov:
“For the Windows and macOS versions of Codecov binaries (where they could use code signing), they do not sign the binaries.”
“They do offer public key verification of the downloads and I highly recommend checking those signatures given the importance of this software,” Pitts told BleepingComputer in an email interview.
For those downloading Codecov’s new uploader binaries, Pitts recommends verifying them with GPG/PGP, and then host them on an internal repository like Artifactory.
This would reduce the risk of external MitM patching.
“All your servers that use Codecov could just pull from Artifactory directly instead of doing the GPG/PGP verification on each download from the Codecov website,” concluded Pitts.
How to verify the integrity of the new uploader?
Codecov has provided simple steps that customers can use to verify the integrity of its new uploader.
Along with the uploader binary, the company provides a checksum (shashum) file which is signed by their public GPG key.
Customers can run a few commands, shown below, to ensure the hash or checksum of the downloaded Uploader matches the hash provided in the checksum file, and that the checksum file is authentic (signed by Codecov’s GPG key).
Codecov’s PGP public key has the following fingerprint and can be downloaded from Keybase or other keyservers:Key ID: ED779869
Key Fingerprint: 2703 4E7F DB85 0E0B BC2C 62FF 806B B28A ED77 9869
The new uploader, along with the corresponding SHA256SUM, and SHA256SUM.sig files can be downloaded from Codecov’s server.
Codecov breach reportedly impacted hundreds of networks
Ever since the Codecov incident was disclosed on April 15th, U.S. federal investigators were quick to step in and claimed that Codecov attackers managed to breach hundreds of customer networks.
Over the next few weeks, as reported by BleepingComputer, multiple companies came forward disclosing impact from this two-month-long supply-chain attack:
The impacted companies include software manufacturer HashiCorp, cloud communications platform Twilio, cloud services provider Confluent, insurance company Coalition, U.S. cybersecurity firm Rapid7, workflow management platform Monday.com, and most recently, e-commerce giant Mercari.
Considering Codecov had over 29,000 customers with many relying on the previously compromised Bash Uploader, the full impact of the incident may continue to unfold in the coming months.
Codecov users should scan their CI/CD environments and networks for any signs of compromise, and as a safeguard, rotate any and all secrets that may have been exposed.
The newly introduced NodeJS uploader that customers are advised to switch to is expected to address some of the concerns with Codecov’s former set of uploaders.
In fact, beginning Nov 1st, 2021, the company will begin performing “random unscheduled brownouts” of its Bash Uploader, purposely making it unavailable at times, and will completely phase it out by February 2022.
Update 09:53 AM: Added comments from software developer Josh Pitts.