Google reckons Spectre and Meltdown were the worst vulnerabilities in a decade

Now that the patches across various platforms for the recently discovered Spectre and Meltdown vulnerabilities have largely been deployed, Google has detailed how it managed to address these threats on its cloud services such as Gmail and Search before the public even knew about them. Hint: It wasn’t easy.

In a lengthy blog post Thursday, Google’s VP of 24/7 operations Ben Treynor Sloss explains how tough these security holes were to patch, and how long it took Google to fully fix all of them, even though it was Google’s own Project Zero team that had discovered them.

According to Sloss, Spectre and Meltdown are actually three different vulnerabilities, one of which — a variant of Spectre — was particularly hard to protect from. One solution involved disabling some CPU features, which would inevitably lead to slower performance. 

“For months, hundreds of engineers across Google and other companies worked continuously to understand these new vulnerabilities and find mitigations for them,” he wrote.

Finally, software engineer Paul Turner created Retpoline, a software that does the job without slowing down the machines it’s applied to.

Sloss said that by December, all Google Cloud Platform services were protected from all variants of these vulnerabilities. The company deployed this solution across its infrastructure and open-sourced it so that others can benefit from it as well.

“This set of vulnerabilities was perhaps the most challenging and hardest to fix in a decade, requiring changes to many layers of the software stack. It also required broad industry collaboration since the scope of the vulnerabilities was so widespread,” wrote Sloss.