Is it time to replace openSSL? We’ll follow up on the Heartbleed story, discuss how attackers got read access to Google’s production servers and then it’s a great batch of your questions and our answers.

All that and much much more…

On this week’s TechSNAP!

Thanks to:

— Show Notes: —

Heartbleed followup

• The developer who introduced the heartbeat feature, that lead to the heartbleed flaw, claims it was an honest mistake and denies the involvement of any intelligence agencies or other mischief
• An investigation and some insider info has allowed the construction of a more much accurate timeline of the events leading up to heartbleed and the breakdown of responsible disclosure
• 2014-03-21 (or earlier): Neel Mehta of Google Security discovers Heartbleed vulnerability
• 2014-03-21: Bodo Moeller and Adam Langley of Google commit a patch for the flaw (based on timestamp of the patch sent to OpenSSL and Redhat)
• 2014-03-31 (or earlier): CloudFlare is notified about the flaw under a non-disclosure agreement
• 2014-04-01: Google notifies OpenSSL of the flaw. OpenSSL was going to release a patch that week, but changed the planned release to April 9th to allow more coordination
• 2014-04-02: Researchers at Codenomicon find the same bug
• 2014-04-03: Codenomicon notifies NCSC-FI
• 2014-04-04: Akamai finds out about the bug from an undisclosed source and patches their servers
• 2014-04-04: Rumors begin about a flaw in OpenSSL, but no details are available
• 2014-04-05: Codenomicon purchases the heartbleed.com domain, OpenSSL team commits the Google patch to their private git repo
• 2014-04-06: NCSC-FI (CERT-FI) requests that CERT/CC (Computer Emergency Response Team Coordination Center) allocate a CVE for the bug, no details given. Mark Cox of OpenSSL notifies Redhat about the issue and asks them to coordinate with the other operating systems since Mark is on holiday. RedHat security engineer sends the email to the private distributions list with no details, distros that agree to the embargo (no notification until April 9th) are given the details. These distros include SuSE (Monday, April 7 at 01:15), Debian (01:16), FreeBSD (01:49) and AltLinux (03:00). Other operating systems, Ubuntu (asked at 04:30), Gentoo (07:14) and Chromium (09:15), request details during the night, but by time the RedHat engineer gets up in the morning, the flaw has become public
• 2014-04-07 (or earlier): Facebook is notified and patches their servers
• 2014-04-07: NCSC-FI notifies OpenSSL that Codenomicon has discovered a flaw. OpenSSL team decides that \”the coincidence of the two finds of the same issue at the same time increases the risk while this issue remained unpatched. OpenSSL therefore released updated packages [later] that day.\”
• 2014-04-09: Facebook and Microsoft donate $US15,000 to Neel Mehta via the Internet Bug Bounty program for finding the OpenSSL bug. Mehta gives the funds to the Freedom of the Press Foundation
• In summary, those who knew about the issue before it was public: Google (March 21 or prior), CloudFlare (March 31 or prior), OpenSSL (April 1), Codenomicon (April 2), National Cyber Security Centre Finland (April 3), Akamai (April 4 or earlier) and Facebook (no date given)
• Those who had a few hours of advanced warning: SuSE, Debian, FreeBSD and AltLinux
• There is also a timeline from the perspective of the OpenSSL team
• Researchers at Lawrence Berkeley National Laboratory find No evidence that heartbleed was exploited before it became public
• Meanwhile, researchers at the University of Michigan setup a honeypot and monitored who tried to heartbleed it. Of the 41 groups that attacked the honeypot, the majority of those groups — 59 percent — were in China.
• What is worse than heartbleed? Bugs in heartbleed detection scripts
• Akamai claimed to be protected from heartbleed by their own patches
• They were proven wrong
• Akamai updated their blog post explaining the fault
• Akamai was forced to rekey all of their customers’ SSL certificates

• The Akamai post raises an interesting questions about EV-SSL certificates, and how much longer they might take to re-key

• Is the IETF to blame for allowing the heartbeat feature to be added to the standard in the first place?

• See BSD Now episode 033 for news on the OpenBSD fork of OpenSSL
• Why I quit writing Internet standards
• Statement by the CRA about 900 SIN numbers leaking due to heartbleed
• RCMP arrests 19 year old from London, Ontario for using heartbleed against the CRA
• CloudFlare launched a challenge to see if anyone could capture their private key
• two or more people were successful, and the certificate has since been revoked
• ACM Queue – PHK: Putting OpenSSL out of its misery
• PHK shares his experience making a living while working on open source in “Raking in the dough on Free and Open Source Software”
• Open source is a thankless job, we do it anyway
• Does the OpenSSL flaw alter the “open source is safer” equation?
• Coverity report finds open source software to be higher quality Coverity\’s analysis found an average defect density of .59 for open source C/C++ projects that leverage the Scan service, compared to an average defect density of .72 for proprietary C/C++ code
• 863 CIOs surveyed, 18% said their orgs are unaffected by Heartbleed because they use anti-virus

How we got read access to Google’s production servers

• A group of researchers decided to target Google
• Looking at the trends in the industry, flaws are most often found in:
• Old and deprecated software
• Unknown and hardly accessible software
• Proprietary software that only a few people have access to
• Alpha/Beta releases and otherwise new technologies
• So they did their homework
• They used the Google search engine, to search for software and companies that Google had acquired, antique systems, and products with very few users
• They found the Google Toolbar button gallery
• The product allows users to customize the toolbar by uploading XML that controls the style etc
• They quickly managed to perform an XXE attack
• They were then able to read files on Google’s production servers, including /etc/passwd, and some custom init scripts that Google uses to manage their cluster of servers
• They likely could have escalated the attack, and possibly accessed Google’s internal servers
• The team reporting the issue to Google, and was awarded a $10,000 bug bounty

Feedback:

• LastPass

• Automated bare-metal backups for a physical server?

• SLAs and Daylight Savings

• pfsense and logs

• Heartbleed

• Memory allocation in OpenSSL

Round Up:

• PSA: Fraudulent Crypto Kickstarter Campaign
• weev (AT&T iPad data incident) conviction vacated, the information was not protected by a password or other security mechanism, so accessing it was not “misuse”
• HAM Radio operators accidently tripping arc fault circuit interrupters, American Radio Relay League labs working with manufacturer on fix
• When Azure breaks: Lessons for every cloud programmer
• Obama: NSA Must Reveal Bugs Like Heartbleed, Unless They Help the NSA
• Canadian Government publishes guidelines for staying safe online
• Comcast PAC gave money to every senator examining Time Warner Cable merger | Ars Technica
• 7 Habits of highly successful UNIX admins
• Over $100K in Bitcoin Was Stolen in a Ridiculously Low-Tech Heist