Apple Pretend Filesystem | TechSNAP 271

Posted on: June 16, 2016

Why didn’t Apple choose ZFS for its new filesystem? We journey through the long history of ZFS at Apple. Plus how the BadTunnel bug can hijack traffic from all versions of Windows & should we worry about Intel’s management tech?

Plus great questions, a huge round up & much more!

Thanks to:

Get Paid to Write for DigitalOcean

Direct Download:

RSS Feeds:

Become a supporter on Patreon:

Show Notes:

BadTunnel bug can hijack traffic from all versions of Windows

“Microsoft has patched a severe security issue in its implementation of the NetBIOS protocol that affected all Windows versions ever released”
“Among the more than three dozen vulnerabilities Microsoft patched on Tuesday was a fix for a bug that the researcher who found it said has “probably the widest impact in the history of Windows.””
“An attacker could leverage this vulnerability to pass as a WPAD or ISATAP server and redirect all the victim’s network traffic through a point controlled by the attacker.”
“The flaw, which he’s called BadTunnel, exposes local area networks to cross-network NetBIOS Name Service spoofing. An attacker can remotely attack a firewall- or NAT-protected LAN and steal network traffic or spoof a network print or file server.”
“The flaw is particularly serious because it affects every version of Windows, including long-unsupported versions of the OS going back to Windows 95.”
“To successfully implement a BadTunnel attack, [you] just need the victim to open a URL (with Internet Explorer or Edge), or open a file (an Office document), or plug in a USB memory stick. [You] even may not need the victim to do anything when the victim is a web server.”
“For example, if a file URI or UNC path is embedded into a shortcut link file (Microsoft’s LNK), the BadTunnel attack can be triggered at the moment the user views the file in the Windows Explorer. It therefore can be exploited via webpage, email, flash drive and many other medias. It can even be effective against servers.”
“Exploitation points remain open for non-supported Windows operating systems such as XP, Windows Server 2003, and others, for which patches have not been released. For these operating systems, and for those that can’t be updated just yet, system administrators should disable NetBIOS.”
Additional Coverage: Threat Post
Official Microsoft Bulletin MS16-077 CVE-2016-3213

ZFS: Apple’s New Filesystem That Wasn’t

Adam Leventhal, a ZFS developer who designed features such as RAID-Z3, and also worked on DTrace, writes a post about Apple’s recent announcement of its new file system, APFS.
This story is mostly about how ZFS was almost the Apple file system, and what happened.
To learn more about Adam and what he has done, check out our BSDNow #122 Interview with him
“I attended my first WWDC in 2006 to participate in Apple’s launch of their DTrace port to the next version of Mac OS X (Leopard). Apple completed all but the fiddliest finishing touches without help from the DTrace team. Even when they did meet with us we had no idea that they were mere weeks away from the finished product being announced to the world. It was a testament both to Apple’s engineering acumen as well as their storied secrecy.”
“At that same WWDC Apple announced Time Machine, a product that would record file system versions through time for backup and recovery. How were they doing this? We were energized by the idea that there might be another piece of adopted Solaris technology. When we launched Solaris 10, DTrace shared the marquee with ZFS, a new filesystem that was to become the standard against which other filesystems are compared. Key among the many features of ZFS were snapshots that made it simple to capture the state of a filesystem, send the changes around, recover data, etc. Time Machine looked for all the world like a GUI on ZFS (indeed the GUI that we had imagined but knew to be well beyond the capabilities of Sun).”
“Of course Time Machine had nothing to do with ZFS. After the keynote we rushed to an Apple engineer we knew. With shame in his voice he admitted that it was really just a bunch of hard links to directories. For those who don’t know a symlink from a symtab this is the moral equivalent of using newspaper as insulation: it’s fine until the completely anticipated calamity destroys everything you hold dear. So there was no ZFS in Mac OS X, at least not yet.”
“A few weeks before WWDC 2007 nerds like me started to lose their minds: Apple really was going to port ZFS to Mac OS X. It was actually going to happen! Beyond the snapshots that would make backing up a cinch, ZFS would dramatically advance the state of data storage for Apple users. HFS was introduced in System 2.1. HFS improved upon the Macintosh File System by adding—wait for it—hierarchy! No longer would files accumulate in a single pile; you could organize them in folders. And that filesystem has limped along for more than 30 years, nudged forward, rewritten to avoid in-kernel Pascal code, but never reimagined or reinvented.”
“ZFS was to bring to Mac OS X data integrity, compression, checksums, redundancy, snapshots, etc, etc etc. But while energizing Mac/ZFS fans, Sun CEO, Jonathan Schwartz, had clumsily disrupted the momentum that ZFS had been gathering in Apple’s walled garden. Apple had been working on a port of ZFS to Mac OS X. They were planning on mentioning it at the upcoming WWDC. Jonathan, brought into the loop either out of courtesy or legal necessity, violated the cardinal rule of the Steve Jobs-era Apple. Only one person at Steve Job’s company announces new products: Steve Jobs.”
“In fact, this week you’ll see that Apple is announcing at their Worldwide Developer Conference that ZFS has become the file system in Mac OS 10,” mused Jonathan at a press event, apparently to bolster Sun’s own credibility. Less than a week later, Apple spoke about ZFS only when it became clear that a port was indeed present in a developer version of Leopard albeit in a nascent form. Yes, ZFS would be there, sort of, but it would be read-only and no one should get their hopes up.
“By the next WWDC (2008) it seemed that Sun had been forgiven. ZFS was featured in the keynotes, it was on the developer disc handed out to attendees, and it was even mentioned on the Mac OS X Server website. Apple had been working on their port since 2006 and now it was functional enough to be put on full display. I took it for a spin myself; it was really real. The feature that everyone wanted (but most couldn’t say why) was coming!”
“By the time Snow Leopard shipped (2009) only a careful examination of the Apple web site would turn up the odd reference to ZFS left unscrubbed. Whatever momentum ZFS had enjoyed within the Mac OS X product team was gone. I’ve heard a couple of theories and anecdotes from people familiar with the situation”
The uncertainty created by Oracle acquiring Sun, and the fact that it took over a year to close the deal, may not have helped
“In the meantime Sun and NetApp had been locked in a lawsuit over ZFS and other storage technologies since mid-2007”, that certainly didn’t help
“Finally, and perhaps most significantly, personal egos and NIH (not invented here) syndrome certainly played a part. I’m told by folks in Apple at the time that certain leads and managers preferred to build their own rather adopting external technology—even technology that was best of breed. They pitched their own project, an Apple project, that would bring modern filesystem technologies to Mac OS X”
“The design center for ZFS was servers, not laptops—and certainly not phones, tablets, and watches—his argument was likely that it would be better to start from scratch than adapt ZFS.”
“Licensing FUD was thrown into the mix; even today folks at Apple see the ZFS license as nefarious and toxic in some way whereas the DTrace license works just fine for them. Note that both use the same license with the same grants and same restrictions.”
By 2010, “Amazingly that wasn’t quite the end for ZFS at Apple. The architect for ZFS at Apple had left, the project had been shelved, but there were high-level conversations between Sun and Apple about reviving the port. Apple would get indemnification and support for their use of ZFS”
“The Apple-ZFS deal was brought for Larry Ellison’s approval, the first born child of the conquered land brought to be blessed by the new king. “I’ll tell you about doing business with my best friend Steve Jobs,” he apparently said, “I don’t do business with my best friend Steve Jobs.””
“Amusingly the version of the story told quietly at WWDC 2016 had the friends reversed with Steve saying that he wouldn’t do business with Larry. Still another version I’ve heard calls into question the veracity of their purported friendship, and has Steve instead suggesting that Larry go f*ck himself.”
“In the 7 years since ZFS development halted at Apple, they’ve worked on a variety of improvements in HFS and Core Storage, and hacked at at least two replacements for HFS that didn’t make it out the door. This week Apple announced their new filesystem, APFS, after 2 years in development. It’s not done; some features are still in development, and they’ve announced the ambitious goal of rolling it out to laptop, phone, watch, and tv within the next 18 months. At Sun we started ZFS in 2001. It shipped in 2005 and that was really the starting line, not the finish line. Since then I’ve shipped the ZFS Storage Appliance in 2008 and Delphix in 2010 and each has required investment in ZFS / OpenZFS to make them ready for prime time. A broadly featured, highly functional filesystem takes a long time.”
“APFS has merits (more in my next post), but it will always disappoint me that Apple didn’t adopt ZFS irrespective of how and why that decision was made. Dedicated members of the OpenZFS community have built and maintain a port. It’s not quite the same as having Apple as a member of that community, embracing and extending ZFS rather than building their own incipient alternative.”
Additional Coverage
Apple’s APFS guide

Intel ME/AMT: The other processor inside your computer

Recent Intel x86 processors implement a rarely discussed powerful control mechanism that runs on a separate chip that no one is allowed to audit or examine.
Many (all?) vPro chipsets (MCHs) have:
An Independent CPU (not IA32!)
Access to dedicated DRAM memory
Special interface to the Network Card (NIC)
Execution environment called Management Engine (ME)
The Intel Management Engine (ME) is a subsystem composed of a special 32-bit ARC microprocessor that’s physically located inside the chipset. It is an extra general purpose computer running a firmware blob that is sold as a management system for big enterprise deployments.
On some chipsets, the firmware running on the ME implements a system called Intel’s Active Management Technology (AMT). This is entirely transparent to the operating system, which means that this extra computer can do its job regardless of which operating system is installed and running on the main CPU.
The purpose of AMT is to provide a way to manage computers remotely.
This is similar to an older system called “Intelligent Platform Management Interface” or IPMI, but more powerful).
It can offer VNC access to the screen (optionally prompting the local user for permission), IDE redirection (Virtual Media, to boot from a remote device), Serial redirection, etc
To achieve this task, the ME is capable of accessing any memory region without the main x86 CPU knowing about the existence of these accesses. It also runs a TCP/IP server on your network interface and packets entering and leaving your machine addresses to the second MAC address bypass any firewall running on your system.
ME is classified by security researchers as “Ring -3”.
Rings of security can be defined as layers of security that affect particular parts of a system, with a smaller ring number corresponding to an area closer to the hardware.
For example, Ring 3 threats are defined as security threats that manifest in “userspace” mode. Ring 0 threats occur in “kernel” level,
Ring -1 threats occur in a “hypervisor” level, one level lower than the kernel
Ring -2 threats occur in a special CPU mode called “SMM” mode. SMM stands for System-Management-Mode, a special mode that Intel CPUs can be put into that runs a separately defined chunk of code. If attackers can modify the SMM code and trigger the mode, they can get arbitrary execution of code on a CPU.
Although the ME firmware is cryptographically protected with RSA 2048, researchers have been able to exploit weaknesses in the ME firmware and take partial control of the ME on early models. This makes ME a huge security loophole, and it has been called a very powerful rootkit mechanism.
On systems newer than the Core2 series, the ME cannot be disabled.
Intel systems that are designed to have ME but lack ME firmware (or whose ME firmware is corrupted) will refuse to boot, or will shut-down shortly after booting.
There is no way for the x86 firmware or operating system to disable ME permanently. Intel keeps most details about ME absolutely secret. There is absolutely no way for the main CPU to tell if the ME on a system has been compromised.
“We also discovered that the critical parts of the ME firmware are stored in a non-standard compressed format, which gets decompressed by a special hardware decompressor. My initial attempts to brute-force the decompression scheme failed miserably. Another group had better success and they have now completed a working decompression routine for all versions of ME up to but not including version 11.”
There are only a few methods to enable AMT, which is disabled by default.
Most require physical presence during the BIOS boot
ME hardware – ME
Intel ME huffman dictionaries – Unhuffme v2.4
Introducing Ring -3 Rootkits PDF

How to Write Service Status Updates

“The lowly incident status update happens to be one of the most essential pieces of communication a company gets to write”
Your company is having a bad time, your customers are hurting. Everyone is busy, scrambling to fix things, but it is still important to communicate clearly, and regularly, with your customers.
“When users navigate to a status page, they’re driven by a heightened sense of urgency (compared to, say, a website, a blog, or a newsletter). Not many words get as dissected, discussed and forwarded as the ones we place on our status page.”
Often times, very little is written, possibly because very little is known. Everything is read with a slant, because you know the company write it to try to minimize how bad they look.
“Now let’s state the obvious. Customers couldn’t care less about a string of words posted on a status update. What they care about is, “am I in good hands?” Every time we publish (or fail to publish) a service status update we are ultimately answering that question.”
Goals:
1. Write frequent status updates — This can mean postly updates hourly, or even more often. It depends how rapidly the situation is developing. There is nothing worse than an acknowledgement that there is a problem from hours ago, with no further updates. Ideally, indicate when to expect the number post at the end of each update.
2. Well written status updates — Write authoritatively and honestly. Avoid “weasel phrases”.
3. Productive Updates — “What we learned early on was that regular and well-written status updates reduce the amount of incoming support requests. Investing the time to get incident updates right was paying productivity dividends for the rest of the team”
“When faced with service interruptions, we drop everything in our hands and perform operational backflips 24×7 until the service is restored for all customers. During this time, over-communication is a good thing. As is transparency, i.e. acknowledging problems and throwing the public light of accountability on all remaining issues until they’re resolved.”
“While the crisis is unfolding we publish short status updates at regular intervals. We stick to the facts, including scope of impact and possible workarounds. We update the status page even if it’s just to say “we’re still looking into it.””
“Once service is resolved, it’s time to turn our focus on the less urgent, but equally important piece of writing: the post mortem. It demonstrates that someone is investing time on their product. That they care enough to sit down and think things through. Most crucially, it also creates the space for our team to learn and grow as a company”
They link to a second post: How to Write a Post Morten
Or you can just not: Apple offers no explanation for 7 hour outage