Ultimate ZFS Overview | TechSNAP 28
Posted on: October 20, 2011

Coming up on this week’s TechSNAP…
Buckle up and prepare for our Ultimate ZFS overview!
Plus, the next generation of Stuxnet is in the wild, but this time is laying low, collecting data.
All that and more, on this week’s TechSNAP!
Direct Download Links:
HD Video | Large Video | Mobile Video | MP3 Audio | OGG Audio | YouTube
Subscribe via RSS and iTunes:
[ad#shownotes]
Show Notes:
Jupiter Broadcasting Gear
https://www.printfection.com/jbgear
- Coupon Code: SuperDuperShip – Free Shipping on Super Saver, International, and Canadian Airmail orders. No minimums
- Coupon Code: SuperSave$10 – $10 off orders with a subtotal of $50+
- Coupon Code: Scary35% – 35% off orders with a subtotal of $100+
Next generation of Stuxnet seen in the wild?
- Called Duqu, the malware appears to be based on the same concepts as Stuxnet, and likely was written by some of the same people, or someone with access to the Stuxnet source code.
- The malware is designed to be stealthy and silent, rather than exploiting the system to some gain, like most malware
- The rootkit loads it self as a validly signed driver. It appears to have been signed by the certificate of a company in Taiwan identified as C-Media Electronics Incorporation. It is possible that their systems were compromised and their private key is being used without their knowledge. The certificate was set to expire on August 2, 2012, but authorities revoked it on Oct. 14
- The malware is not a worm, as it does it spread, and has no destructive payload
- It appears to only gather intelligence and act as a espionage agent, collecting data to be used a future attack.
- Analysts claim it appears to be seeking information on an unidentified industrial control system
- Duqu appears to have been in operation, undetected for more than a year
- Symantec has declined to name the countries where the malware was found, or to identify the specific industries infected, other than to say they are in the manufacturing and critical infrastructure sectors
- Duqu analysis paper
Google switching to SSL for logged in users’ searches
- Users who do a search while logged in, will do the search over SSL, meaning their search query and the results will be protected from snooping by their ISP, Government, Law Enforcement and WiFi hackers.
- This is an important step as google works to personalize your search results more and more.
- An interesting side effect of this is that browsers do not pass referrer headers when you transition from an SSL site. So the sites you visit from the search results page will no longer see what your search query was. Clicks on Adwords and other sponsored links will still pass your search query.
- The primary impediment to SSL for everything is performance, encrypting all traffic on the web would require a great deal more hardware. This is why Google defaults to a weaker encryption for things like search results, than what online merchants typically use.
- Another impediment to SSL is the certificate system, typical setups require a unique IP for each SSL certificate (because the name based virtual hosting typically done by web servers relies on an HTTP header, that is not sent until after the encryption session is started). However modern browsers and web servers support ‘SNI’ (Server Name Indication) to allow that information to be passed as part of the initial encryption setup. There are also solutions such as wildcard certificates (ie, *.google.com) and Unified Communications Certificates (UCC, typically used for MS Exchange servers and the like).
- Google will also provide website owners with the top 1000 search queries that lead visitors to their site via Google Webmaster Tools.
- HTTPS Everywhere | Electronic Frontier Foundation
Feedback:
- TechSnap Question – YouTube
- Typically a solution like this relies on a hard line connection between the two wireless APs so that they do not have to communicate via wireless as well.
- www.dd-wrt.com | Unleash Your Router
- DD-WRT Router Database
- Turn Your $60 Router into a User-Friendly Super-Router with Tomato
- Tomato (firmware)
ZFS Segment
- This week we will be taking a look at ZFS as a storage solution
- ZFS was originally developed by Sun Microsystems to be able to store a zetta byte of data (A zetta byte is equal to 1 billion tera bytes)
- ZFS is both the Volume Manager and the File System. This gives it some unique benefits, including the ability to increase the size of the file system on the fly and improves performance for the ‘scrub’ (integrity check all data) and resilver (recover from a failed disk) operations, as only data blocks that are actually in use need to be rewritten, whereas a hardware RAID controller must resilver the entire disk because it is unaware of the file system.
- ZFS is a ‘Copy-On-Write’ file system, this means that data is not immediately overwritten when it is changed
- Features
- Multiple mount points – You can create various mount points from the same storage pool, allowing you to have different settings for different types of files.
- Passive Integrity Checking (Fletcher Checksum or SHA–2) – As data is read, it is compared against the checksum (or hash, depending on settings). If the data is found to be corrupted, ZFS attempts to recover it (from a mirrored device, RAID Z, or copies). This feature allows ZFS to detect silent corruption that normally goes unnoticed.
- RAID Z – RAID Z works very similar to RAID 5, except without the requirement for a hardware RAID controller. RAID Z2 provides two parity drives, like RAID 6. Recently, RAID Z3 was also introduced, using 3 drives for parity, providing exceptional fault tolerance.
- Compression – Allow you to compress the data stored in this mount point (defaults to lzjb for speed, or you can choose a specific level of gzip). This can be great for storing highly compressible information such as log files
- Deduplication – Since ZFS already knows the hash of your files as it writes them, it can detect that a file with the identical content already exists in your storage pool, and it will simply link the new file to the old one, and because ZFS is copy-on-write, if either file changes, it does not effect the other. ZFS also supports an optional ‘verify’ setting, where even if the checksum/hash matches, it will do a byte-by-byte verification to ensure the files are the same, to avoid a cache collision resulting in data corruption, even though the chances of this happening are around 10^–77. Deduplication uses a lot of ram, so it is recommended that you only use it on datasets where there is a high probability of duplication (It requires 320 bytes per block, meaning 1TB of data in 8kb blocks requires 32GB of ram. ZFS allows blocks up to 128kb). Deduplication will only use up to 25% of ARC memory, after that performance is degraded.
- Purposeful Duplication (Copies) – Allows you to ask ZFS to maintain more than 1 copy of each file in a mount point. This is in addition to any redundancy provided by mirrors/RAID Z etc. Where possible the additional copies are stored on different physical devices. This allows you to get the benefit of a system like RAID Z but only for a specific set of data, while using regular striping for the rest, to maximize your storage capacity. (The ‘Copies’ system was not designed to protect against entire drives failing, just the loss of specific sectors, also this setting only effects newly created files, so you should set it when you create the mount point)
- Snapshots – A read only copy of the file system from a specific point in time, great for backups etc.
- Clones – A writable snapshot. Allows you to create a second copy of the file system that shares all of the same disk space, and any changes to either the original or the clone get saved separately.
- Dynamic Striping – As you add more disks to your ZFS pool, the strips are automatically adjusted to take advantage of the write performance of all available disks.
- Space Reservation – Since all mount points share the same pool of free space, you can set reservations to make sure specific mount points always have access to free space, even if another mount point is trying to use all of the space.
- In summary, ZFS can be a great solution for your home file server, as it allows you the flexibility to add additional storage at any time, deduplicate files, provided limited redundancy without needing RAID and can even provide some Drobo like functionality.
- If you keep at least one SATA port available in your file server, you can replace smaller devices by attaching the newer drive, and using the ‘zpool replace’ command, to copy all of the data to the new device, then remove the smaller one. You can eventually replace every device in the system this way, and the storage pool sizes up automatically.
- RAID Z pools cannot currently have devices added to them, although this feature is in the works. If you create a RAID Z (or Z2/Z3) pool, you can still increase it’s storage capacity by replacing each disk one at a time, and waiting for it to resilver (unlike in non-redundant setups, you do not have to connect the new device before removing the old one). Again, because ZFS is both the Volume Manager and the File System, the resilvering process is faster, because only data that is actually in use needs to be written to the new device.
Round Up:
- Jobs offered ‘nine-digit price’ to buy Dropbox
- NCI, Australia’s largest Supercomputer, confirmed hacked.
- Sesame Street’s YouTube channel hacked, replaced with porn
- Analysis of 250,000 hacker conversations PDF
- Google Music to support peer-to-peer file sharing, says record exec
- MIT researches develop system to record real time video through walls