replication – Jupiter Broadcasting https://www.jupiterbroadcasting.com Open Source Entertainment, on Demand. Wed, 30 Sep 2020 05:39:47 +0000 en-US hourly 1 https://wordpress.org/?v=5.5.3 https://original.jupiterbroadcasting.net/wp-content/uploads/2019/04/cropped-favicon-32x32.png replication – Jupiter Broadcasting https://www.jupiterbroadcasting.com 32 32 Your New Tools | LINUX Unplugged 373 https://original.jupiterbroadcasting.net/142932/your-new-tools-linux-unplugged-373/ Tue, 29 Sep 2020 22:30:00 +0000 https://original.jupiterbroadcasting.net/?p=142932 Show Notes: linuxunplugged.com/373

The post Your New Tools | LINUX Unplugged 373 first appeared on Jupiter Broadcasting.

]]>

Show Notes: linuxunplugged.com/373

The post Your New Tools | LINUX Unplugged 373 first appeared on Jupiter Broadcasting.

]]>
Snapshot Sanity | TechSNAP 402 https://original.jupiterbroadcasting.net/130786/snapshot-sanity-techsnap-402/ Thu, 25 Apr 2019 16:01:50 +0000 https://original.jupiterbroadcasting.net/?p=130786 Show Notes: techsnap.systems/402

The post Snapshot Sanity | TechSNAP 402 first appeared on Jupiter Broadcasting.

]]>

Show Notes: techsnap.systems/402

The post Snapshot Sanity | TechSNAP 402 first appeared on Jupiter Broadcasting.

]]>
Big Network, SmallWall | BSD Now 97 https://original.jupiterbroadcasting.net/84942/big-network-smallwall-bsd-now-97/ Thu, 09 Jul 2015 10:06:09 +0000 https://original.jupiterbroadcasting.net/?p=84942 Coming up this time on the show, we’ll be chatting with Lee Sharp. He’s recently revived the m0n0wall codebase, now known as SmallWall, and we’ll find out what the future holds for this new addition to the BSD family. As usual, we’ve also got answers to your emails and all this week’s news on BSD […]

The post Big Network, SmallWall | BSD Now 97 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Coming up this time on the show, we’ll be chatting with Lee Sharp. He’s recently revived the m0n0wall codebase, now known as SmallWall, and we’ll find out what the future holds for this new addition to the BSD family. As usual, we’ve also got answers to your emails and all this week’s news on BSD Now – the place to B.. SD.

Thanks to:


DigitalOcean


iXsystems


Tarsnap

Direct Download:

Video | HD Video | MP3 Audio | OGG Audio | Torrent | YouTube

RSS Feeds:

MP3 Feed | OGG Feed | iTunes Feed | Video Feed | HD Vid Feed | HD Torrent Feed

– Show Notes: –

Headlines

BSDCan and pkgsrcCon videos


OPNsense 15.7 released

  • The OPNsense team has released version 15.7, almost exactly six months after their initial debut
  • In addition to pulling in the latest security fixes from upstream FreeBSD, 15.7 also includes new integration of an intrusion detection system (and new GUI for it) as well as new blacklisting options for the proxy server
  • Taking a note from upstream PF’s playbook, ALTQ traffic shaping support has finally been retired as of this release (it was deprecated from OpenBSD a few years ago, and the code was completely removed just over a year ago)
  • The LibreSSL flavor has been promoted to production-ready, and users can easily migrate over from OpenSSL via the GUI – switching between the two is simple; no commitment needed
  • Various third party ports have also been bumped up to their latest versions to keep things fresh, and there’s the usual round of bug fixes included
  • Shortly afterwards, 15.7.1 was released with a few more small fixes

NetBSD at Open Source Conference 2015 Okinawa

  • If you liked last week’s episode then you’ll probably know what to expect with this one
  • The NetBSD users group of Japan hit another open source conference, this time in Okinawa
  • This time, they had a few interesting NetBSD machines on display that we didn’t get to see in the interview last week
  • We’d love to see something like this in North America or Europe too – anyone up for installing BSD on some interesting devices and showing them off at a Linux con?

OpenBSD BGP and VRFs

  • VRFs, or in OpenBSD rdomains, are a simple, yet powerful (and sometimes confusing) topic”
  • This article aims to explain both BGP and rdomains, using network diagrams, for some network isolation goodness
  • With multiple rdomains, it’s also possible to have two upstream internet connections, but lock different groups of your internal network to just one of them
  • The idea of a “guest network” can greatly benefit from this separation as well, even allowing for the same IP ranges to be used without issues
  • Combining rdomains with the BGP protocol allows for some very selective and precise blocking/passing of traffic between networks, which is also covered in detail here
  • The BSDCan talk on rdomains expands on the subject a bit more if you haven’t seen it, as well as a few related posts

Interview – Lee Sharp – lee@smallwall.org

SmallWall, a continuation of m0n0wall


News Roundup

Solaris adopts more BSD goodies

  • We mentioned a while back that Oracle developers have begun porting a current version of OpenBSD’s PF firewall to their next version, even contributing back patches for SMP and other bug fixes
  • They recently published an article about PF, talking about what’s different about it on their platform compared to others – not especially useful for BSD users, but interesting to read if you like firewalls
  • Darren Moffat, who was part of originally getting an SSH implementation into Solaris, has a second blog post up about their “SunSSH” fork
  • Going forward, their next version is going to offer a completely vanilla OpenSSH option as well, with the plan being to phase out SunSSH after that
  • The article talks a bit about the history of getting SSH into the OS, forking the code and also lists some of the differences between the two
  • In a third blog post, they talk about a new system call they’re borrowing from OpenBSD, getentropy(2), as well as the addition of arc4random to their libc
  • With an up-to-date and SMP-capable PF, ZFS with native encryption, jail-like Zones, unaltered OpenSSH and secure entropy calls… is Solaris becoming better than us?
  • Look forward to the upcoming “Solaris Now” podcast (not really)

EuroBSDCon 2015 talks and tutorials

  • This year’s EuroBSDCon is set to be held in Sweden at the beginning of October, and the preliminary list of accepted presentations has been published
  • The list looks pretty well-balanced between the different BSDs, something Paul would be happy to see if he was still with us
  • It even includes an interesting DragonFly talk and a couple talks from NetBSD developers, in addition to plenty of FreeBSD and OpenBSD of course
  • There are also a few tutorials planned for the event, some you’ve probably seen already and some you haven’t
  • Registration for the event will be opening very soon (likely this week or next)

Using ZFS replication to improve offsite backups

  • If you take backups seriously, you’re probably using ZFS and probably keeping an offsite copy of the data
  • This article covers doing just that, but with a focus on making use of the replication capability
  • It’ll walk you through taking a snapshot of your pool and then replicating it to another remote system, using “zfs send” and SSH – this has the benefit of only transferring the files that have changed since the last time you did it
  • Steps are also taken to allow a regular user to take and manage snapshots, so you don’t need to be root for the SSH transfer
  • Data integrity is a long process – filesystem-level checksums, resistance to hardware failure, ECC memory, multiple copies in different locations… they all play a role in keeping your files secure; don’t skip out on any of them
  • One thing the author didn’t mention in his post: having an offline copy of the data, ideally sealed in a safe place, is also important

Block encryption in OpenBSD

  • We’ve covered ways to do fully-encrypted installations of OpenBSD (and FreeBSD) before, but that requires dedicating a whole drive or partition to the sensitive data
  • This blog post takes you through the process of creating encrypted containers in OpenBSD, à la TrueCrypt – that is, a file-backed virtual device with an encrypted filesystem
  • It goes through creating a file that looks like random data, pointing vnconfig at it, setting up the crypto and finally using it as a fake storage device
  • The encrypted container method offers the advantage of being a bit more portable across installations than other ways

Docker hits FreeBSD ports

  • The inevitable has happened, and an early FreeBSD port of docker is finally here
  • Some details and directions are available to read if you’d like to give it a try, as well as a list of which features work and which don’t
  • There was also some Hacker News discussion on the topic

Microsoft donates to OpenSSH

  • We’ve talked about big businesses using BSD and contributing back before, even mentioning a few other large public donations – now it’s Microsoft’s turn
  • With their recent decision to integrate OpenSSH into an upcoming Windows release, Microsoft has donated a large sum of money to the OpenBSD foundation, making them a gold-level sponsor
  • They’ve also posted some contract work offers on the OpenSSH mailing list, and say that their changes will be upstreamed if appropriate – we’re always glad to see this

Feedback/Questions


  • Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv
  • We’re always looking for interviews – get in touch if you’re doing anything cool with BSD that you’d like to talk about (or want to suggest someone else)
  • The FreeNAS community recently lost one of their most active members, Marbus90, who has been a big help to them for a long time – rest in peace and thanks for all your work

The post Big Network, SmallWall | BSD Now 97 first appeared on Jupiter Broadcasting.

]]>
Churning Over Btrfs | LINUX Unplugged 88 https://original.jupiterbroadcasting.net/80442/churning-over-btrfs-lup-88/ Tue, 14 Apr 2015 19:03:35 +0000 https://original.jupiterbroadcasting.net/?p=80442 Our discussion of Linux filesystems goes in depth this week as our LUG sorts out the best filesystem for your Linux desktop, server, laptop & mobile. Plus a few corrections from last week & lots of follow up. We also look at the release of Linux 4.0, some of the more humorous press coverage it’s […]

The post Churning Over Btrfs | LINUX Unplugged 88 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Our discussion of Linux filesystems goes in depth this week as our LUG sorts out the best filesystem for your Linux desktop, server, laptop & mobile. Plus a few corrections from last week & lots of follow up.

We also look at the release of Linux 4.0, some of the more humorous press coverage it’s received & the “big feature” Linus could care less about.

Thanks to:

Ting


DigitalOcean


Linux Academy

Direct Download:

MP3 Audio | OGG Audio | Video | HD Video | Torrent | YouTube

RSS Feeds:

MP3 Feed | OGG Feed | iTunes Feed | Video Feed | Torrent Feed | WebM Torrent Feed

Become a supporter on Patreon:

Foo

Show Notes:

Catch Up:

4.0 doesn’t have all that much special. Much have been
made of the new kernel patching infrastructure, but realistically,
that not only wasn’t the reason for the version number change, we’ve
had much bigger changes in other versions. So this is very much a
“solid code progress” release

TING

Dell XPS 13

LinuxFest Northwest 2015

Bellingham, WA • April 25th & 26th

Holy cookies I did it..

I created a Jupiter Broadcasting Meetup group. I was inspired by the idea of how nice it would be to have a rough idea of how many folks will be making it out to LFNW. Just to help with planning.

tl;dr


Linux Academy

The Death of Chris’ #1 Rig

Death of my #1 Rig Captured

Why I think btrfs remains the future for those who want it

Chris on the latest episode of Unplugged took the time to vent his latest frustration with btrfs. The issue in question is a bug that causes arch not to boot on 3.19.1-3, 3.18.9 and 3.14.35. I and quite a few others have experienced this on our Arch btrfs setups and I enjoyed fixing this issue for the same reason I enjoyed fixing libxcursor and radeon bugs that caused my system not to boot.

Easy rollback after update (btrfs and grub).

Hope anyone can find this useful when dealing with bleeding edge breakage 🙂

OS Backups…

How PC-BSD Does it…


  • Stable kernel version 3.19.1+ can cause a deadlock at mount time
    • workaround: boot with older kernel, or run btrfs-zero-log to clear the log (beware of the consequences)
    • fix: scheduled for 3.19.4, or apply 9c4f61f01d269815bb7c37.
    • also affected: 3.14.35+, 3.18.9+
  • Versions from 3.15 up to 3.16.1 suffer from a deadlock that was observed during heavy rsync workloads with compression on, it’s recommended to use 3.16.2 and newer

The server edition of Fedora 22 is using the XFS file-system by default rather than EXT4.

Using the XFS file-system as the default within an LVM has been part of the Fedora Server technical spec while with Fedora 22 it’s finally happened. The default layout for Fedora Server 22 installations is using XFS atop LVM while /boot is outside the LVM setup.

DigitalOcean

Runs Linux from the people:

  • Send in a pic/video of your runs Linux.
  • Please upload videos to YouTube and submit a link via email or the subreddit.

Support Jupiter Broadcasting on Patreon

The post Churning Over Btrfs | LINUX Unplugged 88 first appeared on Jupiter Broadcasting.

]]>
Day-0 of an InfoSec Career | TechSNAP 209 https://original.jupiterbroadcasting.net/80277/day-0-of-an-infosec-career-techsnap-209/ Thu, 09 Apr 2015 19:57:13 +0000 https://original.jupiterbroadcasting.net/?p=80277 Is it possible to make a truly private phone call anymore? The answer might surprise you. Cisco and Level 3 battle a huge SSH botnet & how to Build a successful Information Security career. Plus a great batch of your questions, a rocking round up, and much, much more! Thanks to: Get Paid to Write […]

The post Day-0 of an InfoSec Career | TechSNAP 209 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Is it possible to make a truly private phone call anymore? The answer might surprise you. Cisco and Level 3 battle a huge SSH botnet & how to Build a successful Information Security career.

Plus a great batch of your questions, a rocking round up, and much, much more!

Thanks to:


DigitalOcean


Ting


iXsystems

Direct Download:

HD Video | Mobile Video | MP3 Audio | OGG Audio | YouTube | HD Torrent | Mobile Torrent

RSS Feeds:

HD Video Feed | Mobile Video Feed | MP3 Audio Feed | Ogg Audio Feed | iTunes Feed | Torrent Feed

Become a supporter on Patreon:

Foo

— Show Notes: —

How to make secret phone calls

  • “There’s a lot you can find in the depths of the dark web, but in 2013, photographer and artist Curtis Wallen managed to buy the ingredients of a new identity”
  • “After purchasing a Chromebook with cash, Wallen used Tor, virtual marketplaces, and a bitcoin wallet to purchase a fake driver’s license, insurance card, social security number, and cable bill, among other identifying documents. Wallen saw his new identity, Aaron Brown, as more than just art: Brown was a political statement on the techno-surveillance age.”
  • The article sets out the steps required to conduct untraceable phone calls
  • The instructions are based on looking at how CIA OpSec was compromised by cell phones in the cases of the 2005 extraordinary rendition of Hassan Mustafa Osama in Italy and their surveillance of Lebanese Hezbollah
  • “using a prepaid “burner” phone, posting its phone number publicly on Twitter as an encrypted message, and waiting for your partner to decrypt the message and call you at a later time”
  • Analyze your daily movements, paying special attention to anchor points (basis of operation like home or work) and dormant periods in schedules (8-12 p.m. or when cell phones aren’t changing locations);
  • Leave your daily cell phone behind during dormant periods and purchase a prepaid no-contract cell phone (“burner phone”);
  • After storing burner phone in a Faraday bag, activate it using a clean computer connected to a public Wi-Fi network;
  • Encrypt the cell phone number using a onetime pad (OTP) system and rename an image file with the encrypted code. Using Tor to hide your web traffic, post the image to an agreed upon anonymous Twitter account, which signals a communications request to your partner;
  • Leave cell phone behind, avoid anchor points, and receive phone call from partner on burner phone at 9:30 p.m.—or another pre-arranged “dormant” time—on the following day;
  • Wipe down and destroy handset.
  • “The approach is “very passive” says Wallen. For example, “Posting an image to Twitter is a very common thing to do, [and] it’s also very common for image names to have random numbers and letters as a file name,” he says. “So, if I’ve prearranged an account where I’m going to post an encrypted message, and that message comes in the form of a ‘random’ filename, someone can see that image posted to a public Twitter account, and write down the filename—to decrypt by hand—without ever actually loading the image. Access that Twitter account from Tor, from a public Internet network, and there’s hardly any trace that an interaction even happened.””
  • “This is not easy, of course. In fact, it’s really, comically hard. “If the CIA can’t even keep from getting betrayed by their cell phones, what chance do we have?””
  • “Central to good privacy, says Wallen, is eliminating or reducing anomalies that would pop up on surveillance radars, like robust encryption or SIM card swapping. To understand the risks of bringing unwanted attention to one’s privacy practices, Wallen examined the United States Marine Corps’ “Combat Hunter” program, which deals with threat assessment through observation, profiling, and tracking.”
  • “Anomalies are really bad for what I’m trying to accomplish—that means any overt encryption is bad, because it’s a giant red flag,” Wallen said. “I tried to design the whole system to have as small a footprint as possible, and avoid creating any analyzable links.”
  • “I was going out and actually buying phones, learning about different ways to buy them, to activate them, to store them, and so on,” said Wallen, who eventually bought a burner phone from a Rite Aid. “I kept doing it until I felt like I’d considered it from every angle.”
  • “After consulting on commercially available Faraday bags, Wallen settled on the Ramsey Electronics STP1100
  • Wallen cautions his audience about taking his instructions too literally. The project, he says, “was less about arriving at a necessarily practical system for evading cell phone tracking, than it was about the enjoyment of the ‘game’ of it all. In fact, I think that it is so impractical says a lot.”
  • “Bottom line,” he adds. “If your adversary is a nation state, don’t use a cellphone.”
  • Guide to creating and using One-Time Pads
  • John Oliver: Government Surveillance — Interview with Edward Snowden

Cisco and Level 3 battle a huge SSH botnet

  • “Talos has been monitoring a persistent threat for quite some time, a group we refer to as SSHPsychos or Group 93. This group is well known for creating significant amounts of scanning traffic across the Internet. Although our research efforts help inform and protect Cisco customers globally, sometimes it is our relationships that can multiply this impact. Today Cisco and Level 3 Communications took action to help ensure a significantly larger portion of the Internet is also protected.”
  • “The behavior consists of large amounts of SSH brute force login attempts from 103.41.124.0/23, only attempting to guess the password for the root user, with over 300,000 unique passwords. Once a successful login is achieved the brute forcing stops. The next step involves a login from a completely different IP ranges owned by shared hosting companies based out of the United States. After login is achieved a wget request is sent outbound for a single file which has been identified as a DDoS rootkit. “
  • “Once the rootkit is installed additional instructions are downloaded via an XOR encoded file from one of the C2 servers. The config file is largely constructed of a list of IP addresses that are being denied and filenames, and files to be deleted.”
  • “At times, this single attacker accounted for more than 35% of total Internet SSH traffic”
  • Level 3 then worked to block the malicious traffic
  • “Our goal, when confirming an Internet risk, is to remove it as broadly as possible; however, before removing anything from the Internet, it is important to fully understand the impact that may have to more benign hosts. To do this, we must understand more details of the attacker’s tools and infrastructure.”
  • “As part of the process, Level 3 worked to notify the appropriate providers regarding the change. On March 30th SSHPsychos suddenly pivoted. The original /23 network went from a huge volume of SSH brute force attempts to almost no activity and a new /23 network began large amounts of SSH brute forcing following the exact same behavior associated with SSHPsychos. The new network is 43.255.190.0/23 and its traffic was more than 99% SSH immediately after starting communication. The host serving the malware also changed and a new host (23.234.19.202) was seen providing the same file as discussed before a DDoS Rootkit.”
  • “Based on this sudden shift, immediate action was taken. Talos and Level 3 decided to remove the routing capabilities for 103.41.124.0/23, but also add the new netblock 43.255.190.0/23. The removal of these two netblocks introduced another hurdle for SSHPsychos, and hopefully slows their activity, if only for a short period.”
  • “For those of you who have Linux machines running sshd on the open Internet, be sure to follow the best practice of disabling root login in your sshd config file. That step alone would stop this particular attacker from being successful in your environment.”
  • Remote root login should never be allowed anyway
  • Hopefully this will send a clear message to the providers that allow these type of attackers to operate on their network. If you don’t clean up your act, you’ll find large swaths of your IP space unusable on the public internet.

How to Build a Successful Information Security Career

  • A question I often get is “how do I get into InfoSec”
  • Myself, not actually being an InfoSec professional, and never having really worked in that space, do not have the answer
  • Luckily, someone who is in that space, finally wrote it all down
  • “One of the most important things for any infosec professional is a good set of inputs for news, articles, tools, etc.”
    • So, keep watching TechSNAP
  • Basic Steps:
  • Education (Sysadmin, Networking, Development)
  • Building Your Lab (VMs, VPSs from Digital Ocean)
  • You Are Your Projects (Build something)
  • Have a Presence (Website, Blog, Twitter, etc)
  • Certifications (“Things have the value that others place on them”)
  • Networking With Others (Find a mentor, be an intern)
  • Conferences (Go to Conferences. Speak at them)
  • Mastering Professionalism (Dependability, Well Written, Good Speaker)
  • Understanding the Business (Businesses want to quantify risk so they can decide how much should be spent on mitigating it)
  • Having Passion (90% of being successful is simply getting 100,000 chances to do so. You get chances by showing up)
  • Becoming Guru
  • It is a very good read, broken down into easy to understand steps, with the justification for each requirement, as well as some alternatives, because one size does not fit all
  • Related, but Roundup is already full enough: How to Avoid a Phone Call from Brian Krebs – The Basics of Intrusion Detection and Prevention with Judy Novak

Feedback:


Round Up:


The post Day-0 of an InfoSec Career | TechSNAP 209 first appeared on Jupiter Broadcasting.

]]>
Any Cert Will Do | TechSNAP 208 https://original.jupiterbroadcasting.net/79867/any-cert-will-do-techsnap-208/ Thu, 02 Apr 2015 16:51:51 +0000 https://original.jupiterbroadcasting.net/?p=79867 Why boring technology might be the better choice, Google revokes & China chokes, why you want to create an account at irs.gov before crooks do it for you. Plus your great IT questions, a rocking round up & much, much more! Thanks to: Get Paid to Write for DigitalOcean Direct Download: HD Video | Mobile […]

The post Any Cert Will Do | TechSNAP 208 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Why boring technology might be the better choice, Google revokes & China chokes, why you want to create an account at irs.gov before crooks do it for you.

Plus your great IT questions, a rocking round up & much, much more!

Thanks to:


DigitalOcean


Ting


iXsystems

Direct Download:

HD Video | Mobile Video | MP3 Audio | OGG Audio | YouTube | HD Torrent | Mobile Torrent

RSS Feeds:

HD Video Feed | Mobile Video Feed | MP3 Audio Feed | Ogg Audio Feed | iTunes Feed | Torrent Feed

Become a supporter on Patreon:

Foo

— Show Notes: —

Why you should choose boring technology

  • The basic premise is that in building technology, specifically web sites and web services, there is often a bias towards using the latest and greatest technology, rather than the same old boring stuff
  • This often turns out to bite you in the end. Look at people who based their site or product on FoundationDB, which was recently bought and shutdown by Apple
  • Look at one of the most popular sites on the internet, Facebook, originally written in PHP and MySQL, and still largely remains based on those same old technologies
  • “The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood.”
  • “Anyone who knows me well will understand that it’s only with a overwhelming sense of malaise that I now invoke the spectre of Don Rumsfeld, but I must.“
  • “When choosing technology, you have both known unknowns and unknown unknowns”
  • The Socratic paradox
  • A known unknown is something like: we don’t know what happens when this database hits 100% CPU.
  • An unknown unknown is something like: geez it didn’t even occur to us that writing stats would cause GC pauses.
  • “Both sets are typically non-empty, even for tech that’s existed for decades. But for shiny new technology the magnitude of unknown unknowns is significantly larger, and this is important.”
  • The advantage to using boring technology is that more people understand how it works, more people understand how it fails, more people have come before you, tried to do something similar to what you are doing
  • You won’t find the answer on Stack Overflow if you are the first person to try it
  • “One of the most worthwhile exercises I recommend here is to consider how you would solve your immediate problem without adding anything new. First, posing this question should detect the situation where the “problem” is that someone really wants to use the technology. If that is the case, you should immediately abort.”
  • People like new toys and new challenges
  • Businesses should try to avoid new costs, and new risks
  • Adding a new technology is not a bad thing, but first consider if the goal can be accomplished with what you already have

Google revokes CNNIC root certificate trust

  • On March 20th Google security engineers noticed a number of unauthorized certificates being used for gmail and other google domains
  • The certificates were issued by a subordinate CA, MCS Holdings
  • “Established in 2005, MCS (Mideast Communication Systems) offers Value Added Distribution focusing on Networking and Automation businesses.”
  • MCS Holdings makes Firewalls and other network appliances
  • MCS got its subordinate CA certificate from CNNIC (Chinese Internet Network Information Center)
  • “CNNIC is included in all major root stores and so the misissued certificates would be trusted by almost all browsers and operating systems. Chrome on Windows, OS X, and Linux, ChromeOS, and Firefox 33 and greater would have rejected these certificates because of public-key pinning, although misissued certificates for other sites likely exist.”
  • Google added the MCS certificate to its revocation list so it would no longer be trusted
  • “CNNIC responded on the 22nd to explain that they had contracted with MCS Holdings on the basis that MCS would only issue certificates for domains that they had registered. However, rather than keep the private key in a suitable HSM, MCS installed it in a man-in-the-middle proxy. These devices intercept secure connections by masquerading as the intended destination and are sometimes used by companies to intercept their employees’ secure traffic for monitoring or legal reasons. The employees’ computers normally have to be configured to trust a proxy for it to be able to do this. However, in this case, the presumed proxy was given the full authority of a public CA, which is a serious breach of the CA system”
  • Google accepted the explanation as the truth, but is unsatisfied with the situation
  • “This explanation is congruent with the facts. However, CNNIC still delegated their substantial authority to an organization that was not fit to hold it.”
  • CNNIC has specific obligations it must fulfill in order to be a trusted CA
  • The CA/Browser Forum sets the policies agreed upon for signing new trusted certificates
  • Mozilla has an existing policy that enumerates the possible problems and their immediate and potential consequences
  • “Update – April 1: As a result of a joint investigation of the events surrounding this incident by Google and CNNIC, we have decided that the CNNIC Root and EV CAs will no longer be recognized in Google products. This will take effect in a future Chrome update. To assist customers affected by this decision, for a limited time we will allow CNNIC’s existing certificates to continue to be marked as trusted in Chrome, through the use of a publicly disclosed whitelist. While neither we nor CNNIC believe any further unauthorized digital certificates have been issued, nor do we believe the misissued certificates were used outside the limited scope of MCS Holdings’ test network, CNNIC will be working to prevent any future incidents. CNNIC will implement Certificate Transparency for all of their certificates prior to any request for reinclusion. We applaud CNNIC on their proactive steps, and welcome them to reapply once suitable technical and procedural controls are in place.”
  • CNNIC has released an official statement calling Google’s actions “unacceptable”
  • Mozilla is considering similar actions:
  • Reject certificates chaining to CNNIC with a notBefore date after a threshold date
  • Request that CNNIC provide a list of currently valid certificates and publish that list so that the community can recognize any back-dated certs
  • Allow CNNIC to re-apply for full inclusion, with some additional requirements (to be discussed on this list)
  • If CNNIC’s re-application is unsuccessful, then their root certificates will be removed
  • The Mozilla community feels that CNNIC needs more than a slap on the wrist, to ensure other CAs (and Governments) get the message that this type of behaviour is unacceptable
  • Google reiterates the need for the Certificate Transparency project
  • “Certificate Transparency makes it possible to detect SSL certificates that have been mistakenly issued by a certificate authority or maliciously acquired from an otherwise unimpeachable certificate authority. It also makes it possible to identify certificate authorities that have gone rogue and are maliciously issuing certificates.”
  • Additional Coverage – Ars Technica

Signup for an account at irs.gov before crooks do it for you

  • “If you’re an American and haven’t yet created an account at irs.gov, you may want to take care of that before tax fraudsters create an account in your name and steal your personal and tax data in the process.”
  • “Recently, KrebsOnSecurity heard from Michael Kasper, a 35-year-old reader who tried to obtain a copy of his most recent tax transcript with the Internal Revenue Service (IRS). Kasper said he sought the transcript after trying to file his taxes through the desktop version of TurboTax, and being informed by TurboTax that the IRS had rejected the request because his return had already been filed.”
  • “Kasper said he phoned the IRS’s identity theft hotline (800-908-4490) and was told a direct deposit was being made that very same day for his tax refund — a request made with his Social Security number and address but to be deposited into a bank account that he didn’t recognize.”
  • The fraudster filed the new return using nearly identical data to the correct information that the victim had filed the previous year
  • The victim suspects that the fraudster was able to use the irs.gov portal to view his previous returns and extract information from them to file the fraudulent return
  • The fraudster files a corrected W-2 to adjust the withholding amount, to get a bigger refund
  • The story goes on into details about the case, including the college student that was used as a money mule
  • “The IRS’s process for verifying people requesting transcripts is vulnerable to exploitation by fraudsters because it relies on static identifiers and so-called “knowledge-based authentication” (KBA) — i.e., challenge questions that can be easily defeated with information widely available for sale in the cybercrime underground and/or with a small amount of searching online.”
  • In addition, Americans who have not already created an account at the Social Security Administration under their Social Security number are vulnerable to crooks hijacking SSA benefits now or in the future. For more on how crooks are siphoning Social Security benefits via government sites, check out this story.
  • In Canada, to get access to your CRA Account, a passcode is mailed to you, at the current address the government already has on file for you
  • In order to gain access to your account, you also must answer more specific questions than just KBAs, usually including things like “the number from line 350 of your 2013 tax return”

Feedback:


Round Up:


The post Any Cert Will Do | TechSNAP 208 first appeared on Jupiter Broadcasting.

]]>
Patch and Notify | TechSNAP 197 https://original.jupiterbroadcasting.net/75657/patch-and-notify-techsnap-197/ Thu, 15 Jan 2015 22:21:43 +0000 https://original.jupiterbroadcasting.net/?p=75657 Been putting off that patch? This week we’ll cover how an out of date Joomla install led to a massive breach, Microsoft and Google spar over patch disclosures & picking the right security question… Plus a great batch of your feedback, a rocking round up & much, much more! Thanks to: Get Paid to Write […]

The post Patch and Notify | TechSNAP 197 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Been putting off that patch? This week we’ll cover how an out of date Joomla install led to a massive breach, Microsoft and Google spar over patch disclosures & picking the right security question…

Plus a great batch of your feedback, a rocking round up & much, much more!

Thanks to:


DigitalOcean


Ting


iXsystems

Direct Download:

HD Video | Mobile Video | MP3 Audio | OGG Audio | YouTube | HD Torrent | Mobile Torrent

RSS Feeds:

HD Video Feed | Mobile Video Feed | MP3 Audio Feed | Ogg Audio Feed | iTunes Feed | Torrent Feed

Become a supporter on Patreon:

Foo

— Show Notes: —

Data thieves target parking lots

  • “Late last year, KrebsOnSecurity wrote that two huge swaths of credit card numbers put up for sale in the cybercrime underground had likely been stolen from Park ‘N Fly and from OneStopParking.com, competing airport parking services that lets customers reserve spots in advance of travel via Internet reservation systems. This week, both companies confirmed that they had indeed suffered a breach.”
  • “When contacted by Krebs on Dec. 15, Atlanta-based Park ‘N Fly said while it had recently engaged multiple security firms to investigate breach claims, it had not found any proof of an intrusion. In a statement released Tuesday, however, the company acknowledged that its site was hacked and leaking credit card data, but stopped short of saying how long the breach persisted or how many customers may have been affected”
  • “OneStopParking.com reached via phone this morning, the site’s manager Amer Ghanem said the company recently determined that hackers had broken in to its systems via a vulnerability in Joomla for which patches were made available in Sept. 2014. Unfortunately for OneStopParking.com and its customers, the company put off applying that Joomla update because it broke portions of the site.”
  • “Unlike card data stolen from main street retailers — which can be encoded onto new plastic and used to buy stolen goods in physical retail stores — cards stolen from online transactions can only be used by thieves for fraudulent online purchases. However, most online carding shops that sell stolen card data in underground stores market both types of cards, known in thief-speak as “dumps” and “CVVs,” respectively.”
  • “Interestingly, the disclosure timeline for both of these companies would have been consistent with a new data breach notification law that President Obama called for earlier this week. That proposal would require companies to notify consumers about a breach within 30 days of discovering their information has been hacked.”
  • Krebs also appears to be having fun with the LizzardSquad

Microsoft pushes emergency fixes, blames Google

  • Microsoft and Adobe both released critical patches this week
  • “Leading the batch of Microsoft patches for 2015 is a drama-laden update to fix a vulnerability in Windows 8.1 that Google researchers disclosed just two days ago. Google has a relatively new policy of publicly disclosing flaws 90 days after they are reported to the responsible software vendor — whether or not that vendor has fixed the bug yet. That 90-day period elapsed over the weekend, causing Google to spill the beans and potentially help attackers develop an exploit in advance of Patch Tuesday.”
  • Yahoo recently announced a similar new policy, to disclose all bugs after 90 days
  • This is the result of too many vendors take far too long to resolve bugs after they are notified
  • Researchers have found that need to straddle the line between responsible disclosure, and full disclosure, as it is irresponsible to not notify the public when it doesn’t appear as if the vendor is taking the vulnerability seriously.
  • Microsoft also patched a critical telnet vulnerability
  • “For its part, Microsoft issued a strongly-worded blog post chiding Google for what it called a “gotcha” policy that leaves Microsoft users in the lurch”
  • There is also a new Adobe flash to address multiple issues
  • Krebs notes: “Windows users who browse the Web with anything other than Internet Explorer may need to apply this patch twice, once with IE and again using the alternative browser (Firefox, Opera, e.g.).” because of the way Microsoft bundles flash
  • Infact, if you use Chrome and Firefox on windows, you’ll need to make sure all 3 have properly updated.

What makes a good security question?

  • Safe: cannot be guessed or researched
  • Stable: does not change over time
  • Memorable: you can remember it
  • Simple: is precise, simple, consistent
  • Many: has many possible answers
  • It is important that the answer not be something that could easily be learned by friending you on facebook or twitter
  • Some examples:
  • What is the name of the first beach you visited?
  • What is the last name of the teacher who gave you your first failing grade?
  • What is the first name of the person you first kissed?
  • What was the name of your first stuffed animal or doll or action figure?
  • Too many of the more popular questions are too easy to research now
  • Some examples of ones that might not be so good:
    • In what town was your first job? (Resume, LinkedIn, Facebook)
    • What school did you attend for sixth grade?
    • What is your oldest sibling’s birthday month and year? (e.g., January 1900) (Now it isn’t your facebook, but theirs that might be the leak, you can’t control what information other people expose)
  • Sample question scoring

Feedback:


Round Up:


The post Patch and Notify | TechSNAP 197 first appeared on Jupiter Broadcasting.

]]>
Feedback Blowout #1 | TechSNAP 60 https://original.jupiterbroadcasting.net/20097/feedback-blowout-1-techsnap-60/ Thu, 31 May 2012 17:52:35 +0000 https://original.jupiterbroadcasting.net/?p=20097 Software that’s supposed to get you around censorship, could be logging your activities online, plus we’ve got a classic Social Engineering story for you.

The post Feedback Blowout #1 | TechSNAP 60 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

Software that’s supposed to get you around censorship, could be logging your activities online, plus we’ve got a classic Social Engineering story for you.

And then we clear the decks and answer a ton of your questions, in our feedback blowout!

All that and so much more, in this week’s episode of, TechSNAP.

Thanks to:

Use our codes TechSNAP10 to save 10% at checkout, or TechSNAP20 to save 20% on hosting!

Limited time offer:

$1.99/mo economy hosting for 3 months – special offer!
Code:  199tech
Expires:  June 30, 2012

Direct Download:

HD Video | Mobile Video | MP3 Audio | Ogg Audio | YouTube | Torrent File

RSS Feeds:

HD Video Feed | Mobile Video Feed | MP3 Audio Feed | Ogg Audio Feed | iTunes Feeds | Torrent Feed

 

Support the Show:

Show Notes:

Anti censorship application circulating with backdoor keylogger

  • The anti-censorship application Simurgh, used heavily in Iran and Syria to get around government internet censorship, has been spotted on P2P networks and download sites
  • The official version of the application from the official site is legitimate, however the version being propagated via P2P networks has been modified to log keystrokes and send the data back to a server in the USA on an IP block registered in Saudi Arabia
  • The infected version injects javascript into pages, and removes the windows navigation sounds to prevent the user noticing the automated activity
  • Anyone who has run a compromised version should consider all of their online accounts (email, IM, social networks, banking) compromised

WHMCS databases compromised via Social Engineering

  • WHMCS (Web Hosting Management Complete Solution) is a commonly used billing, help desk and client management system for web hosting companies
  • The attackers called the hosting company where WHMCS has their servers, managed to successfully answer the security questions and have the administrative passwords etc send to them
  • The attackers made off with 1.7GB of data including the usernames, email addresses, hashed passwords, and encrypted credit card details
  • The hashed passwords as not immediately vulnerable, however they can still be brute forced with time (especially if they are plain MD5 rather than salted MD5)
  • It is highly recommended that you change all of your passwords if you were a WHMCS customer
  • The attackers claim they targetted WHMCS because they refused to stop doing business with cyber criminals, specifically, script kiddies selling exploits, malware and running scams while using WHMCS to process the payments
  • Additional Coverage
  • Official Response
  • It seems the group that comprised the data, has since analyzed the source code for WHMCS and found a number of vulnerabilities
  • PHP Register Globals
  • SQL Injection

Cambridge Researchers find backdoor in US Military chips


Feedback:

KatsumeBlisk wrote:

The Blizzard thing is why I use their 2-factor authentication. There’s no reason not to when there’s an app for the major mobile OSes and the $6.50 physical one.

Wayne Merricks asks: How can I replace DFS

Justin Bates asks: Backing up Between two Windows Hosts

Chris Urie asks: How to Setup SSH Keys

Jono asks: Safely Storing Local Passwords

A few of you asked: WHY U NO MIRO?


Round-Up:

The post Feedback Blowout #1 | TechSNAP 60 first appeared on Jupiter Broadcasting.

]]> Server Room Fire | TechSNAP 44 https://original.jupiterbroadcasting.net/16812/server-room-fire-techsnap-44/ Thu, 09 Feb 2012 19:08:08 +0000 https://original.jupiterbroadcasting.net/?p=16812 It’s a worst case scenario, when a server room catches fire in this week’s war story! Plus: The secrets to reliable SQL replication.

The post Server Room Fire | TechSNAP 44 first appeared on Jupiter Broadcasting.

]]>

post thumbnail

It’s a worst case scenario, when a server room catches fire in this week’s war story!

Plus: We’ll share a story that might make you re-think taking advantage of your hard drive warranty, the secrets to reliable SQL replication.

All that and more, in this episode of TechSNAP!

Thanks to:

GoDaddy.com Use our codes TechSNAP10 to save 10% at checkout, or TechSNAP20 to save 20% on hosting!

Super special savings for TechSNAP viewers only. Get a .co domain for only $7.99 (regular $29.99, previously $17.99). Use the GoDaddy Promo Code cofeb8 before February 29, 2012 to secure your own .co domain name for the same price as a .com.

Pick your code and save:
cofeb8: .co domain for $7.99
techsnap7: $7.99 .com
techsnap10: 10% off
techsnap20: 20% off 1, 2, 3 year hosting plans
techsnap40: $10 off $40
techsnap25: 25% off new Virtual DataCenter plans
Deluxe Hosting for the Price of Economy (12+ mo plans)
Code:  hostfeb8
Dates: Feb 1-29

   

Direct Download Links:

HD Video | Large Video | Mobile Video | MP3 Audio | OGG Audio | YouTube

 

Subscribe via RSS and iTunes:

   

Show Notes:

Crypto crack makes satellite phones vulnerable to eavesdropping

  • Researchers at the Ruhr University Bochum in Germany have reverse engineered the GMR–1 and GMR–2 encryption systems used by satellite phones and found serious weaknesses
  • Both algorithms rely on security by obscurity, but by downloading and disassembling the firmware, researchers were able to isolate the cryptographic algorithms
  • “Unlike standard algorithms such as AES and Blowfish—which have been subjected to decades of scrutiny from some of the world’s foremost cryptographers—these secret encryption schemes often rely more on obscurity than mathematical soundness and peer review to rebuff attacks”
  • The GMR–1 encryption system uses an algorithm that closely resembles the proprietary A5/2 encryption system that former protected GSM phone networks, before it was phased out in 2006 due to weaknesses that allowed attackers to decrypt calls in real time
  • The attack against GMR–1 allows anyone with a modest PC and some open source software to decrypt a call in less than an hour. With a cluster of more powerful machines, it is possible to decrypt a call in real time
  • GMR–2 phones are also vulnerable to cracking when there is known plaintext. This is a particularly glaring issue because the datagrams contains predictable headers and other content that can be known by the attacker, making such attacks possible
  • Researchers have not yet reverse engineered the audio codec that is used for voice calls, so a call can be decrypted, but not played back (yet). However other data types that do not use the audio codec (fax, SMS, data), have successfully been intercepted
  • Researchers are only able to intercept communications between the satellite and the user, not communications in the other direction, so only one side of the call can be eavesdropped. This is likely a limitation of the way satellite signals work, to intercept the signal from the phone to the satellite, you would need line of sight, usually requiring an EL-INT aircraft or satellite.

Customer buys refurbished drive from newegg, finds existing partitions and data

  • This story raises a number of questions about used and refurbished drives
  • Everyone knows that they should securely erase their drive before they resell it, we covered some of the techniques on TechSNAP 31 – How Malware Makes Money
  • However, how do you securely erase a drive when it has failed in some way?
  • You send the drive back to the store or the manufacturer in order to receive a replacement drive, however, you must trust to them to securely erase your data, since the drive was not usable when it left you
  • In this case it would seem that the drives we repaired, turned around and sold to another customer, without the data being properly erased
  • It would seem the only option that customers have is to not return the failed drive, which means not taking advantage of their warranty and having to pay full price for the replacement drive

Feedback:

Q: chocamo from the chatroom asks about MySQL Replication

A: MySQL has a few different replication modes built in, the main one being asynchronous replication, where a slave server constantly reads from the binary log of all changes made to the database. So you start with your two servers in a converged state (meaning they have exactly the same data), then then each time an UPDATE or INSERT command is run on the master, the slave runs the same commands, in the same order, and should continue to have the same data.

However, the slave is read only. If you want to do load balancing of more than just reads, you need to do what is called ‘multi-master replication’, In this setup, you have 2 or more servers that are all masters, and each is also the slave of the server in front of it. Something like: A -> B -> C -> A. So when an INSERT is done on server B, server C then executes that same INSERT statement, and then A, and when the query gets back to B, B notices that the query originated at B, and so skips it, preventing a loop. If you attempt an approach such as this, you will also need to adjust the auto_increment settings in MySQL, you will want the auto_increment_increment to be at least as many servers as you have, and then each server should have a different auto_increment_offset. This is to prevent primary key collisions, so that if an INSERT is done on each of the three servers at the same time, each row ends up with a unique key, otherwise replications stops until you solve the primary key collision. In the ScaleEngine setup, we also have 2 real-only slaves, one from server A and one from server C, the first offers read-only access to customers, to be used by applications that support using a read-only slave, and the other is used for taking backups (we pause replication to get a perfectly consistent copy of the entire database, then resume replication to catch back up to real-time data)

MySQL 5.5 also introduces ‘semi-synchronous replication’. In this mode, the MySQL client does not return from the query until the data has been written to not only the master, but at least X of the N slaves. This allows you to ensure that the data has actually been replicated and is safe from the failure of the master server. Normal replication in MySQL is asynchronous, meaning that when you make a change, the client returns a successful result as soon as the data has been written to the server you are connected to, and then replications happens later, this is normally the desired behaviour because it provides the greatest speed, however if the server you wrote to fails before any other servers replicate the change, that change could be lost. Semi-Synchronous replication attempts to solve this issue by allowing you to wait until there is at least 1 or more additional replicas of the data before returning a successful write. Fully synchronous replication is normally undesirable due to the performance impact.

If a table is too large, you can use ‘partitioning’ to break it in to smaller tables. You can also use the MySQL ‘Federation’ feature, to make databases from more than one server appear to be local to a single server, allowing you to move different databases to different physical machines.

War Story:

This week’s features another war story from our good friend Irish_Darkshadow (the other other Alan)


Setting:
IBM has essentially two “faces”, one is the commercial side that deals with all of the clients and the other is a completely internal organisation called the IGA (IBM Global Account) that provides IT infrastructure and support to all parts of IBM engaged with commercial business.

The IBM email system uses Lotus Domino as the server component and Lotus Notes as the client side application. The Domino servers handle the email for the company but also serve as database hosts and applications hosts. At the point in time when this war story took place, each country had their own server farm for these email / database / application servers. Each individual EMEA (Europe / Middle East / Africa) country then routed email from their in-country servers to the two “hubs”, those being Portsmouth (North Harbour) in the UK and Ehningen in Germany.

The events described below took place in the summer of 2004.

War Story:

Well, there I was once more with the 24×7 on-call phone and bouncing through my weekend without a care in the world. Well, sort of I suppose, if you don’t count a German girlfriend with shopping addiction and two kids with the inability to be quiet and give daddy some quality time with his computers. It was a Sunday afternoon and we were at the cinema which I figured was a safer option than what I chose to do for my last was story (getting very drunk).

The on-call phone started to ring almost immediately after we got out of the movie and it was the duty manager telling me that she had been “summoned” to the office to some of the higher ups for the EMEA geography. My first instinct was “and this is my problem, why?” but I resisted the urge to expose my inner bastard and played nice instead. I suspected that she had simply guessed that being called in to the office without any details was likely not a good sign and it might be useful to have some insurance (or a scape goat) beside her for the upcoming call. Apparently as I was the Crit Sit Manager for that week, I was the aforementioned insurance.

Being the devious little git that I am, I decided to bring the kids with me to the office. That would then allow me to counter any requirements on my time there with a need to get the kids home to feed them / wash them / imprison them…whatever fitted best. Essentially they would be my passport to get out of the office and buy myself some time if I needed it.

The Duty Manager that day was one of those people who had graduated to the position despite having absolutely no technical skill or capability but had an uncanny knack of lunching with the right people and “networking” with the right higher ups. Upon arrival in the office I sat in her office with her to chat about any details she had left out during her call to me. I had the kids running up and down the aisles of the call centre with one of the agents I trusted keeping an eye on them.

Nothing new was divulged prior to the big conference call kicking off and even when they started to expain the purpose of the call, details were being kept very very vague. The driver on the call was a guy from Italian Service Management which completely threw me as I had never seen a high level call originate from that part of the organisation.

The key part of the call went something like this :

Italian Guy: We are, eh, here today to eh, discuss a situation in the Vimercate (vim err kaa tay) site. Eh, perhaps we should proceed on that basis.

Duty Manager: Hello there, xxx here. I’m the duty manager for the EMEA CSC this weekend. I’m not sure what the Vimercate site is. Could you please explain ?

Me : *presses mute on the phone
Vimercate is the server farm location for Italy, all of the email and Lotus Notes database / applications for the country are run from there. If that site is down then IBM Italy will be unable to do ANY business for the duration of the outage.
*
unmutes the phone

Italian Guy: It is one of our locations here in Italy that is responsible for some servers.

Duty Manager: Ah ok, thanks for the explanation.

Italian Guy: Well about two hours ago eh….we a, received a call from the cleaning contractors that there was a, some cigarette coming out of the server room. We immediately alerted the rest of Service Management and started dealing with the crisis as a critical situation.

Me: ** rolls about laughing then thinks to telnet to some email servers in that site and nothing was connecting…….the urgency of the call started to dawn on me at this point.

Duty Manager: I’m sorry but I don’t understand what you mean when you say that there was a cigarette coming out of the server room. Did I mishear you?

Italian Guy: Sorry, not cigarette, I mean to say smoke. There was smoke coming out of the server room.

Duty Manager: Oh lord, has anyone been hurt? Is there any emergency service personnel at the site?

Italian Guy: Yes, the fire service were alerted almost immediately and nobody other than the cleaning staff was in the site when the alarm was raised. The fire has spread to other parts of the building and the firemen have been unable to get to the server room yet.

Me: Hi, I’m the crit sit manager here today. Could you please give me a current status on the server room itself? If those servers are not recoverable then we will need to activate the business continuity location and get the backup tapes couriered there. We could be up and running within 12 hours that way.

Italian Guy: Yes, yes, we know all of that. We are service management. We have already started to deal with those things. We invited you onto this call so that you are aware of the issue and can place voice messages on your incoming call lines and have your agents prepared to explain things to our users if they call your help desk. Nothing more.

Me: I have no doubt that you are on top of the situation but in such circumstances the in-country Service Management report in to the EMEA Critical Situations team who then coordinate all actions until there is a satisfactory resolution as per the EOP (Enterprise Operating Procedures). I will be taking point on this for you and liaising with EMEA Service Management for the duration of this situation.

**lots of back and forward, territorial pissing contest arguing took place until it was agree to have a followup call every hour. The second call went something like this :

Me: Good evening folks, how are things progressing on the site now?

Italian Guy: The emergency services are having difficulty due to the age of the building and they have not been able to get to the server room yet. There is nothing else new to say.

Duty Manager: So does that mean the servers are destroyed now or is there still some chance?

Italian Guy: The fire suppression system in the server room activated, that is all that we know right now.

** we adjourned the call and the next two were more of the same until the fifth call :

Italian Guy: The firemen have made it to the server room and have reported that the fire suppression system has not worked correctly. The servers themselves have been fire damaged.

Duty Manager: That’s very unfortunate, how are your efforts to get the backup tapes to the secondary site going?

Italian Guy: Eh, there is a problem with that too. The tape libraries are in the same room as the servers in an enclosure. The firemen have not retrieved them for us yet.

Me: Whoa, hold on a minute. The tapes that we’ve been trying to get into play for the last four hours are actually in the same room with the fire? Why didn’t you tell us that earlier ? If both the servers AND the backup tapes are destroyed then IBM Italy will be offline for days while a secondary site is configured. This completely changes the severity of this situation.

Italian Guy: yes, we believe that both the servers and the tapes have been damaged at this time.

**at this point I resisted the urge to reach my arm through the phone line and throttle this guy.

Duty Manager: So what can we do at this point?

Me: We need to get EMEA Service Management to start prepping a completely fresh site to take over for the ruined server farm. The problem is now that we’ve lost four hours waiting for tapes that were never going to arrive, we could have had the new servers being readied all that time.

So this all continued for a few more calls, I had my girlfriend pick up the kids between the calls and take them home and I just dived in and tried to maintain some momentum in the resolution efforts. Rather than drag it out and bore you to tears, here were the remaining revelations :
Servers were burnt to a crisp.
Backup tapes (which were in the same room) were partially burned but all were smoke damaged.
The fire suppression system simply failed to work
The firemen had to use water due to the composition of the building…WATER…on a room full of electronics.
It took 2 full days to build the new server environment which essentially meant that IBM Italy were unable to do business electronically for that duration.
Nobody ever explained why the tapes were in the server room other than to say – it was an oversight by the IT Manager. Really? an oversight?!?!!
The only bright spot in the entire debacle was that some of the data on the tapes was salvaged and shortened the duration of the outage significantly for some people.

I’m not sure there is a moral to the story or a catchy tag line like “patch your shit” but I suppose that my overriding memory of the whole situation was when I wondered how anyone thought it would be a good idea to put backup tapes in the same physical location as the servers and then neglected to do regular maintenance on an old building that was clearly a fire trap.


Round-Up:

The post Server Room Fire | TechSNAP 44 first appeared on Jupiter Broadcasting.

]]>