et lux in tenebris lucet

Not content to let the Fisher Plaza fire outdo it, the 151 Front Street carrier hotel in Toronto is on fire. As of this writing, whatever happened is apparently with 17 units and being called a 2-alarm fire:

(http://www.toronto.ca/fire/cadinfo/livecad.htm)

Rumor was that Peer1′s suite is on fire, and this has been confirmed as “an issue” with “fire” on their status board. Reports currently range from just their suite to the whole building without power. I’m sure there will be more on this in the morning.

UPDATE: Peer1 originally said the whole building lost power:

But they since redacted that and are now saying just the 7th and 8th floors:

UPDATE 2: A post to the NANOG mailing list confirms that indeed the initial report by Peer1 was incorrect and they still had power (or power from the generator) to their 7th floor suite.

MORNING UPDATE: The problem was completely localized to Peer1 due to one of their UPS systems (most likely a battery) catching fire. Remember kids: keep your UPS systems in a separate room, inspect your batteries often, and install your distribution panels with an external bypass.

There was an electrical fire this morning at Fisher Plaza in Seattle (“the only mission-critical business community in the Northwest” according to their Google search summary as fisherplaza.com is down) resulting in a full outage. The news reports a vault fire, while another source on NANOG says generator transfer switch fire. Could be both as they’re without power at this point. Anyway, one of the affected parties was the well-known credit card processor authorize.net which apparently had a full halt on transactions when the Fisher Plaza equipment went offline.

Here’s what auhtorize.net had to say about the outage on their twitter account:

Uh, no you certainly don’t have redundancy. If it’s not working it’s not “fully redundant”. Period. Yes, you have an alternate site you’re trying to bring online, but it’s obviously not redundant.

UPDATE: A more detailed list of companies affected by the Fisher Plaza outage.

Congressman has bill ready to give FTC veto on ISP data caps

FTA: In a press conference today, Massa reiterated both of these, saying that “volume-based pricing is detrimental to our economy,” and highlighting how doctors who tried to work from home could have been hit hard by the usage fees.

So, internet access should be completely free? I’m cool with not charging my customers anything as long as my upstream transit, electricity, and office space become free too. Just make sure I never have to pay a bill to anyone and we’re good to go with your cunning plan Mr. Massa.

FTC Sues, Shuts Down N. Calif. Web Hosting Firm

And it looks like there’s a noticeable effect, although not as immediate as the McColo shutdown. I never see the mail pool load balancer CPU go this low during the day, especially on a weekday. Bandwidth did not change nearly as noticeably, but the load balancer CPU illustrates the packets-per-second impact.

UPDATE:

The decrease has continued and it feels a bit unsettling; it looks like something is broken, but the board is green, the phones are quiet, and the normal mailing list traffic keeps flowing. I’m sure it will eventually pick back up when the various botnets C&C is restored.

And for fun, here’s a graph of when McColo was shut down on Nov. 11, 2008 (RRD averaged out the fine points):

Working on Linux kernel serial drivers is not something I normally do – or have ever actually done before – but I found myself needing to update a kernel module last month to compile under the 2.6.26 serial driver model. It turned out to be a success, so I’ve decided to share it with anyone else who happens to want to run one of these cards on Linux:

http://www.rollernet.us/opensource/

This patched kernel module driver is being used successfully on a StarTech PCI2S422ISO. This driver should work for any of the PCIPORT family of cards: P584, P588, P514, P518, P220, P232, P422, P984, P924, P985. This card may also be known as a “rayon” card.

I absolutely love this.

UPS Ground packages make it from Las Vegas, NV to Reno, NV by way of Utah.

I wonder if anyone ever drives to Vegas using crazy out of the way routes like this because they stick to interstates. Central Nevada’s roads are actually in damn good condition; typically better than the interstates.

I can across this that’s worth sharing regarding LVM snapshot performance pitfalls. The trick is chunk sizes, although ultimately it’s still copy-on-write.

http://www.nikhef.nl/~dennisvd/lvmcrap.html

I never use LVM snapshots long term anyway; just freeze fs, snapshot, unfreeze fs, do whatever, destroy snapshot. Another trick is to put the snapshot on a physically separate device from the source LV. Although you’ll still see a performance impact, it will be a bit less since you aren’t driving the disk seek rate through the roof.