It’s been a rough year in many regions given the wave of natural disasters. For businesses, this is an opportunity for a conversation about business continuity and what it takes to keep your business communications powered. Today, we kick off a series of discussions on business continuity from guest blogger and Digium partner, Scott Pell, of API Digital. Scott provides a “behind-the-scenes” look at how API and Digium responded to a critical weather event that seriously threatened both companies’ business operations.
Scott Pell, API Digital
When we leave the house to go to work, it never runs through our minds that it might very well be the last day we see our families. On April 27, 2011, the family, friends and neighbors of over 200 people in Alabama were faced with just such a reality. The storms of that day, literally left trenches of wrecked infrastructure that will require years to completely rebuild. For the first time in 50 years, the entire county was without power and with very limited communications, including the PSTN, which eventually went down for nearly 24 hours. Even backup communications through the cellular network was reduced to a nearly unusable level.
During it all, Digium and API Digital were up. Generators for both companies roared to life within seconds of the outage and with the help of dedicated staff most of our customers were completely unaware of what had just taken place in Alabama.
Lesson 1: make sure all your eggs aren’t in one basket. Early in on our long-term relationship, we worked with Digium to ensure redundancy. And, that’s why API Digital provides the Internet over one fiber connection and a separate phone company takes care of the voice over another fiber connection.
Friday morning, I got a call from Steve Burcham, VP of Operations for Digium. Apparently, the voice provider started losing power on their SONET ring, taking down Digium’s PRI. To solve the problem, we quickly contacted Digium’s phone company and had them add our PRIs (on dedicated fiber with the CLEC) as Digium’s secondary trunk group. At first, the CLEC misconfigured the workaround and Digium support calls started showing up in our support queue. Both Digium callers and our engineers were confused, but our quick thinking folks just started taking messages.
Once the CLEC fixed their mistake, we were able to route Digium’s inbound calls to our Asterisk cluster, then send the calls over our IPconnection directly to Digium’s Switchvox. After inbound was fixed, we set ourselves up as an outbound route for the Switchvox. Digium’s PBX was back in business. The best part is that this solution is permanent. If the PRI ever goes down again or fills up, for that matter, calls will automatically get routed through us, then we will send them right over to Digium’s Switchvox.
API Digital has actually been using this technique in Huntsville for a few years. Several large companies use us specifically for voice backup and we also provide this service to all of our Switchvox customers.
Lesson 2: know the rules and use them to your advantage. Case and point: The National ISDN standard for PRI provides for primary and secondary trunks. As long as the customer allows, the secondary trunk can be another company, like us. We can even perform this service for customers that do not have a VoIP-enabled PBX.
Lesson 3: Listen, learn and pass it on. In this lesson, I also pass along my thanks and a personal side note: Hey Mark, I was paying attention when you taught me voice routing. I’ve passed that knowledge on to people who are smarter than me, so that we can continue performing this kind of voice miracle or two. This year at Astricon, maybe we can pull another rabbit out of our hat with a multi-site – Denver, Chicago and Atlanta – SCF demonstration.
Lesson 4: Be prepared for the unexpected. We all learned an important lesson that day: Just because you have a generator doesn’t mean everyone else does, especially your phone company. Many CLEC’s co-locate SONET gear in customer buildings to build their ring topology. It is a great way to build a network, but it only works if everyone is on generator.
AT&T actually had the same problem at all of their remotes. By Saturday morning, their batteries ran out, killing the PSTN and T1 network. AT&T, however, has a special plug on the side of each remote that they can plug in a generator. By Saturday night, all AT&T remotes were powered by a fleet of Katrina-proven mini generators. I still had to explain to our younger employees that a standard, corded phone will work when the power is out.
Stay tuned for the next business continuity discussion…