Facing frequent outages in the DSL (digital subscriber line) network serving 50-plus corporate-owned Gold's Gyms, Bobby Badugu knew it was time for a significant upgrade. After looking into the various options – including satellite, frame relay and a different DSL provider – he opted for a carrier-provided voice VoIP (voice over IP) service. That's when the trouble began.
The problems of the 'big bang' approach
In the months that followed, the vice-president of IT for Gold's Gym, based in Dallas, learned some valuable lessons about not only VoIP, but any major network project that involves upgrades to numerous, far-flung sites. Badugu shared his insights in a presentation at the recent Network World IT Roadmap Conference in Dallas. He said the lessons include using a phased implementation process; establishing clear SLAs (service-level agreements), including penalties; and conducting a thorough technology assessment to identify potential problems.
When the rollout began in June, it was scheduled to take eight weeks. As of mid-November, the data portion of the rollout is complete, but only about 20 percent of the voice lines have been ported to the VoIP network. Badugu remains bullish, expects benefits including 35 to 40 percent savings in voice and data service costs, and already is saving $6,000 to $10,000 (£3,200 to £5,300) on his conference calling. But if he had it to do again, he'd do more than a few things differently.
Gold's Gym started in 1965 with a single facility in Venice, California. Through a series of licensing deals, mergers and acquisitions, the company grew to 660 gyms in 26 countries. Of the 550 US Gold's Gyms, most are franchises; about 50 are corporate-owned. Badugu is responsible for providing network services to those corporate-owned gyms, which are in California, Hawaii, Maryland, Missouri, Texas and the Washington DC area.
Network services are particularly crucial to the gyms, because they rely on a central server for everything from member data to the retail point-of-sale system.
Badugu targeted satellite when he embarked on the upgrade project in February 2006, but quickly learned that it wouldn't provide enough bandwidth. He also wasn't impressed by the available frame relay offerings, which he considered older and less flexible technology, and didn't feel comfortable with a DSL service from a different provider.
Then Badugu recalled a Qwest MPLS project from his consulting days, so he called in Qwest to inquire about VPN (virtual private network) service.
"That's where we started," he said. "From there the proposal evolved to address a lot of our phone needs. That's when the VoIP solution was proposed."
Once voice was on the table, Badugu brought AT&T and Sprint into the discussion to see what they had to offer. But their proposals were too voice-centric and didn't have a strong strategy for combining voice and data, so Qwest won the day.
The big driver for upgrading the voice network was reducing costs for local, long-distance and conference calling. "As we add gyms and continue to grow our business, those costs became astronomical," Badugu said.
At the same time, with its lean IT staff, the company struggled to manage the lines and make sure each was properly used. "We didn't have the staff to monitor it every month and make sure we're doing the right thing for the company," he said. He also didn't have the staff or the desire to build his own VoIP network.
The plan was to install a T-1 line to each gym and implement voice and data service when the circuits were installed. Badugu figured scheduling the T-1s from various LECs (local exchange carriers) would be the most complicated issue.
"That was pretty simple. They gave us a date and when it occurred, they were out there and put the T-1 in place," he said. From there, Qwest subcontractors complicated matters.
At each gym, one contractor would order, configure and install a router. Another configured and installed the firewall and hooked up the VPN tunnel. "As we did each location, we found more and more complexities in that process," Badugu said. "We had six or seven parties involved in putting the phone and data lines in. I really felt like some were pretty much learning the technology or learning the equipment."
Badugu also admitted to shortcomings on the company's part. "One of the things we didn't do correctly was we didn't know our environment," he said. That included the types of phone systems installed in each location and whether they could connect to the T-1 line. Even though he had been installing digital, IP-ready phone systems in all gyms, some didn't have the required T-1 card, which meant an additional expense, and others were still analog. Locating all the wiring closets was another challenge.
For these reasons, Badugu decided to focus on bringing up the data side first, then add voice later. As a result, the original eight-week implementation window passed and the company didn't have all its T-1s installed.
The decision to delay the voice implementation proved prescient, as that side of the equation presented numerous challenges. Chief among them was porting existing phone numbers to the Qwest VoIP service.
Gold's Gym gave Qwest all its working telephone numbers and billing numbers to begin the porting process from various ILECs (incumbent LECs). "Of the first five numbers we submitted, four failed," he said, because the billing telephone numbers that Gold's Gym had were different from those the ILEC had.
"The whole porting process became a real nightmare," Badugu said. In some instances, he tried to forward calls from the old phone system to the new one. But that often was confusing for callers who heard the phone ring once, then heard a short delay before it rang again. If two calls came in at the same time, the second would reach a busy signal.
Now he is porting the main number for each gym, then assigning new numbers for each extension, which typically aren't called directly. Badugu has a schedule from his ILECs for when each number is supposed to be ported, then he co-ordinates with phone technicians to ensure the new system is up and running properly. He is hoping to have all the numbers ported by the end of the year.
Despite the challenging implementation, Badugu began realising a savings of 20 percent on long-distance once he got the T-1s installed and switched all long-distance service to Qwest. Additionally, conference calls now are carried over the Qwest network, obviating the need for a third-party provider and saving at least $6,000 per month. As each gym is brought onto the VoIP network, he gets closer to his projected savings of 25 to 30 percent on local calling costs.
If he had to do it again, however, he would take a more phased approach. "We tried the big-bang theory, doing everything at once," he said. That decision was driven by business needs, because the DSL network was so unreliable, and because Qwest was urging Gold's to sign a contract that included every gym. "Even still, I think we should've done two or three gyms first, and made sure that was okay before we did the rest."
He also advised mapping out details about the steps required to install any new technology and who is responsible for each step, along with contacts and escalation procedures, should things go awry. "So if something fails, you have somewhere to go to keep the process moving," Badugu says.
Another must is technology assessment to identify items such as phone systems that were analogue or lacked T-1 cards.
"A site survey would've been good, with a detailed analysis of the phone systems, the phone numbers, how the hunt groups work, how the phone systems are configured and the number of workstations," Badugu said. "For each location, if we had that up front, it would've made life easier for everybody involved."
For more information, our sister site Techworld has a comprehensive VoIP resource page.