Here’s How the White House Fixed the Obamacare Website

The Affordable Care Act is a piece of legislation meant to be the landmark achievement of Barack Obama’s presidency. But with the software errors and design flaws of the federally created online insurance marketplaces causing hours-long wait times, preventing potential customers from creating accounts and completing the 30-step enrollment process, and sending insurers the wrong information, the administration must first ensure can work as intended before it can ever hope to reverse the tidal wave of criticism the error-riddled, October 1 launch of the insurance exchanges engendered.

Compiled by the Centers for Medicare and Medicaid Services, a progress report of the fixes made in repairing the software errors and design flaws affirmed that the goals set out by the administration were achieved by the deadline of November 30.

That means, the $630 million Web portal that links all 36 federal online insurance marketplaces created by the Affordable Care Act, should be functioning smoothly for most people. From data in the administration’s progress report and other anecdotal evidence, it seems is working much better than it was in October and early November. Although some problems remain, the website can apparently now handle 50,000 concurrent users as promised and 800,000 users in a single day.

As government documents and interviews conducted by The New York Times make evident, the story of how the administration confronted what may have been the most difficult moment of Obama’s presidency pointed to “an insular White House” that initially did not fully comprehend the magnitude of the backlash from the botched debut of the insurance marketplaces. When the extent of the damage was fully appreciated, the administration began scrambling to repair its reputation and the legacy of President Obama.

By October 30, criticism had reached a fever pitch, and the tenuous alliance between Democratic lawmakers and insurance executives was in danger. That day, Obama defended the Affordable Care Act from Boston’s Faneuil Hall, the colonial-era landmark where then-governor Mitt Romney signed Massachusetts’s health care reform into law in 2006. He was also forced to confront another failing that same day.

Contrary to what Obama had promised while campaigning for the Affordable Care Act, insurance companies had started canceling insurance policies, spawning criticism that the president had lied when he promised: “If you like your current health insurance plan you can keep it. Period.” Meanwhile, that mix-up had breathed new life into the Republican Party, which was facing its own share of voter disapproval after the 16-day partial federal government shutdown and budget crisis.

Fixing the botched rollout of the federally facilitated insurance exchanges was of utmost importance. “If we don’t do that,” one senior White House adviser told the Times, “it’s a very serious threat to the success of the legislation and a very serious threat to him. We get that.”

For the administration, fixing the website meant overcoming the dysfunctional relationship between the Department of Health and Human Services and its technology contractors, as well as accepting that officials must take charge of a project that was never fully tested. The site was partly failing because the Centers for Medicare and Medicaid Services had not hired a “systems integrator” to coordinate the construction of the complex parts of the exchange system.

On October 1, it was not immediately obvious how serious the website’s problems were. Three years in the making, the federal exchanges — just like the 14 state-run exchanges — were meant to allow consumers to comparison-shop for health insurance policies in online marketplaces, which were designed to give individual customers collective bargaining power, fostering competition and in theory driving down prices.

Shopping for insurance was meant to be as easy to use, and the administration was confident the marketplaces would be success. On the eve of the rollout, Department of Health and Human Services Secretary Kathleen Sebelius said, “We’re about to make some history,” according to the Times.

When the site went live at midnight, teams of technology experts monitored the activity, and Todd Park, the White House chief technology officer, kept his eye on traffic numbers. By morning, White House Chief of Staff Denis McDonough was relieved to hear that “the traffic is really high,” the newspaper reported. That website traffic was high gave the administration some piece of mind, as enrollment numbers had been a primary concern.

However, technicians at the Herndon, Virginia, offices of CGI Federal — the American subsidiary of the Canadian-based technology firm that built the majority of the federal health care website — knew a problem was developing. It was not high numbers of visitors to website that was causing the glitches to Errors were appearing in many different areas, and the high traffic exacerbated the problems.

The software that assigned particular identities to enrollees and kept their personal information linked to those identities, known internally as EIdM, was overwhelmed by the high traffic numbers, preventing customers from logging in and creating accounts. Park was called in to help, and together, he and the software engineers who built the system attempted to find the source of the glitches. “They kept looking, looking, looking, but there wasn’t anybody moving through the system,” The New York Times learned through a person familiar with the project.

The creation of accounts was under the jurisdiction of Quality Software Services Inc., or QSSI, a subsidiary of UnitedHealth Group (NYSE:UNH) based in Columbia, Maryland. QSSI called upon its subcontractor Oracle (NYSE:ORCL), which sent in a team of software engineers to Washington. Experts could not agree on what went wrong, and it took eight days to resolve the account bottleneck.

While Obama was telling the public that “interest way exceeded expectations,” privately, White House officials were wondering whether the website needed to be taken down in order be fixed, the Times says. But it was eventually decided that the website should remain functioning so that problems could be identified. More importantly, senior White House officials determined that the website was fixable.

Still, potential enrollees could not register to buy insurance — a fact that was alarming insurance executives. While the system was intended to handle 50,000 concurrent users, as few as 500 users could disable it, according to Times sources. “These are not glitches,” one insurance executive said at the time, according to the publication. “The extent of the problems is pretty enormous. At the end of our calls, people say, ‘It’s awful, just awful.’”

By October 9, Jeffrey Zients — a former health care management consultant and Office of Management and Budget official — had been charged with leading the repair efforts, although his appointment would not be publicly announced for several weeks. He would also be named Obama’s top economic adviser, a position he will take up in January. In choosing Zients, the White House turned to an Obama intimate who could offer an independent opinion on the website’s problems. What Zients saw was a technical and personnel mess, according to the Times.

Website crashes continued throughout the month, and more problems materialized. The Web portal’s data center had inadequate capacity and poorly written computer code created by contractors pressured to rapidly adjust to changing government specifications. Many of these problems had not been uncovered because the website had barely been tested before October 1.

A major damage control meeting was held on October 15. The Times reports that the president spent 90 minutes pressing those in attendance for answers and struggling to understand the crisis. “We created this problem we didn’t need to create,” Obama said, according to one adviser interviewed by the newspaper. “And it’s of our own doing, and it’s our most important initiative.”

While federal contractors were pointing fingers at each other and government officials, Zients was deciding that QSSI should take over as “systems integrator” so that the conflicting directions could be resolved. Zients created order, and from the Rose Garden of the White House, Obama announced a tech surge, which was mostly an exercise in public relations.

“Tech surge” suggests an army of software engineers who would fix the website, but in reality, the surge consisted of approximately a half-dozen people who had taken leave from major technology companies in order to help. One such individual told the Times that the use of surge “was probably an overstatement.”

It was late in October when congressional oversight began. On October 30, Sebelius was grilled for three-and-a-half hours by the House Energy and Commerce Committee. She apologized to U.S. consumer for the flawed rollout. During her hearing, the website crashed, displaying a “Please try again later” message, when representatives attempted to log on.

Meanwhile, insurers were still alarmed — after all, billions of dollars hung in the balance. Karen Ignagni, the chief executive of America’s Health Insurance Plans, a insurers’ trade association, gave the administration suggestions for what to do in the event of website malfunctions, but the concerns of the insurance industry fell on deaf ears, according to the Times. Other prominent members of the insurance industry voiced frustration, as well.

There’s so much wrong, you just don’t know what’s broken until you get a lot more of it fixed,” Aetna (NYSE:AET) Chief Executive Mark Bertolini said on CNBC. His criticism was especially tough, given that he supported the health reform from the beginning. Bertolini was convinced that the number of uninsured Americans needed to be reduced, and, unlike rival insurers UnitedHealth and Cigna (NYSE:CI), he prepared for the new law by hiring hundreds of new workers and spending tens of millions of dollars.

A meeting was held in the Roosevelt Room of the White House on October 23 with numerous insurance executives. However, that meeting was largely forgotten once millions of individuals began receiving policy termination notices. The terminations gave Obama another problem to solve.

Not only were insurers alarmed and Congress considering legislation to extend the canceled policies — including the “The Keeping the Affordable Care Act Promise Act” penned by Democratic Sen. Mary L. Landrieu of Louisiana — but the president was “uncomfortable,” one senior adviser told The New York Times. The adviser was upset that so many Americans had received cancellation letters and were angry because “of what the president had said.”

“For the vast majority of people who have health insurance that works, you can keep it,” Obama told a crowd in Boston on October 30. “So if you’re getting one of those letters, just shop around in the new marketplace.” But by November, Democrats were pressuring Obama, as well. Even former President Bill Clinton said the president should allow Americans to keep their insurance.

A week passed, but still no remedy was put forward. Eventually, to amend for the president’s broken promise, the administration announced that individuals could renew those cancelled policies for another year, but tensions between the government and insurers remained high. That announcement came just hours before a vote on a Republican bill that would allow insurers to renew policies and sell similar ones to new customers.

The focus now for Zients and his team, who have assembled in Maryland, is making work smoothly for insurance consumers. That means fixes to the back-end have been put off. The Times reports that Zients tracks problems that show themselves in spikes on one of 16 oversized Samsung televisions screens at the Maryland command center; the screens offer real-time data on any website problems or delays.

The system has paid off, the publication reports. As the December 2 progress report from the Centers for Medicare and Medicaid Services made clear, metrics are improving, error rates are down to less than 1 percent, and page-load times are at an acceptable level of less than one second.

But the true measure of success will not be available until the end of March, when statistics from the six-month enrollment period are available. For now, signups remain slow: Only 106,000 people enrolled via the federally facilitated exchanges during October. Plus, Republicans are likely to continue their opposition of the health care reform as Obama redirects his attention to the issues of immigration and the economy.

Follow Meghan on Twitter @MFoley_WSCS

Don’t Miss: Sebelius: Put Health Care on Your Shopping List.