Healthcare.gov 2013: the launch that didn't scale

01 · CONTEXT

One site, three systems and one front door

HealthCare.gov is the US federal website for buying private health plans, for consumers in states that did not create their own marketplaces. At launch, on October 1, 2013, and for some time afterward, users ran into outages and technical failures. The source of this case is the study of the project by the Office of Inspector General of the US Department of Health and Human Services.

Three pieces matter here. The EIDM let consumers create accounts and verify their identity. The FFM was the core of the system and determined eligibility for plans and for financial assistance. The Hub routed requests for information to other federal agencies and back, such as the IRS and the SSA (Social Security Administration).

1. The consumer enters through the EIDM, the system that creates accounts and verifies identities. It is the site's front door.

2. From there the consumer reaches the site's functions and information, which live in the FFM, the core of the system.

3. The FFM determines eligibility for plans and for financial assistance and queries its own database.

4. When it needs federal data, the request goes through the Hub, which routes it to other agencies, such as the IRS and SSA, and returns the response.

02 · LAUNCH DAY

More people than expected and almost no one enrolled

On October 1, 2013 the site had 250,000 concurrent users, far above the planned capacidade and the outages began within 2 hours of launch, keeping many consumers from logging in and signing up.

The US CTO attributed the problems to volume, five times the expected number of concurrent users. But the investigation itself reports that it soon became clear the problems did not come only from the number of visitors but also from core performance problems. On the first day of the first open enrollment, only six consumers managed to submit their enrollment and choose a plan.

03 · WHAT SET UP THE OUTAGE

A test that did not happen, capacity discovered late and a narrow front door

End-to-end testing is the final, critical stage that simulates consumer use across all functions. CMS never managed to complete it, because the system's pieces arrived too late. The technical team considered launching on schedule a higher priority than testing and fixing performance problems, and one official described the launch itself as a test of the system, with the problems to be fixed afterward.

On September 26, 2013, CMS technicians ran a limited performance test and saw that the site supported far fewer concurrent users than planned. Part of the reason was that the code required more infrastructure than CMS had anticipated. Shortly before launch, CMS asked that the capacidade be doubled in 3 days. Terremark added more than double and, by September 29, the concurrent-user limit had risen considerably.

1. The EIDM's lack of capacity created a bottleneck for consumers trying to reach the site's functions and information.

2. The EIDM contractor reported that it did not know all visitors would pass through it and therefore underestimated the capacity needed.

3. In the FFM, the site's software requested information from the database more than 100 times in an operation that should have required 1 or 2 requests.

4. A CMS technician said the FFM was the bigger problem, but that the scale of the code problems only became visible after the EIDM one was overcome.

04 · NOT SEEING THE SYSTEM

Those who operate it do not see the whole

According to CMS technicians, the monitoramento coordinated effort was not a priority before launch because resources went into building the site. Before launch, each contractor monitored its own systems and could not see the others, even though they worked together.

After launch there were separate command centers, with tools to track outages, response time and errors, but no team with a view of the whole system and the power to act. A member of the technical team who helped with the repair said there was nowhere to see whether the site was up, except CNN.

05 · RECOVERY AND LESSONS

Dashboards, waiting rooms and redundant entry points

Between October 1 and December 1, 2013, CMS bought monitoramento tools with site performance dashboards. It also decided to refactor the existing code instead of rebuilding from scratch and hired QSSI as the systems integrator, to advise on coordinating tasks and technical resources.

1. Monitoring tools began giving the team site performance dashboards.

2. With capacity overloaded, CMS could put consumers in a waiting room. There they could ask for an email to return when capacity was available or call the call center.

3. For redundancy, CMS hired developers to start building a login exclusive to HealthCare.gov, separate from the login used across CMS.

4. A new plan-shopping tool did not require the consumer to create an account.

5. The share of time the site was up went from 42 percent in early November to more than 90 percent in late November, and it began supporting more than 35,000 concurrent users without going down.

CMS restored the site for heavy use within two months. The new contingency policy sought to create redundantes systems to keep the site up if the main ones failed. The table summarizes the change on the points this case tracks.