Rogers Jul 2022: removed route filter takes down a country's network

01 · CONTEXT

One IP core for all services

On July 8, 2022, the Canadian carrier Rogers Communications had a network outage that affected more than 12 million cable internet and cellular network users. This case follows the failure from the technical trigger to recovery, based on the company's letter to the regulator and an independent review commissioned by the regulator itself.

According to the independent review, because the mobile and fixed networks shared a common IP core, the reach of the outage was extreme, with a catastrophic loss of all services. The same review states that this sharing is common among carriers, to balance cost and performance, and that the failure did not stem from an architectural design error.

1. Two sets of services serve customers: mobile (cellular) and fixed internet.

2. Both services pass through the distribution routers and reach the same IP core.

3. The independent review notes that this common core is a frequent choice to balance cost and performance, and that it widened the reach of the outage.

02 · THE TRIGGER

A filter removed in one phase of the upgrade

Rogers was performing an upgrade of its IP core in seven phases, and the filter was removed in the sixth. According to the independent review, employees removed the policy filter (an access control list) from the distribution routers in an effort to clean up files.

According to the company's letter to the regulator, without the filter all possible routes to the internet passed through the routers and exceeded the core's capacity. The independent review describes the result as a core flooded with IP routing information, which eventually crashed.

1. The distribution routers apply the policy filter, the core's only protection against excess routes.

2. During one phase of the upgrade, the filter is removed from the distribution routers.

3. All possible routes pass to the core, which exceeds capacity and crashes.

4. Rogers stops announcing its presence and other networks cannot find it.

5. Mobile services and fixed internet services go down.

03 · THE REACH

Far beyond Rogers' customers

From the outside, Cloudflare saw spikes in BGP updates and withdrawal of IP prefixes. Rogers was not announcing its presence, and so other networks could not find it.

The effect spread to those who depended on Rogers' infrastructure. Interac went down and no business in the country could accept debit cards, even those using another internet provider. The use of 911 from cell phones on the Rogers network was impaired, and some federal government services were affected.

04 · THE RECOVERY

Network management depended on the broken network

The outage also cut internal access to the company's systems, which hampered mobilizing a team and identifying the problem. Some employees managed to connect through emergency SIMs on alternative networks, thanks to reciprocal agreements with other carriers; others traveled to centralized locations to get network access.

The independent review points out that the management network depended on the IP core and that the operations center and other critical remote sites lacked redundant connectivity from alternative providers. At first, the team also had no access to the error logs of the routers that failed and took about 14 hours, from the start of the outage, to identify the root cause.

1. The management network depended on the IP core, which was down.

2. Remote employees cannot access the management network.

3. The operations center had no redundant connectivity from an alternative provider to manage the network.

4. At first the team had no access to the error logs of the routers that failed, and it took about 14 hours to identify the root cause.

5. After the outage, Rogers deployed backup third-party connectivity at the operations center and a separate physical and logical management network.

Outside help was available: the two competitors contacted offered assistance. Rogers, however, concluded it was not possible to transfer its users to other networks, citing the lack of access to the user database and the excessive load this would impose.

05 · WHAT CHANGED

Post-outage measures and design lessons

According to the independent review, Rogers added safeguards to the core routers against renewed overload, deployed a separate management network and backup third-party connectivity. It also changed the risk assessment algorithm and made organizational changes to improve collaboration between operations and engineering. On the regulatory side, the government announced policies requiring mutual assistance among carriers in the event of an outage.