Connect with us


How a coding error caused Rogers outage that left millions without service – The Globe and Mail



People use the wifi inside Toronto’s Fairview Mall on July 8.Yader Guzman/The Globe and Mail

Rogers Communications Inc. RCI-B-T engineers began the sixth step of a seven-step process to upgrade the core infrastructure that supports the company’s wireless and broadband networks at 2:27 a.m. on July 8.

Two hours and 16 minutes later, a coding error was introduced that triggered a cascade of events, resulting in a massive outage that left millions of Canadians without cellphone, internet or home phone service for at least a day.

The shutdown of one of Canada’s dominant telecommunications networks created widespread chaos. Rogers was unable to deliver four emergency alerts to its wireless customers in Saskatchewan, including three tornado warnings and one dangerous person report.

Rogers customers were unable to call 911, and the Interac debit system was also affected, causing issues for both consumers and businesses. In Toronto, the disruption forced Canadian singer-songwriter the Weeknd to postpone a concert that was supposed to have been held at the Rogers Centre that night.

Initially, even Rogers itself was unsure what was causing the service disruption. But weeks later, in a detailed submission in response to questions from the Canadian Radio-television and Telecommunications Commission, the company gave a full account of its version of events.

Opinion: Rogers still has some explaining to do about its outage and the fallout for its Shaw deal

Opinion: Rogers outage a reminder of Canada’s failure to set up a secure wireless network for emergency services

Those documents, which were disclosed publicly by the CRTC in redacted form on Friday, give new details on the outage and provide an early glimpse at the set of facts Rogers executives will draw upon on Monday, when they are expected to testify about the incident in a public hearing before the House of Commons committee on industry and technology.

Like many of its peers, Rogers currently has one core network that supports all of the services it provides. The core is essentially the network’s brain. It receives, processes, transmits and connects all voice, wireless data, internet and television traffic.

The telecom had started the seven-phase process to upgrade the core back in February, after what the company described in its CRTC submission as a comprehensive planning process that included budget and project approvals, risk assessment and testing.

The first five phases had gone smoothly. But, at 4:43 a.m. on July 8, a piece of code was introduced that deleted a routing filter. In telecom networks, packets of data are guided and directed by devices called routers, and filters prevent those routers from becoming overwhelmed, by limiting the number of possible routes that are presented to them.

Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down.

Rogers uses equipment from different manufacturers in its network core, and the two vendors the company buys routers from have different designs and approaches to managing traffic and protecting the equipment from overloading. Those differences are at the core of the outage Rogers experienced, the company said in the documents.

But, in the early hours, the company’s technicians had not yet pinpointed the cause of the catastrophe. Rogers apparently considered the possibility that its networks had been attacked by cybercriminals. At 6 a.m., Jorge Fernandes, who at the time was the company’s chief technology officer, reached out to his counterparts at Telus Corp. T-T and BCE Inc.’s Bell Canada BCE-T to inform them of the outage and warn them to look out for cyberattacks, the company said in its submission.

Although Bell and Telus offered to help, Rogers quickly determined that it would not be able to transfer its customers to its rivals’ networks because certain elements of the Rogers network, such as its centralized user database, were inaccessible as a result of the outage. In any case, the rival networks would not have been able to handle the sudden surge of traffic from Rogers’s 10.2 million wireless subscribers, the telecom said.

Rogers outage may weigh on decision around $26-billion takeover of Shaw, Champagne says

Mr. Fernandes was in Portugal when the outage began, and he immediately started making arrangements to return to Canada, according to two sources familiar with his whereabouts. The Globe is not identifying the sources because they were not authorized to speak publicly about the matter.

Meanwhile, the Rogers network team gathered at the company’s network operations centre in Brampton, Ont., re-established access to the network and started trying to figure out the cause of the outage.

In order to communicate with each other and coordinate the recovery effort, some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.

It wasn’t until 8:54 a.m. – roughly four hours after the start of the outage – that the company publicly acknowledged the situation. “We know how important it is for our customers to stay connected,” the telecom tweeted through its customer service account. “We are aware of issues currently affecting our networks and our teams are fully engaged to resolve the issue as soon as possible. We will continue to keep you updated as we have more information to share.”

The company’s disclosures to the CRTC suggest the delayed reaction might have had to do with problems logging in to online accounts used to communicate with customers. The telecom said that, in the future, it will ensure its crisis response teams have alternative methods of accessing social-media accounts that are protected by two-factor authentication linked to Rogers devices.

It took all day for the network team to restore the network. They had to disconnect the equipment that was causing the problem, redirect traffic and confirm the stability of the network before slowly bringing services back online. The process had to be done methodically to prevent overloading the network and triggering another outage, the company said.

“Our wireless services are starting to recover and our technical teams are working hard to get everyone back online as quickly as possible,” the company tweeted shortly before 10 p.m.

The following morning, Rogers announced that it had restored services for the “vast majority” of its customers. But intermittent issues persisted throughout the weekend.

This Sunday, in an open letter to customers, Rogers CEO Tony Staffieri vowed to invest more in testing, oversight and artificial intelligence to improve the reliability of the company’s networks. He put the price tag of the changes at around $10-billion over three years.

The wireless giant will also physically separate its wireless and wireline core networks to ensure that any future outages don’t affect both services, Mr. Staffieri said.

Last week, the company replaced Mr. Fernandes, a former Vodafone executive, with veteran telecom executive Ron McKenzie. Mr. McKenzie was previously the president of Rogers for Business, the division that offers wireless and internet services to corporate clients.

Mr. McKenzie will kick off his new role with an appearance in front of the House of Commons committee that is studying the outage. The committee, which is made up of members of Parliament from all four major federal parties, is expected to grill him, Mr. Staffieri and Rogers chief regulatory officer Ted Woodhead on the five-day billing credit the company is offering to compensate its customers for the outage. The committee may also ask about the network and operational changes the telecom plans to make in order to prevent future outages.

As all of this is happening, Rogers is awaiting regulatory approval of its contested $26-billion takeover of Shaw Communications Inc., ahead of a July 31 deadline. The Competition Bureau is attempting to block the merger, arguing that it will result in poorer service and higher prices for cellphone customers.

Your time is valuable. Have the Top Business Headlines newsletter conveniently delivered to your inbox in the morning or evening. Sign up today.

Adblock test (Why?)

Source link

Continue Reading


Retail sales fall for 1st time this year as consumers start to tap out – CBC News



Facing sky-high inflation, consumers put away their wallets more often in July, new data revealed Friday, as retail sales fell for the first time since 2021.

Canadian retailers rang up $61.3 billion in sales in July, Statistics Canada reported Friday. That’s a decline of 2.5 per cent from the previous month’s level as lower sales at gas stations and clothing stores led the way down.

Sales at gas stations fell by 14 per cent. A big part of that was lower prices for the fuel itself, but even in volume terms sales were down by seven per cent. Fewer people were filling up during the month, which was in keeping with the vehicle segment overall as auto sales edged down 0.5 per cent. Both new and used car dealers reported declines.

Consumables like food and drink also weren’t flying off the shelves, as supermarkets and grocery stores saw sales slip by 0.9 per cent, while liquor stores saw a decline of 1.2 per cent.

The soft retail sales numbers suggest consumers are starting to put away their wallets in the face of sky-high prices and a gloomy outlook for the economy.

“This retail sales report was unambiguously weak, suggesting that consumers tightened their purse strings in July,” TD Bank economist Ksenia Bushmeneva said of the numbers. “Consumer demand appears to have broadly cooled across most categories of spending.”

“All in all, given the triple headwinds emanating from higher consumer prices, rapidly rising interest rates and a drop in wealth, consumers are becoming more frugal,” she said.

Adblock test (Why?)

Source link

Continue Reading


Dow drops but narrowly avoids confirming bear market status –



Sept 23 (Reuters) – The blue-chip Dow Jones Industrial Average (.DJI)tumbled to its lowest level since November 2020 on Friday, but narrowly missed ending more than 20% below its Jan. 4 closing record high.

A Dow close below 29,439.72 would have confirmed a bear market that began from that record, according to a widely used definition. read more The Dow fell 486.27 points, or 1.62%, to end at 29,590.41.

The Dow is the only one of the three main indexes not to have bear market status. The S&P 500 (.SPX) notched that grim milestone in June and the Nasdaq (.IXIC) in March.

The renewed selling pressure in markets came in a week that saw the U.S. Federal Reserve raise interest rates by three-quarters of a percentage point for a third straight time and a vow to keep it going until inflation is under control.

It has been a tumultuous year for Wall Street, plagued by worries about Russia’s invasion of Ukraine, an energy crisis in Europe and the end of easy money policy globally.

The S&P 500 has lost 23% this year and the Nasdaq has shed 31%.

The last time the three indexes pulled back so sharply was in 2020 during the heights of the pandemic selloff.

Heightened fears of a U.S. economic downturn next year and its impact on corporate profits has prompted brokerages to downgrade their year-end targets for the S&P 500. read more

Dow Jones Industrials bear markets

Dow Jones Industrials bear markets

(This story refiles to fix typo in headline)

Reporting by Medha Singh in Bengaluru; additional reporting by Caroline Valetkevitch in New York; Editing by Shounak Dasgupta, Shinjini Ganguli, Maju Samuel and Diane Craft

Our Standards: The Thomson Reuters Trust Principles.

Adblock test (Why?)

Source link

Continue Reading


TSX slumps as oil falls below $80 and economic gloom settles in – CBC News



Canada’s benchmark stock index dropped heavily on Friday as prospects of a global recession cause investors to sell first and ask questions later.

The S&P/TSX Composite Index was off by more than 520 points or 2.75 per cent to close at 18,480, dragged down by a plunge in the price of oil. That’s the lowest level for the benchmark Canadian stock index since July.

The benchmark price of crude oil in North America lost almost $5 to close at $79.13 a barrel, its lowest price since January. The catalyst for oil’s decline seems to have been central banks signaling this week that they are so committed to reining in inflation that they are willing to create a recession to achieve it.

The U.S. Federal Reserve hiked its benchmark interest rate on Wednesday, and nine other countries around the world followed suit the next day. That will help bring down inflation, but it will likely come at great cost to the economy.

“Clearly what they are saying is they are so determined to bring inflation down that they are going to bring down the economy in the process,” said John Zecher, the founder of Toronto-based money manager J Zechner & Associates. “That’s the way the market is reading it … They aren’t going to stop until the economy turns down.”

Oil price down to lowest since January

A recession would lead to much less demand for energy, which is why oil sold off. About a fifth of the companies on the TSX are in the energy sector, and they were among the biggest losers Friday. Shares in Suncor, Cenovus, MEG Energy and Crescent Point all lost more than eight per cent on the day.

More and more economic indicators are starting to suggest Canada’s economy either already has derailed or is about to. Employment numbers last week showed the economy has lost jobs for three months in a row, and retail sales data on Friday showed that Canadians are putting away their wallets once more.

Stock markets are responding to that gloom, and some analysts think there is a lot more pain to come.

“The lows that we saw recently in the summer months are going to be challenged in the next couple of days to weeks,” said Larry Berman, chief investment officer with Toronto-based money manager QWealth, in an interview.  “The market [isn’t] priced for what the central banks are going to do.”

The Canadian dollar dipped as low as 73.61 cents US, its lowest level in more than two years.

Shares in New York also sold off, with the Dow Jones Industrial Average closing down almost 500 points to 29,590 — its lowest level of the year.

“Over the next couple of weeks, long-term investors may hesitate buying into weakness,” said Edward Moya, an analyst with foreign exchange firm Oanda. “How far we go below the summer lows is anyone’s guess.”

Adblock test (Why?)

Source link

Continue Reading