People use the wifi inside Toronto’s Fairview Mall on July 8.Yader Guzman/The Globe and Mail
Rogers Communications Inc. RCI-B-T engineers began the sixth step of a seven-step process to upgrade the core infrastructure that supports the company’s wireless and broadband networks at 2:27 a.m. on July 8.
Two hours and 16 minutes later, a coding error was introduced that triggered a cascade of events, resulting in a massive outage that left millions of Canadians without cellphone, internet or home phone service for at least a day.
The shutdown of one of Canada’s dominant telecommunications networks created widespread chaos. Rogers was unable to deliver four emergency alertsto its wireless customers in Saskatchewan, including three tornado warnings and one dangerous person report.
Rogers customers were unable to call 911, and the Interac debit system was also affected, causing issues for both consumers and businesses. In Toronto, the disruption forced Canadian singer-songwriter the Weeknd to postpone a concert that was supposed to have been held at the Rogers Centre that night.
Initially, even Rogers itself was unsure what was causing the service disruption. But weeks later, in a detailed submission in response to questions from the Canadian Radio-television and Telecommunications Commission, the company gave a full account of its version of events.
Those documents, which were disclosed publicly by the CRTC in redacted form on Friday, give new details on the outage and provide an early glimpse at the set of facts Rogers executives will draw upon on Monday, when they are expected to testify about the incident in a public hearing before the House of Commons committee on industry and technology.
Like many of its peers, Rogers currently has one core network that supports all of the services it provides. The core is essentially the network’s brain. It receives, processes, transmits and connects all voice, wireless data, internet and television traffic.
The telecom had started the seven-phase process to upgrade the core back in February, after what the company described in its CRTC submission as a comprehensive planning process that included budget and project approvals, risk assessment and testing.
The first five phases had gone smoothly. But, at 4:43 a.m. on July 8, a piece of code was introduced that deleted a routing filter. In telecom networks, packets of data are guided and directed by devices called routers, and filters prevent those routers from becoming overwhelmed, by limiting the number of possible routes that are presented to them.
Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down.
Rogers uses equipment from different manufacturers in its network core, and the two vendors the company buys routers from have different designs and approaches to managing traffic and protecting the equipment from overloading. Those differences are at the core of the outage Rogers experienced, the company said in the documents.
But, in the early hours, the company’s technicians had not yet pinpointed the cause of the catastrophe. Rogers apparently considered the possibility that its networks had been attacked by cybercriminals. At 6 a.m., Jorge Fernandes, who at the time was the company’s chief technology officer, reached out to his counterparts at Telus Corp. T-T and BCE Inc.’s Bell Canada BCE-T to inform them of the outage and warn them to look out for cyberattacks, the companysaid in its submission.
Although Bell and Telus offered to help, Rogers quickly determined that it would not be able to transfer its customers to its rivals’ networks because certain elements of the Rogers network, such as its centralized user database, were inaccessible as a result of the outage. In any case, the rival networks would not have been able to handle the sudden surge of traffic from Rogers’s 10.2 million wireless subscribers, the telecom said.
Mr. Fernandes was in Portugal when the outage began, and he immediately started making arrangements to return to Canada, according to two sources familiar with his whereabouts. The Globe is not identifying the sources because they were not authorized to speak publicly about the matter.
Meanwhile, the Rogers network team gathered at the company’s network operations centre in Brampton, Ont., re-established access to the network and started trying to figure out the cause of the outage.
In order to communicate with each other and coordinate the recovery effort, some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.
It wasn’t until 8:54 a.m. – roughly four hours after the start of the outage – that the company publicly acknowledged the situation. “We know how important it is for our customers to stay connected,” the telecom tweeted through its customer service account. “We are aware of issues currently affecting our networks and our teams are fully engaged to resolve the issue as soon as possible. We will continue to keep you updated as we have more information to share.”
The company’s disclosures to the CRTC suggest the delayed reaction might have had to do with problems logging in to online accounts used to communicate with customers. The telecom said that, in the future, it will ensure its crisis response teams have alternative methods of accessing social-media accounts that are protected by two-factor authentication linked to Rogers devices.
It took all day for the network team to restore the network. They had to disconnect the equipment that was causing the problem, redirect traffic and confirm the stability of the network before slowly bringing services back online. The process had to be done methodically to prevent overloading the network and triggering another outage, the company said.
“Our wireless services are starting to recover and our technical teams are working hard to get everyone back online as quickly as possible,” the company tweeted shortly before 10 p.m.
The following morning, Rogers announced that it had restored services for the “vast majority” of its customers. But intermittent issues persisted throughout the weekend.
This Sunday, in an open letter to customers, Rogers CEO Tony Staffieri vowed to invest more in testing, oversight and artificial intelligence to improve the reliability of the company’s networks. He put the price tag of the changes at around $10-billion over three years.
The wireless giant will also physically separate its wireless and wireline core networks to ensure that any future outages don’t affect both services, Mr. Staffieri said.
Last week, the company replaced Mr. Fernandes, a former Vodafone executive, with veteran telecom executive Ron McKenzie. Mr. McKenzie was previously the president of Rogers for Business, the division that offers wireless and internet services to corporate clients.
Mr. McKenzie will kick off his new role with an appearance in front of the House of Commons committee that is studying the outage. The committee, which is made up of members of Parliament from all four major federal parties, is expected to grill him, Mr. Staffieri and Rogers chief regulatory officer Ted Woodhead on the five-day billing credit the company is offering to compensate its customers for the outage. The committee may also ask about the network and operational changes the telecom plans to make in order to prevent future outages.
As all of this is happening, Rogers is awaiting regulatory approval of its contested $26-billion takeover of Shaw Communications Inc., ahead of a July 31 deadline. The Competition Bureau is attempting to block the merger, arguing that it will result in poorer service and higher prices for cellphone customers.
Your time is valuable. Have the Top Business Headlines newsletter conveniently delivered to your inbox in the morning or evening. Sign up today.
TORONTO – Roots Corp. may have built its brand on all things comfy and cosy, but its CEO says activewear is now “really becoming a core part” of the brand.
The category, which at Roots spans leggings, tracksuits, sports bras and bike shorts, has seen such sustained double-digit growth that Meghan Roach plans to make it a key part of the business’ future.
“It’s an area … you will see us continue to expand upon,” she told analysts on a Friday call.
The Toronto-based retailer’s push into activewear has taken shape over many years and included several turns as the official designer and supplier of Team Canada’s Olympic uniform.
But consumers have had plenty of choice when it comes to workout gear and other apparel suited to their sporting needs. On top of the slew of athletic brands like Nike and Adidas, shoppers have also gravitated toward Lululemon Athletica Inc., Alo and Vuori, ramping up competition in the activewear category.
Roach feels Roots’ toehold in the category stems from the fit, feel and following its merchandise has cultivated.
“Our product really resonates with (shoppers) because you can wear it through multiple different use cases and occasions,” she said.
“We’ve been seeing customers come back again and again for some of these core products in our activewear collection.”
Her remarks came the same day as Roots revealed it lost $5.2 million in its latest quarter compared with a loss of $5.3 million in the same quarter last year.
The company said the second-quarter loss amounted to 13 cents per diluted share for the quarter ended Aug. 3, the same as a year earlier.
In presenting the results, Roach reminded analysts that the first half of the year is usually “seasonally small,” representing just 30 per cent of the company’s annual sales.
Sales for the second quarter totalled $47.7 million, down from $49.4 million in the same quarter last year.
The move lower came as direct-to-consumer sales amounted to $36.4 million, down from $37.1 million a year earlier, as comparable sales edged down 0.2 per cent.
The numbers reflect the fact that Roots continued to grapple with inventory challenges in the company’s Cooper fleece line that first cropped up in its previous quarter.
Roots recently began to use artificial intelligence to assist with daily inventory replenishments and said more tools helping with allocation will go live in the next quarter.
Beyond that time period, the company intends to keep exploring AI and renovate more of its stores.
It will also re-evaluate its design ranks.
Roots announced Friday that chief product officer Karuna Scheinfeld has stepped down.
Rather than fill the role, the company plans to hire senior level design talent with international experience in the outdoor and activewear sectors who will take on tasks previously done by the chief product officer.
This report by The Canadian Press was first published Sept. 13, 2024.
VANCOUVER – Mediated talks between the union representing HandyDART workers in Metro Vancouver and its employer, Transdev, are set to resume today as a strike that has stopped most services drags into a second week.
No timeline has been set for the length of the negotiations, but Joe McCann, president of the Amalgamated Transit Union Local 1724, says they are willing to stay there as long as it takes, even if talks drag on all night.
About 600 employees of the door-to-door transit service for people unable to navigate the conventional transit system have been on strike since last Tuesday, pausing service for all but essential medical trips.
Hundreds of drivers rallied outside TransLink’s head office earlier this week, calling for the transportation provider to intervene in the dispute with Transdev, which was contracted to oversee HandyDART service.
Transdev said earlier this week that it will provide a reply to the union’s latest proposal on Thursday.
A statement from the company said it “strongly believes” that their employees deserve fair wages, and that a fair contract “must balance the needs of their employees, clients and taxpayers.”
This report by The Canadian Press was first published Sept. 12, 2024.
MONTREAL – Travel company Transat AT Inc. reported a loss in its latest quarter compared with a profit a year earlier as its revenue edged lower.
The parent company of Air Transat says it lost $39.9 million or $1.03 per diluted share in its quarter ended July 31.
The result compared with a profit of $57.3 million or $1.49 per diluted share a year earlier.
Revenue in what was the company’s third quarter totalled $736.2 million, down from $746.3 million in the same quarter last year.
On an adjusted basis, Transat says it lost $1.10 per share in its latest quarter compared with an adjusted profit of $1.10 per share a year earlier.
Transat chief executive Annick Guérard says demand for leisure travel remains healthy, as evidenced by higher traffic, but consumers are increasingly price conscious given the current economic uncertainty.
This report by The Canadian Press was first published Sept. 12, 2024.
:format(jpeg)/cloudfront-us-east-1.images.arcpublishing.com/tgam/HWXWE6VUFVFVHHU2F7NNETL4EQ.JPG)
