Wed, April 24, 2024 at 9:35 AM EDT
Business
How a coding error caused Rogers outage that left millions without service – The Globe and Mail
Rogers Communications Inc. RCI-B-T engineers began the sixth step of a seven-step process to upgrade the core infrastructure that supports the company’s wireless and broadband networks at 2:27 a.m. on July 8.
Two hours and 16 minutes later, a coding error was introduced that triggered a cascade of events, resulting in a massive outage that left millions of Canadians without cellphone, internet or home phone service for at least a day.
The shutdown of one of Canada’s dominant telecommunications networks created widespread chaos. Rogers was unable to deliver four emergency alerts to its wireless customers in Saskatchewan, including three tornado warnings and one dangerous person report.
Rogers customers were unable to call 911, and the Interac debit system was also affected, causing issues for both consumers and businesses. In Toronto, the disruption forced Canadian singer-songwriter the Weeknd to postpone a concert that was supposed to have been held at the Rogers Centre that night.
Initially, even Rogers itself was unsure what was causing the service disruption. But weeks later, in a detailed submission in response to questions from the Canadian Radio-television and Telecommunications Commission, the company gave a full account of its version of events.
Opinion: Rogers still has some explaining to do about its outage and the fallout for its Shaw deal
Those documents, which were disclosed publicly by the CRTC in redacted form on Friday, give new details on the outage and provide an early glimpse at the set of facts Rogers executives will draw upon on Monday, when they are expected to testify about the incident in a public hearing before the House of Commons committee on industry and technology.
Like many of its peers, Rogers currently has one core network that supports all of the services it provides. The core is essentially the network’s brain. It receives, processes, transmits and connects all voice, wireless data, internet and television traffic.
The telecom had started the seven-phase process to upgrade the core back in February, after what the company described in its CRTC submission as a comprehensive planning process that included budget and project approvals, risk assessment and testing.
The first five phases had gone smoothly. But, at 4:43 a.m. on July 8, a piece of code was introduced that deleted a routing filter. In telecom networks, packets of data are guided and directed by devices called routers, and filters prevent those routers from becoming overwhelmed, by limiting the number of possible routes that are presented to them.
Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down.
Rogers uses equipment from different manufacturers in its network core, and the two vendors the company buys routers from have different designs and approaches to managing traffic and protecting the equipment from overloading. Those differences are at the core of the outage Rogers experienced, the company said in the documents.
But, in the early hours, the company’s technicians had not yet pinpointed the cause of the catastrophe. Rogers apparently considered the possibility that its networks had been attacked by cybercriminals. At 6 a.m., Jorge Fernandes, who at the time was the company’s chief technology officer, reached out to his counterparts at Telus Corp. T-T and BCE Inc.’s Bell Canada BCE-T to inform them of the outage and warn them to look out for cyberattacks, the company said in its submission.
Although Bell and Telus offered to help, Rogers quickly determined that it would not be able to transfer its customers to its rivals’ networks because certain elements of the Rogers network, such as its centralized user database, were inaccessible as a result of the outage. In any case, the rival networks would not have been able to handle the sudden surge of traffic from Rogers’s 10.2 million wireless subscribers, the telecom said.
Rogers outage may weigh on decision around $26-billion takeover of Shaw, Champagne says
Mr. Fernandes was in Portugal when the outage began, and he immediately started making arrangements to return to Canada, according to two sources familiar with his whereabouts. The Globe is not identifying the sources because they were not authorized to speak publicly about the matter.
Meanwhile, the Rogers network team gathered at the company’s network operations centre in Brampton, Ont., re-established access to the network and started trying to figure out the cause of the outage.
In order to communicate with each other and coordinate the recovery effort, some employees started swapping out their SIM cards for Bell or Telus SIM cards that they had received back in 2015 as part of an emergency contingency plan established between the wireless carriers.
It wasn’t until 8:54 a.m. – roughly four hours after the start of the outage – that the company publicly acknowledged the situation. “We know how important it is for our customers to stay connected,” the telecom tweeted through its customer service account. “We are aware of issues currently affecting our networks and our teams are fully engaged to resolve the issue as soon as possible. We will continue to keep you updated as we have more information to share.”
The company’s disclosures to the CRTC suggest the delayed reaction might have had to do with problems logging in to online accounts used to communicate with customers. The telecom said that, in the future, it will ensure its crisis response teams have alternative methods of accessing social-media accounts that are protected by two-factor authentication linked to Rogers devices.
It took all day for the network team to restore the network. They had to disconnect the equipment that was causing the problem, redirect traffic and confirm the stability of the network before slowly bringing services back online. The process had to be done methodically to prevent overloading the network and triggering another outage, the company said.
“Our wireless services are starting to recover and our technical teams are working hard to get everyone back online as quickly as possible,” the company tweeted shortly before 10 p.m.
The following morning, Rogers announced that it had restored services for the “vast majority” of its customers. But intermittent issues persisted throughout the weekend.
This Sunday, in an open letter to customers, Rogers CEO Tony Staffieri vowed to invest more in testing, oversight and artificial intelligence to improve the reliability of the company’s networks. He put the price tag of the changes at around $10-billion over three years.
The wireless giant will also physically separate its wireless and wireline core networks to ensure that any future outages don’t affect both services, Mr. Staffieri said.
Last week, the company replaced Mr. Fernandes, a former Vodafone executive, with veteran telecom executive Ron McKenzie. Mr. McKenzie was previously the president of Rogers for Business, the division that offers wireless and internet services to corporate clients.
Mr. McKenzie will kick off his new role with an appearance in front of the House of Commons committee that is studying the outage. The committee, which is made up of members of Parliament from all four major federal parties, is expected to grill him, Mr. Staffieri and Rogers chief regulatory officer Ted Woodhead on the five-day billing credit the company is offering to compensate its customers for the outage. The committee may also ask about the network and operational changes the telecom plans to make in order to prevent future outages.
As all of this is happening, Rogers is awaiting regulatory approval of its contested $26-billion takeover of Shaw Communications Inc., ahead of a July 31 deadline. The Competition Bureau is attempting to block the merger, arguing that it will result in poorer service and higher prices for cellphone customers.
Your time is valuable. Have the Top Business Headlines newsletter conveniently delivered to your inbox in the morning or evening. Sign up today.
Continue Reading
Business
Oil Firms Doubtful Trans Mountain Pipeline Will Start Full Service by May 1st
|
Oil companies planning to ship crude on the expanded Trans Mountain pipeline in Canada are concerned that the project may not begin full service on May 1 but they would be nevertheless obligated to pay tolls from that date.
In a letter to the Canada Energy Regulator (CER), Suncor Energy and other shippers including BP and Marathon Petroleum have expressed doubts that Trans Mountain will start full service on May 1, as previously communicated, Reuters reports.
Trans Mountain Corporation, the government-owned entity that completed the pipeline construction, told Reuters in an email that line fill on the expanded pipeline would be completed in early May.
After a series of delays, cost overruns, and legal challenges, the expanded Trans Mountain oil pipeline will open for business on May 1, the company said early this month.
“The Commencement Date for commercial operation of the expanded system will be May 1, 2024. Trans Mountain anticipates providing service for all contracted volumes in the month of May,” Trans Mountain Corporation said in early April.
The expanded pipeline will triple the capacity of the original pipeline to 890,000 barrels per day (bpd) from 300,000 bpd to carry crude from Alberta’s oil sands to British Columbia on the Pacific Coast.
The Federal Government of Canada bought the Trans Mountain Pipeline Expansion (TMX) from Kinder Morgan back in 2018, together with related pipeline and terminal assets. That cost the federal government $3.3 billion (C$4.5 billion) at the time. Since then, the costs for the expansion of the pipeline have quadrupled to nearly $23 billion (C$30.9 billion).
The expansion project has faced continuous delays over the years. In one of the latest roadblocks in December, the Canadian regulator denied a variance request from the project developer to move a small section of the pipeline due to challenging drilling conditions.
The company asked the regulator to reconsider its decision, and received on January 12 a conditional approval, avoiding what could have been another two-year delay to start-up.
Business
Tesla profits cut in half as demand falls
|
Tesla profits slump by more than a half
Tesla has announced its profits fell sharply in the first three months of the year to $1.13bn (£910m), compared with $2.51bn in 2023.
It caps a difficult period for the electric vehicle (EV) maker, which – faced with falling sales – has announced thousands of job cuts.
Boss Elon Musk remains bullish about its prospects, telling investors the launch of new models would be brought forward.
Its share price has risen but analysts say it continues to face significant challenges, including from lower-cost rivals.
The company has suffered from falling demand and competition from cheaper Chinese imports which has led its stock price to collapse by 43% over 2024.
Figures for the first quarter of 2024 revealed revenues of $21.3bn, down on analysts’ predictions of just over $22bn.
But the decision by Tesla to bring forward the launch of new models from the second half of 2025 boosted its shares by nearly 12.5% in after-hours trading.
It did not reveal pricing details for the new vehicles.
However Mr Musk made clear he also grander ambitions, touting Tesla’s AI credentials and plans for self-driving vehicles – even going as far as to say considering it to be just a car company was the “wrong framework.”
“If somebody doesn’t believe Tesla is going to solve autonomy I think they should not be an investor,” he said.
Such sentiments have been questioned by analysts though, with Deutsche Bank saying driverless cars face “technological, regulatory and operational challenges.”
Some investors have called for the company to instead focus on releasing a lower price, mass-market EV.
However, Tesla has already been on a charm offensive, trying to win over new customers by dropping its prices in a series of markets in the face of falling sales.
It also said its situation was not unique.
“Global EV sales continue to be under pressure as many carmakers prioritize hybrids over EVs,” it said.
Despite plans to bring forward new models originally planned for next year the firm is cutting its workforce.
Tesla said it would lose 3,332 jobs in California and 2,688 positions in Texas, starting mid-June.
The cuts in Texas represent 12% of Tesla’s total workforce of almost 23,000 in the area where its gigafactory and headquarters are located.
However, Mr Musk sought to downplay the move.
“Tesla has now created over 30,000 manufacturing jobs in California!” he said in a post on his social media platform X, formerly Twitter, on Tuesday.
Another 285 jobs will be lost in New York.
Tesla’s total workforce stood at more than 140,000 late last year, up from around 100,000 at the end of 2021, according to the company’s filings with US regulators.
Musk’s salary
The car firm is also facing other issues, with a struggle over Mr Musk’s compensation still raging on.
On Wednesday, Tesla asked shareholders to vote for a proposal to accept Mr Musk’s compensation package – once valued at $56bn – which had been rejected by a Delaware judge.
The judge found Tesla’s directors had breached their fiduciary duty to the firm by awarding Mr Musk the pay-out.
Due to the fall in Tesla’s stock value, the compensation package is now estimated to be around $10bn less – but still greater than the GDP of many countries.
In addition, Tesla wants its shareholders to agree to the firm being moved from Delaware to Texas – which Mr Musk called for after the judge rejected his payday.
Business
Stock market today: Nasdaq futures pop, Tesla surges after earnings with more heavyweights on deck
|
Tech stocks rose on Wednesday, outstripping the broader market as investors welcomed Tesla’s (TSLA) cheaper car pledge and waited for the next rush of corporate earnings.
The Nasdaq Composite (^IXIC) rose roughly 0.6%, coming off a sharp closing gain. The S&P 500 (^GSPC) was up 0.2%, continuing a rebound from its longest losing streak of 2024, while the Dow Jones Industrial Average (^DJI) fell 0.1%.
Tesla shares jumped nearly 12% after the EV maker’s vow to speed up the launch of more affordable models eclipsed its quarterly earnings and revenue miss. That cheered up investors worried about growth amid a strategy shift to robotaxis and the planned cancellation of a cheaper model.
The results from the first “Magnificent Seven” to report have intensified the already high hopes for Big Tech earnings, that the megacaps can revive the rally in stocks they powered. The spotlight is now on Meta’s (META) report due after the market close, as the Facebook owner’s shares rose after the Senate voted for a potential ban on rival TikTok. Microsoft (MSFT) and Alphabet (GOOG) next up on Thursday.
Meanwhile, Boeing (BA) reported better than expected first quarter results before the opening bell with a loss per share of $1.13, narrower than the $1.72 estimated by Wall Street. Shares rose about 2% in morning trade.
Live6 updates
-
Health19 hours ago
Remnants of bird flu virus found in pasteurized milk, FDA says
-
Health24 hours ago
Bird flu virus found in grocery milk as officials say supply still safe
-
News16 hours ago
Amid concerns over ‘collateral damage’ Trudeau, Freeland defend capital gains tax change
-
Art20 hours ago
Random: We’re In Awe of Metaphor: ReFantazio’s Box Art
-
Investment24 hours ago
Taxes should not wag the tail of the investment dog, but that’s what Trudeau wants
-
Art13 hours ago
The unmissable events taking place during London’s Digital Art Week
-
Media19 hours ago
Vaughn Palmer: B.C. premier gives social media giants another chance
-
Science19 hours ago
NASA hears from Voyager 1, the most distant spacecraft from Earth, after months of quiet