Tech

SpikeGPT: researcher releases code for largest-ever spiking neural network for language generation – University of California, Santa Cruz

Published

2 years ago

March 7, 2023

Language generators such as ChatGPT are gaining attention for their ability to reshape how we use search engines and change the way we interact with artificial intelligence. But these algorithms are both computationally expensive to run and depend on maintenance from just a few companies to avoid outages.

But UC Santa Cruz Assistant Professor of Electrical and Computer Engineering Jason Eshraghian created a new model for language generation that can address both of these issues. Language models typically use modern deep learning methods called neural networks, but Eshraghian is powering a language model with an alternative algorithm called a spiking neural network (SNN). He and two students have recently released the open-sourced code for the largest language-generating SNN ever, named SpikeGPT, which uses 22 times less energy than a similar model using typical deep learning. Using SNNs for language generation can have huge implications for accessibility, data security, and green computing and energy efficiency within this field.

“Brains are way more efficient than AI algorithms,” Eshraghian said. “Large scale language models rely on ridiculous amounts of compute power, and that’s pretty damn expensive. We’re taking an informed approach to borrowing principles from the brain, copying this idea that neurons are usually quiet and not transmitting anything. Using spikes is a much more efficient way to represent information.”

Neural networks in general are based on emulating how the brain processes information, and spiking neural networks are a variation that try to make the networks more efficient. Instead of constantly transmitting information throughout the network, as non-spiking networks behave, the neurons in SNNs stay in a quiet state unless they are activated, and therefore spike. This introduces a temporal dimension into the equation, because the functions are concerned with how the neurons behave over time.

Spiking neural networks, however, face their own challenges in the training of the models. Many of the optimization strategies that have been developed for regular neural networks and modern deep learning, such as backpropagation and gradient descent, cannot be easily applied to the training of SNNs because the information inputted into the system is not compatible with the training techniques. But Eshraghian has pioneered methods to circumvent these problems and apply the optimization techniques developed for traditional deep learning for the training of SNNs.

Large language models, such as ChatGPT, use a technique called self-attention, taking a sequence of data, such as a string of words, and applying a function to determine how closely each data point is related to each other. The mathematics behind this requires matrix-matrix multiplication, a complexity which is computationally expensive.

When trying to combine self-attention with SNNs, there was a similar complexity problem, until Eshraghian and his incoming graduate student Ruijie Zhu developed a technique to feed each data point in the sequence into the SNN model one by one, eliminating the need to do matrix-matrix multiplication.

“By coming up with a way to break down that backbone of language models into sequences, we completely squashed down that computational complexity without compromising on the ability of the model to generate language,” Eshraghian said. “It was taking the best of both worlds – the low complexity of sequential models and the performance of self-attention.”

In a preprint paper, Eshraghian describes three versions of SpikeGPT. The first is the smallest scale, at 45 million parameters, close in size to the largest-ever SNN that had been developed up to this point. Right now Eshraghian has only released the code for this smallest model, and he is still training the two larger ones.

The medium- and large-size models, at 125 million and 260 million parameters respectively, will likely become the second-largest and largest models when their training is complete and their code is released.

The preprint shows examples of language generation that these two models were able to produce, even in their not-yet fully trained states. Eshraghian found that his small-scale version is significantly more energy efficient than typical deep-learning models, and expects similar results for the other size models.

Using SNNs for language generation to power language generation in a more energy-efficient way can mean a decreased dependency on the large companies that currently dominate the language generation field. Making the technology more accessible will mitigate issues such as those that occur when gigantic servers running ChatGPT go down and render the technology useless for a time.

“If we manage to get this low-power enough to function on a scale comparable with the brain, then that could be something that everyone has locally on their devices, with less reliance on some monopolized entity,” Eshraghian said.

SpikeGPT also offers huge benefits for data security and privacy. With the language generator on a local device, data imputed into the systems are much more secure, protected from potential data-harvesting enterprises.

Eshraghian hopes that his models will show the language generation industry the vast potential of SNNs.

“This work shows that we can actually train models at the same scale with very similar performance, with far, far better energy consumption than what’s currently out there. Showing that in this paper could nudge industry in a direction to be more open to adopting SNNs as a full-fledged technique to address their power-consumption problems.”

However, this transition will require the development of brain-inspired hardware, which is a significant investment. Eshraghian hopes to work with a hardware company such as Intel to host these models, which would allow him to further demonstrate the energy-saving benefits of his SNN.

Since releasing the preprint paper and the code for the SNN, Eshraghian has seen a positive reaction from the research community. Hugging Face, a major company that hosts open-source models that are too large to live on GitHub, offered to host his model. He has also started a Discord server for people to experiment, build chatbots, and share results.

“What’s most appreciated by the community is the fact that we’ve shown it’s actually possible to do language generation with spikes.”

Adblock test (Why?)

Source link

Tech

Ottawa orders TikTok’s Canadian arm to be dissolved

Published

3 days ago

November 6, 2024

Anne Joseph

The federal government is ordering the dissolution of TikTok’s Canadian business after a national security review of the Chinese company behind the social media platform, but stopped short of ordering people to stay off the app.

Industry Minister François-Philippe Champagne announced the government’s “wind up” demand Wednesday, saying it is meant to address “risks” related to ByteDance Ltd.’s establishment of TikTok Technology Canada Inc.

“The decision was based on the information and evidence collected over the course of the review and on the advice of Canada’s security and intelligence community and other government partners,” he said in a statement.

The announcement added that the government is not blocking Canadians’ access to the TikTok application or their ability to create content.

However, it urged people to “adopt good cybersecurity practices and assess the possible risks of using social media platforms and applications, including how their information is likely to be protected, managed, used and shared by foreign actors, as well as to be aware of which country’s laws apply.”

Champagne’s office did not immediately respond to a request for comment seeking details about what evidence led to the government’s dissolution demand, how long ByteDance has to comply and why the app is not being banned.

A TikTok spokesperson said in a statement that the shutdown of its Canadian offices will mean the loss of hundreds of well-paying local jobs.

“We will challenge this order in court,” the spokesperson said.

“The TikTok platform will remain available for creators to find an audience, explore new interests and for businesses to thrive.”

The federal Liberals ordered a national security review of TikTok in September 2023, but it was not public knowledge until The Canadian Press reported in March that it was investigating the company.

At the time, it said the review was based on the expansion of a business, which it said constituted the establishment of a new Canadian entity. It declined to provide any further details about what expansion it was reviewing.

A government database showed a notification of new business from TikTok in June 2023. It said Network Sense Ventures Ltd. in Toronto and Vancouver would engage in “marketing, advertising, and content/creator development activities in relation to the use of the TikTok app in Canada.”

Even before the review, ByteDance and TikTok were lightning rod for privacy and safety concerns because Chinese national security laws compel organizations in the country to assist with intelligence gathering.

Such concerns led the U.S. House of Representatives to pass a bill in March designed to ban TikTok unless its China-based owner sells its stake in the business.

Champagne’s office has maintained Canada’s review was not related to the U.S. bill, which has yet to pass.

Canada’s review was carried out through the Investment Canada Act, which allows the government to investigate any foreign investment with potential to might harm national security.

While cabinet can make investors sell parts of the business or shares, Champagne has said the act doesn’t allow him to disclose details of the review.

Wednesday’s dissolution order was made in accordance with the act.

The federal government banned TikTok from its mobile devices in February 2023 following the launch of an investigation into the company by federal and provincial privacy commissioners.

— With files from Anja Karadeglija in Ottawa

This report by The Canadian Press was first published Nov. 6, 2024.

Source link

Health

Here is how to prepare your online accounts for when you die

Published

2 weeks ago

October 24, 2024

Anne Joseph

LONDON (AP) — Most people have accumulated a pile of data — selfies, emails, videos and more — on their social media and digital accounts over their lifetimes. What happens to it when we die?

It’s wise to draft a will spelling out who inherits your physical assets after you’re gone, but don’t forget to take care of your digital estate too. Friends and family might treasure files and posts you’ve left behind, but they could get lost in digital purgatory after you pass away unless you take some simple steps.

Here’s how you can prepare your digital life for your survivors:

Apple

The iPhone maker lets you nominate a “ legacy contact ” who can access your Apple account’s data after you die. The company says it’s a secure way to give trusted people access to photos, files and messages. To set it up you’ll need an Apple device with a fairly recent operating system — iPhones and iPads need iOS or iPadOS 15.2 and MacBooks needs macOS Monterey 12.1.

For iPhones, go to settings, tap Sign-in & Security and then Legacy Contact. You can name one or more people, and they don’t need an Apple ID or device.

You’ll have to share an access key with your contact. It can be a digital version sent electronically, or you can print a copy or save it as a screenshot or PDF.

Take note that there are some types of files you won’t be able to pass on — including digital rights-protected music, movies and passwords stored in Apple’s password manager. Legacy contacts can only access a deceased user’s account for three years before Apple deletes the account.

Google

Google takes a different approach with its Inactive Account Manager, which allows you to share your data with someone if it notices that you’ve stopped using your account.

When setting it up, you need to decide how long Google should wait — from three to 18 months — before considering your account inactive. Once that time is up, Google can notify up to 10 people.

You can write a message informing them you’ve stopped using the account, and, optionally, include a link to download your data. You can choose what types of data they can access — including emails, photos, calendar entries and YouTube videos.

There’s also an option to automatically delete your account after three months of inactivity, so your contacts will have to download any data before that deadline.

Facebook and Instagram

Some social media platforms can preserve accounts for people who have died so that friends and family can honor their memories.

When users of Facebook or Instagram die, parent company Meta says it can memorialize the account if it gets a “valid request” from a friend or family member. Requests can be submitted through an online form.

The social media company strongly recommends Facebook users add a legacy contact to look after their memorial accounts. Legacy contacts can do things like respond to new friend requests and update pinned posts, but they can’t read private messages or remove or alter previous posts. You can only choose one person, who also has to have a Facebook account.

You can also ask Facebook or Instagram to delete a deceased user’s account if you’re a close family member or an executor. You’ll need to send in documents like a death certificate.

TikTok

The video-sharing platform says that if a user has died, people can submit a request to memorialize the account through the settings menu. Go to the Report a Problem section, then Account and profile, then Manage account, where you can report a deceased user.

Once an account has been memorialized, it will be labeled “Remembering.” No one will be able to log into the account, which prevents anyone from editing the profile or using the account to post new content or send messages.

It’s not possible to nominate a legacy contact on Elon Musk’s social media site. But family members or an authorized person can submit a request to deactivate a deceased user’s account.

Passwords

Besides the major online services, you’ll probably have dozens if not hundreds of other digital accounts that your survivors might need to access. You could just write all your login credentials down in a notebook and put it somewhere safe. But making a physical copy presents its own vulnerabilities. What if you lose track of it? What if someone finds it?

Instead, consider a password manager that has an emergency access feature. Password managers are digital vaults that you can use to store all your credentials. Some, like Keeper,Bitwarden and NordPass, allow users to nominate one or more trusted contacts who can access their keys in case of an emergency such as a death.

But there are a few catches: Those contacts also need to use the same password manager and you might have to pay for the service.

___

Is there a tech challenge you need help figuring out? Write to us at onetechtip@ap.org with your questions.

Source link

Tech

Google’s partnership with AI startup Anthropic faces a UK competition investigation

Published

2 weeks ago

October 24, 2024

Anne Joseph

LONDON (AP) — Britain’s competition watchdog said Thursday it’s opening a formal investigation into Google’s partnership with artificial intelligence startup Anthropic.

The Competition and Markets Authority said it has “sufficient information” to launch an initial probe after it sought input earlier this year on whether the deal would stifle competition.

The CMA has until Dec. 19 to decide whether to approve the deal or escalate its investigation.

“Google is committed to building the most open and innovative AI ecosystem in the world,” the company said. “Anthropic is free to use multiple cloud providers and does, and we don’t demand exclusive tech rights.”

San Francisco-based Anthropic was founded in 2021 by siblings Dario and Daniela Amodei, who previously worked at ChatGPT maker OpenAI. The company has focused on increasing the safety and reliability of AI models. Google reportedly agreed last year to make a multibillion-dollar investment in Anthropic, which has a popular chatbot named Claude.

Anthropic said it’s cooperating with the regulator and will provide “the complete picture about Google’s investment and our commercial collaboration.”

“We are an independent company and none of our strategic partnerships or investor relationships diminish the independence of our corporate governance or our freedom to partner with others,” it said in a statement.

The U.K. regulator has been scrutinizing a raft of AI deals as investment money floods into the industry to capitalize on the artificial intelligence boom. Last month it cleared Anthropic’s $4 billion deal with Amazon and it has also signed off on Microsoft’s deals with two other AI startups, Inflection and Mistral.

Source link