Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.


In a research paper (via VentureBeat), Apple outlines a new system for how large language models tackle reference resolution, which involves deciphering ambiguous references to on-screen entities, as well as understanding conversational and background context. As a result, ReALM could lead to more intuitive and natural interactions with devices.

Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple’s ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.

ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen’s content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI’s GPT-4.

ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.

See also  How to stop your iPhone tracking your in iOS 17 (Video)

Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.

Popular Stories

Top Stories: WWDC 2024 Announced, New iPads Delayed, and More

Apple’s WWDC 2024 dates have been announced, giving us timing for the unveiling of the company’s next round of major operating system updates and likely some other announcements. This week also saw some disappointing news on the iPad front, with update timing for the iPad Pro and iPad Air pushed back from previous rumors. We did hear some new tidbits about what might be coming in iOS 18 and…

Apple to Launch New iPad Pro and iPad Air Models in May

Apple will introduce new iPad Pro and iPad Air models in early May, according to Bloomberg’s Mark Gurman. Gurman previously suggested the new iPads would come out in March, and then April, but the timeline has been pushed back once again. Subscribe to the MacRumors YouTube channel for more videos. Apple is working on updates to both the iPad Pro and iPad Air models. The iPad Pro models will…

Apple Says iPhone 6 Plus Now ‘Obsolete’ and iPad Mini 4 Now ‘Vintage’

Apple today added a handful of devices to its public-facing vintage and obsolete products list, including some older iPhone and iPad models. Apple now considers the iPhone 6 Plus to be “obsolete” worldwide, meaning that Apple Stores and Apple Authorized Service Providers no longer offer repairs or other hardware service for the device. Apple says it considers a product “obsolete” once seven…

See also  Accumulated Deposits for Apple Card Savings Surpass $10 Billion Milestone

What to Expect From iOS 17.5

Apple has yet to release the first beta of iOS 17.5 for the iPhone, but two changes are already expected with the upcoming software update. iOS 17.5 will likely allow iPhone users in the EU to download apps directly from the websites of eligible developers, and the update might include some changes to how Apple ID recovery contacts work. More details about these potential changes follow. W…

Criminals in Montreal Using AirTags to Steal Vehicles

Thieves in Montreal, Canada have been using Apple’s AirTags to facilitate vehicle theft, according to a report from Vermont news sites WCAX and NBC5 (via 9to5Mac). Police officers in Burlington, Vermont have issued a warning about AirTags for drivers who recently visited Canada. Two Burlington residents found Apple AirTags in their vehicles after returning from trips to Montreal, and these…

Google Reveals When to Expect RCS Support on iPhone for Improved Texting With Android Users

In November, Apple announced that the iPhone would support the cross-platform messaging standard RCS (Rich Communication Services) in the Messages app starting “later” in 2024, and Google has now revealed a more narrow timeframe. In a since-deleted section of the revamped Google Messages web page, spotted by 9to5Google, Google said that Apple would be adopting RCS on the iPhone in the “fall…

Apple Card Savings Account to Receive First-Ever Interest Rate Decrease

Nearly one year after it launched in the U.S., the Apple Card’s high-yield savings account will be receiving its first-ever interest rate decrease. Starting on April 3, the Apple Card savings account’s annual percentage yield (APY) will be lowered to 4.4%, according to data on Apple’s backend discovered by MacRumors contributor Aaron Perris. The account currently has a 4.5% APY. 4.4% will …

See also  Tarsier 3D VR180 8K spatial video UHD camera with live streaming