Microsoft’s AI can now read your screen — or rather, the websites you’re browsing.
On Thursday, the company began rolling out a limited, U.S.-only preview of Copilot Vision, a tool that can understand and respond to questions about sites you’re visiting using Microsoft Edge. Gated behind Copilot Labs, an opt-in program for experimental AI capabilities, Copilot Vision can analyze text and images on web pages to answer queries like “What’s the recipe for this lasagna?”
Note that Copilot Labs requires a subscription to Microsoft’s Copilot Pro plan, which costs $20 per month.
Beyond answering questions, Copilot Vision can summarize and translate text, and handle tasks like spotlighting discounted products in a store catalog. It can also serve as a game assistant, for example offering pointers during matches on Chess.com.
“When you choose to enable Copilot Vision, it sees the page you’re on, it reads along with you, and you can talk through the problem you’re facing together,” Microsoft wrote in the blog post shared with TechCrunch. “It’s a new way to invite AI along with you as you navigate the web, tucked neatly into the bottom of your Edge browser whenever you want to ask for help.”
No doubt eager to avoid more bad press from AI privacy fumbles, Microsoft is stressing that Copilot Vision deletes data following every session. Processed audio, images or text aren’t stored or used to train models, the company says — at least not in this preview release.
Copilot Vision is also limited in the types of websites it can view. For the time being, Microsoft’s blocking the feature from working on paywalled and “sensitive” content, limiting Vision to a pre-approved list of “popular” sites.
What’s “sensitive” entail, exactly? Porn? Graphic violence? Microsoft wouldn’t say.
“The list of allowed websites is determined by category and on a case-by-case basis,” a Microsoft spokesperson said. “We’re starting with a small list of sites we’ve thoroughly tested and will continue to add more sites over time.”
Microsoft’s cautious approach is partly the product of legal disputes with news outlets. In one ongoing suit, The New York Times alleges that Microsoft let users get around its paywall by serving NY Times articles through the Copilot chatbot on Bing.
Many major publishers have opted to block AI tools from trawling their websites not only out of fear their data will be used without permission, but also to prevent these tools from sending their server costs soaring. Microsoft said that Copilot Vision will respect sites’ “machine-readable controls on AI,” like rules that disallow bots from scraping data for AI training. However, the company hasn’t said precisely which controls Vision will respect; there are several in use.
If the current trend holds, Copilot Vision may not work on some of the web’s top news sites. But Microsoft said it’s committed to “taking feedback” to allay publishers’ concerns.
“Some of those we’ve collaborated with are third-party publishers, who help us understand how Vision could be used to help people better engage and make decisions on their pages,” the company wrote in the blog. “These inputs will help us design Vision to be more helpful in how it interacts with web pages.”
TechCrunch has an AI-focused newsletter! Sign up here to get it in your inbox every Wednesday.
