Copilot Vision on Windows 11 Transmits Data to Microsoft
Microsoft's Latest AI Innovations in Windows 11
Microsoft is once again leveraging artificial intelligence to enhance the Windows 11 experience, introducing a range of new features that aim to make the operating system more intuitive and powerful. Among these innovations is Copilot Vision, a successor to the controversial Recall feature, which has sparked both excitement and concern among users.
The company claims that AI is transforming how people interact with their PCs, and while some may agree, others are skeptical about whether this change is beneficial. Windows 11 is being positioned as the ultimate home for AI, offering an expansive array of capabilities through Copilot+ PCs. These include exclusive features like Recall (in preview), Click to Do (in preview), improved Windows search, and groundbreaking interactions such as Copilot Vision on Windows.
Understanding Copilot Vision
Copilot Vision is designed to analyze everything users do on their computers by capturing constant screenshots and feeding them to an optical character recognition system and a large language model for analysis. Unlike Recall, which operated locally, Copilot Vision sends data to Microsoft servers for processing. According to a Microsoft spokesperson, user data will not be stored long-term, except for transcripts of conversations with the Copilot assistant, and these are not used for model training or ad personalization.
The vision for Copilot Vision is to eventually become a "true companion," offering a deeper understanding of user goals and providing clear, step-by-step guidance. While screen snooping only occurs when the user activates it during a Copilot session, the feature is designed to be more proactive than previous releases, reminiscent of Clippy from earlier versions of Microsoft Office.
Expanding AI Features Globally
At the time of writing, Copilot Vision was only available in the United States, with plans to roll out to non-European countries soon. This move appears to be a response to the European Union's AI Act. However, even outside the EU, users can expect other AI features in the latest Windows 11 update. One notable addition is the operating system's first "agentic" AI, which can manage system settings on behalf of the user.
This AI, based on a local language model called Mu, is available on Copilot+ systems running on Qualcomm Snapdragon hardware, with support for Intel and AMD coming soon. The agent accepts natural language instructions, such as "connect to my Bluetooth device" or "change my screen resolution to 1920 x 1080," and performs tasks seamlessly.
Challenges and Considerations
Despite its promise, Microsoft has not addressed the unresolved issue of "hallucination" in language models, where the output can be detached from reality. For chatbot-style LLM implementations, this means incorrect answers; for agentic AI, which takes action itself, it can lead to serious consequences.
Microsoft has also introduced new actions for its "Click to Do" feature, available as a preview in countries outside the European Economic Area. These actions include listening to users read passages aloud to improve reading skills, using Copilot's generative-AI capabilities in Microsoft Word, activating an "Immersive Reader" mode, and triggering messages and scheduling requests in Microsoft Teams.
Additional AI Enhancements
The company's AI push extends beyond these features. The Photos app now includes an AI-powered "Relight" feature that simulates up to three virtual light sources added to existing photographs. Even the classic Paint application gets an AI update with a "sticker generator" and "object select" tools. The Snipping Tool also receives a Copilot+ exclusive "perfect screenshot" feature that automatically resizes the capture area based on screen content.
Beyond AI: New Features in Windows 11
The latest Windows 11 update isn't all about AI, though it's hard to ignore the company's focus on this technology. It also marks the end of the classic Blue Screen of Death (BSoD), replacing it with a Black Screen of Death. This new error screen is less verbose and more readable, along with a system called "quick machine recovery" that automates fixes and quickly gets users back to a productive state.
Microsoft is rolling out these features through its "controlled feature rollout" system, ensuring that updates are deployed gradually to prevent widespread issues. Users who want to access the latest updates immediately can enable the "Get the latest updates as soon as they're available" checkbox in Windows Update.
New Hardware Launches
Alongside its software advancements, Microsoft unveiled new hardware, including the Surface Laptop 5G, which features Copilot+ support. This device comes with a 13.8" display, Intel Core Ultra processors, and an optional built-in 5G cellular modem, allowing users to stay continuously connected to Microsoft 365 Copilot and other cloud tools.