| November 16, 2024
OpenAI’s ChatGPT desktop app for macOS has taken a significant step forward, introducing the ability to work seamlessly with popular coding tools. Announced on November 14, 2024, the feature, called “Work with Apps,” enables ChatGPT to access code directly from developer-focused apps such as VS Code, Xcode, TextEdit, Terminal, and iTerm2.
Revolutionizing Workflow for Developers
This new integration eliminates the need for developers to copy and paste code into ChatGPT manually. Instead, when activated, the feature allows ChatGPT to automatically fetch sections of code from the selected app and use them as context for your prompts.
- Supported apps: Coding environments like Xcode and VS Code.
- Automatic context sharing: Up to the last 200 lines of code or content from the active window are sent to ChatGPT.
- Highlight focus: Developers can select specific code snippets for more precise assistance.
Despite this advancement, ChatGPT cannot yet write code directly into these apps, unlike tools like GitHub Copilot or Cursor.
A Step Toward AI Agent Capabilities
While “Work with Apps” is not a fully-fledged AI agent, OpenAI sees it as a foundational feature for more advanced systems. OpenAI’s Alexander Embiricos highlighted its importance, stating, “This idea that ChatGPT understands or can work with all the content you have is a really key building block.”
Currently, the feature is limited to text-based interactions and relies on macOS’s Accessibility API for reading text. However, it cannot process visual elements like images or videos, limiting its scope.
How It Works
In apps like Xcode, ChatGPT can analyze code in real-time. In a demo with TechCrunch, an OpenAI employee demonstrated adding a missing planet to a solar system project in Xcode. After selecting the app tab within ChatGPT, the employee prompted the AI to “add the missing planets,” and the bot generated the required code snippet.
This feature requires:
- A special extension for apps like VS Code.
- Integration with Apple’s screen reader, which has been reliable for most applications over the years.
Expanding the Vision
The “Work with Apps” feature underscores OpenAI’s focus on making AI tools more collaborative and accessible. It marks a deliberate move toward general-purpose AI agents, with reports suggesting OpenAI’s upcoming agent, codenamed “Operator,” could launch in 2025. Operator is expected to compete with tools like Anthropic’s Computer Use and Google’s rumored “Jarvis” agent.
Unlike Anthropic’s approach, which uses screenshots to analyze desktops, OpenAI relies on APIs for more accurate and responsive interaction with apps.
Availability and Future Updates
The feature is currently available to Plus and Teams users on macOS, with plans to roll it out to Enterprise and Edu users in the coming weeks. However, a Windows version remains under development despite Microsoft’s close ties with OpenAI.
Looking ahead, OpenAI plans to extend the feature to other types of apps, particularly those used for writing and text-based tasks.
