Autonomous Visual Studio Code Desktop Automation using Computer Use Agent & PyAutoGUI

Project Overview

The system replicates a developer’s workflow by autonomously launching VS Code, configuring the environment, and generating code via GitHub Copilot Agent mode. This automation is ideal for scenarios where:

1. Codebase is not hosted on GitHub – GitHub Agents on github.com are not applicable.
2. GitHub Codespaces cannot access the code – VS Code in the browser is ruled out.
3. Desktop automation is required – Must work with VS Code desktop client on a local computer.

Key Innovation: CUA Model + PyAutoGUI Integration

The project demonstrates the synergy between CUA model and PyAutoGUI:

• PyAutoGUI: Executes desktop actions (launching apps, typing, clicking).
• CUA Model: Analyzes screenshots and determines next actions.
• Deterministic Outcomes: Enables autonomous detection and correction at each step.

Complete Autonomous Workflow

Phase 1: Environment Setup

1. Launch VS Code desktop application.
2. Navigate to the project workspace directory.
3. Maximize VS Code window.
4. Open integrated PowerShell terminal.

Phase 2: Intelligent Package Management

1. Read requirements.txt.
2. Execute pip install -r requirements.txt.
3. Monitor progress:
   
   • Capture screenshots.
   • CUA analyzes terminal output.
   • Detects “in_progress” vs “complete” states.
   • PyAutoGUI waits for CUA confirmation.

Phase 3: GitHub Copilot Integration

1. Open GitHub Copilot chat panel (Ctrl+Shift+I).
2. Switch to Agent mode.
3. Submit developer prompt.
4. Monitor status:
   
   • Capture screenshots every 3 seconds.
   • CUA checks “Keep” button status.
   • Accept generated code when ready.

CUA Model Decision Points

Package Installation Monitoring

• In Progress: Detects progress bars, downloads.
• Complete: Identifies empty prompt ready for input.

Code Generation Monitoring

• Disabled: Code generation ongoing.
• Enabled: Ready for acceptance

Automation Flow in Action

The following are the sequence of steps that are automated in the flow

1. Launch VS Code → Open project workspace.
2. Open terminal → Execute pip install.
3. CUA monitors → Detects completion.
4. Open Copilot → Activate Agent mode.
5. Submit prompt → Begin code generation.
6. Monitor “Keep” button → Detect enabled state.
7. Accept code → Automation complete.

Real-World Applications

Development Scenarios

• Clone repositories.
• Create branches.
• Generate code.
• Run tests and validations.

Regression Testing Use Cases

• Routine code generation.
• Test case creation.
• Documentation updates.
• Refactoring tasks.

Note: Requires a dedicated computer to run VS Code desktop. Use responsibly with safeguards and monitoring.

Refer to the GitHub Repo here. Refer to the video of this sample in action below:

Limitations:
With pyautogui, the App on which the automation needs to be done has to be on the foreground. It cannot choose a particular app from amongst other apps on the Desktop to monitor the visual state. There are other options to consider for that, which I will cover in another article later