Microsoft shipped an experimental desktop app that watches you…
Microsoft shipped an experimental desktop app that watches you perform a task once on screen and reconstructs it into a reusable agent skill — 835 stars and 90 forks in four days.
What is the idea?
Records a screen work session (optionally with spoken narration), hands it to the GitHub Copilot CLI, and reconstructs it as an intent plus ordered steps, then emits a reusable Skill or Automation.
Who would use it, and what do they get?
Nobody yet — and that is the finding. The person this would serve does one fiddly ten-minute computer chore every month (download the statement, rename it, drop it in the right folder; re-key the same form; file the same three attachments) and can neither script it nor describe it precisely enough for an agent — but can always just do it once. What they get is that chore already done next month, with the screen recording never leaving the machine.
Why would anyone bother?
The repeated point-and-click chores ordinary people do monthly are exactly the things they can neither script nor describe precisely enough for an agent — but they can always just do them.
What is missing today?
It is wired to Copilot CLI, Microsoft Scout and Copilot Studio, ships as an Electron source install, and is aimed at enterprise workflow authors. The ordinary-person version — 'show it once, it does it for you from then on' with nothing to configure — does not exist, and nothing exists for the Chinese desktop at all.
What makes this one different?
Treated here as a capability signal, not a business: the opportunity is the consumer wedge — one narrow recurring household chore taught by demonstration, running locally, with the screen recording never leaving the machine. Cold start (Chinese): Xiaohongshu and Bilibili comment sections under 办公自动化 / 重复劳动 tutorials, where people describe the exact monthly ritual they hate. Cold start (English): r/Automate and r/productivity, search 'automate this thing I do every month'.
What is the sharpest thing about it?
It inverts how you teach a computer to do a chore: you do it once while it watches, instead of describing or scripting it. The team is also building on-device detection and redaction of sensitive details in the recording.
What could kill it?
Microsoft is occupying this directly and has the distribution; a demonstration is also a lossy spec, and the maintainers' own issue tracker notes recordings that miss a step produce a broken skill. Forks are 11% of stars, so this is still developer curiosity rather than proven demand.
Who pays, and for what?
Not a business as shipped; the consumer version would be a paid local desktop app.
What is the main risk?
Screen recordings are the most sensitive capture surface there is; anything less than fully local processing is unsellable.
How much work is it?
not a build — logged as a capability signal
Where does the evidence come from?
https://github.com/microsoft/skill-recorder https://github.com/microsoft/skill-recorder/issues/25 https://github.com/microsoft/skill-recorder/issues/32
← Turning your handwriting into a real font went to one photo and… · An AI receptionist that answers a solo business's phone after hours… →