Brainstorming: AI-powered context-aware desktop assistant

[Weiwu Zhang]

Hey all, I've been playing around with some ideas lately and wanted to
bounce them off the community.

These days I have llama3 and QWEN running locally on consumer hardware
(think RTX 4060 with 12GB VRAM), and I've produced command line utilities
for it to accept pipe input and output to formatting tools like pandocs and
find it very useful to link AI into commandline works, and I wonder if this
can be pushed to desktop.

I'm wondering if we could soup up our desktop environments with some AI
magic. Here's what I'm thinking:

 1. Context-aware screen reader on steroids: Imagine a GNOME extension that
learns what you're up to and serves up relevant quick actions. Writing an
email? Bam, here's a shortcut to your contact list. Working on project
docs? As you type "while it's not sure if Google supports xxx, we can
conduct experiments" the smart screen reader already displays that "google
does support xxx" with a draw down menu.

2. Smart workspace organizer: An AI that groups your billion browser tabs
into project-specific windows. No more tab hell! Has anyone tried something
like this before? I know there've been attempts at AI assistants, but I'm
talking about something more integrated with the DE itself, because DE has
locally operating AI models. This didn't work before because old desktop
organiser doesn't understand what is in each window and what them the user
is working on. We need something that can read tab titles and operate
locally.

'm not married to any specific implementation, but I reckon this could be a
game-changer for productivity.Anyway, just wanted to throw this out there
and see if it sticks. If anyone has thoughts on potential pitfalls
(privacy, resource hogging, etc.), I'm all ears. Cheers!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/xdg/attachments/20240719/ed9ff6b4/attachment.htm>