Brainstorming: AI-powered context-aware desktop assistant

[Aleix Pol]

On Fri, Jul 19, 2024 at 4:22 AM Weiwu Zhang <a at colourful.land> wrote:
>
> Hey all, I've been playing around with some ideas lately and wanted to bounce them off the community.
>
> These days I have llama3 and QWEN running locally on consumer hardware (think RTX 4060 with 12GB VRAM), and I've produced command line utilities for it to accept pipe input and output to formatting tools like pandocs and find it very useful to link AI into commandline works, and I wonder if this can be pushed to desktop.
>
> I'm wondering if we could soup up our desktop environments with some AI magic. Here's what I'm thinking:
>
>  1. Context-aware screen reader on steroids: Imagine a GNOME extension that learns what you're up to and serves up relevant quick actions. Writing an email? Bam, here's a shortcut to your contact list. Working on project docs? As you type "while it's not sure if Google supports xxx, we can conduct experiments" the smart screen reader already displays that "google does support xxx" with a draw down menu.
>
> 2. Smart workspace organizer: An AI that groups your billion browser tabs into project-specific windows. No more tab hell! Has anyone tried something like this before? I know there've been attempts at AI assistants, but I'm talking about something more integrated with the DE itself, because DE has locally operating AI models. This didn't work before because old desktop organiser doesn't understand what is in each window and what them the user is working on. We need something that can read tab titles and operate locally.
>
> 'm not married to any specific implementation, but I reckon this could be a game-changer for productivity.Anyway, just wanted to throw this out there and see if it sticks. If anyone has thoughts on potential pitfalls (privacy, resource hogging, etc.), I'm all ears. Cheers!

Sounds like a good idea but is this really the mailing list where to
discuss this? It sounds like something very tied to the shell and the
information management that this one will do. If anything, it could
make sense to standardise by the XDG (i.e. cross-desktop group) which
ways to collaborate on such topics. To do something like that I guess
we first need a couple of implementations brewing.

Best,
Aleix