When refresh_sources starts, it needs to check all files in memory store to find missing vectors #5

Closed
opened 2026-09-15 19:54:32 -06:00 by tangent · 2 comments
Owner

This is because an error during a run "pollutes" the database because the embeddings don't get saved.

This also means file names won't be associated, which is also a problem.

I guess I need to make an option for whether to allow these source-less items to be kept. (Also, as it is checking files, if it can find a broken association, IT SHOULD ADD IT BACK.)

This is because an error during a run "pollutes" the database because the embeddings don't get saved. This also means file names won't be associated, which is also a problem. I guess I need to make an option for whether to allow these source-less items to be kept. (Also, as it is checking files, if it can find a broken association, IT SHOULD ADD IT BACK.)
Author
Owner

The difficulty is when a file needs to be chunked, this will try to process the whole file and chunks in a non deterministic manner, causing duplication of work.

The result will be shockingly similar to the refresh_sources function itself.

The difficulty is when a file needs to be chunked, this will try to process the whole file and chunks in a non deterministic manner, causing duplication of work. The result will be shockingly similar to the refresh_sources function itself.
Author
Owner

This is done in 470901d9b9.

This is done in 470901d9b9.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: tools/llm-tricks#5