102 lines
3.2 KiB
Markdown
102 lines
3.2 KiB
Markdown
# LLM Tricks
|
|
Finding useful things to do with local LLMs.
|
|
|
|
All scripts work with data within `PRIVATE_DATA/` so that private data can't be
|
|
accidentally committed.
|
|
|
|
### `cosine_similarity.lua`
|
|
Opens `embeddings.json` and creates `similarities.json` with a sorted list of
|
|
comparisons.
|
|
|
|
### `generate_embeddings.lua`
|
|
Opens every file on its whitelist within the `notebook` directory, and generates
|
|
embeddings (placed in `embeddings.json`). When files are too long, it truncates
|
|
them.
|
|
|
|
### `least_similar.lua`
|
|
Opens `similarities.json`, reverses the sort order, and saves it as
|
|
`differences.json`.
|
|
|
|
### `refresh_sources.lua`
|
|
Creates/Maintains a store of chunked data with embeddings based on configurable
|
|
`sources.json`. Example:
|
|
|
|
```json
|
|
{
|
|
"source name":{
|
|
"embedding_model":"qwen3-embedding:0.6b",
|
|
"filters":{
|
|
"blacklist":[".git"],
|
|
"extension_whitelist":["md"]
|
|
},
|
|
"initialize_command":"git clone REMOTE .",
|
|
"max_chunk_size":32768,
|
|
"path":"will be created before initialize_command is run",
|
|
"strip_frontmatter":true,
|
|
"target_chunk_size":3072,
|
|
"refresh_command":"git fetch origin && git reset --hard origin/main"
|
|
}
|
|
}
|
|
```
|
|
|
|
The filters are based on `utility.tree`'s filter options (optional).
|
|
`initialize_command` is only run the first time (optional),
|
|
while `refresh_command` is run each time (optional).
|
|
Commands will be run in the specified `path`.
|
|
`strip_frontmatter` will remove YAML frontmatter (common in Markdown files).
|
|
`embedding_model` and `max_chunk_size` are self-explanatory (optional).
|
|
`target_chunk_size` (optional) allows for finer-grained chunks to be generated
|
|
alongside the maximum overviews for more precise retrieval.
|
|
|
|
The embeddings are stored like so:
|
|
|
|
```json
|
|
{
|
|
"files":{
|
|
"PRIVATE_DATA/source_path/path/to/file.ext":["sha512sum", "another sum"]
|
|
},
|
|
"vectors":{
|
|
"sha512sum":[0.5, 0, 1, -0.5, -1, ...]
|
|
}
|
|
}
|
|
```
|
|
|
|
SHA2 512-bit sums are used to link a text chunk with its embedding. All file
|
|
names reference a list of text chunks so they can handle being too large. Every
|
|
file that is too large for a single chunk has a whole-file embedding calculated
|
|
first (it is the first element of the array), so that if a whole file becomes
|
|
relevant, it can still show up instead of only chunks.
|
|
|
|
### `synopsis_generator.lua`
|
|
Chooses a random file within `notebook`, and generates a novel synopsis from it.
|
|
|
|
Arguments:
|
|
- `refresh_file_list`: Refreshes the cached file list to choose from.
|
|
- `export_ordered_list_of_prompts`: Makes an epub to review generated synopses.
|
|
|
|
## JSON config
|
|
This repo uses my utility library's config system, using a `config.json` file in
|
|
the repo root that is excluded from commits.
|
|
|
|
```json
|
|
{
|
|
"models":{
|
|
"embedding":{
|
|
"model":"qwen3-embedding:0.6b",
|
|
"max_chunk_size":32768
|
|
},
|
|
"initialized_sources":{}
|
|
}
|
|
}
|
|
```
|
|
|
|
`refresh_sources.lua` uses `models` to store default embedding model information
|
|
and `initialized_sources` to store which sources have been initialized.
|
|
|
|
## Tasks
|
|
- [ ] The whitelisting/blacklisting of tree should be in list too.
|
|
- [ ] synopsis_generator should be able to blacklist files it already tried?
|
|
- [ ] `refresh_sources.lua` doesn't check for defined files that don't exist
|
|
anymore, does it?
|
|
- [ ] I think it checks for every other possibility, but this needs checking.
|