# LLM Tricks
Finding useful things to do with local LLMs.

All scripts work with data within `PRIVATE_DATA/` so that private data can't be
accidentally committed.

### `cosine_similarity.lua`
Opens `embeddings.json` and creates `similarities.json` with a sorted list of
comparisons.

### `generate_embeddings.lua`
Opens every file on its whitelist within the `notebook` directory, and generates
embeddings (placed in `embeddings.json`). When files are too long, it truncates
them.

### `least_similar.lua`
Opens `similarities.json`, reverses the sort order, and saves it as
`differences.json`.

### `refresh_sources.lua`
Creates/Maintains a store of chunked data with embeddings based on configurable
sources. Example:

```json
{
  "source name":{
    "filters":{
      "blacklist":[".git"],
      "extension_whitelist":["md"]
    },
    "initialize_command":"git clone REMOTE .",
    "path":"will be created before initialize_command is run",
    "strip_frontmatter":true,
    "refresh_command":"git reset --hard origin/main"
  }
}
```

The filters are based on `utility.tree`'s filter options (optional).
`initialize_command` is only run the first time (optional),
while `refresh_command` is run each time (optional).
Commands will be run in the specified `path`.
`strip_frontmatter` will remove YAML frontmatter (common in Markdown files).

The embeddings are stored like so:

```json
{
  "files":{
    "PRIVATE_DATA/source_path/path/to/file.ext":["sha512sum", "another sum"]
  },
  "vectors":{
    "sha512sum":[0.5, 0, 1, -0.5, -1, ...]
  }
}
```

TODO

### `synopsis_generator.lua`
Chooses a random file within `notebook`, and generates a novel synopsis from it.

Arguments:
- `refresh_file_list`: Refreshes the cached file list to choose from.
- `export_ordered_list_of_prompts`: Makes an epub to review generated synopses.

## Tasks
- [ ] The whitelisting/blacklisting of tree should be in list too.
- [ ] synopsis_generator should be able to blacklist files it already tried?