diff --git a/ReadMe.md b/ReadMe.md index 6c4beb6..b889130 100644 --- a/ReadMe.md +++ b/ReadMe.md @@ -4,20 +4,44 @@ Finding useful things to do with local LLMs. All scripts work with data within `PRIVATE_DATA/` so that private data can't be accidentally committed. -###### `cosine_similarity.lua` +### `cosine_similarity.lua` Opens `embeddings.json` and creates `similarities.json` with a sorted list of comparisons. -###### `generate_embeddings.lua` +### `generate_embeddings.lua` Opens every file on its whitelist within the `notebook` directory, and generates embeddings (placed in `embeddings.json`). When files are too long, it truncates them. -###### `least_similar.lua` +### `least_similar.lua` Opens `similarities.json`, reverses the sort order, and saves it as `differences.json`. -###### `synopsis_generator.lua` +### `refresh_sources.lua` +Creates/Maintains a store of chunked data with embeddings based on configurable +sources. Example: + +```json +{ + "source name (unused by the script)":{ + "filters":{ + "blacklist":".git", + "extension_whitelist":"md" + }, + "initialize_command":"git clone REMOTE", + "path":"will be created before initialize_command is run", + "strip_frontmatter":true, + "refresh_command":"git reset --hard origin/main" + } +} +``` + +The filters are based on `utility.tree`'s filter options. `initialize_command` +is only run the first time (optional), while `refresh_command` is run each time +(optional). Commands will be run in the specified `path`. `strip_frontmatter` is +for removing YAML frontmatter (common in Markdown files) if truthy. + +### `synopsis_generator.lua` Chooses a random file within `notebook`, and generates a novel synopsis from it. Arguments: