Organized
This commit is contained in:
Jason Thistlethwaite
@@ -0,0 +1,290 @@
|
||||
# KarpathyWiki Obsidian Plugin Inspection
|
||||
|
||||
**Date:** 2026-07-09
|
||||
**Plugin:** `karpathywiki` / `green-dalii/obsidian-llm-wiki`
|
||||
**Repo inspected:** `https://github.com/green-dalii/obsidian-llm-wiki`
|
||||
**Fresh clone verification:** commit `0979d50`
|
||||
**License:** Apache-2.0
|
||||
**Version observed:** 1.24.0
|
||||
|
||||
## Why inspected
|
||||
|
||||
Jason observed that the Obsidian Karpathy LLM Wiki plugin is doing much of what the Redmine ticket decomposition process should do: entity/concept extraction, knowledge graph page generation, links, indexes, and conversational query. The question was whether we can trigger it headlessly/CLI or mirror its logic.
|
||||
|
||||
## Repository / structure findings
|
||||
|
||||
The plugin is a compact TypeScript Obsidian plugin.
|
||||
|
||||
Important files:
|
||||
|
||||
```text
|
||||
src/main.ts
|
||||
src/wiki/wiki-engine.ts
|
||||
src/wiki/source-analyzer.ts
|
||||
src/wiki/page-factory.ts
|
||||
src/wiki/prompts/ingestion.ts
|
||||
src/wiki/prompts/generation.ts
|
||||
src/wiki/prompts/merge.ts
|
||||
src/wiki/prompts/lint.ts
|
||||
src/wiki/query-engine/
|
||||
src/core/
|
||||
```
|
||||
|
||||
Package metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "karpathywiki",
|
||||
"version": "1.24.0",
|
||||
"main": "main.js"
|
||||
}
|
||||
```
|
||||
|
||||
Fresh clone verification confirmed `package.json` has no `bin` entry:
|
||||
|
||||
```text
|
||||
PACKAGE {'name': 'karpathywiki', 'version': '1.24.0', 'main': 'main.js', 'bin': None, 'license': 'Apache-2.0'}
|
||||
```
|
||||
|
||||
There is no package `bin` entry and no obvious standalone CLI exposed by the project.
|
||||
|
||||
## Obsidian command entry points
|
||||
|
||||
`src/main.ts` registers Obsidian commands:
|
||||
|
||||
```text
|
||||
ingest-source
|
||||
ingest-folder
|
||||
ingest-multiple-files
|
||||
query-wiki
|
||||
lint-wiki
|
||||
regenerate-index
|
||||
cancel-ingestion
|
||||
ingest-active-file
|
||||
view-ingestion-history
|
||||
recreate-welcome-note
|
||||
```
|
||||
|
||||
These are Obsidian command palette commands, not shell commands.
|
||||
|
||||
If Obsidian is running, it may be possible to trigger commands through Obsidian/Advanced URI style automation, but that would still depend on Obsidian being open and the plugin being loaded. It is not a true headless pipeline.
|
||||
|
||||
## Core engine dependency
|
||||
|
||||
The core class is `WikiEngine` in `src/wiki/wiki-engine.ts`.
|
||||
|
||||
It is separate from the plugin UI, but its constructor requires Obsidian types and APIs:
|
||||
|
||||
```ts
|
||||
constructor(
|
||||
app: App,
|
||||
settings: LLMWikiSettings,
|
||||
getLLMClient: () => LLMClient | null,
|
||||
schemaManager: SchemaManager,
|
||||
onFileWrite?: (path: string) => void,
|
||||
onProgress?: (message: string) => void,
|
||||
onDone?: (report: IngestReport) => void
|
||||
)
|
||||
```
|
||||
|
||||
It uses:
|
||||
|
||||
- `app.vault.read`
|
||||
- `app.vault.getMarkdownFiles`
|
||||
- `app.vault.process`
|
||||
- `app.vault.create`
|
||||
- `app.vault.getAbstractFileByPath`
|
||||
- `app.metadataCache.getFileCache`
|
||||
- `TFile`, `TFolder`, `Notice`, `normalizePath`
|
||||
|
||||
Therefore, the engine cannot be called directly from Node/Python CLI without either:
|
||||
|
||||
1. running inside Obsidian, or
|
||||
2. writing an adapter/shim that emulates enough of Obsidian's `App/Vault/metadataCache/TFile` behavior.
|
||||
|
||||
## Portable logic worth mirroring
|
||||
|
||||
The useful logic is not locked to Obsidian conceptually. It can be mirrored or adapted:
|
||||
|
||||
### Source analysis
|
||||
|
||||
`src/wiki/source-analyzer.ts`:
|
||||
|
||||
- reads source content
|
||||
- checks blank/duplicate content
|
||||
- prompts LLM for structured JSON
|
||||
- extracts entities and concepts
|
||||
- includes aliases
|
||||
- collects verbatim quotes / provenance
|
||||
- identifies contradictions, related pages, key points
|
||||
- normalizes irregular LLM JSON
|
||||
|
||||
The core ingestion prompt is in `src/wiki/prompts/ingestion.ts`.
|
||||
|
||||
Notable extraction output shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_title": "...",
|
||||
"summary": "...",
|
||||
"entities": [
|
||||
{
|
||||
"name": "...",
|
||||
"type": "person|organization|project|product|event|place|other",
|
||||
"aliases": [],
|
||||
"summary": "...",
|
||||
"mentions_in_source": [],
|
||||
"mentions_with_provenance": [],
|
||||
"related_entities": [],
|
||||
"related_concepts": []
|
||||
}
|
||||
],
|
||||
"concepts": [],
|
||||
"contradictions": [],
|
||||
"related_pages": [],
|
||||
"key_points": []
|
||||
}
|
||||
```
|
||||
|
||||
### Page generation
|
||||
|
||||
`src/wiki/page-factory.ts`:
|
||||
|
||||
- slugifies entities/concepts
|
||||
- checks existing pages
|
||||
- resolves duplicates via slug/alias/LLM semantic match
|
||||
- generates entity/concept pages
|
||||
- merges into existing pages
|
||||
- protects reviewed pages
|
||||
- corrects polluted wikilinks
|
||||
- injects provenance/mentions
|
||||
|
||||
### Prompts
|
||||
|
||||
`src/wiki/prompts/generation.ts` includes high-value generation prompts for:
|
||||
|
||||
- entity pages
|
||||
- concept pages
|
||||
- source summary pages
|
||||
- preserving reviewed pages
|
||||
|
||||
### Query / graph retrieval
|
||||
|
||||
The README claims zero-embedding graph retrieval via Personalized PageRank over the wiki link graph. Source references in `WikiEngine` show:
|
||||
|
||||
```ts
|
||||
buildGraphFromContent(...)
|
||||
getOrBuildGraph(...)
|
||||
```
|
||||
|
||||
and query pipeline files exist under:
|
||||
|
||||
```text
|
||||
src/wiki/query-engine/pipeline/
|
||||
```
|
||||
|
||||
This is useful design inspiration, but not required for the first Redmine decomposition tool because gbrain already supplies search/indexing and the vault graph itself supplies wikilinks.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Do **not** depend on Obsidian running for Redmine ticket decomposition.
|
||||
|
||||
Instead:
|
||||
|
||||
1. Mirror the plugin's source-analysis and page-generation logic in our own CLI-oriented tool.
|
||||
2. Reuse its prompt structure and output schema ideas.
|
||||
3. Write directly to the filesystem vault, but outside the plugin-owned `wiki/` tree by default.
|
||||
4. Use gbrain for indexing/search after files are written.
|
||||
5. Keep Redmine contact updates as a separate explicit apply phase.
|
||||
6. Preserve compatibility with Obsidian/KarpathyWiki by using normal Markdown, wikilinks, and frontmatter fields the plugin understands where practical.
|
||||
|
||||
Compatibility rule: the plugin owns `wiki/entities`, `wiki/concepts`, `wiki/sources`, `wiki/index.md`, and `wiki/log.md`. The headless decomposer should default to LDR-owned paths like `Timeline/`, `entities/organizations/`, and `Processes/Redmine/`, linking to `wiki/...` pages when useful but not editing them.
|
||||
|
||||
This gives us the same broad effect without requiring a GUI Obsidian session and without competing with the existing plugin state.
|
||||
|
||||
## Possible implementation paths
|
||||
|
||||
### Path A — Thin TypeScript headless port
|
||||
|
||||
Build a Node/TypeScript CLI in `/home/jason/projects/redmine` or a sibling project that imports/copied logic from the plugin and supplies a fake Obsidian adapter.
|
||||
|
||||
Pros:
|
||||
|
||||
- Closer to upstream code.
|
||||
- Easier to vendor future prompt improvements.
|
||||
- Can reuse zod/AI SDK provider setup.
|
||||
|
||||
Cons:
|
||||
|
||||
- Need to emulate enough of Obsidian `Vault`, `TFile`, metadata cache, and settings.
|
||||
- Adds Node/TypeScript toolchain complexity to Redmine workflow.
|
||||
|
||||
### Path B — Python mirror using existing Redmine tooling
|
||||
|
||||
Implement the same conceptual pipeline in Python alongside `redmine_tickets.py`:
|
||||
|
||||
- source normalization
|
||||
- LLM JSON extraction
|
||||
- entity/concept dedup via filesystem search + slug rules
|
||||
- page generation templates
|
||||
- timeline/entity writers
|
||||
- gbrain sync hooks
|
||||
|
||||
Pros:
|
||||
|
||||
- Fits current Redmine tooling.
|
||||
- Easier to integrate with ticket cache and contact updates.
|
||||
- No Obsidian runtime needed.
|
||||
|
||||
Cons:
|
||||
|
||||
- We do not directly reuse TypeScript implementation.
|
||||
- Need to maintain our own prompt/schema logic.
|
||||
|
||||
### Path C — Obsidian-driven automation
|
||||
|
||||
Use Obsidian command automation / Advanced URI to trigger plugin commands.
|
||||
|
||||
Pros:
|
||||
|
||||
- Minimal code if it works.
|
||||
|
||||
Cons:
|
||||
|
||||
- Requires Obsidian running.
|
||||
- Harder to make deterministic/headless/cron-safe.
|
||||
- Harder to feed Redmine tickets and capture structured contact-update proposals.
|
||||
|
||||
## Best fit for LDR Redmine decomposition
|
||||
|
||||
Path B is the best first implementation path.
|
||||
|
||||
Reasons:
|
||||
|
||||
- Redmine tickets are already fetched/cached in Python.
|
||||
- We need Redmine contact updates and safety gates.
|
||||
- We need Onyx URLs and gbrain sync hooks.
|
||||
- We need headless/cron/agent execution.
|
||||
- We already proved manual note generation format in the decomposition pilot.
|
||||
|
||||
Use the plugin as a reference for:
|
||||
|
||||
- `src/wiki/prompts/ingestion.ts` — source analysis prompt and entity/concept output shape
|
||||
- `src/wiki/prompts/generation.ts` — entity/concept/source page generation prompts
|
||||
- `src/wiki/prompts/merge.ts` — classify-then-route merge strategy for updating existing pages
|
||||
- `src/wiki/prompts/lint.ts` — wiki quality checks and contradiction review ideas
|
||||
- `src/wiki/source-analyzer.ts` — JSON normalization, duplicate/blank source handling, provenance mentions
|
||||
- `src/wiki/page-factory.ts` — slug/alias resolution, cross-type collision handling, merge-vs-append routing
|
||||
- `src/wiki/query-engine/` — graph retrieval ideas for later, not first pass
|
||||
|
||||
Do not try to invoke it via Obsidian as the primary production pipeline; use the source as a design reference for a headless Python mirror.
|
||||
|
||||
## Immediate next action
|
||||
|
||||
Update `ticket_decomposition_knowledge_ingestion_plan.md` so Phase 2 says:
|
||||
|
||||
- use KarpathyWiki plugin prompt/schema as reference
|
||||
- implement a Python headless mirror first
|
||||
- consider TypeScript adapter later only if Python mirror becomes too costly
|
||||
|
||||
Then implement a `decompose --dry-run` engine using the pilot tickets as fixtures.
|
||||
Reference in New Issue
Block a user