ldr/redmine
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Organized

Browse Source
This commit is contained in:
Jason Thistlethwaite
2026-10-17 20:50:35 -04:00
parent bd26c8894f
commit 9ee9eb3ec6
130 changed files with 24172 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ticket_decomposition_knowledge_ingestion_plan.md
+788
View File
@@ -0,0 +1,788 @@
# Ticket Decomposition & Knowledge Ingestion Implementation Plan
**Date:** 2026-07-09
**Project repo:** `/home/jason/projects/redmine`
**Primary vault:** `/home/jason/lilaBuild/vaults/obsidian-ldr`
**Primary Redmine project to start:** `customer-service`
**Production Redmine:** `https://rm.ldrprep.com`
**Onyx vault URL prefix:** `https://onyx.int.ldrprep.com/projects/obsidian-ldr/`
## Goal
Build a ticket decomposition process that converts meaningful Redmine tickets into durable organizational memory: timeline event notes, entity/contact notes, Redmine contact enrichment, cross-links, tags, and gbrain-ingestable vault content.
The immediate wedge is Redmine tickets, especially `customer-service`, but the final interface should support:
```text
decompose <target>
```
where `<target>` may be:
- Redmine issue ID / Redmine project query
- email text
- PDF/document file
- pasted text
- cached ticket JSON
## Scope
This plan is about **knowledge ingestion and decomposition**, not spam closure. It should build on the ticket sync substrate, not replace it.
Existing substrate to reuse:
- `/home/jason/projects/redmine/redmine_tickets.py`
- `/home/jason/projects/redmine/redmine_contacts.py`
- Redmine issue API with `include=journals,helpdesk`
- Obsidian LDR vault
- Onyx browser links
- gbrain source/index tooling
---
## Ticket Buckets Worth Decomposing
A ticket should be eligible for decomposition if it appears to fit one or more of these buckets:
### 1. Definite customer ticket with meaningful event
Customer ticket where:
- a decision was made
- a complaint was involved
- a new project started
- a workflow/process implication was exposed
- an account/service/pricing change happened
Examples:
- customer complaint about missing photos, missing package, wrong condition, wrong item
- pricing/service inquiry that caused a meaningful answer or decision
- customer onboarding/project kickoff
- customer-specific exception or policy decision
### 2. Durable purchase receipt
Receipt about a durable item or business asset worth remembering.
Examples:
- equipment
- tools
- technology/hardware
- warehouse/office asset
- anything likely to matter for warranty, accounting, maintenance, or future context
### 3. Regular utility / recurring service receipt or invoice
Receipt/invoice/payment confirmation about a regular service.
Examples:
- utilities
- insurance
- phone/VoIP
- shipping/billing services
- cloud/SaaS subscriptions
- recurring vendors
This may not need rich event notes every time, but should enrich vendor/entity records and optionally create monthly financial/operations timeline entries.
### 4. Partnership / inbound opportunity / buy-from-us request
Not a customer and not a sales person trying to sell us something, but someone proposing a relationship or wanting to buy something **from us**.
Examples:
- buy Gaylords
- buy liquidation
- advertise/affiliate/blog partnership
- ask about partnership or marketplace cooperation
- use LDR as a source/supplier
These should become opportunity/entity notes, not spam.
### 5. Any meaningful operational event
Catch-all for tickets that refer to a meaningful event:
- policy change
- operational incident
- vendor problem
- staff/customer decision
- process drift or known-failure-method recurrence
- new obligation or future follow-up
---
## Non-Goals / Safety Boundaries
1. Do not ingest every Redmine ticket into the vault as a polished note. Use cache/archive for raw data; decompose only meaningful tickets.
2. Do not overwrite existing entity pages destructively.
3. Do not invent facts. Use evidence blocks and confidence levels.
4. Do not send customer-visible Helpdesk replies.
5. Do not update Redmine contacts until the proposed changes are explicit and reviewable.
6. Do not let spam-cleanup closures trigger decomposition unless the ticket also has meaningful non-spam content.
---
## Vault Information Architecture
### Timeline root
Create a new timeline tree:
```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Timeline/
```
Monthly folders:
```text
Timeline/2026-07/
Timeline/2026-08/
```
Monthly index:
```text
Timeline/2026-07/index.md
```
Event notes:
```text
Timeline/2026-07/2026-07-09_customer-name-short-subject.md
```
Filename rules:
- kebab-case subject slug
- no spaces
- prefix `YYYY-MM-DD_`
- keep short but human-readable
Example:
```text
Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry.md
Timeline/2026-07/2026-07-09_u2wear-returns-provider-followup.md
Timeline/2026-07/2026-07-09_cyberstorm-order-return-confirmation.md
```
### Monthly index format
`Timeline/YYYY-MM/index.md` should contain:
```markdown
# YYYY-MM Timeline Index
## Key events
- [[2026-07-09_repackify-gaylord-inquiry]] — Partnership / buy-from-us inquiry. Redmine #41424.
- [[2026-07-09_u2wear-returns-provider-followup]] — Existing/prospective customer follow-up. Redmine #41448.
## Decisions
- ...
## Complaints / service failures
- ...
## Purchases / receipts
- ...
## Partnerships / opportunities
- ...
```
### Entity notes
Use the existing vault `entities/` tree as the default entity home.
Proposed paths:
```text
entities/people/<person-slug>.md
entities/organizations/<organization-slug>.md
entities/concepts/<concept-slug>.md
entities/places/<place-slug>.md
```
If the repo prefers flat `entities/<slug>.md`, the first implementation can create flat notes and later migrate. But the decomposition code should centralize path decisions in one function, e.g. `entity_note_path(entity)`.
### Entity note frontmatter
Minimum frontmatter for Redmine contacts:
```yaml
---
type: organization
name: RePackify
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
tags:
- redmine-contact
- maybe-customer
- partnership-opportunity
aliases:
- RePackify
---
```
For a person:
```yaml
---
type: person
name: Sarah Johnson
organization: [[repackify]]
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/people/sarah-johnson.md
tags:
- redmine-contact
- contact-person
---
```
### Entity body format
```markdown
# Entity Name
## Summary
Brief factual summary with confidence level.
## Redmine links
- Contact: https://rm.ldrprep.com/contacts/4470
- Related tickets:
- https://rm.ldrprep.com/issues/41424
## Timeline
- [[Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry]]
## Known facts
- Fact with source link.
## Open questions
- Question that needs human/agent follow-up.
```
---
## Event Note Format
Each event note should be evidence-first and dialectic.
```markdown
---
type: timeline-event
date: 2026-07-09
source_type: redmine-ticket
redmine_issue_id: 41424
redmine_url: https://rm.ldrprep.com/issues/41424
helpdesk_contact_id: 4470
entities:
- [[repackify]]
tags:
- timeline
- redmine
- partnership-opportunity
- buy-from-us
confidence: medium
---
# 2026-07-09 — RePackify Gaylord Inquiry
## One-line summary
RePackify asked whether LDR has used Gaylord boxes available to purchase.
## Source evidence
- Redmine: https://rm.ldrprep.com/issues/41424
- From: support@repackify.com
- Subject: Gaylord inquiry-RePackify
- Key excerpt:
> We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.
## Classification
- Bucket: partnership / buy-from-us request
- Customer status: not confirmed customer
- Suggested contact tag: maybe-customer or partnership-opportunity
## Dialectic analysis
### Claim
This is not spam; it is a potential inbound business/opportunity ticket.
### Supporting evidence
- The sender wants to buy something from LDR, not sell LDR a generic service.
- The request references a concrete asset class: used Gaylord boxes.
### Counterpoints / uncertainty
- It may be a one-off opportunistic buyer, not a durable customer relationship.
- Needs human judgment on whether LDR wants to sell Gaylords.
### Decision / outcome
- No final business decision captured yet.
- Needs follow-up if LDR wants a liquidation/Gaylord sale process.
## Entities to update
- [[repackify]]
## Redmine contact enrichment proposal
```json
{
"contact_id": 4470,
"add_tags": ["maybe-customer", "partnership-opportunity", "buy-from-us"],
"obsidian_article_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md"
}
```
## Follow-up candidates
- Decide whether LDR sells used Gaylords.
- If yes, define a standard response/process.
```
---
## Dialectic Reasoning Requirements
Each decomposition should include:
1. **Claim** — what seems to be true or meaningful.
2. **Evidence** — exact ticket excerpts, sender, date, Redmine link.
3. **Counterpoints** — why the interpretation might be wrong.
4. **Decision/outcome** — what was decided or what remains undecided.
5. **Recommended graph updates** — timeline note, entity notes, contact updates, tags.
6. **Confidence** — low/medium/high.
This prevents the vault from becoming a pile of confident hallucinated summaries. A small mercy in a cruel world.
---
## Redmine Contact Cross-Linking
### Vault → Redmine
Entity frontmatter should include:
```yaml
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
```
### Redmine → Vault
The Redmine contact should be updated with an Onyx URL, not an `obsidian://` link.
Preferred URL format:
```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/<vault-relative-path>.md
```
Example:
```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```
Implementation detail:
- If a Redmine contact custom field exists for `Obsidian Article`, use `redmine_contacts.py --set-custom-field`.
- If no suitable custom field exists, append a clearly delimited line to contact background/notes, e.g.:
```text
Obsidian article: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```
Do not guess custom field IDs. Discover available custom fields from `get_contact()` payloads first.
---
## `decompose <target>` Architecture
### KarpathyWiki plugin reference
Jason identified the Obsidian `karpathywiki` plugin as a close match for the desired behavior. Inspection notes are saved at:
```text
/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md
```
Finding: the plugin does not expose a true standalone CLI. It registers Obsidian commands and its `WikiEngine` depends on Obsidian `App/Vault/TFile/metadataCache` APIs. However, its logic is highly relevant and portable as a design reference:
- source analysis prompt / JSON schema
- entity vs concept decision tree
- alias handling
- duplicate/semantic entity resolution
- entity/concept/source page generation prompts
- contradiction and related-page fields
- graph-based query/retrieval ideas
Recommended implementation path for Redmine decomposition: build a Python headless mirror integrated with `redmine_tickets.py`, using KarpathyWiki's prompts/schema as reference. Consider a TypeScript adapter later only if maintaining a Python mirror becomes too costly.
### Compatibility contract with the Obsidian KarpathyWiki plugin
The LDR vault already contains a plugin-managed wiki tree:
```text
wiki/entities/
wiki/concepts/
wiki/sources/
wiki/index.md
wiki/log.md
```
The headless Redmine decomposer must be a **companion**, not a competing owner of that tree.
Default rules:
1. Do **not** write into `wiki/` by default.
2. Do **not** edit `wiki/index.md` or `wiki/log.md`; those are plugin-owned.
3. Do **not** overwrite or reformat plugin-generated `wiki/entities/*`, `wiki/concepts/*`, or `wiki/sources/*` pages.
4. Write Redmine-derived timeline/event artifacts into LDR-owned paths:
- `Timeline/YYYY-MM/...`
- `entities/organizations/...`
- `entities/people/...`
- `Processes/Redmine/...`
5. Use Obsidian-compatible wikilinks so plugin-generated pages and headless-generated pages can link to each other.
6. If linking to plugin pages, use explicit paths such as `[[wiki/concepts/returns-management|returns management]]` rather than ambiguous bare names.
7. Use plugin-compatible frontmatter concepts where practical (`type`, `created`, `updated`, `tags`, `sources`, `confidence`, `contradictions`) but allow LDR-specific types such as `timeline-event` outside `wiki/`.
8. If we ever want the plugin itself to ingest Redmine-derived material, create a deliberate handoff/export file outside `wiki/` first and let the plugin ingest it through Obsidian; do not silently write into plugin-owned `sources/`.
This keeps Obsidian’s KarpathyWiki plugin useful in the desktop vault while allowing Redmine decomposition to run headlessly/cron/agent-side without fighting plugin state.
Concrete upstream files to mirror first:
- `src/wiki/prompts/ingestion.ts` — extraction prompt / JSON schema
- `src/wiki/prompts/generation.ts` — page generation prompts
- `src/wiki/prompts/merge.ts` — merge/update triage prompts
- `src/wiki/source-analyzer.ts` — source analysis normalization/provenance behavior
- `src/wiki/page-factory.ts` — slug/alias/page merge behavior
The first Python implementation should not attempt full parity. It should clone only the useful minimum:
1. analyze source into JSON entities/concepts/key points/contradictions;
2. resolve entity slugs against existing vault files;
3. generate or update entity/event notes;
4. produce a reviewable contact-update proposal;
5. leave graph retrieval to Obsidian/gbrain for now.
### Input abstraction
Create a common normalized input model:
```python
@dataclass
class DecompositionSource:
source_type: str # redmine-ticket, email, pdf, text, file
source_id: str | None
title: str
body: str
date: str | None
sender: str | None
redmine_issue_id: int | None
redmine_contact_id: int | None
attachments: list[Path]
metadata: dict[str, Any]
```
Adapters:
- `from_redmine_issue(issue_json)`
- `from_email_text(raw_text)`
- `from_pdf(path)`
- `from_plain_text(text)`
- `from_file(path)`
### Decomposition output model
```python
@dataclass
class DecompositionPlan:
source: DecompositionSource
buckets: list[str]
timeline_events: list[TimelineEvent]
entities: list[EntityUpdate]
redmine_contact_updates: list[ContactUpdate]
tags: list[str]
gbrain_actions: list[GBrainAction]
confidence: str
review_required: bool
```
### Apply phases
1. `decompose --dry-run` — produce JSON/Markdown plan only.
2. `decompose --write-vault` — write timeline/entity notes.
3. `decompose --update-redmine-contact` — update Redmine contact with Onyx article link/tags.
4. `decompose --gbrain-sync` — run/queue gbrain sync/index.
Each phase should be separately idempotent.
---
## CLI Proposal
### Decompose a Redmine ticket
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
decompose \
--issue-id 41424 \
--dry-run
```
### Decompose from cached tickets matching a bucket
```bash
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_prod_meaningful_20260709 \
--project customer-service \
decompose-cached \
--bucket partnership-opportunity \
--dry-run
```
### Decompose arbitrary text
```bash
./decompose.py text \
--title "Supplier proposal" \
--input /tmp/message.txt \
--dry-run
```
### Write vault notes after review
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
decompose \
--issue-id 41424 \
--write-vault
```
---
## gbrain Integration
Initial approach should be simple and explicit:
1. Write/update Obsidian notes.
2. Run or queue gbrain sync for the LDR vault source.
3. Verify that the new notes are searchable.
Commands need source discovery first:
```bash
gbrain sources list --json
gbrain stats
```
Then likely:
```bash
gbrain sync --source obsidian-ldr --no-embed --json
gbrain embed --stale
gbrain extract --stale
gbrain search "RePackify Gaylord inquiry"
```
Do not assume the exact source ID is `obsidian-ldr` until verified by `gbrain sources list --json`.
---
## Implementation Phases
### Phase 0 — Immediate schema/docs checkpoint
**Goal:** Preserve this architecture and make future implementation straightforward.
Tasks:
1. Save this plan.
2. Link it from Redmine `HANDOFF.md`.
3. Optionally create an Obsidian process note at:
```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md
```
Definition of done:
- Plan exists.
- Handoff links it.
- Next worker can start implementation without transcript context.
### Phase 1 — Manual pilot decomposition for 3 tickets
**Goal:** Prove note structure and cross-linking manually before automating.
Suggested pilot tickets:
- RePackify Gaylord inquiry (`#41424`)
- u2wear returns-provider follow-up (`#41448`)
- Snuggle Puppy / 3PL & FBA Returns (`#41504`)
Tasks:
1. Fetch each ticket with `include=journals,helpdesk`.
2. Write one timeline event note per ticket.
3. Create/update entity notes.
4. Prepare Redmine contact update payloads, dry-run first.
5. Review Onyx links.
6. Optionally run gbrain sync/search smoke test.
Definition of done:
- 3 timeline notes exist.
- Monthly index links them.
- Entity notes exist or are updated.
- Redmine contact update payloads are reviewed.
- No customer-visible Redmine replies are sent.
### Phase 2 — Implement decomposition dry-run engine
**Goal:** Generate decomposition plans from cached Redmine tickets without writing vault/Redmine.
**Status — 2026-07-10:** First deterministic slice implemented in `redmine_tickets.py decompose`. It supports `--issue-id`, `--input-json`, and `--format json|markdown`. It produces dry-run plans for the three pilot fixtures and respects the KarpathyWiki compatibility boundary by planning writes outside `wiki/`. A KarpathyWiki-style structured extraction layer now cleans Helpdesk/forwarded-message noise and emits `analysis.summary`, `analysis.entities`, `analysis.concepts`, `analysis.key_points`, `analysis.contradictions`, `analysis.related_pages`, and `analysis.clean_text`. Tests pass (`23` tests). Current limitation: extraction is still deterministic/rule-backed rather than LLM-backed; next slice should add prompt-driven JSON extraction for broader ticket types before vault-write mode.
Files:
- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Add tests: `/home/jason/projects/redmine/tests/test_ticket_decomposition.py` or extend `test_redmine_tickets.py`
Tasks:
1. Add `DecompositionSource` and `DecompositionPlan` models.
2. Add `source_from_redmine_issue(issue)`.
3. Add bucket classifier:
- customer-decision
- customer-complaint
- customer-new-project
- durable-purchase-receipt
- recurring-utility-receipt
- partnership-buy-from-us
- meaningful-event
4. Add dialectic plan generator.
5. Add JSON/Markdown dry-run output.
6. Add KarpathyWiki-inspired source cleanup and LLM JSON extraction for higher-quality summaries/entities.
Definition of done:
- Unit tests cover each bucket.
- Dry-run output includes source evidence, entities, proposed notes, and proposed contact updates.
- Output quality is comparable to the hand-written pilot notes for the first three fixtures.
### Phase 3 — Implement vault writer
**Goal:** Write idempotent timeline/entity notes.
Tasks:
1. Add path builder for `Timeline/YYYY-MM/YYYY-MM-DD_slug.md`.
2. Add monthly index updater.
3. Add entity path resolver.
4. Add frontmatter merge/update helper.
5. Add idempotency marker using source Redmine issue ID in frontmatter.
Definition of done:
- Re-running same decomposition does not duplicate notes or index bullets.
- Existing entity notes are updated, not overwritten.
### Phase 4 — Implement Redmine contact updater
**Goal:** Update Redmine contacts with Onyx article links and useful tags.
Tasks:
1. Discover contact custom fields.
2. Decide field for Obsidian/Onyx article URL.
3. Add dry-run contact update report.
4. Add `--apply-contact-updates` using `rebot` only when approved.
5. Verify contact after update.
Definition of done:
- Contact has vault article URL.
- Vault article has `redmine_contact_id`.
- Writes are impersonated appropriately.
### Phase 5 — gbrain sync/index integration
**Goal:** Make decomposed notes searchable in gbrain.
Tasks:
1. Discover LDR vault gbrain source ID.
2. Add optional command wrapper or instruction to run sync/embed/extract.
3. Verify search for a newly created entity/event.
Definition of done:
- gbrain search finds the decomposed event/entity.
### Phase 6 — Generalize `decompose <target>`
**Goal:** Support non-ticket inputs with the same decomposition outputs.
Tasks:
1. Add plain-text adapter.
2. Add email adapter.
3. Add PDF adapter using existing OCR/document pipeline if needed.
4. Keep output model the same.
Definition of done:
- Same event/entity/timeline/contact note logic works for non-ticket sources.
---
## Recommended First Execution Step
Do **Phase 1 manual pilot** before writing more automation.
Use the 3 known “meaningful but not spam” tickets from the recent spam-cleanup work:
```text
#41424 RePackify Gaylord inquiry
#41448 u2wear returns-provider follow-up
#41504 Snuggle Puppy / 3PL & FBA Returns
```
That gives us immediate signal on:
- note format quality
- entity matching
- Onyx link usefulness
- Redmine contact update shape
- gbrain ingestion friction
Only after that should we automate the decomposition engine. Otherwise we risk building a very elegant machine for generating annoying notes. We have enough of those already.