redmine/docs/ticket_decomposition_knowledge_ingestion_plan.md

# Ticket Decomposition & Knowledge Ingestion Implementation Plan

**Date:** 2026-07-09  
**Project repo:** `/home/jason/projects/redmine`  
**Primary vault:** `/home/jason/lilaBuild/vaults/obsidian-ldr`  
**Primary Redmine project to start:** `customer-service`  
**Production Redmine:** `https://rm.ldrprep.com`  
**Onyx vault URL prefix:** `https://onyx.int.ldrprep.com/projects/obsidian-ldr/`  

## Goal

Build a ticket decomposition process that converts meaningful Redmine tickets into durable organizational memory: timeline event notes, entity/contact notes, Redmine contact enrichment, cross-links, tags, and gbrain-ingestable vault content.

The immediate wedge is Redmine tickets, especially `customer-service`, but the final interface should support:

```text
decompose <target>
```

where `<target>` may be:

- Redmine issue ID / Redmine project query
- email text
- PDF/document file
- pasted text
- cached ticket JSON

## Scope

This plan is about **knowledge ingestion and decomposition**, not spam closure. It should build on the ticket sync substrate, not replace it.

Existing substrate to reuse:

- `/home/jason/projects/redmine/redmine_tickets.py`
- `/home/jason/projects/redmine/redmine_contacts.py`
- Redmine issue API with `include=journals,helpdesk`
- Obsidian LDR vault
- Onyx browser links
- gbrain source/index tooling

---

## Ticket Buckets Worth Decomposing

A ticket should be eligible for decomposition if it appears to fit one or more of these buckets:

### 1. Definite customer ticket with meaningful event

Customer ticket where:

- a decision was made
- a complaint was involved
- a new project started
- a workflow/process implication was exposed
- an account/service/pricing change happened

Examples:

- customer complaint about missing photos, missing package, wrong condition, wrong item
- pricing/service inquiry that caused a meaningful answer or decision
- customer onboarding/project kickoff
- customer-specific exception or policy decision

### 2. Durable purchase receipt

Receipt about a durable item or business asset worth remembering.

Examples:

- equipment
- tools
- technology/hardware
- warehouse/office asset
- anything likely to matter for warranty, accounting, maintenance, or future context

### 3. Regular utility / recurring service receipt or invoice

Receipt/invoice/payment confirmation about a regular service.

Examples:

- utilities
- insurance
- phone/VoIP
- shipping/billing services
- cloud/SaaS subscriptions
- recurring vendors

This may not need rich event notes every time, but should enrich vendor/entity records and optionally create monthly financial/operations timeline entries.

### 4. Partnership / inbound opportunity / buy-from-us request

Not a customer and not a sales person trying to sell us something, but someone proposing a relationship or wanting to buy something **from us**.

Examples:

- buy Gaylords
- buy liquidation
- advertise/affiliate/blog partnership
- ask about partnership or marketplace cooperation
- use LDR as a source/supplier

These should become opportunity/entity notes, not spam.

### 5. Any meaningful operational event

Catch-all for tickets that refer to a meaningful event:

- policy change
- operational incident
- vendor problem
- staff/customer decision
- process drift or known-failure-method recurrence
- new obligation or future follow-up

---

## Non-Goals / Safety Boundaries

1. Do not ingest every Redmine ticket into the vault as a polished note. Use cache/archive for raw data; decompose only meaningful tickets.
2. Do not overwrite existing entity pages destructively.
3. Do not invent facts. Use evidence blocks and confidence levels.
4. Do not send customer-visible Helpdesk replies.
5. Do not update Redmine contacts until the proposed changes are explicit and reviewable.
6. Do not let spam-cleanup closures trigger decomposition unless the ticket also has meaningful non-spam content.

---

## Vault Information Architecture

### Timeline root

Create a new timeline tree:

```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Timeline/
```

Monthly folders:

```text
Timeline/2026-07/
Timeline/2026-08/
```

Monthly index:

```text
Timeline/2026-07/index.md
```

Event notes:

```text
Timeline/2026-07/2026-07-09_customer-name-short-subject.md
```

Filename rules:

- kebab-case subject slug
- no spaces
- prefix `YYYY-MM-DD_`
- keep short but human-readable

Example:

```text
Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry.md
Timeline/2026-07/2026-07-09_u2wear-returns-provider-followup.md
Timeline/2026-07/2026-07-09_cyberstorm-order-return-confirmation.md
```

### Monthly index format

`Timeline/YYYY-MM/index.md` should contain:

```markdown
# YYYY-MM Timeline Index

## Key events

- [[2026-07-09_repackify-gaylord-inquiry]] — Partnership / buy-from-us inquiry. Redmine #41424.
- [[2026-07-09_u2wear-returns-provider-followup]] — Existing/prospective customer follow-up. Redmine #41448.

## Decisions

- ...

## Complaints / service failures

- ...

## Purchases / receipts

- ...

## Partnerships / opportunities

- ...
```

### Entity notes

Use the existing vault `entities/` tree as the default entity home.

Proposed paths:

```text
entities/people/<person-slug>.md
entities/organizations/<organization-slug>.md
entities/concepts/<concept-slug>.md
entities/places/<place-slug>.md
```

If the repo prefers flat `entities/<slug>.md`, the first implementation can create flat notes and later migrate. But the decomposition code should centralize path decisions in one function, e.g. `entity_note_path(entity)`.

### Entity note frontmatter

Minimum frontmatter for Redmine contacts:

```yaml
---
type: organization
name: RePackify
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
tags:
  - redmine-contact
  - maybe-customer
  - partnership-opportunity
aliases:
  - RePackify
---
```

For a person:

```yaml
---
type: person
name: Sarah Johnson
organization: [[repackify]]
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/people/sarah-johnson.md
tags:
  - redmine-contact
  - contact-person
---
```

### Entity body format

```markdown
# Entity Name

## Summary

Brief factual summary with confidence level.

## Redmine links

- Contact: https://rm.ldrprep.com/contacts/4470
- Related tickets:
  - https://rm.ldrprep.com/issues/41424

## Timeline

- [[Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry]]

## Known facts

- Fact with source link.

## Open questions

- Question that needs human/agent follow-up.
```

---

## Event Note Format

Each event note should be evidence-first and dialectic.

```markdown
---
type: timeline-event
date: 2026-07-09
source_type: redmine-ticket
redmine_issue_id: 41424
redmine_url: https://rm.ldrprep.com/issues/41424
helpdesk_contact_id: 4470
entities:
  - [[repackify]]
tags:
  - timeline
  - redmine
  - partnership-opportunity
  - buy-from-us
confidence: medium
---

# 2026-07-09 — RePackify Gaylord Inquiry

## One-line summary

RePackify asked whether LDR has used Gaylord boxes available to purchase.

## Source evidence

- Redmine: https://rm.ldrprep.com/issues/41424
- From: support@repackify.com
- Subject: Gaylord inquiry-RePackify
- Key excerpt:
  > We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.

## Classification

- Bucket: partnership / buy-from-us request
- Customer status: not confirmed customer
- Suggested contact tag: maybe-customer or partnership-opportunity

## Dialectic analysis

### Claim

This is not spam; it is a potential inbound business/opportunity ticket.

### Supporting evidence

- The sender wants to buy something from LDR, not sell LDR a generic service.
- The request references a concrete asset class: used Gaylord boxes.

### Counterpoints / uncertainty

- It may be a one-off opportunistic buyer, not a durable customer relationship.
- Needs human judgment on whether LDR wants to sell Gaylords.

### Decision / outcome

- No final business decision captured yet.
- Needs follow-up if LDR wants a liquidation/Gaylord sale process.

## Entities to update

- [[repackify]]

## Redmine contact enrichment proposal

```json
{
  "contact_id": 4470,
  "add_tags": ["maybe-customer", "partnership-opportunity", "buy-from-us"],
  "obsidian_article_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md"
}
```

## Follow-up candidates

- Decide whether LDR sells used Gaylords.
- If yes, define a standard response/process.
```

---

## Dialectic Reasoning Requirements

Each decomposition should include:

1. **Claim** — what seems to be true or meaningful.
2. **Evidence** — exact ticket excerpts, sender, date, Redmine link.
3. **Counterpoints** — why the interpretation might be wrong.
4. **Decision/outcome** — what was decided or what remains undecided.
5. **Recommended graph updates** — timeline note, entity notes, contact updates, tags.
6. **Confidence** — low/medium/high.

This prevents the vault from becoming a pile of confident hallucinated summaries. A small mercy in a cruel world.

---

## Redmine Contact Cross-Linking

### Vault → Redmine

Entity frontmatter should include:

```yaml
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
```

### Redmine → Vault

The Redmine contact should be updated with an Onyx URL, not an `obsidian://` link.

Preferred URL format:

```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/<vault-relative-path>.md
```

Example:

```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```

Implementation detail:

- If a Redmine contact custom field exists for `Obsidian Article`, use `redmine_contacts.py --set-custom-field`.
- If no suitable custom field exists, append a clearly delimited line to contact background/notes, e.g.:

```text
Obsidian article: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```

Do not guess custom field IDs. Discover available custom fields from `get_contact()` payloads first.

---

## `decompose <target>` Architecture

### KarpathyWiki plugin reference

Jason identified the Obsidian `karpathywiki` plugin as a close match for the desired behavior. Inspection notes are saved at:

```text
/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md
```

Finding: the plugin does not expose a true standalone CLI. It registers Obsidian commands and its `WikiEngine` depends on Obsidian `App/Vault/TFile/metadataCache` APIs. However, its logic is highly relevant and portable as a design reference:

- source analysis prompt / JSON schema
- entity vs concept decision tree
- alias handling
- duplicate/semantic entity resolution
- entity/concept/source page generation prompts
- contradiction and related-page fields
- graph-based query/retrieval ideas

Recommended implementation path for Redmine decomposition: build a Python headless mirror integrated with `redmine_tickets.py`, using KarpathyWiki's prompts/schema as reference. Consider a TypeScript adapter later only if maintaining a Python mirror becomes too costly.

### Compatibility contract with the Obsidian KarpathyWiki plugin

The LDR vault already contains a plugin-managed wiki tree:

```text
wiki/entities/
wiki/concepts/
wiki/sources/
wiki/index.md
wiki/log.md
```

The headless Redmine decomposer must be a **companion**, not a competing owner of that tree.

Default rules:

1. Do **not** write into `wiki/` by default.
2. Do **not** edit `wiki/index.md` or `wiki/log.md`; those are plugin-owned.
3. Do **not** overwrite or reformat plugin-generated `wiki/entities/*`, `wiki/concepts/*`, or `wiki/sources/*` pages.
4. Write Redmine-derived timeline/event artifacts into LDR-owned paths:
   - `Timeline/YYYY-MM/...`
   - `entities/organizations/...`
   - `entities/people/...`
   - `Processes/Redmine/...`
5. Use Obsidian-compatible wikilinks so plugin-generated pages and headless-generated pages can link to each other.
6. If linking to plugin pages, use explicit paths such as `[[wiki/concepts/returns-management|returns management]]` rather than ambiguous bare names.
7. Use plugin-compatible frontmatter concepts where practical (`type`, `created`, `updated`, `tags`, `sources`, `confidence`, `contradictions`) but allow LDR-specific types such as `timeline-event` outside `wiki/`.
8. If we ever want the plugin itself to ingest Redmine-derived material, create a deliberate handoff/export file outside `wiki/` first and let the plugin ingest it through Obsidian; do not silently write into plugin-owned `sources/`.

This keeps Obsidian’s KarpathyWiki plugin useful in the desktop vault while allowing Redmine decomposition to run headlessly/cron/agent-side without fighting plugin state.

Concrete upstream files to mirror first:

- `src/wiki/prompts/ingestion.ts` — extraction prompt / JSON schema
- `src/wiki/prompts/generation.ts` — page generation prompts
- `src/wiki/prompts/merge.ts` — merge/update triage prompts
- `src/wiki/source-analyzer.ts` — source analysis normalization/provenance behavior
- `src/wiki/page-factory.ts` — slug/alias/page merge behavior

The first Python implementation should not attempt full parity. It should clone only the useful minimum:

1. analyze source into JSON entities/concepts/key points/contradictions;
2. resolve entity slugs against existing vault files;
3. generate or update entity/event notes;
4. produce a reviewable contact-update proposal;
5. leave graph retrieval to Obsidian/gbrain for now.

### Input abstraction

Create a common normalized input model:

```python
@dataclass
class DecompositionSource:
    source_type: str  # redmine-ticket, email, pdf, text, file
    source_id: str | None
    title: str
    body: str
    date: str | None
    sender: str | None
    redmine_issue_id: int | None
    redmine_contact_id: int | None
    attachments: list[Path]
    metadata: dict[str, Any]
```

Adapters:

- `from_redmine_issue(issue_json)`
- `from_email_text(raw_text)`
- `from_pdf(path)`
- `from_plain_text(text)`
- `from_file(path)`

### Decomposition output model

```python
@dataclass
class DecompositionPlan:
    source: DecompositionSource
    buckets: list[str]
    timeline_events: list[TimelineEvent]
    entities: list[EntityUpdate]
    redmine_contact_updates: list[ContactUpdate]
    tags: list[str]
    gbrain_actions: list[GBrainAction]
    confidence: str
    review_required: bool
```

### Apply phases

1. `decompose --dry-run` — produce JSON/Markdown plan only.
2. `decompose --write-vault` — write timeline/entity notes.
3. `decompose --update-redmine-contact` — update Redmine contact with Onyx article link/tags.
4. `decompose --gbrain-sync` — run/queue gbrain sync/index.

Each phase should be separately idempotent.

---

## CLI Proposal

### Decompose a Redmine ticket

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  decompose \
  --issue-id 41424 \
  --dry-run
```

### Decompose from cached tickets matching a bucket

```bash
./redmine_tickets.py \
  --cache-dir .cache/redmine_tickets_prod_meaningful_20260709 \
  --project customer-service \
  decompose-cached \
  --bucket partnership-opportunity \
  --dry-run
```

### Decompose arbitrary text

```bash
./decompose.py text \
  --title "Supplier proposal" \
  --input /tmp/message.txt \
  --dry-run
```

### Write vault notes after review

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  decompose \
  --issue-id 41424 \
  --write-vault
```

---

## gbrain Integration

Initial approach should be simple and explicit:

1. Write/update Obsidian notes.
2. Run or queue gbrain sync for the LDR vault source.
3. Verify that the new notes are searchable.

Commands need source discovery first:

```bash
gbrain sources list --json
gbrain stats
```

Then likely:

```bash
gbrain sync --source obsidian-ldr --no-embed --json
gbrain embed --stale
gbrain extract --stale
gbrain search "RePackify Gaylord inquiry"
```

Do not assume the exact source ID is `obsidian-ldr` until verified by `gbrain sources list --json`.

---

## Implementation Phases

### Phase 0 — Immediate schema/docs checkpoint

**Goal:** Preserve this architecture and make future implementation straightforward.

Tasks:

1. Save this plan.
2. Link it from Redmine `HANDOFF.md`.
3. Optionally create an Obsidian process note at:

```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md
```

Definition of done:

- Plan exists.
- Handoff links it.
- Next worker can start implementation without transcript context.

### Phase 1 — Manual pilot decomposition for 3 tickets

**Goal:** Prove note structure and cross-linking manually before automating.

Suggested pilot tickets:

- RePackify Gaylord inquiry (`#41424`)
- u2wear returns-provider follow-up (`#41448`)
- Snuggle Puppy / 3PL & FBA Returns (`#41504`)

Tasks:

1. Fetch each ticket with `include=journals,helpdesk`.
2. Write one timeline event note per ticket.
3. Create/update entity notes.
4. Prepare Redmine contact update payloads, dry-run first.
5. Review Onyx links.
6. Optionally run gbrain sync/search smoke test.

Definition of done:

- 3 timeline notes exist.
- Monthly index links them.
- Entity notes exist or are updated.
- Redmine contact update payloads are reviewed.
- No customer-visible Redmine replies are sent.

### Phase 2 — Implement decomposition dry-run engine

**Goal:** Generate decomposition plans from cached Redmine tickets without writing vault/Redmine.

**Status — 2026-07-10:** First deterministic slice implemented in `redmine_tickets.py decompose`. It supports `--issue-id`, `--input-json`, and `--format json|markdown`. It produces dry-run plans for the three pilot fixtures and respects the KarpathyWiki compatibility boundary by planning writes outside `wiki/`. A KarpathyWiki-style structured extraction layer now cleans Helpdesk/forwarded-message noise and emits `analysis.summary`, `analysis.entities`, `analysis.concepts`, `analysis.key_points`, `analysis.contradictions`, `analysis.related_pages`, and `analysis.clean_text`. Tests pass (`23` tests). Current limitation: extraction is still deterministic/rule-backed rather than LLM-backed; next slice should add prompt-driven JSON extraction for broader ticket types before vault-write mode.

Files:

- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Add tests: `/home/jason/projects/redmine/tests/test_ticket_decomposition.py` or extend `test_redmine_tickets.py`

Tasks:

1. Add `DecompositionSource` and `DecompositionPlan` models.
2. Add `source_from_redmine_issue(issue)`.
3. Add bucket classifier:
   - customer-decision
   - customer-complaint
   - customer-new-project
   - durable-purchase-receipt
   - recurring-utility-receipt
   - partnership-buy-from-us
   - meaningful-event
4. Add dialectic plan generator.
5. Add JSON/Markdown dry-run output.
6. Add KarpathyWiki-inspired source cleanup and LLM JSON extraction for higher-quality summaries/entities.

Definition of done:

- Unit tests cover each bucket.
- Dry-run output includes source evidence, entities, proposed notes, and proposed contact updates.
- Output quality is comparable to the hand-written pilot notes for the first three fixtures.

### Phase 3 — Implement vault writer

**Goal:** Write idempotent timeline/entity notes.

Tasks:

1. Add path builder for `Timeline/YYYY-MM/YYYY-MM-DD_slug.md`.
2. Add monthly index updater.
3. Add entity path resolver.
4. Add frontmatter merge/update helper.
5. Add idempotency marker using source Redmine issue ID in frontmatter.

Definition of done:

- Re-running same decomposition does not duplicate notes or index bullets.
- Existing entity notes are updated, not overwritten.

### Phase 4 — Implement Redmine contact updater

**Goal:** Update Redmine contacts with Onyx article links and useful tags.

Tasks:

1. Discover contact custom fields.
2. Decide field for Obsidian/Onyx article URL.
3. Add dry-run contact update report.
4. Add `--apply-contact-updates` using `rebot` only when approved.
5. Verify contact after update.

Definition of done:

- Contact has vault article URL.
- Vault article has `redmine_contact_id`.
- Writes are impersonated appropriately.

### Phase 5 — gbrain sync/index integration

**Goal:** Make decomposed notes searchable in gbrain.

Tasks:

1. Discover LDR vault gbrain source ID.
2. Add optional command wrapper or instruction to run sync/embed/extract.
3. Verify search for a newly created entity/event.

Definition of done:

- gbrain search finds the decomposed event/entity.

### Phase 6 — Generalize `decompose <target>`

**Goal:** Support non-ticket inputs with the same decomposition outputs.

Tasks:

1. Add plain-text adapter.
2. Add email adapter.
3. Add PDF adapter using existing OCR/document pipeline if needed.
4. Keep output model the same.

Definition of done:

- Same event/entity/timeline/contact note logic works for non-ticket sources.

---

## Recommended First Execution Step

Do **Phase 1 manual pilot** before writing more automation.

Use the 3 known “meaningful but not spam” tickets from the recent spam-cleanup work:

```text
#41424 RePackify Gaylord inquiry
#41448 u2wear returns-provider follow-up
#41504 Snuggle Puppy / 3PL & FBA Returns
```

That gives us immediate signal on:

- note format quality
- entity matching
- Onyx link usefulness
- Redmine contact update shape
- gbrain ingestion friction

Only after that should we automate the decomposition engine. Otherwise we risk building a very elegant machine for generating annoying notes. We have enough of those already.