redmine/docs/ticket_decomposition_knowledge_ingestion_plan.md

Ticket Decomposition & Knowledge Ingestion Implementation Plan

Date: 2026-07-09
Project repo: /home/jason/projects/redmine
Primary vault: /home/jason/lilaBuild/vaults/obsidian-ldr
Primary Redmine project to start: customer-service
Production Redmine: https://rm.ldrprep.com
Onyx vault URL prefix: https://onyx.int.ldrprep.com/projects/obsidian-ldr/

Goal

Build a ticket decomposition process that converts meaningful Redmine tickets into durable organizational memory: timeline event notes, entity/contact notes, Redmine contact enrichment, cross-links, tags, and gbrain-ingestable vault content.

The immediate wedge is Redmine tickets, especially customer-service, but the final interface should support:

decompose <target>

where <target> may be:

• Redmine issue ID / Redmine project query
• email text
• PDF/document file
• pasted text
• cached ticket JSON

Scope

This plan is about knowledge ingestion and decomposition, not spam closure. It should build on the ticket sync substrate, not replace it.

Existing substrate to reuse:

• /home/jason/projects/redmine/redmine_tickets.py
• /home/jason/projects/redmine/redmine_contacts.py
• Redmine issue API with include=journals,helpdesk
• Obsidian LDR vault
• Onyx browser links
• gbrain source/index tooling

---

Ticket Buckets Worth Decomposing

A ticket should be eligible for decomposition if it appears to fit one or more of these buckets:

1. Definite customer ticket with meaningful event

Customer ticket where:

• a decision was made
• a complaint was involved
• a new project started
• a workflow/process implication was exposed
• an account/service/pricing change happened

Examples:

• customer complaint about missing photos, missing package, wrong condition, wrong item
• pricing/service inquiry that caused a meaningful answer or decision
• customer onboarding/project kickoff
• customer-specific exception or policy decision

2. Durable purchase receipt

Receipt about a durable item or business asset worth remembering.

Examples:

• equipment
• tools
• technology/hardware
• warehouse/office asset
• anything likely to matter for warranty, accounting, maintenance, or future context

3. Regular utility / recurring service receipt or invoice

Receipt/invoice/payment confirmation about a regular service.

Examples:

• utilities
• insurance
• phone/VoIP
• shipping/billing services
• cloud/SaaS subscriptions
• recurring vendors

This may not need rich event notes every time, but should enrich vendor/entity records and optionally create monthly financial/operations timeline entries.

4. Partnership / inbound opportunity / buy-from-us request

Not a customer and not a sales person trying to sell us something, but someone proposing a relationship or wanting to buy something from us.

Examples:

• buy Gaylords
• buy liquidation
• advertise/affiliate/blog partnership
• ask about partnership or marketplace cooperation
• use LDR as a source/supplier

These should become opportunity/entity notes, not spam.

5. Any meaningful operational event

Catch-all for tickets that refer to a meaningful event:

• policy change
• operational incident
• vendor problem
• staff/customer decision
• process drift or known-failure-method recurrence
• new obligation or future follow-up

---

Non-Goals / Safety Boundaries

1. Do not ingest every Redmine ticket into the vault as a polished note. Use cache/archive for raw data; decompose only meaningful tickets.
2. Do not overwrite existing entity pages destructively.
3. Do not invent facts. Use evidence blocks and confidence levels.
4. Do not send customer-visible Helpdesk replies.
5. Do not update Redmine contacts until the proposed changes are explicit and reviewable.
6. Do not let spam-cleanup closures trigger decomposition unless the ticket also has meaningful non-spam content.

---

Vault Information Architecture

Timeline root

Create a new timeline tree:

/home/jason/lilaBuild/vaults/obsidian-ldr/Timeline/

Monthly folders:

Timeline/2026-07/
Timeline/2026-08/

Monthly index:

Timeline/2026-07/index.md

Event notes:

Timeline/2026-07/2026-07-09_customer-name-short-subject.md

Filename rules:

• kebab-case subject slug
• no spaces
• prefix YYYY-MM-DD_
• keep short but human-readable

Example:

Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry.md
Timeline/2026-07/2026-07-09_u2wear-returns-provider-followup.md
Timeline/2026-07/2026-07-09_cyberstorm-order-return-confirmation.md

Monthly index format

Timeline/YYYY-MM/index.md should contain:

# YYYY-MM Timeline Index

## Key events

- [[2026-07-09_repackify-gaylord-inquiry]] — Partnership / buy-from-us inquiry. Redmine #41424.
- [[2026-07-09_u2wear-returns-provider-followup]] — Existing/prospective customer follow-up. Redmine #41448.

## Decisions

- ...

## Complaints / service failures

- ...

## Purchases / receipts

- ...

## Partnerships / opportunities

- ...

Entity notes

Use the existing vault entities/ tree as the default entity home.

Proposed paths:

entities/people/<person-slug>.md
entities/organizations/<organization-slug>.md
entities/concepts/<concept-slug>.md
entities/places/<place-slug>.md

If the repo prefers flat entities/<slug>.md, the first implementation can create flat notes and later migrate. But the decomposition code should centralize path decisions in one function, e.g. entity_note_path(entity).

Entity note frontmatter

Minimum frontmatter for Redmine contacts:

---
type: organization
name: RePackify
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
tags:
  - redmine-contact
  - maybe-customer
  - partnership-opportunity
aliases:
  - RePackify
---

For a person:

---
type: person
name: Sarah Johnson
organization: [[repackify]]
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/people/sarah-johnson.md
tags:
  - redmine-contact
  - contact-person
---

Entity body format

# Entity Name

## Summary

Brief factual summary with confidence level.

## Redmine links

- Contact: https://rm.ldrprep.com/contacts/4470
- Related tickets:
  - https://rm.ldrprep.com/issues/41424

## Timeline

- [[Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry]]

## Known facts

- Fact with source link.

## Open questions

- Question that needs human/agent follow-up.

---

Event Note Format

Each event note should be evidence-first and dialectic.

---
type: timeline-event
date: 2026-07-09
source_type: redmine-ticket
redmine_issue_id: 41424
redmine_url: https://rm.ldrprep.com/issues/41424
helpdesk_contact_id: 4470
entities:
  - [[repackify]]
tags:
  - timeline
  - redmine
  - partnership-opportunity
  - buy-from-us
confidence: medium
---

# 2026-07-09 — RePackify Gaylord Inquiry

## One-line summary

RePackify asked whether LDR has used Gaylord boxes available to purchase.

## Source evidence

- Redmine: https://rm.ldrprep.com/issues/41424
- From: support@repackify.com
- Subject: Gaylord inquiry-RePackify
- Key excerpt:
  > We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.

## Classification

- Bucket: partnership / buy-from-us request
- Customer status: not confirmed customer
- Suggested contact tag: maybe-customer or partnership-opportunity

## Dialectic analysis

### Claim

This is not spam; it is a potential inbound business/opportunity ticket.

### Supporting evidence

- The sender wants to buy something from LDR, not sell LDR a generic service.
- The request references a concrete asset class: used Gaylord boxes.

### Counterpoints / uncertainty

- It may be a one-off opportunistic buyer, not a durable customer relationship.
- Needs human judgment on whether LDR wants to sell Gaylords.

### Decision / outcome

- No final business decision captured yet.
- Needs follow-up if LDR wants a liquidation/Gaylord sale process.

## Entities to update

- [[repackify]]

## Redmine contact enrichment proposal

```json
{
  "contact_id": 4470,
  "add_tags": ["maybe-customer", "partnership-opportunity", "buy-from-us"],
  "obsidian_article_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md"
}

Follow-up candidates

• Decide whether LDR sells used Gaylords.
• If yes, define a standard response/process.

---

## Dialectic Reasoning Requirements

Each decomposition should include:

1. **Claim** — what seems to be true or meaningful.
2. **Evidence** — exact ticket excerpts, sender, date, Redmine link.
3. **Counterpoints** — why the interpretation might be wrong.
4. **Decision/outcome** — what was decided or what remains undecided.
5. **Recommended graph updates** — timeline note, entity notes, contact updates, tags.
6. **Confidence** — low/medium/high.

This prevents the vault from becoming a pile of confident hallucinated summaries. A small mercy in a cruel world.

---

## Redmine Contact Cross-Linking

### Vault → Redmine

Entity frontmatter should include:

```yaml
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470

Redmine → Vault

The Redmine contact should be updated with an Onyx URL, not an obsidian:// link.

Preferred URL format:

https://onyx.int.ldrprep.com/projects/obsidian-ldr/<vault-relative-path>.md

Example:

https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md

Implementation detail:

• If a Redmine contact custom field exists for Obsidian Article, use redmine_contacts.py --set-custom-field.
• If no suitable custom field exists, append a clearly delimited line to contact background/notes, e.g.:

Obsidian article: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md

Do not guess custom field IDs. Discover available custom fields from get_contact() payloads first.

---

decompose <target> Architecture

KarpathyWiki plugin reference

Jason identified the Obsidian karpathywiki plugin as a close match for the desired behavior. Inspection notes are saved at:

/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md

Finding: the plugin does not expose a true standalone CLI. It registers Obsidian commands and its WikiEngine depends on Obsidian App/Vault/TFile/metadataCache APIs. However, its logic is highly relevant and portable as a design reference:

• source analysis prompt / JSON schema
• entity vs concept decision tree
• alias handling
• duplicate/semantic entity resolution
• entity/concept/source page generation prompts
• contradiction and related-page fields
• graph-based query/retrieval ideas

Recommended implementation path for Redmine decomposition: build a Python headless mirror integrated with redmine_tickets.py, using KarpathyWiki's prompts/schema as reference. Consider a TypeScript adapter later only if maintaining a Python mirror becomes too costly.

Compatibility contract with the Obsidian KarpathyWiki plugin

The LDR vault already contains a plugin-managed wiki tree:

wiki/entities/
wiki/concepts/
wiki/sources/
wiki/index.md
wiki/log.md

The headless Redmine decomposer must be a companion, not a competing owner of that tree.

Default rules:

1. Do not write into wiki/ by default.
2. Do not edit wiki/index.md or wiki/log.md; those are plugin-owned.
3. Do not overwrite or reformat plugin-generated wiki/entities/*, wiki/concepts/*, or wiki/sources/* pages.
4. Write Redmine-derived timeline/event artifacts into LDR-owned paths:
   • Timeline/YYYY-MM/...
   • entities/organizations/...
   • entities/people/...
   • Processes/Redmine/...
5. Use Obsidian-compatible wikilinks so plugin-generated pages and headless-generated pages can link to each other.
6. If linking to plugin pages, use explicit paths such as [[wiki/concepts/returns-management|returns management]] rather than ambiguous bare names.
7. Use plugin-compatible frontmatter concepts where practical (type, created, updated, tags, sources, confidence, contradictions) but allow LDR-specific types such as timeline-event outside wiki/.
8. If we ever want the plugin itself to ingest Redmine-derived material, create a deliberate handoff/export file outside wiki/ first and let the plugin ingest it through Obsidian; do not silently write into plugin-owned sources/.

This keeps Obsidian’s KarpathyWiki plugin useful in the desktop vault while allowing Redmine decomposition to run headlessly/cron/agent-side without fighting plugin state.

Concrete upstream files to mirror first:

• src/wiki/prompts/ingestion.ts — extraction prompt / JSON schema
• src/wiki/prompts/generation.ts — page generation prompts
• src/wiki/prompts/merge.ts — merge/update triage prompts
• src/wiki/source-analyzer.ts — source analysis normalization/provenance behavior
• src/wiki/page-factory.ts — slug/alias/page merge behavior

The first Python implementation should not attempt full parity. It should clone only the useful minimum:

1. analyze source into JSON entities/concepts/key points/contradictions;
2. resolve entity slugs against existing vault files;
3. generate or update entity/event notes;
4. produce a reviewable contact-update proposal;
5. leave graph retrieval to Obsidian/gbrain for now.

Input abstraction

Create a common normalized input model:

@dataclass
class DecompositionSource:
    source_type: str  # redmine-ticket, email, pdf, text, file
    source_id: str | None
    title: str
    body: str
    date: str | None
    sender: str | None
    redmine_issue_id: int | None
    redmine_contact_id: int | None
    attachments: list[Path]
    metadata: dict[str, Any]

Adapters:

• from_redmine_issue(issue_json)
• from_email_text(raw_text)
• from_pdf(path)
• from_plain_text(text)
• from_file(path)

Decomposition output model

@dataclass
class DecompositionPlan:
    source: DecompositionSource
    buckets: list[str]
    timeline_events: list[TimelineEvent]
    entities: list[EntityUpdate]
    redmine_contact_updates: list[ContactUpdate]
    tags: list[str]
    gbrain_actions: list[GBrainAction]
    confidence: str
    review_required: bool

Apply phases

1. decompose --dry-run — produce JSON/Markdown plan only.
2. decompose --write-vault — write timeline/entity notes.
3. decompose --update-redmine-contact — update Redmine contact with Onyx article link/tags.
4. decompose --gbrain-sync — run/queue gbrain sync/index.

Each phase should be separately idempotent.

---

CLI Proposal

Decompose a Redmine ticket

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  decompose \
  --issue-id 41424 \
  --dry-run

Decompose from cached tickets matching a bucket

./redmine_tickets.py \
  --cache-dir .cache/redmine_tickets_prod_meaningful_20260709 \
  --project customer-service \
  decompose-cached \
  --bucket partnership-opportunity \
  --dry-run

Decompose arbitrary text

./decompose.py text \
  --title "Supplier proposal" \
  --input /tmp/message.txt \
  --dry-run

Write vault notes after review

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  decompose \
  --issue-id 41424 \
  --write-vault

---

gbrain Integration

Initial approach should be simple and explicit:

1. Write/update Obsidian notes.
2. Run or queue gbrain sync for the LDR vault source.
3. Verify that the new notes are searchable.

Commands need source discovery first:

gbrain sources list --json
gbrain stats

Then likely:

gbrain sync --source obsidian-ldr --no-embed --json
gbrain embed --stale
gbrain extract --stale
gbrain search "RePackify Gaylord inquiry"

Do not assume the exact source ID is obsidian-ldr until verified by gbrain sources list --json.

---

Implementation Phases

Phase 0 — Immediate schema/docs checkpoint

Goal: Preserve this architecture and make future implementation straightforward.

Tasks:

1. Save this plan.
2. Link it from Redmine HANDOFF.md.
3. Optionally create an Obsidian process note at:

/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md

Definition of done:

• Plan exists.
• Handoff links it.
• Next worker can start implementation without transcript context.

Phase 1 — Manual pilot decomposition for 3 tickets

Goal: Prove note structure and cross-linking manually before automating.

Suggested pilot tickets:

• RePackify Gaylord inquiry (#41424)
• u2wear returns-provider follow-up (#41448)
• Snuggle Puppy / 3PL & FBA Returns (#41504)

Tasks:

1. Fetch each ticket with include=journals,helpdesk.
2. Write one timeline event note per ticket.
3. Create/update entity notes.
4. Prepare Redmine contact update payloads, dry-run first.
5. Review Onyx links.
6. Optionally run gbrain sync/search smoke test.

Definition of done:

• 3 timeline notes exist.
• Monthly index links them.
• Entity notes exist or are updated.
• Redmine contact update payloads are reviewed.
• No customer-visible Redmine replies are sent.

Phase 2 — Implement decomposition dry-run engine

Goal: Generate decomposition plans from cached Redmine tickets without writing vault/Redmine.

Status — 2026-07-10: First deterministic slice implemented in redmine_tickets.py decompose. It supports --issue-id, --input-json, and --format json|markdown. It produces dry-run plans for the three pilot fixtures and respects the KarpathyWiki compatibility boundary by planning writes outside wiki/. A KarpathyWiki-style structured extraction layer now cleans Helpdesk/forwarded-message noise and emits analysis.summary, analysis.entities, analysis.concepts, analysis.key_points, analysis.contradictions, analysis.related_pages, and analysis.clean_text. Tests pass (23 tests). Current limitation: extraction is still deterministic/rule-backed rather than LLM-backed; next slice should add prompt-driven JSON extraction for broader ticket types before vault-write mode.

Files:

• Modify: /home/jason/projects/redmine/redmine_tickets.py
• Add tests: /home/jason/projects/redmine/tests/test_ticket_decomposition.py or extend test_redmine_tickets.py

Tasks:

1. Add DecompositionSource and DecompositionPlan models.
2. Add source_from_redmine_issue(issue).
3. Add bucket classifier:
   • customer-decision
   • customer-complaint
   • customer-new-project
   • durable-purchase-receipt
   • recurring-utility-receipt
   • partnership-buy-from-us
   • meaningful-event
4. Add dialectic plan generator.
5. Add JSON/Markdown dry-run output.
6. Add KarpathyWiki-inspired source cleanup and LLM JSON extraction for higher-quality summaries/entities.

Definition of done:

• Unit tests cover each bucket.
• Dry-run output includes source evidence, entities, proposed notes, and proposed contact updates.
• Output quality is comparable to the hand-written pilot notes for the first three fixtures.

Phase 3 — Implement vault writer

Goal: Write idempotent timeline/entity notes.

Tasks:

1. Add path builder for Timeline/YYYY-MM/YYYY-MM-DD_slug.md.
2. Add monthly index updater.
3. Add entity path resolver.
4. Add frontmatter merge/update helper.
5. Add idempotency marker using source Redmine issue ID in frontmatter.

Definition of done:

• Re-running same decomposition does not duplicate notes or index bullets.
• Existing entity notes are updated, not overwritten.

Phase 4 — Implement Redmine contact updater

Goal: Update Redmine contacts with Onyx article links and useful tags.

Tasks:

1. Discover contact custom fields.
2. Decide field for Obsidian/Onyx article URL.
3. Add dry-run contact update report.
4. Add --apply-contact-updates using rebot only when approved.
5. Verify contact after update.

Definition of done:

• Contact has vault article URL.
• Vault article has redmine_contact_id.
• Writes are impersonated appropriately.

Phase 5 — gbrain sync/index integration

Goal: Make decomposed notes searchable in gbrain.

Tasks:

1. Discover LDR vault gbrain source ID.
2. Add optional command wrapper or instruction to run sync/embed/extract.
3. Verify search for a newly created entity/event.

Definition of done:

• gbrain search finds the decomposed event/entity.

Phase 6 — Generalize decompose <target>

Goal: Support non-ticket inputs with the same decomposition outputs.

Tasks:

1. Add plain-text adapter.
2. Add email adapter.
3. Add PDF adapter using existing OCR/document pipeline if needed.
4. Keep output model the same.

Definition of done:

• Same event/entity/timeline/contact note logic works for non-ticket sources.

---

Recommended First Execution Step

Do Phase 1 manual pilot before writing more automation.

Use the 3 known “meaningful but not spam” tickets from the recent spam-cleanup work:

#41424 RePackify Gaylord inquiry
#41448 u2wear returns-provider follow-up
#41504 Snuggle Puppy / 3PL & FBA Returns

That gives us immediate signal on:

• note format quality
• entity matching
• Onyx link usefulness
• Redmine contact update shape
• gbrain ingestion friction

Only after that should we automate the decomposition engine. Otherwise we risk building a very elegant machine for generating annoying notes. We have enough of those already.