Organized

docs/raw-notes/features-for-redmine_tickets-sync.md

@@ -0,0 +1,332 @@
+# Features / Roadmap: power-user Redmine client (`redmine_tickets` plane)
+
+> [!todo] Keep this file as the **index**: status, priority, and short summaries.  
+> Long implementation detail lives in `docs/raw-notes/features/*` and is linked from here.  
+> Update checkboxes when work lands; do not re-expand full specs into this file.
+
+**Last status refresh:** 2026-07-15 (post browse-loop + attachments + entities + contacts bridge)
+
+---
+
+## Product intent (what this is)
+
+A **scriptable power-user Redmine client** for humans and agents:
+
+- durable local ticket (+ contact) cache as the **ops workbench**
+- property-based local search, saved filters, filesystem queues
+- CLI / TUI-lite read path (Markdown + `glow`/`bat`/`fzf`) then **queued** writes
+- place to invent/test filters, tools, and agent loops without living in the web UI
+
+**Not** the system of record (Redmine is).  
+**Not** the vault/wiki (curated memory).  
+**Not** Qdrant (optional later semantic layer).  
+
+First project focus: **`customer-service`**.
+
+| Env | Redmine | Cache | Wrapper |
+|-----|---------|--------|---------|
+| Prod | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` | `bin/rtl` |
+| Dev / mutation testing | `http://192.168.50.170` | `/home/jason/projects/redmine/cache-dev/redmine_tickets` | `bin/rtd` |
+
+**Safety defaults**
+
+- Writes/applies prefer LAN; prod needs parent `--allow-prod-writes`
+- `note` never emails; `customer-email` always emails (ugly confirm flag)
+- Browse/actions **queue drafts** by default (apply later)
+- Dual tools OK for now: tickets workbench vs full `redmine_contacts.py` CRM CLI, with bridges
+
+---
+
+## Priority stack (current)
+
+| Priority | Area | Status |
+|----------|------|--------|
+| P0 | Durable incremental ticket sync + remote filters | **Done** |
+| P0 | Cron for CS refresh (+ queue rematerialize) | **Running** (6h window) |
+| P1 | Property-based local search (`--where`) | **Done** |
+| P1 | Markdown view + walk | **Done** |
+| P1 | Browse loop (fzf → pager → action menu) | **MVP done** (polish open) |
+| P2 | Saved filters + symlink queues | **Done** |
+| P2 | Queued notes / customer-email / set-fields | **Done** |
+| P2 | Config wrappers (`--config`, `rtl`/`rtd`) | **Minimal done** |
+| P3 | Local control tags + suggest + JSONL export | **Done** (LLM refine open) |
+| P3 | Contacts mirror + contact-set + ticket-contact | **Done** (cache unify optional) |
+| P4 | Attachments list/download/upload | **Done** |
+| P4 | Entity extract (patterns) + NMI bridge CLI | **MVP done** |
+| Later | Full curses TUI / nested config DSL / master layout | Deferred |
+| Adjacent | Semantic index / vault decompose / OCR rename skill | Separate tracks |
+
+---
+
+## Capability map (what you can run today)
+
+| Need | Command(s) |
+|------|------------|
+| Refresh tickets | `sync` · cron: `cron-scripts/sync-live-customer-service.sh` |
+| Search cache | `search --where '…'` · `@saved` · `--control-tag` · `--has-attachments` |
+| Read ticket | `view ID` · `walk` · **`browse`** (loop) |
+| Internal note | `note` (draft → `draft apply`) |
+| Customer email | `customer-email` + confirm flag |
+| Fields | `set-fields` (`--status Closed`, etc.) · `meta-refresh` |
+| Local labels | `control-tag add|suggest|export|…` |
+| Contacts | `contacts-sync` · `contacts-search` · `contact-show` · `contact-set` · `ticket-contact --fetch` |
+| Files | `attachments list|download|upload` |
+| Tracking/order hints | `entities show|scan` · `entities nmi` · browse `m` |
+| OCR scan rename | `ocr-rename --issue-id` / `--where '@scanned-document'` |
+| Categories | `categories list [--refresh]` |
+| Queues on disk | `materialize-queues --config queues.example.json` |
+
+Cookbook: **`docs/cli_power_user_client.md`**
+
+---
+
+## P0 — Sync foundation
+
+**Detail:** [[features/sync-foundation]]
+
+- [x] Stable cache path (not per-run timestamped dirs)
+- [x] Incremental full-fetch skip when `updated_on` unchanged
+- [x] Server-side `updated_on` + `status_id` (remote flags)
+- [x] Local post-filters retained
+- [x] Sync progress + `--quiet` for cron
+- [x] Cache never auto-deletes old ticket JSON
+- [x] Run-stats ring buffer (`--run-log`)
+- [x] Cron: live + dev CS scripts; **6h** remote window; rematerialize queues after sync
+- [x] Optional contacts cron: `cron-scripts/sync-live-contacts.sh`
+- [ ] Confirm long-term cadence (currently ~every few minutes) is still desired
+- [x] Full fetch includes `attachments` metadata (not just journals/helpdesk)
+
+```bash
+./bin/rtl sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet
+```
+
+---
+
+## P1 — Local search
+
+**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_query.py`  
+**Historical thin search:** [[../2026-07-14_localsearch_before_sanity]]
+
+- [x] JSON path filters + `includes()` / `since()` / AND/OR / parens
+- [x] Nested helpdesk/contact paths
+- [x] `--ids-only`, ordering, max caps
+- [x] Table columns: assignee, contact, control tags
+- [x] `--has-attachments` (cache metadata)
+- [x] **No default status filter** — closed tickets appear if cached
+- [ ] Optional default “open-ish” profile (if desired later)
+
+---
+
+## P1 — Read path + browse loop
+
+**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_view.py` · `redmine_ticket_browse.py`
+
+- [x] Ticket → Markdown (`view`) + MD cache under `markdown/`
+- [x] Pagers: glow / bat / batcat / cat
+- [x] `walk` (n/p/q)
+- [x] **`browse` loop:** fzf (or numbered) → pager → action menu → list
+  - Actions: view again, queue note/private note, queue fields, control-tag, contact handoff, entities, print id
+  - Mutations **queue only**
+  - `--once`, `--pager`, filters same family as search
+- [ ] Browse polish (user-noted “minor things”) — collect below under **Open polish**
+- [ ] Full curses/TUI (linear-tui style) still **later**, not blocking ops
+
+```bash
+./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --pager glow
+```
+
+---
+
+## P2 — Filters, queues, mutations
+
+**Detail:** [[features/local-search-and-cli-client]] · drafts in `redmine_ticket_drafts.py`
+
+- [x] `saved-filters/*.expr` + `@name`
+- [x] `materialize-queues` symlink folders
+- [x] `note` vs `customer-email` (hard separation; Helpdesk `email_note` path)
+- [x] Draft list/cancel/apply; draft binds `base_url`; obvious `draft_id=`
+- [x] `set-fields` + name→id (`meta-refresh`)
+- [x] Parent `--allow-prod-writes` (not subparser-shadowed)
+- [x] Minimal `--config` + `bin/rtl` / `bin/rtd` / `bin/qaa`
+- [ ] Nested config defaults (`search.max-tickets` etc.) — [[features/config-files]]
+- [ ] Note body from file + optional pandoc→Textile (steering list)
+
+```bash
+./bin/rtl note --issue-id 41934 --body 'staff context' --private
+./bin/rtl customer-email --issue-id 41934 --body '…' --i-understand-this-emails-the-customer
+./bin/rtl --allow-prod-writes draft apply <id>
+```
+
+---
+
+## P3 — Control tags
+
+**Detail:** [[features/control-tags]] · export: `redmine_ticket_control_export.py`
+
+- [x] Local wipeable store under `control-tags/tags.json`
+- [x] add / remove / list / show / for-issue / wipe
+- [x] suggest (`--where` rules, optional `--apply` local)
+- [x] `export` → JSONL for training / refine
+- [ ] LLM refine loop (consume export, propose tags, human correct/incorrect)
+
+---
+
+## P3 — Contacts
+
+**Detail:** [[features/contacts-sync-integration]] · `redmine_ticket_contacts.py`  
+**Full CRM still:** `redmine_contacts.py` (separate cache/CLI by design for now)
+
+- [x] Durable mirror: `<cache>/contacts/`
+- [x] `contacts-sync` / `contacts-search` / `contact-show`
+- [x] `contact-set` draft/apply (tags, company, names, `--set`)
+- [x] `ticket-contact` bridge; **`--fetch` implies show+refresh** (bugfix)
+- [x] Contact column in ticket search
+- [ ] Optional unify on-disk layout with `redmine_contacts.py` (only if dual-cache pain grows)
+
+---
+
+## P4 — Attachments
+
+**Detail:** [[features/attachments]] · `redmine_ticket_attachments.py`
+
+- [x] `attachments list` (cache-first, `--refresh`)
+- [x] `attachments download` (one / `--all`)
+- [x] `attachments upload --apply` (POST `/uploads.json` + issue `uploads`; no email)
+- [x] OCR rename workflow CLI (`ocr-rename`) — [[features/ocr-rename]]
+
+---
+
+## P4 — Entities + NMI
+
+**Detail:** [[features/entity-type-mapper]] · `redmine_ticket_entities.py` · `redmine_ticket_nmi.py`
+
+- [x] Pattern extract: UPS/USPS, order/SKU/account + LDR `item_no`
+- [x] Shown in Markdown view
+- [x] `entities show|scan`
+- [x] `entities nmi` bridge to `mrscClient-php bin/nmi` (cs-brief / resolve)
+- [x] Noise filters on medium-confidence `order_id` hits
+- [ ] Richer NMI membership index (known order/item windows) if CLI alone isn’t enough
+- [x] One-key NMI from browse menu (`m` → private note draft)
+
+---
+
+## Open polish / small bugs (browse + ops)
+
+- [x] Browse: fzf header/keybinding discoverability
+- [x] Browse: optional preview window in fzf (`--no-preview` to disable)
+- [x] Browse: louder draft queue banner (`announce_draft`)
+- [x] Browse: `m` → entities nmi brief → private note draft
+- [x] Entity noise filters + LDR 9-digit `item_no`
+- [x] Note `--body-file` (already) + `--to-textile` (pandoc)
+- [x] `search`/`browse`/`walk` `--open-only`
+- [x] OCR attachment → rename (`ocr-rename`) MVP
+- [ ] OCR cron pack for HR “Scanned document” inboxes
+- [ ] categories create/update (list only for now)
+
+---
+
+## Steering / next decisions (pick a lane)
+
+### A — Daily-driver polish — **done for now**
+Reopen if browse UX still nags.
+
+### B — CS frontage / NMI / OCR — **MVP+ done**
+Optional: OCR cron for HR; NMI membership index.
+
+### C — CRM / spam tooling **← good next lane**
+1. Deeper `contact-set` (custom fields) or keep routing to `redmine_contacts.py`  
+2. Control-tag LLM refine from JSONL export  
+3. Issue categories create/update for triage taxonomy  
+
+### D — Bigger product bets (defer unless intentional)
+- Full curses TUI  
+- Unified contact cache  
+- Composable remote/local filter DSL  
+- Redmine project/version/custom-field admin  
+
+---
+
+## Explicitly out of this index (or later)
+
+| Item | Notes |
+|------|--------|
+| Semantic index v2 | `semantic_index/` — feed from this cache |
+| Vault decompose | separate track |
+| Redmine admin: versions / custom field *definitions* / project modules | not ticket ops |
+| Master global JSON + symlink org | nice-to-have |
+
+---
+
+## Related docs
+
+- `docs/cli_power_user_client.md` — **usage cookbook**
+- `docs/ticket_sync_tool_plan.md` — original sync plan
+- `docs/2026-07-14_localsearch_before_sanity.md` — historical
+- `docs/ticket_spam_cleanup_handoff_2026-07-09.md` — spam consumers
+- `docs/raw-notes/features/*.md` — detail files
+- `configs/live.json`, `configs/dev.json` · `bin/rtl` · `bin/rtd`
+
+### Feature detail files
+
+| File | Topic |
+|------|--------|
+| [[features/sync-foundation]] | Sync / cache / remote filters |
+| [[features/local-search-and-cli-client]] | Search, view, walk, queues, writes |
+| [[features/control-tags]] | Local control tags |
+| [[features/contacts-sync-integration]] | Contacts mirror + bridge |
+| [[features/attachments]] | File attachments |
+| [[features/entity-type-mapper]] | Entities + NMI |
+| [[features/nmi-bridge]] | NMI env + browse `m` |
+| [[features/ocr-rename]] | OCR subject rename |
+| [[features/browse-loop]] | fzf browse loop |
+| [[features/config-files]] | Config / aliases |
+
+---
+
+## Maintenance rule
+
+1. Put full detail in `docs/raw-notes/features/<name>.md`  
+2. Leave status + short bullets + link here  
+3. Prefer updating the feature file over re-pasting long prose into this index  
+4. When fixing a bug from **Bug reports**, check the box and leave the minimal repro snippet  
+
+---
+
+## Bug reports and QOL (resolved log)
+
+### --allow-prod-writes unrecognized / ignored
+- [x] Parent flag only; subparser copies removed (child defaults overwrote parent)
+
+### Draft id not obvious
+- [x] `draft_id=` / `apply_hint=` key=value output
+
+### Drafts ignore create-time base-url
+- [x] Apply prefers draft `base_url`; conflict needs `--allow-base-url-override`
+
+### Configs / short aliases
+- [x] Minimal `--config` + `bin/rtl|rtd|qaa`
+
+### `ticket-contact --fetch` alone did nothing
+- [x] `--fetch` implies show + live refresh into contacts cache
+
+---
+
+## Modules (code map)
+
+| Module | Role |
+|--------|------|
+| `redmine_tickets.py` | CLI entry + sync + routing |
+| `redmine_ticket_query.py` | Local AST filters |
+| `redmine_ticket_view.py` | Markdown render / pager |
+| `redmine_ticket_browse.py` | fzf loop + action menu |
+| `redmine_ticket_filters.py` | Saved filters / queues |
+| `redmine_ticket_drafts.py` | Draft queue + write safety hosts |
+| `redmine_ticket_meta.py` | Name→id meta cache |
+| `redmine_ticket_control_tags.py` | Local tags store |
+| `redmine_ticket_control_export.py` | Tag JSONL export |
+| `redmine_ticket_contacts.py` | Contact mirror |
+| `redmine_ticket_attachments.py` | Upload/download helpers |
+| `redmine_ticket_entities.py` | Pattern extract |
+| `redmine_ticket_nmi.py` | NMI CLI bridge |
+| `redmine_ticket_ocr.py` | Attachment OCR + subject suggest |
+| `redmine_contacts.py` | Full CRM CLI (sibling tool) |