Organized

This commit is contained in:
Jason Thistlethwaite
2026-10-17 20:50:35 -04:00
parent bd26c8894f
commit 9ee9eb3ec6
130 changed files with 24172 additions and 140 deletions
+685
View File
@@ -0,0 +1,685 @@
# Power-user Redmine client — usage (`redmine_tickets.py`)
Scriptable local client for **humans and agents**: durable ticket cache, property search, Markdown view, and interactive walk.
| Env | Redmine | Cache |
|-----|---------|--------|
| **Prod** | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` |
| **Dev / mutation testing** | LAN (e.g. `192.168.50.170` / `retired.int.ldrprep.com`) | `/home/jason/projects/redmine/cache-dev/redmine_tickets` |
**Safety:** treat prod cache as read/sync. Test any **writes** on LAN first. Mutations impersonate `rebot` when implemented.
Related:
- Roadmap index: [raw-notes/features-for-redmine_tickets-sync.md](raw-notes/features-for-redmine_tickets-sync.md)
- Pre-MVP search snapshot: [2026-07-14_localsearch_before_sanity.md](2026-07-14_localsearch_before_sanity.md)
- Sync plan: [ticket_sync_tool_plan.md](ticket_sync_tool_plan.md)
- Cron scripts: `cron-scripts/sync-live-customer-service.sh`, `cron-scripts/sync-dev-customer-service.sh`
---
## Setup
```bash
cd /home/jason/projects/redmine
export REDMINE_API_KEY=… # required for sync / live API only
CACHE=/home/jason/projects/redmine/cache/redmine_tickets
PROJECT=customer-service
```
Shorthand used below:
```bash
RT="./redmine_tickets.py --project $PROJECT --cache-dir $CACHE"
```
---
## Sync (fill / refresh cache)
Remote filters hit Redmine; local cache is updated only for tickets whose `updated_on` changed.
```bash
# Initial / catch-up window
$RT sync --remote-status-id '*' --remote-updated-since 30d --run-log
# Frequent refresh (preferred after warm cache)
$RT sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet
```
| Flag | Meaning |
|------|---------|
| `--remote-status-id` / `--status-id` | Server: `*`, `open`, `closed`, or numeric |
| `--remote-updated-since` | Server: `30d`, `2h`, `YYYY-MM-DD`, … → `updated_on >= …` |
| `--remote-updated-on` | Server: raw Redmine operator string |
| `--run-log` | Ring-buffer stats → `$CACHE/run-stats.json` |
| `--run-log-keep N` | Keep last N runs (default 3; cron uses 20) |
| `--quiet` | Less progress noise |
**Do not** use local `--updated-since` as a substitute for remote filters on large projects (local only post-filters stubs already listed).
Cron note: live scripts currently still use `30d` on a short interval (~3–4s after warm-up). **Tighten later** to `2h`/`6h`.
---
## Search (local cache only)
No API key required. Only sees tickets already on disk.
## Search table columns
Default table (tab-separated):
```text
#id updated_on status assignee contact category control_tags subject
```
Contact comes from `helpdesk_ticket.contact.email` (or name/company/from_address).
### Classic filters
```bash
$RT search --status New --max-tickets 50
$RT search --updated-since 2026-07-01 --order newest-updated-first
```
### Property queries (`--where`)
Engine: `redmine_ticket_query.py`. Repeatable `--where` = **AND**.
```bash
$RT search --where 'subject.includes("account")' --max-tickets 20
$RT search --where 'assigned_to IS NULL' --max-tickets 10
$RT search \
--where 'helpdesk_ticket.contact.email == helpdesk_ticket.contact.company' \
--ids-only
$RT search \
--where '(subject.includes("verification code") OR subject.includes("magic link")) AND assigned_to IS NULL'
```
| Construct | Example |
|-----------|---------|
| Equality | `status.name == "New"` |
| Inequality | `assigned_to != "Jason T."` |
| Null | `assigned_to IS NULL` / `IS NOT NULL` |
| Substring | `subject.includes("rate sheet")` |
| Relative time | `updated_on.since("5 days")` |
| Path == path | `helpdesk_ticket.contact.email == helpdesk_ticket.contact.company` |
| Boolean | `AND` `OR` `&&` `||` + `()` |
Nested dicts compared to strings use `.name` (or email/id) when present.
### Output modes
| Flag | Output |
|------|--------|
| (default) | Table: id, updated_on, status, category, subject |
| `--json` | Full issue JSON array |
| `--jsonl` | One full issue JSON per line |
| `--ids-only` | Issue ids only (best for scripts) |
Match count goes to **stderr** on table mode so stdout stays pipe-clean for `--ids-only` / jsonl.
### Automation pattern
```bash
$RT search --where 'status.name == "New"' --where 'assigned_to IS NULL' --ids-only \
| while read -r id; do
$RT view "$id" --print > "/tmp/ticket-$id.md"
done
```
---
## Saved filters (`saved-filters/`)
Put one expression per file (comments with `#` allowed):
```text
saved-filters/unassigned.expr
saved-filters/helpdeskEmailMatchCompany.expr
saved-filters/triage-needed-5d.expr
```
Reference them from CLI:
```bash
$RT search --where '@unassigned' --max-tickets 20
$RT search --where 'alias.helpdeskEmailMatchCompany' --ids-only
$RT walk --where '@triage-needed-5d' --max-tickets 10 --pager cat
```
Bare names that match a file also work: `--where unassigned`.
### Short configs / wrappers
```bash
./bin/rtl search --where 'assigned_to IS NULL' --max-tickets 10 # prod CS
./bin/rtd note --issue-id 39305 --body 'LAN' --apply # dev CS
./redmine_tickets.py --config configs/live.json ...
```
`--allow-prod-writes` is a **parent** flag (before the subcommand):
```bash
./bin/rtl --allow-prod-writes draft apply <draft_id>
```
---
## Named queue folders (`materialize-queues`)
Config: [queues.example.json](../queues.example.json)
```bash
$RT materialize-queues --config queues.example.json
# writes: $CACHE/queues/<name>/*.json → symlinks into $CACHE/$PROJECT/<id>.json
```
Agents that only understand files can list `$CACHE/queues/unassigned/`.
Dry-run:
```bash
$RT materialize-queues --config queues.example.json --dry-run
```
Refresh after cron sync if you want queues kept current (optional second cron step).
---
## View (single ticket → Markdown)
Cache-first. Does not call Redmine unless you add that later.
```bash
$RT view 41934 --print
$RT view --issue-id 41934 --pager auto # glow → bat → batcat → cat
$RT view 41934 --write-md --pager cat
```
| Flag | Meaning |
|------|---------|
| `--print` | Markdown to stdout (no pager) |
| `--pager auto\|glow\|bat\|batcat\|cat` | How to display when not `--print` |
| `--write-md` | Force write under markdown cache |
| `--force-md` | Regen even if `updated_on` stamp matches |
Markdown cache path:
```text
$CACHE/$PROJECT/markdown/<id>.md
$CACHE/$PROJECT/markdown/<id>.md.updated_on
```
Regenerates when ticket `updated_on` changes (stamp file).
---
## Walk (interactive feed)
```bash
# From property search
$RT walk --where 'assigned_to IS NULL' --max-tickets 10 --pager cat
# Explicit ids
$RT walk --ids 41934,41899 --pager cat
```
Controls: `n` / Enter = next, `p` = previous, `q` = quit, `r` = redraw.
Same filter flags as search (`--where`, `--status`, `--max-tickets`, …). `--ids` overrides search selection.
---
## Internal note vs customer email (critical)
These are **different Redmine/Helpdesk operations**. The CLI uses **separate commands** so they are hard to confuse.
| | `note` | `customer-email` |
|--|--------|------------------|
| **What it is** | Internal journal note on the issue | Helpdesk “send note” / customer reply |
| **Emails customer?** | **No** | **Yes** (when applied) |
| **API** | `PUT /issues/:id.json` with `notes` (+ optional `private_notes`) | `POST /helpdesk/email_note.json` with `message.content` |
| **Private?** | Optional `--private` | **Never** (private cannot email) |
| **Confirm flag** | None needed | **Required:** `--i-understand-this-emails-the-customer` |
| **Default** | Queue local draft | Queue local draft |
| **Apply now** | `--apply` | `--apply` (+ confirm flag already required) |
| **Prod writes** | Need `--allow-prod-writes` | Need `--allow-prod-writes` |
| **Impersonation** | `--impersonate-user rebot` (default) on apply | same |
### Queue an internal note (safe default — no email)
```bash
$RT note --issue-id 41934 --body 'Staff-only follow-up' --private
# later:
$RT draft apply <draft-id> # still does NOT email
```
### Apply internal note immediately (LAN/test preferred)
```bash
$RT --base-url http://192.168.50.170 note --issue-id 123 \
--body 'LAN test note' --apply
```
### Queue a customer email (explicit, scary name + confirm flag)
```bash
$RT customer-email --issue-id 41934 \
--body 'Thanks for writing — we received your request.' \
--i-understand-this-emails-the-customer
# later:
$RT draft apply <draft-id> --i-understand-this-emails-the-customer
```
### Send customer email immediately
```bash
$RT --base-url http://192.168.50.170 customer-email --issue-id 123 \
--body 'LAN test customer reply' \
--i-understand-this-emails-the-customer \
--apply
```
### Production writes
Writes to hosts outside the LAN allow-list (e.g. `https://rm.ldrprep.com`) are **refused** unless you pass `--allow-prod-writes`. Prefer testing on LAN first.
### Draft list
```bash
$RT draft list
# columns include EMAILS_CUSTOMER vs no-email
```
**Do not** use `note` when you mean to email a customer. **Do not** use `customer-email` for staff chatter.
---
## set-fields (status / assignee / priority / category / due date)
Does **not** email the customer. Default queues a draft; `--apply` writes (LAN-safe by default).
```bash
# By numeric id
$RT set-fields --issue-id 41934 --status-id 5 --assigned-to-id 0
# By human name (uses meta cache; run meta-refresh once)
$RT meta-refresh --force
$RT set-fields --issue-id 41934 --status Closed --assigned-to unassigned \
--notes 'Unassign + close' --private
```
| Flag | Meaning |
|------|---------|
| `--status` / `--status-id` | name or numeric status |
| `--assigned-to` / `--assigned-to-id` | name/login or id; `unassigned` / `0` clears |
| `--priority` / `--priority-id` | name or id |
| `--category` / `--category-id` | name or id; `none` / `0` clears |
| `--due-date` | `YYYY-MM-DD` |
| `--subject` | replace subject |
| `--notes` / `--private` | optional internal journal note with the update |
| `--refresh-meta` | force name→id cache refresh |
Meta cache: `$CACHE/meta/<project>/`
---
## Contacts (local mirror + light mutations)
Separate from full `redmine_contacts.py` CRM CLI on purpose (different cache + surface), but **workflows can hand off**.
```bash
# Mirror / search / show
$RT contacts-sync --max-contacts 100
$RT contacts-search --query bavaria --max-contacts 20
$RT contact-show --contact-id 1890
# Ticket → contact bridge (starts on ticket, ends on contact tools)
$RT ticket-contact --issue-id 41934
$RT ticket-contact --issue-id 41934 --show --fetch
# Contact mutation (draft/apply, same safety as tickets; does NOT email)
$RT contact-set --contact-id 4515 --add-tag maybe-customer
$RT --allow-prod-writes draft apply <id> # if draft was for prod
# LAN apply:
$RT --base-url http://192.168.50.170 contact-set --contact-id 2284 \
--add-tag maybe-customer --apply --refresh-cache
```
| Piece | Path |
|-------|------|
| Local mirror | `$CACHE/contacts/<id>.json` |
| Full CRM (batch/loops/custom fields) | `redmine_contacts.py` |
| Ticket workbench contacts | `contacts-*`, `contact-set`, `ticket-contact` |
Store: `$CACHE/contacts/<id>.json` + `manifest.json`.
---
## Local control tags (not Redmine tags)
```bash
$RT control-tag add --tag bavaria --issue-id 41934 --property subject --annotation '…'
$RT control-tag list
$RT search --control-tag bavaria
$RT control-tag suggest --tag account --where 'subject.includes("account")' --max-tickets 20
$RT control-tag suggest --tag account --where 'subject.includes("account")' --apply
$RT control-tag wipe --i-understand-this-wipes-local-control-tags
```
Store: `$CACHE/control-tags/tags.json`
---
## Attachments
```bash
$RT attachments list --issue-id 39305 --refresh
$RT attachments download --issue-id 39305 --all
$RT attachments upload --issue-id 39305 --file ./doc.pdf # dry-run
$RT --base-url http://192.168.50.170 attachments upload \
--issue-id 39305 --file ./doc.pdf --apply --notes 'attached scan'
```
Default download dir: `$CACHE/$PROJECT/attachments/<issue_id>/`.
Does **not** email the customer. Prod write needs `--allow-prod-writes`.
---
## Entities (tracking / order patterns)
```bash
$RT entities show --issue-id 41394
$RT entities scan --where 'subject.includes("1Z")' --max-tickets 20
$RT entities scan --type tracking_ups --ids-only
```
Heuristic only — not NMI-backed. Also appears in `view` Markdown.
---
## Browse (interactive loop: list → pager → actions)
```bash
# Default: fzf (preview pane) → glow/bat pager → action menu → back to fzf
./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --open-only
# Explicit pager / no preview
./bin/rtl browse --where '@unassigned' --pager glow
./bin/rtl browse --status New --no-preview
# One-shot (pick once, page, exit — no menu/loop)
./bin/rtl browse --where 'subject.includes("invoice")' --once
```
### Loop keys
| Step | What happens |
|------|----------------|
| **List** | fzf list (Esc quits). Header lists actions. Preview shows ticket snippet. |
| **Enter** | Open ticket Markdown in glow/bat/batcat (`--pager`) |
| **After pager** | Action menu for that ticket |
### Action menu
| Key | Action |
|-----|--------|
| Enter / `v` | View again |
| `n` | Queue **internal note** (no email) |
| `p` | Queue **private** internal note |
| `f` | Queue **field update** (status/assignee by name) |
| `t` | Add **local control-tag** |
| `c` | Show ticket→contact handoff |
| `e` | Extract entities (patterns) |
| `m` | **NMI brief** → confirm → queue as **private** note draft |
| `i` | Print issue id |
| `l` | Back to list |
| `q` | Quit browse |
Mutations from browse **queue drafts only** (loud `draft_id=` banner). Apply later with `draft apply`.
Customer email is **not** in the menu (use CLI deliberately).
```bash
# Notes from file (+ optional Textile via pandoc)
./bin/rtl note --issue-id 41934 --body-file ./reply.md --private
./bin/rtl note --issue-id 41934 --body-file ./reply.md --to-textile --private
# Open tickets only (search/browse)
./bin/rtl search --open-only --where 'assigned_to IS NULL' --max-tickets 20
```
Non-interactive (no TTY): prints list lines and exits.
---
## OCR rename (scanned attachments → better subject)
```bash
# One ticket
./bin/rtl ocr-rename --issue-id 12345
# Scan for generic "Scanned document…" tickets
./bin/rtl ocr-rename --where '@scanned-document' --only-generic-subjects --max-tickets 15
# Apply subject on LAN immediately (note still queued private by default)
./bin/rtd ocr-rename --issue-id 12345 --apply-subject
```
Uses `pdftotext`, else `pdftoppm`+`tesseract`. Queues subject `fields` draft + private OCR note (disable with `--no-queue-subject` / `--no-queue-note`).
---
## Categories
```bash
./bin/rtl categories list
./bin/rtl categories list --refresh
```
---
## Entities + NMI
```bash
./bin/rtl entities show --issue-id 41394
./bin/rtl entities nmi --issue-id 41394
# Env: NMI_CLI, NMI_ENV_FILE, NMI_MODE — see docs/raw-notes/features/nmi-bridge.md
# Browse action `m` queues NMI brief as private note draft
```
---
## Local drafts (queue + apply)
```bash
$RT draft list
$RT draft cancel <id>
$RT draft cancel --all-pending
$RT draft apply <id> # internal notes
$RT draft apply <id> --i-understand-this-emails-the-customer # customer_email only
```
Stored under `$CACHE/drafts/*.json`.
---
## Other subcommands (ops)
| Command | Role |
|---------|------|
| `spam-check` / `close-if-spam` | Deterministic spam tooling (LAN for apply) |
| `known-spam-scan` / `known-spam-close` | Profile-based spam |
| `tag-maybe-customer` | Contact tagging from inquiry evidence |
| `decompose` | Dry-run vault-oriented plan for one ticket |
---
## Modules
| File | Role |
|------|------|
| `redmine_tickets.py` | CLI entry: sync, search, view, walk, spam, … |
| `redmine_ticket_query.py` | Property expression compiler |
| `redmine_ticket_view.py` | Markdown render + pager |
| `redmine_contacts.py` | Shared Redmine client / contacts |
Tests: `tests/test_redmine_ticket_query_and_view.py`, `tests/test_redmine_tickets_remote_filters.py`.
---
## What is **not** here yet
- Queued notes / helpdesk replies / field mutations (P2 remaining)
- Free-text semantic search (Qdrant / semantic_index — separate track)
- SQLite projection layer (planned architecture, not shipped)
**Shipped for queues/filters:** saved-filters + `materialize-queues` (see above).
---
## Quick “is the client alive?” check
```bash
$RT search --max-tickets 3
$RT search --where 'assigned_to IS NULL' --max-tickets 3 --ids-only
ID=$($RT search --max-tickets 1 --ids-only | head -1)
$RT view "$ID" --print | head -30
```
---
## TUI (curses keyboard-driven batch processor) [NEW]
Fast single-screen TUI for the exact workflow you described: queue a template of changes, scroll/cycle tickets, paint the template onto many tickets with one key, then one button to write everything.
```bash
./bin/rtl tui --where 'assigned_to IS NULL' --max-tickets 60 --open-only
./bin/rtl tui --control-tag triage-needed --max-tickets 30
```
**Workflow in the TUI:**
1. **Set template once** (optional fields; menus or type+filter):
- `c` : Category picker (from meta cache; type to filter)
- `a` : Assignee picker (unassigned=0)
- `s` : Status picker
- `p` : Priority (type a number, e.g. 2; hotkeys 1/2/3 also work)
- `P` : Project ID (for moving the ticket)
- `t` : Local control tag (string)
- `C` : Add contact tag (repeatable; creates if new)
- `n` : Private note (applies to painted tickets)
- `e` / `m` : Toggle entity-extraction / NMI-brief flags (become control tags + notes)
2. **Navigate the list**:
- `j` / `k` / arrows : move
- `/` : filter the visible list (live)
- `r` : redraw
3. **Paint / apply**:
- `SPACE` or `Enter` on a ticket: **apply the current template** to it (records per-ticket pending: fields, note, tags, toggles). Shows marker [FNTEM] etc.
4. **Commit**:
- `A` : Queue drafts for all painted tickets (safe default).
- (If you passed `--apply` and have `--allow-prod-writes` on LAN/prod, `S` for direct apply — use with care.)
Other keys: `x` clear pending on current, `R` force meta refresh, `q` quit (prompts if pending), `?` help.
**What gets written on commit:**
- Field updates (status/assignee/priority/category/project) via KIND_FIELDS draft (or direct).
- Private note (per ticket or from template).
- Local control tags (via control-tag add).
- Contact tags (via contact_fields drafts for the helpdesk contact).
- Special toggles → control tags ("entity-extract") + placeholder NMI note if needed.
All the safety rules (drafts vs apply, prod gate, impersonate rebot) still apply. The TUI just removes the per-ticket menu tax.
**Tips:**
- Run `meta-refresh --force` once for full picker lists.
- Combine with saved filters: `./bin/rtl tui --where '@unassigned'`
- The TUI loads from the same cache/search logic as browse/walk.
## TUI Commit behavior update (auto-apply on A)
As of latest change:
- Pressing **A** in the TUI now:
1. Shows a **single** "Are you SURE?" confirmation for the whole batch.
2. Queues the drafts for every painted ticket.
3. Immediately applies those drafts (using the normal apply_draft_to_redmine + mark_applied path).
This removes the need to manually run `draft apply <id>` for each ticket.
`S` remains available for the old direct-apply (bypass draft) path if you ever want it.
All normal safety (LAN/prod gate, --allow-prod-writes, impersonate rebot, no customer emails from this path) remains in force.
See the module docstring in redmine_ticket_tui.py or run the TUI and press `?` for current keys.
## NMI Enrichment Improvements (entities nmi)
The `entities nmi` command (and the NMI path used by browse 'm' and TUI) now:
- Automatically generates rich deep links when it sees item numbers, tracking, request ids, order ids, accounts:
- Item Manager + Item Statement for item_no
- Box History (boxFinder) for tracking numbers
- Order summary and package/request history pages
- User Info for account numbers
(See also the reference note: [[INBOX/WIP/Linking to Entities Within NMI]] )
- When an item_no is resolved, it calls the new `getItemHistory` API endpoint via `nmi item history --item-no=... --snapshot`:
- Returns merged timeline of product_history (moves, condition changes), service_fees (billings), and comments.
- The private note / cs-brief output now includes a "=== NMI Deep Links ===" section and a "=== Item History (getItemHistory) ===" section (truncated for notes; full in --format json).
Usage:
```
rtl entities nmi --issue-id 123456 --format note
rtl entities nmi --issue-id 123456 --format json # full history payload
rtl entities nmi --issue-id 123456 --no-history
```
The same enrichment logic feeds the TUI and browse NMI quick action.
---
## Post-apply cache refresh (anti-staleness)
After any mutation that hits live Redmine (TUI `A`, `draft apply`, `set-fields --apply`, direct notes, etc.) the affected ticket's JSON is immediately re-fetched and overwritten in the local cache.
This means:
- Next `rtl browse`, `rtl tui`, or search sees the new status / assignee / journals / updated_on right away.
- No need to wait for the next full sync for the tickets you just touched.
In the TUI:
- `r` → local redraw (fast)
- `R` → force live refresh of the **currently highlighted** ticket (pulls fresh from Redmine + updates cache + view)
This directly supports the rule: "update the existing cached file when the ticket has changed" and "agents update the ticket when their work is done."
### TUI "S" key (direct apply)
`S` is the **direct/bypass** path (still requires write permission).
- It skips the draft queue for field changes and does an immediate `update_issue`.
- We now also call the same live cache refresh that `A` uses, so the TUI and later `browse` see the change.
- Behavior difference from `A`:
- No "are you sure?" batch confirmation.
- No draft records created for the main fields.
- Still gated by `--allow-prod-writes` (or `--apply`).
Current recommended path for most batch work remains **A** (one confirm, goes through drafts for auditability, then auto-applies).
### TUI safety / UX improvements (2026-07)
- If you start without `--allow-prod-writes`, a clear **READ-ONLY** banner appears near the top.
- Bottom status/help line is now wrapped to avoid `_curses.error` on narrow/short terminals (common when ssh'ing from Windows Terminal, Tabby, or split panes).
- After confirming `A`, the screen immediately shows "COMMIT + APPLY IN PROGRESS" feedback so it doesn't look hung while the writes happen.
- `S` now shows "fields updated DIRECTLY" + cache sync messages when it actually performs a live write.