Organized
This commit is contained in:
Jason Thistlethwaite
@@ -0,0 +1,191 @@
|
||||
# Ticket Sync Tool — Architecture & Implementation Plan
|
||||
|
||||
**Status**: Phase 1–3 tracer implemented and LAN-smoke-tested (2026-07-09)
|
||||
**Goal**: Build a reliable, incremental ticket sync + query foundation (`redmine_tickets.py`) that replaces the obsolete RebotSlick `PullHelpdeskInfo` pattern with the modern `include=helpdesk` capability. This becomes the core data layer for all future agentic ticket workflows (spam cleanup, enrichment, dropped-ball detection, etc.).
|
||||
|
||||
**Primary target**: `customer-service` project (currently >1700 tickets). Tool must work for any project.
|
||||
|
||||
**Key constraints**
|
||||
- Do **not** modify any NMI components.
|
||||
- Reuse existing `RedmineClient` and helpdesk methods from `redmine_contacts.py`.
|
||||
- Leverage the `include=helpdesk` (and `include=journals,helpdesk`) patch already deployed.
|
||||
- Cache must be idempotent and cheap to re-run.
|
||||
- Tool must support “do nothing but sync what’s new” when run with no action.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Requirements (from user spec)
|
||||
|
||||
1. **Incremental sync**
|
||||
- Recursively fetch tickets with optional criteria (`--updated-since`, `--category`, `--status`, `--project`, `--with-tag`, etc.).
|
||||
- Store full enriched payload (issue + journals + helpdesk_ticket + messages) as local JSON.
|
||||
- Only re-fetch when the ticket’s `updated_on` is newer than the cached file mtime (or use a small manifest).
|
||||
|
||||
2. **No-op default behavior**
|
||||
- Running with no filters/action does a safe, incremental sync of everything (or a sensible default window).
|
||||
|
||||
3. **Filter/search mode**
|
||||
- When filters are supplied but no action is chosen, behave as a powerful search/report tool (list matching tickets, optionally write JSONL).
|
||||
|
||||
4. **Idempotent action layer (future)**
|
||||
- Higher-level workers (spam-check, close, enrich, etc.) must be able to mark a ticket as “already processed” so repeated runs skip it.
|
||||
- Suggested mechanism: sidecar manifest or lightweight SQLite table (`ticket_id`, `action_name`, `last_run`, `result`).
|
||||
|
||||
5. **Helpdesk enrichment**
|
||||
- Always request `include=journals,helpdesk` (or configurable) so every cached ticket already contains the rich contact/source metadata.
|
||||
|
||||
---
|
||||
|
||||
## 2. Proposed Architecture
|
||||
|
||||
### 2.1 Tool location & reuse
|
||||
- New script: `/home/jason/projects/redmine/redmine_tickets.py`
|
||||
- Reuses the existing `RedmineClient` class (especially `issue_with_helpdesk`, `get_issue`, list methods).
|
||||
- Same argument patterns, dry-run safety, `--apply`, error handling, and completion scripts as `redmine_contacts.py`.
|
||||
|
||||
### 2.2 Cache layout (local filesystem)
|
||||
Recommended structure (under a configurable `--cache-dir`, default `/home/jason/projects/redmine/cache/tickets/` or `/tmp/redmine-tickets-cache/`):
|
||||
|
||||
```
|
||||
cache/tickets/
|
||||
├── manifest.json # { "ticket_id": { "updated_on": "...", "last_synced": "...", "actions": {...} } }
|
||||
├── customer-service/
|
||||
│ ├── 12345.json # full enriched payload
|
||||
│ ├── 12346.json
|
||||
│ └── ...
|
||||
└── other-project/
|
||||
└── ...
|
||||
```
|
||||
|
||||
- One JSON file per ticket (stable, human-inspectable, easy to diff).
|
||||
- `manifest.json` tracks `updated_on` + per-action idempotency state.
|
||||
- Simple mtime + `updated_on` comparison is sufficient for v1; manifest makes it robust.
|
||||
- Implementation note: ticket payload storage and ticket/action state storage are separated in code. `ManifestTicketStateStore` is the current implementation, but `TicketCache` only depends on a small state-store method set (`get_ticket_state`, `upsert_ticket_state`, `has_action`, `record_action`). If the manifest becomes cumbersome with thousands of tickets, replace it with SQLite or another backend without rewriting sync/search/action logic.
|
||||
|
||||
### 2.3 CLI structure (v1 scope)
|
||||
|
||||
```bash
|
||||
./redmine_tickets.py \
|
||||
--base-url http://192.168.50.170 \
|
||||
--project customer-service \
|
||||
[sync|search] \
|
||||
[--updated-since YYYY-MM-DD] \
|
||||
[--updated-before YYYY-MM-DD] \
|
||||
[--status open|closed|*] \
|
||||
[--category "Spam"] \
|
||||
[--max-tickets N] \
|
||||
[--order newest-updated-first] \
|
||||
[--cache-dir ...] \
|
||||
[--dry-run] [--apply] \
|
||||
[--json] [--jsonl]
|
||||
```
|
||||
|
||||
**Commands**
|
||||
- `sync` (default if omitted): Perform incremental fetch + cache update.
|
||||
- `search`: Same selection logic, but only output matching tickets (no cache write unless `--write-cache`).
|
||||
|
||||
**Safety**
|
||||
- Dry-run by default for any write.
|
||||
- `--apply` required to persist cache or perform mutations.
|
||||
|
||||
### 2.4 Idempotency & action hooks (v1 + v2)
|
||||
|
||||
- Manifest stores per-ticket action state:
|
||||
```json
|
||||
"actions": {
|
||||
"spam-check": { "last_run": "2026-07-09", "result": "not-spam" },
|
||||
"close-if-spam": { "last_run": "...", "result": "skipped" }
|
||||
}
|
||||
```
|
||||
- Future workers read the manifest before acting and update it after.
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation Phases
|
||||
|
||||
### Phase 1 — Core sync + cache (implemented tracer)
|
||||
- Implement `RedmineClient` enhancements if needed (list issues with `include=journals,helpdesk`, pagination).
|
||||
- Basic `sync` command that:
|
||||
- Accepts date/status/project filters.
|
||||
- Fetches pages of issues.
|
||||
- For each issue, writes `cache/<project>/<id>.json` only if newer than manifest entry.
|
||||
- Updates manifest.
|
||||
- `--max-tickets` safety limit.
|
||||
- Dry-run shows what would be fetched/updated.
|
||||
|
||||
**Verified 2026-07-09**: `redmine_tickets.py sync` pulled 10 open `customer-service` tickets from LAN Redmine (`192.168.50.170`), wrote enriched per-ticket JSON, and a second identical run fetched 0 / skipped 10 unchanged cached tickets.
|
||||
|
||||
### Phase 2 — Search / filter-only mode (implemented tracer)
|
||||
- `search` subcommand that reuses the same selection engine.
|
||||
- Output options: human table, `--json`, `--jsonl` (for downstream agents).
|
||||
- Ability to pipe results into later tools.
|
||||
|
||||
**Verified 2026-07-09**: cached search over the 10-ticket LAN cache listed the same newest open tickets seen in the UI, including promo candidates like `#41788` and `#41789`.
|
||||
|
||||
### Phase 3 — Idempotency manifest + action skeleton (implemented tracer)
|
||||
- Add manifest read/write helpers.
|
||||
- Skeleton `--action` interface (initially just records “action X run on ticket Y”).
|
||||
- Example action: `spam-check` (calls the logic we started in `spam_promo_scorer.py` but at ticket level).
|
||||
|
||||
**Verified 2026-07-09**: `spam-check` identified `#41788` and `#41789` as SPAM based on promotional subject language, recorded local manifest idempotency with `--apply`, and a second `spam-check` run checked 0 tickets. `close-if-spam` dry-run planned closures for those two tickets and reviewed sender `contact_id=621`, finding 4/10 recent tickets scored as spam (`likely_spammer=True`). No Redmine close mutation has been applied yet.
|
||||
|
||||
### Phase 4 — Spam cleanup worker (next)
|
||||
- Concrete `spam-check` + `close-if-spam` actions.
|
||||
- Rules based on sender patterns, subject language, lack of human replies, existing contact tags, etc.
|
||||
- Produces a report + optional `--apply` to close tickets and add `spam` tag to contact.
|
||||
|
||||
Preferred spam policy:
|
||||
1. When a ticket is high-confidence spam, close that ticket.
|
||||
2. Use `helpdesk_ticket.contact.id` to fetch/analyze the last 5–10 tickets from the same contact.
|
||||
3. Decide whether the sender/contact appears to be a recurring spammer or whether only the specific ticket/email is spam.
|
||||
4. Only mark/tag the contact as a spammer when the contact-level lookback supports that conclusion.
|
||||
5. Record both `close-if-spam` and `contact-spam-review` idempotency markers so repeated runs do not re-close/re-review the same work.
|
||||
6. Treat Redmine impersonation like `sudo`: do not use it for ordinary reads or dry-run planning, but all spam-cleanup write calls should impersonate `rebot` / “Harry Botter” so Redmine activity logs clearly attribute automated cleanup work.
|
||||
|
||||
---
|
||||
|
||||
## 4. Non-Goals (for this phase)
|
||||
|
||||
- Do not replace or modify any NMI cron scripts.
|
||||
- Do not build full agent orchestration yet (this is the data layer).
|
||||
- Do not implement complex NLP classifiers in v1 (keep scoring simple and reviewable).
|
||||
- Cache does not need to be a database (filesystem + manifest is sufficient and inspectable).
|
||||
|
||||
---
|
||||
|
||||
## 5. Success Criteria
|
||||
|
||||
- Running the tool twice in a row with the same filters produces zero API calls on the second run (everything is cached and up-to-date).
|
||||
- A filter-only search for “tickets updated in last 30 days with no human reply” returns results in < 5 seconds after initial sync.
|
||||
- The same tool can be pointed at any project and produce usable enriched ticket JSON.
|
||||
- Higher-level workers can be written as thin Python scripts that consume the cache/manifest without needing their own Redmine client logic.
|
||||
|
||||
---
|
||||
|
||||
## 6. Next Immediate Action
|
||||
|
||||
The implementation tracer is complete through dry-run `close-if-spam`.
|
||||
|
||||
Next safe execution sequence:
|
||||
|
||||
1. Confirm LAN issue `status_id=5` is the intended closed status.
|
||||
2. Run `close-if-spam --apply` against the LAN copy only, impersonating `rebot` (the CLI default).
|
||||
3. Re-sync or inspect the affected LAN tickets.
|
||||
4. Verify:
|
||||
- `#41788` and `#41789` are closed,
|
||||
- Redmine activity author is `rebot` / Harry Botter,
|
||||
- local state records `close-if-spam` and `contact-spam-review`,
|
||||
- a second `close-if-spam --apply` does nothing.
|
||||
5. Only then expand the batch size beyond the initial 10-ticket cache.
|
||||
|
||||
Detailed handoff: `docs/ticket_spam_cleanup_handoff_2026-07-09.md`.
|
||||
|
||||
---
|
||||
|
||||
**References**
|
||||
- `docs/redmine_issue_api_helpdesk_include.md` (the key enabler)
|
||||
- `redmine_contacts.py` (client patterns, filtering, dry-run UX)
|
||||
- `docs/contact_triage_loop_*` (overall workflow philosophy)
|
||||
- RebotSlick `RebotSlick.php` (historical cache idea — modernized)
|
||||
|
||||
This plan deliberately keeps scope tight while giving us a clean, extensible foundation for the spam-reduction work that is currently blocking the customer-service UI.
|
||||
Reference in New Issue
Block a user