redmine/docs/ticket_sync_tool_plan.md

# Ticket Sync Tool — Architecture & Implementation Plan

**Status**: Phase 1–3 tracer implemented and LAN-smoke-tested (2026-07-09)  
**Goal**: Build a reliable, incremental ticket sync + query foundation (`redmine_tickets.py`) that replaces the obsolete RebotSlick `PullHelpdeskInfo` pattern with the modern `include=helpdesk` capability. This becomes the core data layer for all future agentic ticket workflows (spam cleanup, enrichment, dropped-ball detection, etc.).

**Primary target**: `customer-service` project (currently >1700 tickets). Tool must work for any project.

**Key constraints**
- Do **not** modify any NMI components.
- Reuse existing `RedmineClient` and helpdesk methods from `redmine_contacts.py`.
- Leverage the `include=helpdesk` (and `include=journals,helpdesk`) patch already deployed.
- Cache must be idempotent and cheap to re-run.
- Tool must support “do nothing but sync what’s new” when run with no action.

---

## 1. Core Requirements (from user spec)

1. **Incremental sync**
   - Recursively fetch tickets with optional criteria (`--updated-since`, `--category`, `--status`, `--project`, `--with-tag`, etc.).
   - Store full enriched payload (issue + journals + helpdesk_ticket + messages) as local JSON.
   - Only re-fetch when the ticket’s `updated_on` is newer than the cached file mtime (or use a small manifest).

2. **No-op default behavior**
   - Running with no filters/action does a safe, incremental sync of everything (or a sensible default window).

3. **Filter/search mode**
   - When filters are supplied but no action is chosen, behave as a powerful search/report tool (list matching tickets, optionally write JSONL).

4. **Idempotent action layer (future)**
   - Higher-level workers (spam-check, close, enrich, etc.) must be able to mark a ticket as “already processed” so repeated runs skip it.
   - Suggested mechanism: sidecar manifest or lightweight SQLite table (`ticket_id`, `action_name`, `last_run`, `result`).

5. **Helpdesk enrichment**
   - Always request `include=journals,helpdesk` (or configurable) so every cached ticket already contains the rich contact/source metadata.

---

## 2. Proposed Architecture

### 2.1 Tool location & reuse
- New script: `/home/jason/projects/redmine/redmine_tickets.py`
- Reuses the existing `RedmineClient` class (especially `issue_with_helpdesk`, `get_issue`, list methods).
- Same argument patterns, dry-run safety, `--apply`, error handling, and completion scripts as `redmine_contacts.py`.

### 2.2 Cache layout (local filesystem)
Recommended structure (under a configurable `--cache-dir`, default `/home/jason/projects/redmine/cache/tickets/` or `/tmp/redmine-tickets-cache/`):

```
cache/tickets/
├── manifest.json                 # { "ticket_id": { "updated_on": "...", "last_synced": "...", "actions": {...} } }
├── customer-service/
│   ├── 12345.json                # full enriched payload
│   ├── 12346.json
│   └── ...
└── other-project/
    └── ...
```

- One JSON file per ticket (stable, human-inspectable, easy to diff).
- `manifest.json` tracks `updated_on` + per-action idempotency state.
- Simple mtime + `updated_on` comparison is sufficient for v1; manifest makes it robust.
- Implementation note: ticket payload storage and ticket/action state storage are separated in code. `ManifestTicketStateStore` is the current implementation, but `TicketCache` only depends on a small state-store method set (`get_ticket_state`, `upsert_ticket_state`, `has_action`, `record_action`). If the manifest becomes cumbersome with thousands of tickets, replace it with SQLite or another backend without rewriting sync/search/action logic.

### 2.3 CLI structure (v1 scope)

```bash
./redmine_tickets.py \
  --base-url http://192.168.50.170 \
  --project customer-service \
  [sync|search] \
  [--updated-since YYYY-MM-DD] \
  [--updated-before YYYY-MM-DD] \
  [--status open|closed|*] \
  [--category "Spam"] \
  [--max-tickets N] \
  [--order newest-updated-first] \
  [--cache-dir ...] \
  [--dry-run] [--apply] \
  [--json] [--jsonl]
```

**Commands**
- `sync` (default if omitted): Perform incremental fetch + cache update.
- `search`: Same selection logic, but only output matching tickets (no cache write unless `--write-cache`).

**Safety**
- Dry-run by default for any write.
- `--apply` required to persist cache or perform mutations.

### 2.4 Idempotency & action hooks (v1 + v2)

- Manifest stores per-ticket action state:
  ```json
  "actions": {
    "spam-check": { "last_run": "2026-07-09", "result": "not-spam" },
    "close-if-spam": { "last_run": "...", "result": "skipped" }
  }
  ```
- Future workers read the manifest before acting and update it after.

---

## 3. Implementation Phases

### Phase 1 — Core sync + cache (implemented tracer)
- Implement `RedmineClient` enhancements if needed (list issues with `include=journals,helpdesk`, pagination).
- Basic `sync` command that:
  - Accepts date/status/project filters.
  - Fetches pages of issues.
  - For each issue, writes `cache/<project>/<id>.json` only if newer than manifest entry.
  - Updates manifest.
- `--max-tickets` safety limit.
- Dry-run shows what would be fetched/updated.

**Verified 2026-07-09**: `redmine_tickets.py sync` pulled 10 open `customer-service` tickets from LAN Redmine (`192.168.50.170`), wrote enriched per-ticket JSON, and a second identical run fetched 0 / skipped 10 unchanged cached tickets.

### Phase 2 — Search / filter-only mode (implemented tracer)
- `search` subcommand that reuses the same selection engine.
- Output options: human table, `--json`, `--jsonl` (for downstream agents).
- Ability to pipe results into later tools.

**Verified 2026-07-09**: cached search over the 10-ticket LAN cache listed the same newest open tickets seen in the UI, including promo candidates like `#41788` and `#41789`.

### Phase 3 — Idempotency manifest + action skeleton (implemented tracer)
- Add manifest read/write helpers.
- Skeleton `--action` interface (initially just records “action X run on ticket Y”).
- Example action: `spam-check` (calls the logic we started in `spam_promo_scorer.py` but at ticket level).

**Verified 2026-07-09**: `spam-check` identified `#41788` and `#41789` as SPAM based on promotional subject language, recorded local manifest idempotency with `--apply`, and a second `spam-check` run checked 0 tickets. `close-if-spam` dry-run planned closures for those two tickets and reviewed sender `contact_id=621`, finding 4/10 recent tickets scored as spam (`likely_spammer=True`). No Redmine close mutation has been applied yet.

### Phase 4 — Spam cleanup worker (next)
- Concrete `spam-check` + `close-if-spam` actions.
- Rules based on sender patterns, subject language, lack of human replies, existing contact tags, etc.
- Produces a report + optional `--apply` to close tickets and add `spam` tag to contact.

Preferred spam policy:
1. When a ticket is high-confidence spam, close that ticket.
2. Use `helpdesk_ticket.contact.id` to fetch/analyze the last 5–10 tickets from the same contact.
3. Decide whether the sender/contact appears to be a recurring spammer or whether only the specific ticket/email is spam.
4. Only mark/tag the contact as a spammer when the contact-level lookback supports that conclusion.
5. Record both `close-if-spam` and `contact-spam-review` idempotency markers so repeated runs do not re-close/re-review the same work.
6. Treat Redmine impersonation like `sudo`: do not use it for ordinary reads or dry-run planning, but all spam-cleanup write calls should impersonate `rebot` / “Harry Botter” so Redmine activity logs clearly attribute automated cleanup work.

---

## 4. Non-Goals (for this phase)

- Do not replace or modify any NMI cron scripts.
- Do not build full agent orchestration yet (this is the data layer).
- Do not implement complex NLP classifiers in v1 (keep scoring simple and reviewable).
- Cache does not need to be a database (filesystem + manifest is sufficient and inspectable).

---

## 5. Success Criteria

- Running the tool twice in a row with the same filters produces zero API calls on the second run (everything is cached and up-to-date).
- A filter-only search for “tickets updated in last 30 days with no human reply” returns results in < 5 seconds after initial sync.
- The same tool can be pointed at any project and produce usable enriched ticket JSON.
- Higher-level workers can be written as thin Python scripts that consume the cache/manifest without needing their own Redmine client logic.

---

## 6. Next Immediate Action

The implementation tracer is complete through dry-run `close-if-spam`.

Next safe execution sequence:

1. Confirm LAN issue `status_id=5` is the intended closed status.
2. Run `close-if-spam --apply` against the LAN copy only, impersonating `rebot` (the CLI default).
3. Re-sync or inspect the affected LAN tickets.
4. Verify:
   - `#41788` and `#41789` are closed,
   - Redmine activity author is `rebot` / Harry Botter,
   - local state records `close-if-spam` and `contact-spam-review`,
   - a second `close-if-spam --apply` does nothing.
5. Only then expand the batch size beyond the initial 10-ticket cache.

Detailed handoff: `docs/ticket_spam_cleanup_handoff_2026-07-09.md`.

---

**References**
- `docs/redmine_issue_api_helpdesk_include.md` (the key enabler)
- `redmine_contacts.py` (client patterns, filtering, dry-run UX)
- `docs/contact_triage_loop_*` (overall workflow philosophy)
- RebotSlick `RebotSlick.php` (historical cache idea — modernized)

This plan deliberately keeps scope tight while giving us a clean, extensible foundation for the spam-reduction work that is currently blocking the customer-service UI.