redmine/docs/ticket_sync_tool_plan.md

Ticket Sync Tool — Architecture & Implementation Plan

Status: Phase 1–3 tracer implemented and LAN-smoke-tested (2026-07-09)
Goal: Build a reliable, incremental ticket sync + query foundation (redmine_tickets.py) that replaces the obsolete RebotSlick PullHelpdeskInfo pattern with the modern include=helpdesk capability. This becomes the core data layer for all future agentic ticket workflows (spam cleanup, enrichment, dropped-ball detection, etc.).

Primary target: customer-service project (currently >1700 tickets). Tool must work for any project.

Key constraints

• Do not modify any NMI components.
• Reuse existing RedmineClient and helpdesk methods from redmine_contacts.py.
• Leverage the include=helpdesk (and include=journals,helpdesk) patch already deployed.
• Cache must be idempotent and cheap to re-run.
• Tool must support “do nothing but sync what’s new” when run with no action.

---

1. Core Requirements (from user spec)

1. Incremental sync

   • Recursively fetch tickets with optional criteria (--updated-since, --category, --status, --project, --with-tag, etc.).
   • Store full enriched payload (issue + journals + helpdesk_ticket + messages) as local JSON.
   • Only re-fetch when the ticket’s updated_on is newer than the cached file mtime (or use a small manifest).

2. No-op default behavior

   • Running with no filters/action does a safe, incremental sync of everything (or a sensible default window).

3. Filter/search mode

   • When filters are supplied but no action is chosen, behave as a powerful search/report tool (list matching tickets, optionally write JSONL).

4. Idempotent action layer (future)

   • Higher-level workers (spam-check, close, enrich, etc.) must be able to mark a ticket as “already processed” so repeated runs skip it.
   • Suggested mechanism: sidecar manifest or lightweight SQLite table (ticket_id, action_name, last_run, result).

5. Helpdesk enrichment

   • Always request include=journals,helpdesk (or configurable) so every cached ticket already contains the rich contact/source metadata.

---

2. Proposed Architecture

2.1 Tool location & reuse

• New script: /home/jason/projects/redmine/redmine_tickets.py
• Reuses the existing RedmineClient class (especially issue_with_helpdesk, get_issue, list methods).
• Same argument patterns, dry-run safety, --apply, error handling, and completion scripts as redmine_contacts.py.

2.2 Cache layout (local filesystem)

Recommended structure (under a configurable --cache-dir, default /home/jason/projects/redmine/cache/tickets/ or /tmp/redmine-tickets-cache/):

cache/tickets/
├── manifest.json                 # { "ticket_id": { "updated_on": "...", "last_synced": "...", "actions": {...} } }
├── customer-service/
│   ├── 12345.json                # full enriched payload
│   ├── 12346.json
│   └── ...
└── other-project/
    └── ...

• One JSON file per ticket (stable, human-inspectable, easy to diff).
• manifest.json tracks updated_on + per-action idempotency state.
• Simple mtime + updated_on comparison is sufficient for v1; manifest makes it robust.
• Implementation note: ticket payload storage and ticket/action state storage are separated in code. ManifestTicketStateStore is the current implementation, but TicketCache only depends on a small state-store method set (get_ticket_state, upsert_ticket_state, has_action, record_action). If the manifest becomes cumbersome with thousands of tickets, replace it with SQLite or another backend without rewriting sync/search/action logic.

2.3 CLI structure (v1 scope)

./redmine_tickets.py \
  --base-url http://192.168.50.170 \
  --project customer-service \
  [sync|search] \
  [--updated-since YYYY-MM-DD] \
  [--updated-before YYYY-MM-DD] \
  [--status open|closed|*] \
  [--category "Spam"] \
  [--max-tickets N] \
  [--order newest-updated-first] \
  [--cache-dir ...] \
  [--dry-run] [--apply] \
  [--json] [--jsonl]

Commands

• sync (default if omitted): Perform incremental fetch + cache update.
• search: Same selection logic, but only output matching tickets (no cache write unless --write-cache).

Safety

• Dry-run by default for any write.
• --apply required to persist cache or perform mutations.

2.4 Idempotency & action hooks (v1 + v2)

• Manifest stores per-ticket action state:

  "actions": {
    "spam-check": { "last_run": "2026-07-09", "result": "not-spam" },
    "close-if-spam": { "last_run": "...", "result": "skipped" }
  }
  

• Future workers read the manifest before acting and update it after.

---

3. Implementation Phases

Phase 1 — Core sync + cache (implemented tracer)

• Implement RedmineClient enhancements if needed (list issues with include=journals,helpdesk, pagination).
• Basic sync command that:
  • Accepts date/status/project filters.
  • Fetches pages of issues.
  • For each issue, writes cache/<project>/<id>.json only if newer than manifest entry.
  • Updates manifest.
• --max-tickets safety limit.
• Dry-run shows what would be fetched/updated.

Verified 2026-07-09: redmine_tickets.py sync pulled 10 open customer-service tickets from LAN Redmine (192.168.50.170), wrote enriched per-ticket JSON, and a second identical run fetched 0 / skipped 10 unchanged cached tickets.

Phase 2 — Search / filter-only mode (implemented tracer)

• search subcommand that reuses the same selection engine.
• Output options: human table, --json, --jsonl (for downstream agents).
• Ability to pipe results into later tools.

Verified 2026-07-09: cached search over the 10-ticket LAN cache listed the same newest open tickets seen in the UI, including promo candidates like #41788 and #41789.

Phase 3 — Idempotency manifest + action skeleton (implemented tracer)

• Add manifest read/write helpers.
• Skeleton --action interface (initially just records “action X run on ticket Y”).
• Example action: spam-check (calls the logic we started in spam_promo_scorer.py but at ticket level).

Verified 2026-07-09: spam-check identified #41788 and #41789 as SPAM based on promotional subject language, recorded local manifest idempotency with --apply, and a second spam-check run checked 0 tickets. close-if-spam dry-run planned closures for those two tickets and reviewed sender contact_id=621, finding 4/10 recent tickets scored as spam (likely_spammer=True). No Redmine close mutation has been applied yet.

Phase 4 — Spam cleanup worker (next)

• Concrete spam-check + close-if-spam actions.
• Rules based on sender patterns, subject language, lack of human replies, existing contact tags, etc.
• Produces a report + optional --apply to close tickets and add spam tag to contact.

Preferred spam policy:

1. When a ticket is high-confidence spam, close that ticket.
2. Use helpdesk_ticket.contact.id to fetch/analyze the last 5–10 tickets from the same contact.
3. Decide whether the sender/contact appears to be a recurring spammer or whether only the specific ticket/email is spam.
4. Only mark/tag the contact as a spammer when the contact-level lookback supports that conclusion.
5. Record both close-if-spam and contact-spam-review idempotency markers so repeated runs do not re-close/re-review the same work.
6. Treat Redmine impersonation like sudo: do not use it for ordinary reads or dry-run planning, but all spam-cleanup write calls should impersonate rebot / “Harry Botter” so Redmine activity logs clearly attribute automated cleanup work.

---

4. Non-Goals (for this phase)

• Do not replace or modify any NMI cron scripts.
• Do not build full agent orchestration yet (this is the data layer).
• Do not implement complex NLP classifiers in v1 (keep scoring simple and reviewable).
• Cache does not need to be a database (filesystem + manifest is sufficient and inspectable).

---

5. Success Criteria

• Running the tool twice in a row with the same filters produces zero API calls on the second run (everything is cached and up-to-date).
• A filter-only search for “tickets updated in last 30 days with no human reply” returns results in < 5 seconds after initial sync.
• The same tool can be pointed at any project and produce usable enriched ticket JSON.
• Higher-level workers can be written as thin Python scripts that consume the cache/manifest without needing their own Redmine client logic.

---

6. Next Immediate Action

The implementation tracer is complete through dry-run close-if-spam.

Next safe execution sequence:

1. Confirm LAN issue status_id=5 is the intended closed status.
2. Run close-if-spam --apply against the LAN copy only, impersonating rebot (the CLI default).
3. Re-sync or inspect the affected LAN tickets.
4. Verify:
   • #41788 and #41789 are closed,
   • Redmine activity author is rebot / Harry Botter,
   • local state records close-if-spam and contact-spam-review,
   • a second close-if-spam --apply does nothing.
5. Only then expand the batch size beyond the initial 10-ticket cache.

Detailed handoff: docs/ticket_spam_cleanup_handoff_2026-07-09.md.

---

References

• docs/redmine_issue_api_helpdesk_include.md (the key enabler)
• redmine_contacts.py (client patterns, filtering, dry-run UX)
• docs/contact_triage_loop_* (overall workflow philosophy)
• RebotSlick RebotSlick.php (historical cache idea — modernized)

This plan deliberately keeps scope tight while giving us a clean, extensible foundation for the spam-reduction work that is currently blocking the customer-service UI.