redmine/docs/known_spam_fast_close_plan.md

# Known-Spam Fast Close Implementation Plan

**Date:** 2026-07-09  
**Project:** `/home/jason/projects/redmine`  
**Primary tool:** `redmine_tickets.py`  
**Target Redmine project:** `customer-service`  
**Production base URL:** `https://rm.ldrprep.com`  
**Agent write attribution:** `rebot` / Harry Botter  

## Goal

Build a reliable two-prong production spam-cleanup workflow:

1. **Slow/reviewed path** — the current classifier-driven `sync → spam-check → close-if-spam` flow, used for discovering and validating new spam patterns.
2. **Fast known-profile path** — a verified-profile bulk closer that quickly locates all open `customer-service` tickets matching known spam sender/profile rules and closes them in one bounded, idempotent production operation.

The immediate business goal is to close known promotional/spam clutter quickly without risking customer tickets.

---

## Current Reality

The slow path already works in production:

- `redmine_tickets.py sync` caches enriched tickets using `include=journals,helpdesk`.
- `spam-check` categorizes tickets into classes such as:
  - `spam-promotional`
  - `automated-operational`
  - `unknown-review`
- `close-if-spam` closes reviewed `spam-promotional` tickets using `--impersonate-user rebot`.
- Writes are verified by API as `status=Closed` and close-note author `Harry B.`.
- Idempotency is recorded via local action markers.
- Customer-contact guard is active: before spam-close, the tool fetches the Helpdesk contact and skips if the contact has tag `Customer`.

Validated production cleanup so far:

- First production pass closed 20 reviewed promotional-junk tickets.
- Current 200-ticket cache has a reviewed 27-ticket candidate set pending apply.
- Customer pre-tagging has been performed for confirmed customers including Cyberstorm, Esire-family contacts, Shinkiya, Bohoflow, BavariaTrachten, Solride, and u2wear.

Known safety rule:

> Absence of `Customer` does **not** imply spam. Presence of `Customer` blocks automated spam closure.

---

## Desired Two-Prong Architecture

```text
                              ┌────────────────────────────┐
                              │ redmine_tickets.py substrate│
                              │ sync/cache/helpdesk/actions │
                              └──────────────┬─────────────┘
                                             │
                 ┌───────────────────────────┴───────────────────────────┐
                 │                                                       │
      Slow / reviewed path                                      Fast / known profile path
      discover new patterns                                     close already-verified spam
                 │                                                       │
 sync → spam-check → review → close-if-spam          known-spam-profile scan → dry-run → apply
                 │                                                       │
       produces new safe profiles                              uses frozen allowlisted rules
```

### Slow path responsibilities

Use for:

- New sender types.
- Ambiguous business/vendor/customer cases.
- Classifier development.
- False-positive discovery.
- Contact/customer enrichment.

### Fast path responsibilities

Use for:

- Previously reviewed spam senders/domains/campaigns.
- High-volume obvious junk such as known promo newsletters and sales blasts.
- Rapid production cleanup once a profile has been approved.

The fast path must not rediscover intent. It should only act on a curated profile registry.

---

## Fast Known-Spam Profile Design

### Risk Model Update — Closed Is Recoverable

Jason clarified an important Redmine/Helpdesk behavior:

- Closed tickets are not deleted.
- If the other party sends a new message to a closed ticket, Redmine reopens it and status changes to `New Message`.

That means accidental closure is usually recoverable if the other party still cares enough to reply. The larger risk is **re-closing a ticket after the sender already reopened it**, because that means the system is ignoring renewed human intent.

Policy consequence:

1. If a ticket is clearly `spam-promotional` and stale, especially `updated_on` older than 30 days, it is reasonable to close automatically once other safety gates pass.
2. If a ticket has evidence that it was previously closed and later reopened by a new Helpdesk message, do **not** close it automatically, even if it matches spam.
3. Instead, escalate it for human attention: bump priority one level if possible and optionally set a due date 1–3 days out.

### Terminology

A **known spam profile** is a conservative, reviewed rule bundle that identifies tickets we are comfortable closing automatically, subject to safety gates.

Example profiles:

```yaml
profiles:
  globalindustrial-promos:
    senders:
      - reply@globalindustrial.com
    subject_patterns:
      - "(?i)\\b(clearance|sale|save|% off|bestsellers|fans|blowers|cooling|storage|power tools|kee safety)\\b"
    category: spam-promotional
    confidence: high
    notes: "Repeated promotional vendor emails; reviewed in LAN + prod broom passes."

  vistaprint-promos:
    senders:
      - noreply@mail.vistaprint.com
    subject_patterns:
      - "(?i)\\b(sale|save|% off|email exclusive|ends|qualifying orders|favorite sale|customer faves)\\b"
    category: spam-promotional
    confidence: high

  qnap-newsletters:
    senders:
      - enews-noreply@qnap.com
    subject_patterns:
      - "(?i)\\b(qnap|photodirector|nas|security advisory|product of the year|jc-star|file backup)\\b"
    category: spam-promotional
    confidence: high

  business-journals-promos:
    senders:
      - replycincinnati@engaged.bizjournals.com
      - replycincinnati@partner.bizjournals.com
      - acollett@bizjournals.com
    subject_patterns:
      - "(?i)\\b(events|nominations|business summit|brand next to key business lists|chambercomp)\\b"
    category: spam-promotional
    confidence: high
```

### Safety gates always applied before close

Fast path must still apply these universal gates:

1. Ticket must be open.
2. Ticket must be in `customer-service` unless explicitly overridden.
3. Ticket Helpdesk contact must be fetched.
4. Contact with tag `Customer` must be skipped.
5. Optional: contact with tag `maybe-customer` should be skipped by default unless `--allow-maybe-customer` is explicitly passed.
6. Ticket must not already have local `known-spam-close` / `close-if-spam` marker.
7. If the ticket was previously closed and then reopened by a new Helpdesk message, skip spam-close and escalate instead.
8. Apply writes must impersonate `rebot`.
9. Apply must produce a verification report.

### Reopened-ticket escalation behavior

Add a helper such as `ticket_was_reopened_after_close(issue)` that inspects journals/details for evidence of a transition from Closed/Resolved into `New Message` or another open status after a Helpdesk reply.

When true, do not close as spam. Instead, planned behavior should be:

```json
{
  "priority_id": "one level higher if known/mappable",
  "due_date": "today + 1 to 3 days",
  "notes": "Matched spam profile, but ticket appears to have been reopened after closure. Escalating for human review instead of auto-closing."
}
```

This should be its own idempotent action marker, e.g. `spam-reopen-escalation`, so the ticket is not bumped repeatedly.

---

## Files to Modify

- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Modify: `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- Create: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json`
- Create: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md`
- Create/Update: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md`
- Update: `/home/jason/projects/redmine/HANDOFF.md`
- Update: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- Optional later: `/home/jason/projects/redmine/docs/known_spam_profile_registry.md`

---

## Proposed CLI

### Dry-run known-spam scan

```bash
cd /home/jason/projects/redmine

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
  known-spam-scan \
  --profile profiles/known-spam/customer-service.yaml \
  --status-id open \
  --max-tickets 2000
```

Expected output:

```text
Known-spam scan complete: seen=1732 matched=214 skipped_customer=3 skipped_maybe_customer=5 planned=206 apply=False
profile=globalindustrial-promos issue=... sender=... subject=...
...
Dry run only. Re-run known-spam-close --apply using this cache/profile after review.
```

### Apply known-spam close

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
  known-spam-close \
  --profile profiles/known-spam/customer-service.yaml \
  --close-status-id 5 \
  --impersonate-user rebot \
  --apply
```

Expected output:

```text
Known-spam close complete: planned=206 closed=206 skipped_customer=0 skipped_existing_marker=0 apply=True
```

### Verify apply

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
  verify-known-spam-close
```

Expected output:

```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```

---

## Implementation Tasks

### Task 1: Add profile loader

**Objective:** Load known-spam YAML profiles safely.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
- Create: `profiles/known-spam/customer-service.yaml`

**Implementation notes:**

- Use stdlib if PyYAML is not guaranteed? Existing environment may support YAML via project deps, but avoid installing packages.
- Safer first version: store profile as JSON instead of YAML if dependency uncertainty matters.
- Recommended file if avoiding deps:

```text
profiles/known-spam/customer-service.json
```

**Test cases:**

- valid profile loads
- missing required fields fails
- invalid regex fails with helpful error

**Acceptance:**

```bash
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```

---

### Task 2: Implement known-profile matcher

**Objective:** Determine whether a cached ticket matches a known spam profile.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Rules:**

A ticket matches when:

- sender matches one of profile senders/domains, and
- subject/body pattern matches if provided, and
- category is `spam-promotional`, and
- it does not match profile exclusions.

**Test examples:**

- `reply@globalindustrial.com` + `Clearance Sale: Top Steel Storage And Power Tools` matches `globalindustrial-promos`.
- `reply@globalindustrial.com` + a real invoice/order/customer thread does **not** match if exclusion pattern present.
- `enews-noreply@qnap.com` + `QNAP Security Advisory` matches only if we decide QNAP security advisories are clutter; this profile should be explicit so no one later acts surprised.

---

### Task 3: Add customer/maybe-customer universal skip gates

**Objective:** Ensure known-spam fast path has the same safety guard as slow path.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Tests:**

- contact tagged `Customer` is skipped even if ticket matches known profile.
- contact tagged `maybe-customer` is skipped by default.
- `--allow-maybe-customer` permits maybe-customer if explicitly requested.
- absence of tags does not block known spam profile matching.

**Acceptance:**

Fast path must never bypass customer safety.

---

### Task 3B: Detect reopened-after-close tickets and escalate instead of closing

**Objective:** Prevent the cleaner from repeatedly closing a ticket after the other party has reopened it with a new message.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Implementation notes:**

- Add `ticket_was_reopened_after_close(issue)`.
- Inspect `issue['journals']` and their `details` for status changes.
- Treat evidence of Closed/Resolved → New Message/open status as reopened-after-close.
- If exact status IDs are easier than names, map known status names/IDs from existing Redmine data before hardcoding.
- Do not rely on current status alone; the point is historical reopening.

**Tests:**

- Journal with status change Closed → New Message returns true.
- Journal with only New → Closed returns false.
- Known-spam close skips reopened-after-close tickets.
- Apply path records `spam-reopen-escalation` instead of `known-spam-close`.

**Escalation behavior:**

- Dry-run reports the escalation plan.
- Apply should update priority/due date only when explicit `--apply-escalations` or equivalent is provided; do not mix escalation writes into spam-close apply without a clear flag.

---

### Task 4: Add `known-spam-scan` command

**Objective:** Sync/search open tickets and report matches without closing.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Behavior:**

- Fetch open `customer-service` tickets with `include=journals,helpdesk`.
- Cache ticket payloads.
- Evaluate known spam profiles.
- Fetch contacts for matched tickets only.
- Report:
  - matched profile
  - issue ID
  - sender
  - subject
  - contact ID
  - age / `updated_on`
  - skipped reason if skipped
- Mark tickets older than 30 days and matching a verified `spam-promotional` profile as strong fast-close candidates.
- Newer tickets may still be candidates, but the report should separate them from stale 30+ day candidates.
- Record local action marker:
  - `known-spam-scan = matched|matched-stale|skipped-customer|skipped-maybe-customer|skipped-reopened|not-matched`

**Dry-run only:** This command should never write Redmine.

---

### Task 5: Add `known-spam-close` command

**Objective:** Close the previously scanned/planned known-spam tickets.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Behavior:**

- Reads cache + scan markers.
- Only closes tickets with marker `known-spam-scan = matched`.
- Re-checks contact tags before close. Do not rely only on stale scan state.
- Updates ticket with:

```json
{
  "status_id": 5,
  "notes": "Closed as known promotional spam by redmine_tickets.py known-spam-close. Profile: <profile-name>."
}
```

- Writes as `rebot`.
- Records:
  - `known-spam-close = closed`
  - profile name
  - close payload
  - verification pending

**Tests:**

- apply closes only matched markers
- second apply closes zero
- customer tag recheck blocks closure even if scan marker was matched
- impersonation defaults to `rebot`

---

### Task 6: Add verification command/report

**Objective:** Verify production writes after apply.

**Files:**

- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`

**Behavior:**

For every `known-spam-close = closed` marker:

- fetch issue with `include=journals,helpdesk`
- verify `status.name == Closed`
- verify close note exists
- verify note author is `Harry B.`
- print failures clearly

**Output:**

```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```

---

### Task 7: Seed first known-spam profile registry from verified production data

**Objective:** Convert already-validated spam senders into conservative profiles.

**Initial profile groups:**

- Global Industrial promos
- Vistaprint promos
- QNAP newsletters/promos
- Business Journals / BizJournals promo/events
- TikTok Shop seller promos
- Popmenu restaurant promos
- Samsung promo newsletters
- Cintas mop/product promos
- Wings and Rings restaurant promos
- SellerEssentials promos

**Do not include yet without explicit decision:**

- `sales@supermarketdistribution.com` / stock/MOQ updates — may be vendor-ish; review first.
- QNAP security advisories — likely clutter, but security language can look operational; make it explicit in profile if included.
- Any sender with `Customer` or `maybe-customer` contacts.

---

### Task 8: Production rollout procedure

**Objective:** Define exact operator sequence for today.

**Step A: Commit or checkpoint local code**

Before production bulk close, ensure current code/tests/docs are checkpointed in git or at least `git diff` saved.

```bash
cd /home/jason/projects/redmine
git status --short
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```

**Step B: First fast-path dry run**

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
  known-spam-scan \
  --profile profiles/known-spam/customer-service.json \
  --status-id open \
  --max-tickets 2000
```

**Step C: Human review**

Review summary counts and sample candidate rows. If any candidate looks like customer/prospect/operational context, do not apply; add exclusion/profile refinement.

**Step D: Apply**

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir <reviewed-cache> \
  known-spam-close \
  --profile profiles/known-spam/customer-service.json \
  --close-status-id 5 \
  --impersonate-user rebot \
  --apply
```

**Step E: Verify**

```bash
./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir <reviewed-cache> \
  verify-known-spam-close
```

**Step F: Handoff update**

Update:

- `HANDOFF.md`
- `docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- optionally `docs/known_spam_profile_registry.md`

---

## Acceptance Criteria

The implementation is acceptable when:

1. Slow path still works and tests pass.
2. Fast path can dry-run all open production customer-service tickets without writes.
3. Fast path can report exact match profile per candidate.
4. Fast path skips contacts tagged `Customer`.
5. Fast path skips contacts tagged `maybe-customer` by default.
6. Fast path writes only when `--apply` is passed.
7. Fast path writes as `rebot`.
8. Second apply is idempotent.
9. Verification command proves status closed + Harry B. note author.
10. Profile registry is conservative and versioned in the repo.

---

## Short-Term Recommendation for Today

Before building the full fast-profile command, there is still an immediate safe action available:

1. Apply the currently reviewed 27-ticket production close set from:

```text
.cache/redmine_tickets_prod_200_promotional_eval_1783612885
```

2. Verify closed / Harry B. / idempotency.
3. Then implement the fast known-spam profile path above.

This keeps the broom moving while the turbo-broom gets brakes installed. Very advanced broom science.

---

## Legacy Rebot Rule Extraction Status

Completed source inspection of old Rebot spam/routing logic and captured it in:

```text
docs/rebot_legacy_spam_rules.md
profiles/known-spam/customer-service.json
```

Validated profile seed:

```text
JSON_OK
profiles 7
globalindustrial-promos spam-promotional 1 sender regex / 1 subject regex
vistaprint-promos spam-promotional 1 / 1
qnap-newsletters spam-promotional 1 / 1
business-journals-promos spam-promotional 3 / 1
legacy-rebot-promotional-senders spam-promotional 16 / 0
legacy-rebot-operational-clutter automated-operational 4 / 10
legacy-rebot-receipts-routing-reference routing-reference 4 / 1
```

Important split:

- `legacy-rebot-promotional-senders` can be considered for fast close after dry-run review.
- `legacy-rebot-operational-clutter` preserves old Rebot close rules but should remain report-only until Jason explicitly approves operational clutter closing.
- `legacy-rebot-receipts-routing-reference` should not be closed as spam; it is future routing/reference material.