redmine/docs/known_spam_fast_close_plan.md

Known-Spam Fast Close Implementation Plan

Date: 2026-07-09
Project: /home/jason/projects/redmine
Primary tool: redmine_tickets.py
Target Redmine project: customer-service
Production base URL: https://rm.ldrprep.com
Agent write attribution: rebot / Harry Botter

Goal

Build a reliable two-prong production spam-cleanup workflow:

1. Slow/reviewed path — the current classifier-driven sync → spam-check → close-if-spam flow, used for discovering and validating new spam patterns.
2. Fast known-profile path — a verified-profile bulk closer that quickly locates all open customer-service tickets matching known spam sender/profile rules and closes them in one bounded, idempotent production operation.

The immediate business goal is to close known promotional/spam clutter quickly without risking customer tickets.

---

Current Reality

The slow path already works in production:

• redmine_tickets.py sync caches enriched tickets using include=journals,helpdesk.
• spam-check categorizes tickets into classes such as:
  • spam-promotional
  • automated-operational
  • unknown-review
• close-if-spam closes reviewed spam-promotional tickets using --impersonate-user rebot.
• Writes are verified by API as status=Closed and close-note author Harry B..
• Idempotency is recorded via local action markers.
• Customer-contact guard is active: before spam-close, the tool fetches the Helpdesk contact and skips if the contact has tag Customer.

Validated production cleanup so far:

• First production pass closed 20 reviewed promotional-junk tickets.
• Current 200-ticket cache has a reviewed 27-ticket candidate set pending apply.
• Customer pre-tagging has been performed for confirmed customers including Cyberstorm, Esire-family contacts, Shinkiya, Bohoflow, BavariaTrachten, Solride, and u2wear.

Known safety rule:

Absence of Customer does not imply spam. Presence of Customer blocks automated spam closure.

---

Desired Two-Prong Architecture

                              ┌────────────────────────────┐
                              │ redmine_tickets.py substrate│
                              │ sync/cache/helpdesk/actions │
                              └──────────────┬─────────────┘
                                             │
                 ┌───────────────────────────┴───────────────────────────┐
                 │                                                       │
      Slow / reviewed path                                      Fast / known profile path
      discover new patterns                                     close already-verified spam
                 │                                                       │
 sync → spam-check → review → close-if-spam          known-spam-profile scan → dry-run → apply
                 │                                                       │
       produces new safe profiles                              uses frozen allowlisted rules

Slow path responsibilities

Use for:

• New sender types.
• Ambiguous business/vendor/customer cases.
• Classifier development.
• False-positive discovery.
• Contact/customer enrichment.

Fast path responsibilities

Use for:

• Previously reviewed spam senders/domains/campaigns.
• High-volume obvious junk such as known promo newsletters and sales blasts.
• Rapid production cleanup once a profile has been approved.

The fast path must not rediscover intent. It should only act on a curated profile registry.

---

Fast Known-Spam Profile Design

Risk Model Update — Closed Is Recoverable

Jason clarified an important Redmine/Helpdesk behavior:

• Closed tickets are not deleted.
• If the other party sends a new message to a closed ticket, Redmine reopens it and status changes to New Message.

That means accidental closure is usually recoverable if the other party still cares enough to reply. The larger risk is re-closing a ticket after the sender already reopened it, because that means the system is ignoring renewed human intent.

Policy consequence:

1. If a ticket is clearly spam-promotional and stale, especially updated_on older than 30 days, it is reasonable to close automatically once other safety gates pass.
2. If a ticket has evidence that it was previously closed and later reopened by a new Helpdesk message, do not close it automatically, even if it matches spam.
3. Instead, escalate it for human attention: bump priority one level if possible and optionally set a due date 1–3 days out.

Terminology

A known spam profile is a conservative, reviewed rule bundle that identifies tickets we are comfortable closing automatically, subject to safety gates.

Example profiles:

profiles:
  globalindustrial-promos:
    senders:
      - reply@globalindustrial.com
    subject_patterns:
      - "(?i)\\b(clearance|sale|save|% off|bestsellers|fans|blowers|cooling|storage|power tools|kee safety)\\b"
    category: spam-promotional
    confidence: high
    notes: "Repeated promotional vendor emails; reviewed in LAN + prod broom passes."

  vistaprint-promos:
    senders:
      - noreply@mail.vistaprint.com
    subject_patterns:
      - "(?i)\\b(sale|save|% off|email exclusive|ends|qualifying orders|favorite sale|customer faves)\\b"
    category: spam-promotional
    confidence: high

  qnap-newsletters:
    senders:
      - enews-noreply@qnap.com
    subject_patterns:
      - "(?i)\\b(qnap|photodirector|nas|security advisory|product of the year|jc-star|file backup)\\b"
    category: spam-promotional
    confidence: high

  business-journals-promos:
    senders:
      - replycincinnati@engaged.bizjournals.com
      - replycincinnati@partner.bizjournals.com
      - acollett@bizjournals.com
    subject_patterns:
      - "(?i)\\b(events|nominations|business summit|brand next to key business lists|chambercomp)\\b"
    category: spam-promotional
    confidence: high

Safety gates always applied before close

Fast path must still apply these universal gates:

1. Ticket must be open.
2. Ticket must be in customer-service unless explicitly overridden.
3. Ticket Helpdesk contact must be fetched.
4. Contact with tag Customer must be skipped.
5. Optional: contact with tag maybe-customer should be skipped by default unless --allow-maybe-customer is explicitly passed.
6. Ticket must not already have local known-spam-close / close-if-spam marker.
7. If the ticket was previously closed and then reopened by a new Helpdesk message, skip spam-close and escalate instead.
8. Apply writes must impersonate rebot.
9. Apply must produce a verification report.

Reopened-ticket escalation behavior

Add a helper such as ticket_was_reopened_after_close(issue) that inspects journals/details for evidence of a transition from Closed/Resolved into New Message or another open status after a Helpdesk reply.

When true, do not close as spam. Instead, planned behavior should be:

{
  "priority_id": "one level higher if known/mappable",
  "due_date": "today + 1 to 3 days",
  "notes": "Matched spam profile, but ticket appears to have been reopened after closure. Escalating for human review instead of auto-closing."
}

This should be its own idempotent action marker, e.g. spam-reopen-escalation, so the ticket is not bumped repeatedly.

---

Files to Modify

• Modify: /home/jason/projects/redmine/redmine_tickets.py
• Modify: /home/jason/projects/redmine/tests/test_redmine_tickets.py
• Create: /home/jason/projects/redmine/profiles/known-spam/customer-service.json
• Create: /home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md
• Create/Update: /home/jason/projects/redmine/docs/known_spam_fast_close_plan.md
• Update: /home/jason/projects/redmine/HANDOFF.md
• Update: /home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md
• Optional later: /home/jason/projects/redmine/docs/known_spam_profile_registry.md

---

Proposed CLI

Dry-run known-spam scan

cd /home/jason/projects/redmine

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
  known-spam-scan \
  --profile profiles/known-spam/customer-service.yaml \
  --status-id open \
  --max-tickets 2000

Expected output:

Known-spam scan complete: seen=1732 matched=214 skipped_customer=3 skipped_maybe_customer=5 planned=206 apply=False
profile=globalindustrial-promos issue=... sender=... subject=...
...
Dry run only. Re-run known-spam-close --apply using this cache/profile after review.

Apply known-spam close

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
  known-spam-close \
  --profile profiles/known-spam/customer-service.yaml \
  --close-status-id 5 \
  --impersonate-user rebot \
  --apply

Expected output:

Known-spam close complete: planned=206 closed=206 skipped_customer=0 skipped_existing_marker=0 apply=True

Verify apply

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
  verify-known-spam-close

Expected output:

Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0

---

Implementation Tasks

Task 1: Add profile loader

Objective: Load known-spam YAML profiles safely.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py
• Create: profiles/known-spam/customer-service.yaml

Implementation notes:

• Use stdlib if PyYAML is not guaranteed? Existing environment may support YAML via project deps, but avoid installing packages.
• Safer first version: store profile as JSON instead of YAML if dependency uncertainty matters.
• Recommended file if avoiding deps:

profiles/known-spam/customer-service.json

Test cases:

• valid profile loads
• missing required fields fails
• invalid regex fails with helpful error

Acceptance:

python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py

---

Task 2: Implement known-profile matcher

Objective: Determine whether a cached ticket matches a known spam profile.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Rules:

A ticket matches when:

• sender matches one of profile senders/domains, and
• subject/body pattern matches if provided, and
• category is spam-promotional, and
• it does not match profile exclusions.

Test examples:

• reply@globalindustrial.com + Clearance Sale: Top Steel Storage And Power Tools matches globalindustrial-promos.
• reply@globalindustrial.com + a real invoice/order/customer thread does not match if exclusion pattern present.
• enews-noreply@qnap.com + QNAP Security Advisory matches only if we decide QNAP security advisories are clutter; this profile should be explicit so no one later acts surprised.

---

Task 3: Add customer/maybe-customer universal skip gates

Objective: Ensure known-spam fast path has the same safety guard as slow path.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Tests:

• contact tagged Customer is skipped even if ticket matches known profile.
• contact tagged maybe-customer is skipped by default.
• --allow-maybe-customer permits maybe-customer if explicitly requested.
• absence of tags does not block known spam profile matching.

Acceptance:

Fast path must never bypass customer safety.

---

Task 3B: Detect reopened-after-close tickets and escalate instead of closing

Objective: Prevent the cleaner from repeatedly closing a ticket after the other party has reopened it with a new message.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Implementation notes:

• Add ticket_was_reopened_after_close(issue).
• Inspect issue['journals'] and their details for status changes.
• Treat evidence of Closed/Resolved → New Message/open status as reopened-after-close.
• If exact status IDs are easier than names, map known status names/IDs from existing Redmine data before hardcoding.
• Do not rely on current status alone; the point is historical reopening.

Tests:

• Journal with status change Closed → New Message returns true.
• Journal with only New → Closed returns false.
• Known-spam close skips reopened-after-close tickets.
• Apply path records spam-reopen-escalation instead of known-spam-close.

Escalation behavior:

• Dry-run reports the escalation plan.
• Apply should update priority/due date only when explicit --apply-escalations or equivalent is provided; do not mix escalation writes into spam-close apply without a clear flag.

---

Task 4: Add known-spam-scan command

Objective: Sync/search open tickets and report matches without closing.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Behavior:

• Fetch open customer-service tickets with include=journals,helpdesk.
• Cache ticket payloads.
• Evaluate known spam profiles.
• Fetch contacts for matched tickets only.
• Report:
  • matched profile
  • issue ID
  • sender
  • subject
  • contact ID
  • age / updated_on
  • skipped reason if skipped
• Mark tickets older than 30 days and matching a verified spam-promotional profile as strong fast-close candidates.
• Newer tickets may still be candidates, but the report should separate them from stale 30+ day candidates.
• Record local action marker:
  • known-spam-scan = matched|matched-stale|skipped-customer|skipped-maybe-customer|skipped-reopened|not-matched

Dry-run only: This command should never write Redmine.

---

Task 5: Add known-spam-close command

Objective: Close the previously scanned/planned known-spam tickets.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Behavior:

• Reads cache + scan markers.
• Only closes tickets with marker known-spam-scan = matched.
• Re-checks contact tags before close. Do not rely only on stale scan state.
• Updates ticket with:

{
  "status_id": 5,
  "notes": "Closed as known promotional spam by redmine_tickets.py known-spam-close. Profile: <profile-name>."
}

• Writes as rebot.
• Records:
  • known-spam-close = closed
  • profile name
  • close payload
  • verification pending

Tests:

• apply closes only matched markers
• second apply closes zero
• customer tag recheck blocks closure even if scan marker was matched
• impersonation defaults to rebot

---

Task 6: Add verification command/report

Objective: Verify production writes after apply.

Files:

• Modify: redmine_tickets.py
• Test: tests/test_redmine_tickets.py

Behavior:

For every known-spam-close = closed marker:

• fetch issue with include=journals,helpdesk
• verify status.name == Closed
• verify close note exists
• verify note author is Harry B.
• print failures clearly

Output:

Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0

---

Task 7: Seed first known-spam profile registry from verified production data

Objective: Convert already-validated spam senders into conservative profiles.

Initial profile groups:

• Global Industrial promos
• Vistaprint promos
• QNAP newsletters/promos
• Business Journals / BizJournals promo/events
• TikTok Shop seller promos
• Popmenu restaurant promos
• Samsung promo newsletters
• Cintas mop/product promos
• Wings and Rings restaurant promos
• SellerEssentials promos

Do not include yet without explicit decision:

• sales@supermarketdistribution.com / stock/MOQ updates — may be vendor-ish; review first.
• QNAP security advisories — likely clutter, but security language can look operational; make it explicit in profile if included.
• Any sender with Customer or maybe-customer contacts.

---

Task 8: Production rollout procedure

Objective: Define exact operator sequence for today.

Step A: Commit or checkpoint local code

Before production bulk close, ensure current code/tests/docs are checkpointed in git or at least git diff saved.

cd /home/jason/projects/redmine
git status --short
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py

Step B: First fast-path dry run

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
  known-spam-scan \
  --profile profiles/known-spam/customer-service.json \
  --status-id open \
  --max-tickets 2000

Step C: Human review

Review summary counts and sample candidate rows. If any candidate looks like customer/prospect/operational context, do not apply; add exclusion/profile refinement.

Step D: Apply

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir <reviewed-cache> \
  known-spam-close \
  --profile profiles/known-spam/customer-service.json \
  --close-status-id 5 \
  --impersonate-user rebot \
  --apply

Step E: Verify

./redmine_tickets.py \
  --base-url https://rm.ldrprep.com \
  --project customer-service \
  --cache-dir <reviewed-cache> \
  verify-known-spam-close

Step F: Handoff update

Update:

• HANDOFF.md
• docs/ticket_spam_cleanup_handoff_2026-07-09.md
• optionally docs/known_spam_profile_registry.md

---

Acceptance Criteria

The implementation is acceptable when:

1. Slow path still works and tests pass.
2. Fast path can dry-run all open production customer-service tickets without writes.
3. Fast path can report exact match profile per candidate.
4. Fast path skips contacts tagged Customer.
5. Fast path skips contacts tagged maybe-customer by default.
6. Fast path writes only when --apply is passed.
7. Fast path writes as rebot.
8. Second apply is idempotent.
9. Verification command proves status closed + Harry B. note author.
10. Profile registry is conservative and versioned in the repo.

---

Short-Term Recommendation for Today

Before building the full fast-profile command, there is still an immediate safe action available:

1. Apply the currently reviewed 27-ticket production close set from:

.cache/redmine_tickets_prod_200_promotional_eval_1783612885

2. Verify closed / Harry B. / idempotency.
3. Then implement the fast known-spam profile path above.

This keeps the broom moving while the turbo-broom gets brakes installed. Very advanced broom science.

---

Legacy Rebot Rule Extraction Status

Completed source inspection of old Rebot spam/routing logic and captured it in:

docs/rebot_legacy_spam_rules.md
profiles/known-spam/customer-service.json

Validated profile seed:

JSON_OK
profiles 7
globalindustrial-promos spam-promotional 1 sender regex / 1 subject regex
vistaprint-promos spam-promotional 1 / 1
qnap-newsletters spam-promotional 1 / 1
business-journals-promos spam-promotional 3 / 1
legacy-rebot-promotional-senders spam-promotional 16 / 0
legacy-rebot-operational-clutter automated-operational 4 / 10
legacy-rebot-receipts-routing-reference routing-reference 4 / 1

Important split:

• legacy-rebot-promotional-senders can be considered for fast close after dry-run review.
• legacy-rebot-operational-clutter preserves old Rebot close rules but should remain report-only until Jason explicitly approves operational clutter closing.
• legacy-rebot-receipts-routing-reference should not be closed as spam; it is future routing/reference material.