ldr/redmine
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Organized

Browse Source
This commit is contained in:
Jason Thistlethwaite
2026-10-17 20:50:35 -04:00
parent bd26c8894f
commit 9ee9eb3ec6
130 changed files with 24172 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known_spam_fast_close_plan.md
+623
View File
@@ -0,0 +1,623 @@
# Known-Spam Fast Close Implementation Plan
**Date:** 2026-07-09
**Project:** `/home/jason/projects/redmine`
**Primary tool:** `redmine_tickets.py`
**Target Redmine project:** `customer-service`
**Production base URL:** `https://rm.ldrprep.com`
**Agent write attribution:** `rebot` / Harry Botter
## Goal
Build a reliable two-prong production spam-cleanup workflow:
1. **Slow/reviewed path** — the current classifier-driven `sync → spam-check → close-if-spam` flow, used for discovering and validating new spam patterns.
2. **Fast known-profile path** — a verified-profile bulk closer that quickly locates all open `customer-service` tickets matching known spam sender/profile rules and closes them in one bounded, idempotent production operation.
The immediate business goal is to close known promotional/spam clutter quickly without risking customer tickets.
---
## Current Reality
The slow path already works in production:
- `redmine_tickets.py sync` caches enriched tickets using `include=journals,helpdesk`.
- `spam-check` categorizes tickets into classes such as:
- `spam-promotional`
- `automated-operational`
- `unknown-review`
- `close-if-spam` closes reviewed `spam-promotional` tickets using `--impersonate-user rebot`.
- Writes are verified by API as `status=Closed` and close-note author `Harry B.`.
- Idempotency is recorded via local action markers.
- Customer-contact guard is active: before spam-close, the tool fetches the Helpdesk contact and skips if the contact has tag `Customer`.
Validated production cleanup so far:
- First production pass closed 20 reviewed promotional-junk tickets.
- Current 200-ticket cache has a reviewed 27-ticket candidate set pending apply.
- Customer pre-tagging has been performed for confirmed customers including Cyberstorm, Esire-family contacts, Shinkiya, Bohoflow, BavariaTrachten, Solride, and u2wear.
Known safety rule:
> Absence of `Customer` does **not** imply spam. Presence of `Customer` blocks automated spam closure.
---
## Desired Two-Prong Architecture
```text
┌────────────────────────────┐
│ redmine_tickets.py substrate│
│ sync/cache/helpdesk/actions │
└──────────────┬─────────────┘
│
┌───────────────────────────┴───────────────────────────┐
│ │
Slow / reviewed path Fast / known profile path
discover new patterns close already-verified spam
│ │
sync → spam-check → review → close-if-spam known-spam-profile scan → dry-run → apply
│ │
produces new safe profiles uses frozen allowlisted rules
```
### Slow path responsibilities
Use for:
- New sender types.
- Ambiguous business/vendor/customer cases.
- Classifier development.
- False-positive discovery.
- Contact/customer enrichment.
### Fast path responsibilities
Use for:
- Previously reviewed spam senders/domains/campaigns.
- High-volume obvious junk such as known promo newsletters and sales blasts.
- Rapid production cleanup once a profile has been approved.
The fast path must not rediscover intent. It should only act on a curated profile registry.
---
## Fast Known-Spam Profile Design
### Risk Model Update — Closed Is Recoverable
Jason clarified an important Redmine/Helpdesk behavior:
- Closed tickets are not deleted.
- If the other party sends a new message to a closed ticket, Redmine reopens it and status changes to `New Message`.
That means accidental closure is usually recoverable if the other party still cares enough to reply. The larger risk is **re-closing a ticket after the sender already reopened it**, because that means the system is ignoring renewed human intent.
Policy consequence:
1. If a ticket is clearly `spam-promotional` and stale, especially `updated_on` older than 30 days, it is reasonable to close automatically once other safety gates pass.
2. If a ticket has evidence that it was previously closed and later reopened by a new Helpdesk message, do **not** close it automatically, even if it matches spam.
3. Instead, escalate it for human attention: bump priority one level if possible and optionally set a due date 1–3 days out.
### Terminology
A **known spam profile** is a conservative, reviewed rule bundle that identifies tickets we are comfortable closing automatically, subject to safety gates.
Example profiles:
```yaml
profiles:
globalindustrial-promos:
senders:
- reply@globalindustrial.com
subject_patterns:
- "(?i)\\b(clearance|sale|save|% off|bestsellers|fans|blowers|cooling|storage|power tools|kee safety)\\b"
category: spam-promotional
confidence: high
notes: "Repeated promotional vendor emails; reviewed in LAN + prod broom passes."
vistaprint-promos:
senders:
- noreply@mail.vistaprint.com
subject_patterns:
- "(?i)\\b(sale|save|% off|email exclusive|ends|qualifying orders|favorite sale|customer faves)\\b"
category: spam-promotional
confidence: high
qnap-newsletters:
senders:
- enews-noreply@qnap.com
subject_patterns:
- "(?i)\\b(qnap|photodirector|nas|security advisory|product of the year|jc-star|file backup)\\b"
category: spam-promotional
confidence: high
business-journals-promos:
senders:
- replycincinnati@engaged.bizjournals.com
- replycincinnati@partner.bizjournals.com
- acollett@bizjournals.com
subject_patterns:
- "(?i)\\b(events|nominations|business summit|brand next to key business lists|chambercomp)\\b"
category: spam-promotional
confidence: high
```
### Safety gates always applied before close
Fast path must still apply these universal gates:
1. Ticket must be open.
2. Ticket must be in `customer-service` unless explicitly overridden.
3. Ticket Helpdesk contact must be fetched.
4. Contact with tag `Customer` must be skipped.
5. Optional: contact with tag `maybe-customer` should be skipped by default unless `--allow-maybe-customer` is explicitly passed.
6. Ticket must not already have local `known-spam-close` / `close-if-spam` marker.
7. If the ticket was previously closed and then reopened by a new Helpdesk message, skip spam-close and escalate instead.
8. Apply writes must impersonate `rebot`.
9. Apply must produce a verification report.
### Reopened-ticket escalation behavior
Add a helper such as `ticket_was_reopened_after_close(issue)` that inspects journals/details for evidence of a transition from Closed/Resolved into `New Message` or another open status after a Helpdesk reply.
When true, do not close as spam. Instead, planned behavior should be:
```json
{
"priority_id": "one level higher if known/mappable",
"due_date": "today + 1 to 3 days",
"notes": "Matched spam profile, but ticket appears to have been reopened after closure. Escalating for human review instead of auto-closing."
}
```
This should be its own idempotent action marker, e.g. `spam-reopen-escalation`, so the ticket is not bumped repeatedly.
---
## Files to Modify
- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Modify: `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- Create: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json`
- Create: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md`
- Create/Update: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md`
- Update: `/home/jason/projects/redmine/HANDOFF.md`
- Update: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- Optional later: `/home/jason/projects/redmine/docs/known_spam_profile_registry.md`
---
## Proposed CLI
### Dry-run known-spam scan
```bash
cd /home/jason/projects/redmine
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
known-spam-scan \
--profile profiles/known-spam/customer-service.yaml \
--status-id open \
--max-tickets 2000
```
Expected output:
```text
Known-spam scan complete: seen=1732 matched=214 skipped_customer=3 skipped_maybe_customer=5 planned=206 apply=False
profile=globalindustrial-promos issue=... sender=... subject=...
...
Dry run only. Re-run known-spam-close --apply using this cache/profile after review.
```
### Apply known-spam close
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
known-spam-close \
--profile profiles/known-spam/customer-service.yaml \
--close-status-id 5 \
--impersonate-user rebot \
--apply
```
Expected output:
```text
Known-spam close complete: planned=206 closed=206 skipped_customer=0 skipped_existing_marker=0 apply=True
```
### Verify apply
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
verify-known-spam-close
```
Expected output:
```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```
---
## Implementation Tasks
### Task 1: Add profile loader
**Objective:** Load known-spam YAML profiles safely.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
- Create: `profiles/known-spam/customer-service.yaml`
**Implementation notes:**
- Use stdlib if PyYAML is not guaranteed? Existing environment may support YAML via project deps, but avoid installing packages.
- Safer first version: store profile as JSON instead of YAML if dependency uncertainty matters.
- Recommended file if avoiding deps:
```text
profiles/known-spam/customer-service.json
```
**Test cases:**
- valid profile loads
- missing required fields fails
- invalid regex fails with helpful error
**Acceptance:**
```bash
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```
---
### Task 2: Implement known-profile matcher
**Objective:** Determine whether a cached ticket matches a known spam profile.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Rules:**
A ticket matches when:
- sender matches one of profile senders/domains, and
- subject/body pattern matches if provided, and
- category is `spam-promotional`, and
- it does not match profile exclusions.
**Test examples:**
- `reply@globalindustrial.com` + `Clearance Sale: Top Steel Storage And Power Tools` matches `globalindustrial-promos`.
- `reply@globalindustrial.com` + a real invoice/order/customer thread does **not** match if exclusion pattern present.
- `enews-noreply@qnap.com` + `QNAP Security Advisory` matches only if we decide QNAP security advisories are clutter; this profile should be explicit so no one later acts surprised.
---
### Task 3: Add customer/maybe-customer universal skip gates
**Objective:** Ensure known-spam fast path has the same safety guard as slow path.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Tests:**
- contact tagged `Customer` is skipped even if ticket matches known profile.
- contact tagged `maybe-customer` is skipped by default.
- `--allow-maybe-customer` permits maybe-customer if explicitly requested.
- absence of tags does not block known spam profile matching.
**Acceptance:**
Fast path must never bypass customer safety.
---
### Task 3B: Detect reopened-after-close tickets and escalate instead of closing
**Objective:** Prevent the cleaner from repeatedly closing a ticket after the other party has reopened it with a new message.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Implementation notes:**
- Add `ticket_was_reopened_after_close(issue)`.
- Inspect `issue['journals']` and their `details` for status changes.
- Treat evidence of Closed/Resolved → New Message/open status as reopened-after-close.
- If exact status IDs are easier than names, map known status names/IDs from existing Redmine data before hardcoding.
- Do not rely on current status alone; the point is historical reopening.
**Tests:**
- Journal with status change Closed → New Message returns true.
- Journal with only New → Closed returns false.
- Known-spam close skips reopened-after-close tickets.
- Apply path records `spam-reopen-escalation` instead of `known-spam-close`.
**Escalation behavior:**
- Dry-run reports the escalation plan.
- Apply should update priority/due date only when explicit `--apply-escalations` or equivalent is provided; do not mix escalation writes into spam-close apply without a clear flag.
---
### Task 4: Add `known-spam-scan` command
**Objective:** Sync/search open tickets and report matches without closing.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
- Fetch open `customer-service` tickets with `include=journals,helpdesk`.
- Cache ticket payloads.
- Evaluate known spam profiles.
- Fetch contacts for matched tickets only.
- Report:
- matched profile
- issue ID
- sender
- subject
- contact ID
- age / `updated_on`
- skipped reason if skipped
- Mark tickets older than 30 days and matching a verified `spam-promotional` profile as strong fast-close candidates.
- Newer tickets may still be candidates, but the report should separate them from stale 30+ day candidates.
- Record local action marker:
- `known-spam-scan = matched|matched-stale|skipped-customer|skipped-maybe-customer|skipped-reopened|not-matched`
**Dry-run only:** This command should never write Redmine.
---
### Task 5: Add `known-spam-close` command
**Objective:** Close the previously scanned/planned known-spam tickets.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
- Reads cache + scan markers.
- Only closes tickets with marker `known-spam-scan = matched`.
- Re-checks contact tags before close. Do not rely only on stale scan state.
- Updates ticket with:
```json
{
"status_id": 5,
"notes": "Closed as known promotional spam by redmine_tickets.py known-spam-close. Profile: <profile-name>."
}
```
- Writes as `rebot`.
- Records:
- `known-spam-close = closed`
- profile name
- close payload
- verification pending
**Tests:**
- apply closes only matched markers
- second apply closes zero
- customer tag recheck blocks closure even if scan marker was matched
- impersonation defaults to `rebot`
---
### Task 6: Add verification command/report
**Objective:** Verify production writes after apply.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
For every `known-spam-close = closed` marker:
- fetch issue with `include=journals,helpdesk`
- verify `status.name == Closed`
- verify close note exists
- verify note author is `Harry B.`
- print failures clearly
**Output:**
```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```
---
### Task 7: Seed first known-spam profile registry from verified production data
**Objective:** Convert already-validated spam senders into conservative profiles.
**Initial profile groups:**
- Global Industrial promos
- Vistaprint promos
- QNAP newsletters/promos
- Business Journals / BizJournals promo/events
- TikTok Shop seller promos
- Popmenu restaurant promos
- Samsung promo newsletters
- Cintas mop/product promos
- Wings and Rings restaurant promos
- SellerEssentials promos
**Do not include yet without explicit decision:**
- `sales@supermarketdistribution.com` / stock/MOQ updates — may be vendor-ish; review first.
- QNAP security advisories — likely clutter, but security language can look operational; make it explicit in profile if included.
- Any sender with `Customer` or `maybe-customer` contacts.
---
### Task 8: Production rollout procedure
**Objective:** Define exact operator sequence for today.
**Step A: Commit or checkpoint local code**
Before production bulk close, ensure current code/tests/docs are checkpointed in git or at least `git diff` saved.
```bash
cd /home/jason/projects/redmine
git status --short
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```
**Step B: First fast-path dry run**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
known-spam-scan \
--profile profiles/known-spam/customer-service.json \
--status-id open \
--max-tickets 2000
```
**Step C: Human review**
Review summary counts and sample candidate rows. If any candidate looks like customer/prospect/operational context, do not apply; add exclusion/profile refinement.
**Step D: Apply**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir <reviewed-cache> \
known-spam-close \
--profile profiles/known-spam/customer-service.json \
--close-status-id 5 \
--impersonate-user rebot \
--apply
```
**Step E: Verify**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir <reviewed-cache> \
verify-known-spam-close
```
**Step F: Handoff update**
Update:
- `HANDOFF.md`
- `docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- optionally `docs/known_spam_profile_registry.md`
---
## Acceptance Criteria
The implementation is acceptable when:
1. Slow path still works and tests pass.
2. Fast path can dry-run all open production customer-service tickets without writes.
3. Fast path can report exact match profile per candidate.
4. Fast path skips contacts tagged `Customer`.
5. Fast path skips contacts tagged `maybe-customer` by default.
6. Fast path writes only when `--apply` is passed.
7. Fast path writes as `rebot`.
8. Second apply is idempotent.
9. Verification command proves status closed + Harry B. note author.
10. Profile registry is conservative and versioned in the repo.
---
## Short-Term Recommendation for Today
Before building the full fast-profile command, there is still an immediate safe action available:
1. Apply the currently reviewed 27-ticket production close set from:
```text
.cache/redmine_tickets_prod_200_promotional_eval_1783612885
```
2. Verify closed / Harry B. / idempotency.
3. Then implement the fast known-spam profile path above.
This keeps the broom moving while the turbo-broom gets brakes installed. Very advanced broom science.
---
## Legacy Rebot Rule Extraction Status
Completed source inspection of old Rebot spam/routing logic and captured it in:
```text
docs/rebot_legacy_spam_rules.md
profiles/known-spam/customer-service.json
```
Validated profile seed:
```text
JSON_OK
profiles 7
globalindustrial-promos spam-promotional 1 sender regex / 1 subject regex
vistaprint-promos spam-promotional 1 / 1
qnap-newsletters spam-promotional 1 / 1
business-journals-promos spam-promotional 3 / 1
legacy-rebot-promotional-senders spam-promotional 16 / 0
legacy-rebot-operational-clutter automated-operational 4 / 10
legacy-rebot-receipts-routing-reference routing-reference 4 / 1
```
Important split:
- `legacy-rebot-promotional-senders` can be considered for fast close after dry-run review.
- `legacy-rebot-operational-clutter` preserves old Rebot close rules but should remain report-only until Jason explicitly approves operational clutter closing.
- `legacy-rebot-receipts-routing-reference` should not be closed as spam; it is future routing/reference material.