ldr/redmine
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Organized

Browse Source
This commit is contained in:
Jason Thistlethwaite
2026-10-17 20:50:35 -04:00
parent bd26c8894f
commit 9ee9eb3ec6
130 changed files with 24172 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HANDOFF.md
+77
View File
@@ -0,0 +1,77 @@
# redmine handoff
## Where to find things
- Project rules: `/home/jason/projects/redmine/PROJECT.md`
- Active checklist: `/home/jason/projects/redmine/PLAN.md`
- Main script touched: `/home/jason/projects/redmine/redmine_contacts.py`
- Ticket sync script: `/home/jason/projects/redmine/redmine_tickets.py`
- Tests added: `/home/jason/projects/redmine/tests/test_redmine_contacts.py`
- Ticket sync tests: `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- Ticket spam cleanup handoff: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- Known-spam fast close plan: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md`
- Rebot legacy spam rules: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md`
- Known-spam profile seed: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json`
- Ticket decomposition plan: `/home/jason/projects/redmine/docs/ticket_decomposition_knowledge_ingestion_plan.md`
- Obsidian process note: `/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md`
- Decomposition pilot contact proposals: `/home/jason/projects/redmine/docs/decomposition_pilot_contact_update_proposals_2026-07-09.json`
- KarpathyWiki plugin inspection: `/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md`
## Last completed milestones
1. Applied the shared PROJECT template into this repo with redmine-specific requirements and continuity-doc expectations.
2. Implemented and tested `redmine_contacts.py` upgrades:
- date/tag filtering helpers used by `fetch` + `search`
- `show` command (cache/api source)
- `issue-full` command (issue + helpdesk context)
- `update` tag and custom-field ergonomic options (`--add-tag`, `--remove-tag`, `--set-custom-field`)
3. Added unit tests for new filter/tag/custom-field behavior and verified:
- `python3 -m unittest discover -s tests -p 'test_redmine_contacts.py' -v`
- `python3 -m py_compile redmine_contacts.py`
4. Completed LAN integration validation on `192.168.50.170`:
- validated `show --source api` and `issue-full`
- executed safe `update --apply` on contact `14` with temporary tag/custom-field, then rolled it back to original state
- locked fetch behavior: default `fetch` caches full results; filtered cache requires `--cache-mode filtered`
5. Implemented and verified a dry-run-first `batch-update` command:
- supports date/tag filters + `--max-contacts`
- supports shared update payload options (`--set`, `--add-tag`, `--remove-tag`, `--set-custom-field`)
- defaults to dry-run output and only writes when `--apply` is passed
6. Implemented deterministic batch ordering + UX upgrades:
- added `batch-update --order` with choices:
- `as-is`
- `oldest-updated-first` / `newest-updated-first`
- `oldest-created-first` / `newest-created-first`
- `id-asc` / `id-desc`
- added tests for oldest-updated-first ordering and missing-timestamp behavior
- expanded argparse `--help` descriptions/options across commands
- added Bash completion script at `/home/jason/projects/redmine/redmine_contacts_completion.sh` and verified with `bash -n`
7. Captured Jason's broader contact triage loop vision in:
- `/home/jason/projects/redmine/docs/contact_triage_loop_workflow.md`
- includes workflow explanation, scoring rubric examples, queue-tag architecture, and a mermaid loop diagram
- linked the document from `/home/jason/projects/redmine/README.md` for discoverability
8. Adjusted `fetch` filter behavior to match expected UX:
- `fetch --cache-mode auto` (default) now caches filtered results when any filter flag is supplied
- `--cache-mode full` remains available for canonical full-snapshot caching
- added tests for `resolve_fetch_cache_mode` and verified against LAN command example
9. Implemented and verified ticket-centric sync/search foundation:
- new `/home/jason/projects/redmine/redmine_tickets.py`
- new `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- `sync` command caches enriched issue payloads using standard `/issues` API with `include=journals,helpdesk`
- per-project cache under `.cache/redmine_tickets_lan/customer-service/*.json`
- `manifest.json` tracks `updated_on`, local cache paths, and per-ticket action idempotency markers
- `search` command filters cached issues without API calls
- `spam-check` action scores cached tickets for promotional/spam signals and can record local idempotency with `--apply`
10. LAN validation completed on `192.168.50.170` / `customer-service`:
- dry-run: `seen=10 fetched=10 skipped_cached=0 written=0`
- cache write: `seen=10 fetched=10 skipped_cached=0 written=10`
- immediate second sync: `seen=10 fetched=0 skipped_cached=10 written=0`
- cached search showed newest open tickets matching UI, including `#41788` / `#41789` “Up to 20% Off Select Fans & Blowers”
- `spam-check` flagged `#41788` and `#41789` as `SPAM` (`score=2`) and left the other 8 as not-spam/low-score
- `spam-check --apply` recorded local manifest action markers; a second `spam-check` checked 0 tickets
- `close-if-spam` dry-run planned closures for `#41788` and `#41789` with payload `{status_id: 5, notes: ...}`
- `close-if-spam` fetched/reviewed sender `contact_id=621` history and found `4/10` recent tickets scoring as spam, so `likely_spammer=True`
- no Redmine ticket close mutation has been applied yet; `close-if-spam --apply` is pending explicit approval
- `close-if-spam --apply` defaults write impersonation to `--impersonate-user rebot`; reads/dry-runs do not impersonate
- tests verified: `python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v` (`23` tests)
- compile verified: `python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py`
## Next step and why
Jason identified the Obsidian KarpathyWiki plugin as close to the desired decomposition behavior. Inspection found it has no standalone CLI and depends on Obsidian `App/Vault/TFile` APIs, but its prompts/schema are strong references for a headless mirror. Compatibility rule: the plugin owns `obsidian-ldr/wiki/` (`wiki/entities`, `wiki/concepts`, `wiki/sources`, `wiki/index.md`, `wiki/log.md`), so the Python decomposer must not write there by default; use LDR-owned paths like `Timeline/`, `entities/organizations/`, and `Processes/Redmine/`, with explicit wikilinks to plugin pages when useful. Current implementation: `redmine_tickets.py decompose --issue-id/--input-json --format json|markdown` produces dry-run plans and includes a KarpathyWiki-style structured extraction layer (`summary`, `entities`, `concepts`, `key_points`, `contradictions`, `clean_text`) for the three pilot tickets. Tests pass (`23` tests). Dry-run outputs live under `.cache/decomposition-pilot-20260709/dry-run-output/`. Next: add prompt-driven/LLM JSON extraction for broader ticket types, then compare output quality against hand-written pilot notes before adding vault-write mode.
PLAN.md
+49
View File
@@ -0,0 +1,49 @@
# redmine contacts + project-standardization plan
Scope: standardize this repository around the shared `PROJECT.md` workflow and improve `redmine_contacts.py` so contact review/update work can be done in controlled batches with less API/UI friction.
Target stopping point for this phase: deliver tested CLI upgrades for filtering/show/update ergonomics, plus research documentation for duplicate-contact merge feasibility.
## Checklist
### Contact tooling foundation
- [x] Apply shared project template to `/home/jason/projects/redmine/PROJECT.md` with redmine-specific rules.
- [x] Create continuity docs (`PLAN.md`, `HANDOFF.md`) for ongoing milestone handoff.
- [x] Add tests for new `redmine_contacts.py` contact-filter + tag/custom-field patch behavior.
- [x] Implement date/tag contact filtering utilities used by `fetch` and `search`.
- [x] Add `show` command with cache/api source control.
- [x] Add `issue-full` command that composes full Redmine issue data with helpdesk ticket/messages.
- [x] Add `update` enhancements:
- [x] `--add-tag`
- [x] `--remove-tag`
- [x] `--set-custom-field <kebab-case-name>=<value>`
- [x] Verify new unit tests pass and script compiles.
- [x] Document duplicate-contact merge feasibility (API support vs scripted browser-equivalent merge path).
- [x] Add config-vault project index entry for redmine workstream.
- [x] Decide and implement default fetch cache behavior: `fetch` now caches full fetched contacts by default; `--cache-mode filtered` opt-in stores only filtered subset.
- [x] Add live integration validation against LAN Redmine copy (`192.168.50.170`) for new commands requiring API responses.
- [x] Add dry-run-first `batch-update` command for "N contacts matching filters" workflows.
- [x] Add controlled ordering options for batch runs (`--order` including oldest-updated-first) to support systematic sweep-through campaigns.
- [x] Expand argparse `--help` output with clearer descriptions/option help across commands.
- [x] Add Bash completion script: `/home/jason/projects/redmine/redmine_contacts_completion.sh`.
- [x] Capture the broader contact triage loop workflow in project docs (including scoring concept + diagram).
- [x] Make `fetch` intuitive: with filters and default `--cache-mode auto`, cache filtered results by default (override with `--cache-mode full`).
- [x] Implement Phase 1 contact loop tracer: per-contact evidence pull (3-5 recent tickets/messages) + scoring output in dry-run mode.
### Ticket-centric spam cleanup substrate
- [x] Plan ticket-centric sync/search/action architecture in `docs/ticket_sync_tool_plan.md`.
- [x] Implement `redmine_tickets.py` incremental sync using standard Redmine issue API with `include=journals,helpdesk`.
- [x] Implement per-ticket JSON cache plus state-store abstraction (`ManifestTicketStateStore` now, SQLite-capable seam later).
- [x] Implement cached `search` command.
- [x] Implement idempotent `spam-check` action.
- [x] Implement dry-run-first `close-if-spam` action with recent sender/contact lookback.
- [x] Ensure Redmine write calls can impersonate users via `X-Redmine-Switch-User`.
- [x] Default spam cleanup writes to `--impersonate-user rebot` (Harry Botter), while reads/dry-runs do not impersonate.
- [x] Verify LAN dry-run against `192.168.50.170` / `customer-service`: `#41788` and `#41789` flagged and closure-planned.
- [x] Document complete handoff in `docs/ticket_spam_cleanup_handoff_2026-07-09.md`.
- [ ] Confirm LAN issue `status_id=5` is the desired closed status.
- [ ] Run `close-if-spam --apply` on LAN for the currently planned spam tickets only.
- [ ] Re-sync/inspect LAN tickets to verify status closed, activity author `rebot`, and second apply is idempotent.
- [ ] After LAN apply verification, expand cache/sync batch size beyond the first 10 tickets.
PROJECT.md
+171
View File
@@ -0,0 +1,171 @@
---
tags:
- projects
- redmine
- contacts
- helpdesk
description: Legacy Redmine tooling and local plugin forks for reliable contact/search/helpdesk operations without requiring a risky platform-wide upgrade.
title: redmine
---
## Project rules
The following rules must be strictly adhered to when working on this project. Deviations from these rules are permitted only when the user explicitly requests or authorizes it.
### Understand the expectation of the user and what they are doing while you work
The user is expecting you to work continuously for long blocks of time, such as 20+ minutes minimum, during which time they plan to be away from the computer. They only intend to be highly-available at the outset of the work (during planning / clarifications), at the beginning/end of major milestones, or where, prior to starting a particular segment of work, you warn them you expect their involvement will be needed more often for some reason.
Keep this in mind when planning your work, and try to structure your work to meet the user's expectations about this.
#### For commands you need the user to run/approve, consider shell scripts
At the outset of work or the beginning of each phase, carefully think through what commands or file access you may need to perform, which is likely to require approval from the user. Also, think about any commands or file edits you will likely require, but would not be able to do yourself for some reason (e.g. you'd be tempted to instruct the user what to do, asking them to confirm results, instead of doing it yourself).
#### Best practices for handling these requirements
1. If a command will need to be run often, particularly with only slight variations, consider creating a temporary (project-specific) shell script or similar, which the user can approve once, and then you can use it repeatedly without needing more approvals.
2. If you will need to do something that requires limited root or sudo access, such as restarting a service or making edits to a root-owned file, do not plan on asking the user for approval each time. Instead, suggest the following:
1. Ask for limited/scoped sudo access which doesn't require a password, and is limited to exactly the commands you'll need to run.
2. Ask for a temporary change in file permissions so you can edit the file(s) without root access until the work is finished.
3. If working with remote hosts and ssh is available, do not plan on asking the user to shell into the remote host themselves and run commands on your behalf. Instead, suggest the creation and setup of a temporary ssh public key you can use that won't require a password each time you use it. Then, if the user agrees to that, verify you'll be able to connect via ssh without user assistance or approval for what you anticipate you'll need to do.
4. **Browser or similar tool isn't listening/running as expected**: If you need to use a tool like a web browser (like camofox or chrome with CDP) which seems like it should already be running and available to you, but when you try to use it, it doesn't work... try to solve that yourself without immediately running to the user for help. Valid solutions or troubleshooting for this consists of or should include:
1. Try starting the thing yourself (like camofox or a browser)
2. Check if an alternative method is available, and use it instead (e.g. you want to use Firecrawl, but the API connection doesn't work. For what you're trying to do, if camofox would work okay and it's available... just use that instead).
#### Avoid stopping mid-segment to ask for preferred implementation methods
During the course of work, you might run into situations where the project requirements could be implemented or understood more than one way. If that happens, your normal behavior tends to be stopping to ask the user which they would prefer, and presenting pros/cons. If you can, try to keep questions like that isolated to the outset of the work or the beginning/end of major milestones. The user should not be left with the impression you will busy yourself with work for 20+ minutes, so they leave and come back a half hour later, only to find that you stopped after 5 minutes to ask a question like this, especially not if you could have reasonably anticipated the question.
If an implementation detail like this arises unexpectedly, you should do the following if you can:
1. Choose the implementation path that is the lowest effort to _undo_ if it's not what the user wanted, and which has the lowest chance of turning into a "rabbit hole".
### Avoid "rabbit holes" or "wild goose chases"
Do not send the user on a quest of running various commands or actions to help you debug or test something, without first _thoroughly_ researching the likely causes of whatever problem has been encountered.
As much as possible, do not tell the user what commands to run or what files to edit. Instead, run the commands or edit the files yourself, unless there is a prevailing reason you prefer to have the user run the commands or make the edits. Valid reasons to ask the user to make edits or run commands on your behalf include:
- File permissions or access controls prevent you from doing it yourself
- The user specified they want to be involved / run things on your behalf
- Taking action yourself would involve opening/reading files that are very likely to contain passwords, API keys, or other secrets.
#### What is meant by a "rabbit hole" or "wild goose chase"
During the course of the plan, you encounter an error message or maybe a command doesn't work the way you expected. You proceed to ask the user to troubleshoot the problem with you, by directing them to do things like run commands, reboot things, or change files. It becomes a back-and-forth with the user, where instead of progressing on the overall goals of the project, we become stuck on troubleshooting one thing for an extended period of time.
**Example of what a rabbit hole can look like:** A project being worked on includes 15+ different elements to the plan, one of which is setting up apache2 to properly act as a reverse proxy to serve opencode. You encounter an error that indicates opencode isn't working correctly, and then you start troubleshooting it by asking the user to run various commands, copy-pasting the results to you, making edits to various files, etc. The user isn't in a position to understand or estimate how much of this will actually be needed to get the problem fixed, so they continue working with you in this manner, when they had originally planned to go work on something unrelated for an hour, while you worked on this project. So, they keep going back and forth with you to troubleshoot this opencode problem, hoping with each turn, you'll finally be done asking them to run commands. Before the user realizes it, an hour and half have transpired, and the project really hasn't moved forward at all because we've been fixated on fussing with that one bullet point about opencode, instead of the wider project.
A chain of events similar to the above is what is meant by a "rabbit hole" or "wild goose chase".
### Maintain a PLAN.md and HANDOFF.md
The PLAN.md should begin with a short statement that explains the scope of the project we're working on, including _why_ we're working on it, and the expected timeframe it needs to be at a stopping point. The rest of the document should read like a checklist of project goals or milestones. As you work, you should update the PLAN.md to check off things as they are implemented.
If you realize a particular checklist item in the plan is likely to become a "rabbit hole", is overly broad, or the expected implementation didn't actually resolve it, do the following:
1. Edit the PLAN.md with one or more sub-points added, breaking the goal/task down into smaller steps or goals that are more achievable.
1. If a decision is needed which the user isn't available for or has asked you to save for later, add making that decision as a task/goal of it's own.
2. Create a separate Markdown file containing the context for what decision needs to be made and what the options are, and explicitly reference that file in the sub-point.
2. Move on to the next part of the plan you can actually complete.
3. To indicate statuses with the "[ ]" fields of the markdown, use the following:
1. "x" : fully completed
2. "~": partially completed
3. "?": blocked until a decision is made or issue troubleshooted
#### HANDOFF.md
Each time you reach a major milestone with the project, update HANDOFF.md so that it reflects enough information that a different agent or worker could pick up where you left off. It should briefly explain:
1. Where to locate _this_ file.
2. Where to find the PLAN.md
3. The last 2-3 things you finished with the project.
4. The thing you were planning on doing next, and why.
### Use and update "global" docs and knowledge bases
Two different shared obsidian vaults exist, each with a different function. You should make liberal usage of them. The user may also have a private obsidian vault, named obsidian-private (but they might not).
- **obsidian-ldr**: Company knowledge-base about processes and standards, business practices, workplace rules or policies, employees or other contacts, etc.
- Business policies or processes, service/product definitions or promotional materials
- Workplace rules, best practices, SOPs, etc.
- Records of key business decisions, clients, employees, vendors, etc.
- **configs**: Technical docs about systems, infrastructure, tools, or software the company uses or is planning. May include troubleshooting info, coding standards, etc.
- Network configs, info about services available, etc.
- List of projects being worked on and where to find them
- Preferences about tooling, configs, fixtures, etc.
- **When to add things to this vault:**
- You changed a shared configuration on a company network or server
- Setup or installed a new server/service
- Started a new project
- Learned a new fact about company systems or infrastructure not already documented within the vault
- *obsidian-private*: Notes and info that's personal or private to the user. It should/may contain information that's not useful to other people, is private to the user, or moderately sensitive.
- **If it exists, typically used for:**
- User's _personal_ preferences or opinions
- Personal projects other people don't use
- Personalized configs for various tools/systems
- Personal schedules, todo lists, or journals
- **Typical location:**
- `~/obsidian-private`
#### Leverage the knowledge bases:
When something comes up you're not familiar with, and which plausibly seems to be related to company-specific jargon, methods, people, or similar, check the relevant vault before asking the user to explain what it is, and before you act on assumptions. Then, ask clarifying questions if needed.
If in the course of doing your work or asking the user questions, you learn things which seem like a future person/agent working on this or another project might also have questions about, _add what you learned to the appropriate vault_.
##### If unsure what to add or whether to add it, do the following:
Within the most appropriate vault's `INBOX` folder, create a markdown file (or using obsidian's tooling for this) named like YYYY-MM-projectName if it doesn't already exist. Then, append your input or observations to that file.
### Make liberal, appropriate use of delegation and background processes
You should have the ability to delegate work to sub-agents or even to opencode. Make intelligent use of these capabilities.
#### Here are some suggested uses for that:
- **Information Lookup**: When information might exist in multiple places, considering doing things like delegating one agent to check Obsidian and another to search the internet, then compare/contrast results to make a decision.
## If unclear where the project should live, make the home as follows
You might be provided instructions within a chat interface, or you might be pointed to a copy of this file somewhere on disk that isn't really a project folder. For instance, you might be pointed to this file within the INBOX section of an obsidian vault like `obsidian-ldr` or `configs`. In a case like this, you should do the following:
1. First, confirm with the user what this "project" is called, and whether it actually does have a "home" directory it should live within. If they say that it does, just use that. Otherwise, proceed with this list.
2. If the user is available, suggest 3-5 potential codenames for this project. Ask them to choose a name. Try to provide:
1. 1-2 functional, boring, but descriptive names
2. 1-3 creative, mythical, or narrative names
3. If the user isn't available, just pick a descriptive name for the project.
4. **Create and setup project directory:**
1. Convert the selected name to kebab-case or camel-case.
2. Within the user's `~/projects/` directory, create a directory for the project using the converted project name.
3. Within that directory:
1. Place a copy of this file inside of that project, and name it PROJECT.md.
2. Create a README.md within that directory, and add a succinct description to the top which summarizes the scope/purpose of the project.
3. Initialize the project directory as a git repository.
4. **If `goto` is available, add the project to it**
1. Check if the shell command `goto` is available. If it is, add the project to goto.
5. **Add it to the project index in `configs`**:
1. Within the `configs` Obsidian vault, check inside of `projects/` for a markdown article about this project. If one does not exist, create it.
2. Within that file, include a short section that explains where the actual project directory is, and a basic gist (very short) explaining what the project is (scope, problem it solves, etc).
### Troubleshooting
#### Obsidian vault not found or not working
If the vault isn't found, try the following:
- If `goto` is available, run `goto list` to see if it locates it
- Check these paths:
- `~/projects/` : for instance, `~/projects/obsidian-ldr` or `~/projects/configs`
If the vault isn't "working", consider how you're trying to access it:
- If you're using an MCP tool, that tool might be connected to the wrong vault, or it my rely on the obsidian GUI to be running.
- Check to see if one of the following methods will work, and if so, use one of these. Raise the issue about the vault not working correctly _after_ you reach the next project milestone.
- obsidian cli
- Usage of `fd`, `rg`, and similar shell commands
***
# Actual project instructions
The above information and rules govern how you should approach the following project. Below this section are the redmine-specific requirements.
## redmine project-specific requirements
- Project name is **redmine**.
- Project home is **`/home/jason/projects/redmine/`** (already exists; do not relocate).
- Primary objective: safe operational tooling around legacy Redmine CRM/Helpdesk data so support, triage, and automation workflows can move faster without a platform rewrite.
- Prioritize `redmine_contacts.py` ergonomics for data-use outcomes:
- filtering contacts by date/tag combinations for batch cleanup workflows,
- full issue retrieval with helpdesk context,
- precise, non-destructive contact updates (tags/custom fields).
- Maintain continuity docs in-repo:
- `PLAN.md` (active checklist using `[x]`, `[~]`, `[?]`)
- `HANDOFF.md` (last milestones + next step)
- Keep plugin source-of-truth in tracked `plugins/` paths. Do not treat ignored `redmine-copy/` as canonical source.
- Prefer dry-run/read-only validation where possible before mutating live-like test data.
- When documenting research or API feasibility, store findings in `docs/` with concrete route, method, auth, and risk notes.
README.md
+51 -3
View File
@@ -395,12 +395,26 @@ Project docs:
- [docs/helpdesk_smoke_test.md](docs/helpdesk_smoke_test.md)
- [docs/test_instance_post_import.md](docs/test_instance_post_import.md)
- [docs/pre_existing_issues.md](docs/pre_existing_issues.md)
- [docs/contact_triage_loop_workflow.md](docs/contact_triage_loop_workflow.md)
- [docs/contact_triage_loop_raw_notes_2026-07-04.md](docs/contact_triage_loop_raw_notes_2026-07-04.md)
Tooling:
- [redmine_contacts.py](redmine_contacts.py)
- [redmine_tickets.py](redmine_tickets.py) — local power-user client (sync, property search, view, walk)
- [redmine_helpdesk_search.py](redmine_helpdesk_search.py)
Power-user client (sync + CLI):
- [docs/cli_power_user_client.md](docs/cli_power_user_client.md) — **usage for humans/agents**
- [docs/raw-notes/features-for-redmine_tickets-sync.md](docs/raw-notes/features-for-redmine_tickets-sync.md) — roadmap index
- [docs/2026-07-14_localsearch_before_sanity.md](docs/2026-07-14_localsearch_before_sanity.md) — old thin-search baseline
Ticket-centric spam cleanup docs:
- [docs/ticket_sync_tool_plan.md](docs/ticket_sync_tool_plan.md)
- [docs/ticket_spam_cleanup_handoff_2026-07-09.md](docs/ticket_spam_cleanup_handoff_2026-07-09.md)
Local plugin work:
- [redmine-copy/plugins/redmine_event_outbox](redmine-copy/plugins/redmine_event_outbox)
@@ -408,7 +422,39 @@ Local plugin work:
## Current Recommended Next Steps
If continuing this project, the next best work is:
### Current priority: power-user local client + automations
Durable CS cache + CLI for search/view/walk (see [docs/cli_power_user_client.md](docs/cli_power_user_client.md)).
```bash
cd /home/jason/projects/redmine
CACHE=/home/jason/projects/redmine/cache/redmine_tickets
./redmine_tickets.py --project customer-service --cache-dir "$CACHE" \
search --where 'assigned_to IS NULL' --max-tickets 10
./redmine_tickets.py --project customer-service --cache-dir "$CACHE" \
view <id> --print
```
Cron: `cron-scripts/sync-live-customer-service.sh` (still `30d` window — tighten later).
Spam cleanup remains available on LAN before prod apply — see handoff doc below.
### Ticket-centric spam cleanup (secondary)
- [redmine_tickets.py](redmine_tickets.py)
- [docs/ticket_sync_tool_plan.md](docs/ticket_sync_tool_plan.md)
- [docs/ticket_spam_cleanup_handoff_2026-07-09.md](docs/ticket_spam_cleanup_handoff_2026-07-09.md)
Next safe sequence for spam apply:
1. Confirm LAN issue `status_id=5` is the intended closed status.
2. Run `redmine_tickets.py close-if-spam --apply` on the LAN copy only, using default `--impersonate-user rebot`.
3. Re-sync/inspect affected LAN tickets.
4. Verify status closed, activity author `rebot` / Harry Botter, and idempotency on a second apply.
### Other Redmine indexing / outbox workstream
If continuing the older outbox/search-index project, the next best work is:
1. Run the post-import validator and Helpdesk smoke test to establish a clean
baseline.
@@ -427,10 +473,12 @@ Fetch contacts:
./redmine_contacts.py fetch
```
Search contacts:
Search contacts. Short examples: `./redmine_contacts.py search --help`. What the CLI can do: [docs/redmine_contacts_search.md](docs/redmine_contacts_search.md). Situations it cannot do yet: [docs/redmine_contacts_search_proposals.md](docs/redmine_contacts_search_proposals.md).
```sh
./redmine_contacts.py search "customer name or phone"
./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod"
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search promotional-spam --with-tag promotional-spam --refresh
```
Preview a contact update:
Rebuild or something.md
+1
View File
@@ -0,0 +1 @@
I had a _really powerful_ program that had been `redmine_tickets.py`, `bin/rtl`, and the other tools it connected with. The curses-based TUI was amazingly fast and powerful. Then, for
bin/checkin_truth.md
+16
View File
@@ -0,0 +1,16 @@
reliable "change sku" API endpoint that corrects the SKU assigned to one or more individual_product entries owned by the same user.
## What usually causes the problem:
**Not confident to choose/verify SKU**: While receiving and checking in packages from someone like Moses, there we receive one or more items we can't confidently match to one of his products. Then, either:
a) We choose an "AUTO-" sku or "add unexpected item", and type a description of what we have.
b) We make a guess or a mistake, and and it turns out to be incorrect.
On the employee-facing side of the portal, we have a tool for fixing this from the [[Refurbishment Panel]] and also [[Item Manager]]. Customers like Moses who want to correct the SKU chosen for an item seem to lack an intended, easy-to-use way to accomplish that.
## Benefits and Tradeoffss about AUTO SKUs and our receiving procedures
### Our intention should be maximal truth within a predictable timespan and price, pertaining to all handling of inventory. That includes confirming and documenting we received somethinmg addressed to the customer, even if it wasn't expected by anyone.
AUTO skus are usually left in a "non-final condition" like [[Customer Service Needed]] until somebody does something about it. In practice, we have a persistent rumor that setting "CSN" notifies customers, most customers decide they don't want the items, and Danny is the person who deals with that.
bin/cli-sync.sh
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/bin/bash
cd /home/jason/projects/redmine/
BURL="https://rm.ldrprep.com/"
TCACHE="./cache/redmine_tickets"
./redmine_contacts.py --base-url $BURL --project customer-service --cache ./cache/redmine_contacts/ fetch
./redmine_tickets.py --base-url $BURL --project customer-service --cache-dir "$TCACHE" sync \
--remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20
./redmine_tickets.py --base-url $BURL --project quality-tracker --cache-dir "$TCACHE" sync \
--remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20
./redmine_tickets.py --base-url $BURL --project meetings --cache-dir "$TCACHE" sync \
--remote-status-id '*' --remote-updated-since 48h --run-log --run-log-keep 20
bin/qaa
Executable
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Short alias example (qaa) → same as rtl (prod CS). Rename/copy for more namespaces later.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
exec "$ROOT/bin/rtl" "$@"
bin/rtd
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Muscle-memory wrapper: LAN/dev customer-service cache + base URL.
# Usage: rtd note --issue-id 39305 --body 'test' --apply
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
exec "$ROOT/redmine_tickets.py" --config "$ROOT/configs/dev.json" "$@"
bin/rtl
Executable
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
# Muscle-memory wrapper: prod customer-service cache + base URL.
# Usage: rtl search --where 'assigned_to IS NULL' --max-tickets 10
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
# rtl is the production Redmine wrapper. Keep an explicitly supplied mode, but
# never silently default its NMI briefs to the dev profile.
export NMI_MODE="${NMI_MODE:-production}"
exec "$ROOT/redmine_tickets.py" --config "$ROOT/configs/live.json" "$@"
contact_enrichment.py
+441
View File
@@ -0,0 +1,441 @@
#!/usr/bin/env python3
"""Contact enrichment: map Redmine CRM contacts to a target CRM schema.
This module is intentionally decoupled from ``redmine_contacts.py`` so the sync
module can import it without pulling in CLI / cache logic. All enrichment is
deterministic and side-effect free; external lookups are accepted via a
configurable provider so callers can inject rate-limited clients.
"""
from __future__ import annotations
import json
import re
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Callable
EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE_RE = re.compile(r"[\d\s\-().+]{7,}")
URL_RE = re.compile(r"https?://[^\s<>\"']+|www\.[^\s<>\"']+")
def _utc_now() -> datetime:
return datetime.now(timezone.utc)
@dataclass
class EnrichmentResult:
"""Output of enriching a single Redmine contact."""
contact: dict[str, Any]
enriched: dict[str, Any]
changed_fields: list[str]
warnings: list[str] = field(default_factory=list)
skipped: bool = False
reason: str = ""
@dataclass
class EnrichmentConfig:
"""Configuration for the enrichment pipeline."""
source: str = "redmine"
confidence_threshold: float = 0.7
default_country_code: str = "US"
tags_from_redmine: list[str] = field(default_factory=lambda: ["from-redmine"])
external_lookup: Callable[[str], dict[str, Any]] | None = None
domain_to_company: dict[str, str] = field(default_factory=dict)
class EnrichmentError(RuntimeError):
pass
class RateLimitError(EnrichmentError):
"""Raised when an external lookup exhausts retries or hits a hard limit."""
pass
def enrich_contact(
redmine_contact: dict[str, Any],
config: EnrichmentConfig | None = None,
) -> EnrichmentResult:
"""Map a Redmine contact to the target CRM schema and fill missing fields."""
config = config or EnrichmentConfig()
contact_id = redmine_contact.get("id")
if contact_id is None:
return EnrichmentResult(
contact=redmine_contact,
enriched={},
changed_fields=[],
skipped=True,
reason="Missing Redmine contact id",
)
warnings: list[str] = []
emails = _extract_emails(redmine_contact, warnings)
phones = _extract_phones(redmine_contact, config.default_country_code, warnings)
company = _enrich_company(redmine_contact, emails, config, warnings)
name_parts = _name_parts(redmine_contact)
address = _extract_address(redmine_contact)
notes = _build_notes(redmine_contact)
tags = _build_tags(redmine_contact, config)
enriched: dict[str, Any] = {
"crm_id": None,
"redmine_contact_id": contact_id,
"full_name": name_parts["full_name"],
"first_name": name_parts["first_name"],
"middle_name": name_parts.get("middle_name"),
"last_name": name_parts["last_name"],
"company": company,
"title": _clean_str(redmine_contact.get("job_title")),
"emails": emails,
"phones": phones,
"website": _clean_str(redmine_contact.get("website")),
"birthday": _format_birthday(redmine_contact.get("birthday")),
"address": address,
"notes": notes,
"tags": tags,
"source": config.source,
"synced_at": _utc_now().isoformat(),
"enriched_at": _utc_now().isoformat(),
"last_redmine_update": _format_timestamp(redmine_contact.get("updated_at", redmine_contact.get("updated_on"))),
}
changed_fields = [key for key, value in enriched.items() if value not in (None, "", [], {})]
if not emails and not company:
warnings.append("No email or company; record may be unmatchable in CRM")
return EnrichmentResult(
contact=redmine_contact,
enriched=enriched,
changed_fields=changed_fields,
warnings=warnings,
)
def _clean_str(value: Any) -> str | None:
if value is None:
return None
cleaned = str(value).strip()
return cleaned or None
def _extract_emails(redmine_contact: dict[str, Any], warnings: list[str]) -> list[dict[str, Any]]:
emails: list[dict[str, Any]] = []
primary = _clean_str(redmine_contact.get("email"))
if primary:
emails.append({"value": primary.lower(), "type": "work"})
# Search background and custom fields for additional emails when primary missing.
if not emails:
haystack = _contact_text_haystack(redmine_contact)
for match in EMAIL_RE.findall(haystack):
candidate = match.lower()
if not any(existing["value"] == candidate for existing in emails):
emails.append({"value": candidate, "type": "work"})
if not emails:
warnings.append("No email found")
return emails
def _extract_phones(
redmine_contact: dict[str, Any],
default_country_code: str,
warnings: list[str],
) -> list[dict[str, Any]]:
phones: list[dict[str, Any]] = []
raw_phone = _clean_str(redmine_contact.get("phone"))
if raw_phone:
normalized = _normalize_phone(raw_phone, default_country_code)
if normalized:
phones.append({"value": normalized, "type": _classify_phone_type(raw_phone)})
# Do not mine background for phones unless the explicit field is empty.
if not phones:
haystack = _contact_text_haystack(redmine_contact)
for match in PHONE_RE.findall(haystack):
normalized = _normalize_phone(match, default_country_code)
if normalized and not any(existing["value"] == normalized for existing in phones):
phones.append({"value": normalized, "type": _classify_phone_type(match)})
if len(phones) >= 3:
break
if not phones:
warnings.append("No phone found")
return phones
def _normalize_phone(raw: str, default_country_code: str) -> str | None:
digits = re.sub(r"\D", "", raw)
if len(digits) < 7:
return None
if len(digits) == 10 and default_country_code == "US":
digits = "1" + digits
if digits.startswith("1") and len(digits) == 11:
return f"+{digits}"
return f"+{digits}"
def _classify_phone_type(raw: str) -> str:
lowered = raw.lower()
if "mobile" in lowered or "cell" in lowered or "m:" in lowered:
return "mobile"
if "home" in lowered or "h:" in lowered:
return "home"
return "work"
def _enrich_company(
redmine_contact: dict[str, Any],
emails: list[dict[str, Any]],
config: EnrichmentConfig,
warnings: list[str],
) -> str | None:
company = _clean_str(redmine_contact.get("company"))
if company:
return company
if config.external_lookup is not None:
try:
lookup = config.external_lookup(str(redmine_contact.get("id")))
company = _clean_str(lookup.get("company"))
if company:
return company
except RateLimitError:
warnings.append("External company lookup hit rate limit")
except Exception as exc: # noqa: BLE001
warnings.append(f"External company lookup failed: {exc}")
if emails:
domain = emails[0]["value"].split("@", 1)[-1]
company = config.domain_to_company.get(domain)
if company:
return company
company = _company_from_domain(domain)
if company:
return company
warnings.append("Company could not be enriched")
return None
def _company_from_domain(domain: str) -> str | None:
# Simple deterministic heuristic; keeps component dependency-free.
if not domain or "." not in domain:
return None
name = domain.rsplit(".", 1)[0]
name = re.sub(r"[^a-zA-Z0-9]+", " ", name).strip()
if not name:
return None
return name.capitalize() + " Inc"
def _name_parts(redmine_contact: dict[str, Any]) -> dict[str, Any]:
first = _clean_str(redmine_contact.get("first_name"))
middle = _clean_str(redmine_contact.get("middle_name"))
last = _clean_str(redmine_contact.get("last_name"))
if redmine_contact.get("is_company"):
company = _clean_str(redmine_contact.get("company")) or first or ""
return {"full_name": company, "first_name": company, "middle_name": None, "last_name": None}
parts = [part for part in (first, middle, last) if part]
full_name = " ".join(parts)
# If no structured name, try to infer from display-style strings.
if not full_name:
full_name = _clean_str(redmine_contact.get("name")) or ""
if full_name:
split = full_name.split()
first = split[0] if split else None
last = " ".join(split[1:]) if len(split) > 1 else None
return {"full_name": full_name or None, "first_name": first, "middle_name": middle, "last_name": last}
def _extract_address(redmine_contact: dict[str, Any]) -> dict[str, Any]:
attrs = redmine_contact.get("address_attributes") or {}
if not attrs:
return {}
return {
"street": _clean_str(attrs.get("street")),
"city": _clean_str(attrs.get("city")),
"region": _clean_str(attrs.get("state")),
"country": _clean_str(attrs.get("country")),
"postal_code": _clean_str(attrs.get("zip")),
}
def _build_notes(redmine_contact: dict[str, Any]) -> str:
parts: list[str] = []
background = _clean_str(redmine_contact.get("background"))
if background:
parts.append(background)
custom_fields = redmine_contact.get("custom_fields") or []
for field in custom_fields:
name = field.get("name")
value = field.get("value")
if name and value:
parts.append(f"{name}: {value}")
contact_id = redmine_contact.get("id")
if contact_id is not None:
parts.append(f"Redmine contact: {contact_id}")
return "\n".join(parts)
def _build_tags(redmine_contact: dict[str, Any], config: EnrichmentConfig) -> list[str]:
raw_tags = redmine_contact.get("tag_list") or []
if isinstance(raw_tags, str):
tags = {part.strip().lower() for part in raw_tags.split(",") if part.strip()}
elif isinstance(raw_tags, list):
tags = {str(tag).strip().lower() for tag in raw_tags if str(tag).strip()}
else:
tags = set()
for tag in config.tags_from_redmine:
tags.add(tag.lower())
project_id = redmine_contact.get("project_id")
if project_id is not None:
tags.add(f"project-{project_id}")
return sorted(tags)
def _contact_text_haystack(redmine_contact: dict[str, Any]) -> str:
pieces: list[str] = []
background = _clean_str(redmine_contact.get("background"))
if background:
pieces.append(background)
for field in redmine_contact.get("custom_fields") or []:
value = field.get("value")
if value:
pieces.append(str(value))
return "\n".join(pieces)
def _format_birthday(value: Any) -> str | None:
cleaned = _clean_str(value)
if not cleaned:
return None
for fmt in ("%Y-%m-%d", "%Y/%m/%d", "%m/%d/%Y", "%d/%m/%Y"):
try:
return datetime.strptime(cleaned, fmt).date().isoformat()
except ValueError:
continue
return None
def _format_timestamp(value: Any) -> str | None:
if value is None:
return None
raw = str(value).strip()
if not raw:
return None
normalized = raw.replace("Z", "+00:00")
try:
dt = datetime.fromisoformat(normalized)
except ValueError:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc).isoformat()
def build_sync_payload(
redmine_contacts_iter: list[dict[str, Any]],
config: EnrichmentConfig | None = None,
) -> dict[str, Any]:
"""Enrich a batch and return a summary suitable for a sync module."""
config = config or EnrichmentConfig()
results = [enrich_contact(contact, config) for contact in redmine_contacts_iter]
return {
"enriched_at": _utc_now().isoformat(),
"count": len([r for r in results if not r.skipped]),
"records": [result.enriched for result in results if not result.skipped],
"skipped": [
{"redmine_contact_id": result.contact.get("id"), "reason": result.reason}
for result in results
if result.skipped
],
"warnings": [
{"redmine_contact_id": result.contact.get("id"), "warning": warning}
for result in results
for warning in result.warnings
],
}
def with_rate_limit(
func: Callable[..., Any],
*,
max_calls: int = 10,
per_seconds: float = 1.0,
max_retries: int = 5,
backoff_base: float = 1.0,
backoff_cap: float = 30.0,
) -> Callable[..., Any]:
"""Wrap a callable with a simple token-bucket rate limit and retry policy.
The wrapper is stateful; create one per external provider session. It sleeps
using ``time.sleep`` so it is safe for sync scripts but should not be used
inside async loops.
"""
import time
tokens = float(max_calls)
last_refill = time.monotonic()
lock_retried = 0
def _refill() -> None:
nonlocal tokens, last_refill
now = time.monotonic()
elapsed = now - last_refill
tokens = min(float(max_calls), tokens + elapsed * (max_calls / per_seconds))
last_refill = now
def wrapper(*args: Any, **kwargs: Any) -> Any:
nonlocal tokens, lock_retried
for attempt in range(max_retries + 1):
_refill()
if tokens < 1:
sleep_time = (1 - tokens) * (per_seconds / max_calls)
time.sleep(max(0.0, sleep_time))
_refill()
tokens -= 1
try:
return func(*args, **kwargs)
except RateLimitError:
raise
except Exception as exc: # noqa: BLE001
if attempt == max_retries:
raise
wait = min(backoff_cap, backoff_base * (2 ** attempt))
time.sleep(wait)
raise EnrichmentError("Rate-limited wrapper exhausted retries")
return wrapper
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Enrich Redmine CRM contacts for sync.")
parser.add_argument("contacts", type=json.loads, help="JSON-encoded Redmine contact or list of contacts.")
parser.add_argument("--source", default="redmine")
parser.add_argument("--confidence-threshold", type=float, default=0.7)
args = parser.parse_args()
raw = args.contacts
contacts = raw if isinstance(raw, list) else [raw]
config = EnrichmentConfig(source=args.source, confidence_threshold=args.confidence_threshold)
print(json.dumps(build_sync_payload(contacts, config), indent=2, sort_keys=True))
cron-scripts/sync-dev-customer-service.sh
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# Dev/LAN CS ticket sync → cache-dev, then rematerialize queues.
set -euo pipefail
cd /home/jason/projects/redmine
export REDMINE_API_KEY="${REDMINE_API_KEY:?REDMINE_API_KEY required}"
CACHE=/home/jason/projects/redmine/cache-dev/redmine_tickets
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir "$CACHE" \
sync \
--remote-status-id '*' \
--remote-updated-since 6h \
--run-log \
--run-log-keep 20 \
--quiet
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir "$CACHE" \
materialize-queues \
--config queues.example.json \
>/dev/null
cron-scripts/sync-live-contacts.sh
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Optional slower contacts mirror refresh (prod cache). Run less often than ticket sync.
set -euo pipefail
cd /home/jason/projects/redmine
#export REDMINE_API_KEY="${REDMINE_API_KEY:?REDMINE_API_KEY required}"
./bin/rtl contacts-sync --quiet
cron-scripts/sync-live-customer-service.sh
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Prod CS ticket sync → durable cache, then rematerialize local queues.
# Window: 6h after warm cache (bump to 30d only for catch-up).
set -euo pipefail
cd /home/jason/projects/redmine
#CACHE=/home/jason/projects/redmine/cache/redmine_tickets
./bin/rtl \
sync \
--remote-status-id '*' \
--remote-updated-since 1d \
--run-log \
--run-log-keep 20 \
--quiet
# Refresh symlink queues for external agents/scripts (local-only).
./bin/rtl \
materialize-queues \
# --config queues.example.json \
>/dev/null
dispatch-dry-run-2026-09-19.md
+142
View File
@@ -0,0 +1,142 @@
# Dispatch dry-run report: customer-service New/unassigned (updated since 2026-09-18)
Generated: 2026-09-19 19:12 EDT | Tickets examined: 17 | Cache: /home/jason/projects/redmine/cache/redmine_tickets
## Summary
- Obvious spam/promotional: 6
- Automated operational notifications (Amazon/AM Shipping receipts, invoices): 9
- Inbound business inquiries needing review: 0
- Tickets with extractable entities (tracking/order/item): 9
- High attention score (>=40): 17
## Per-ticket details
### #43690 — Product Session: Moses / Esire — 90-day ticket themes (from #43412)
- Status: New | Category: Product Session | Assignee: — | Updated: 2026-09-19T11:52
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **84.0** (status:new; unassigned+open; kw:complaint; kw:missing; kw:wrong; has-entities)
- Entities: item_or_sku:corrections(medium); item_or_sku:still(medium); item_or_sku:verification(medium); item_or_sku:match(medium); item_or_sku:backlog(medium); item_or_sku:reference(medium); item_or_sku:sampling(medium); item_or_sku:stills(medium); item_no:200230889(medium); item_no:200222683(medium); item_no:200226501(medium); item_no:200243595(medium); account_ref:structure(medium); account_ref:request(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43711 — Don’t miss out on your work perks
- Status: New | Category: — | Assignee: — | Updated: 2026-09-19T10:28
- Spam: `spam-promotional` score=3 — sender has promotional/automated hint; body/journal contains promotional language; ticket text contains promotional sender hint
- Triage score: **57.0** (stale>8h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43694 — On Hand Now: GOLDENROD 2-Packs at $16
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:55
- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint
- Triage score: **57.0** (stale>22h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43693 — Nature2 3-Packs: $40 + FREE Poly Prep
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:54
- Spam: `spam-promotional` score=3 — subject contains promotional language; ticket text contains promotional sender hint
- Triage score: **57.0** (stale>22h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43692 — Re: #25643: New message from our team may require attention
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T20:40
- Spam: `unknown-review` score=0 — no signals
- Triage score: **57.0** (stale>22h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: High attention: needs Dispatch containment review now.
### #43683 — Looking for 3 Prep Centers
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T14:04
- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint
- Triage score: **70.0** (stale>29h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43682 — One customer generated $9,176.07 in 7 days
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T14:04
- Spam: `spam-promotional` score=2 — body/journal contains promotional language; ticket text contains promotional sender hint
- Triage score: **70.0** (stale>29h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43681 — Your call transcription from 937-632-8358 to 513-930-2899
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T13:47
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **70.0** (stale>29h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.
### #43676 — Special Low MOQ Friday Offer!
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T12:45
- Spam: `spam-promotional` score=4 — sender has promotional/automated hint; subject contains promotional language; ticket text contains promotional sender hint
- Triage score: **70.0** (stale>30h; status:new; unassigned+open)
- Entities: none detected
- Suggested action: Close as spam (spam-check → close-if-spam).; High attention: needs Dispatch containment review now.
### #43675 — Order ID: 1091195 - an inquiry about a parcel ref. tracking # 1ZY228K59096949586 (UPS)
- Status: New | Category: — | Assignee: — | Updated: 2026-09-18T12:34
- Spam: `unknown-review` score=0 — no signals
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: tracking_ups:1ZY228K59096949586; order_id:1091195; order_id:PW9yG(medium)
- Suggested action: High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43550 — Invoice 341570 from AM Shipping Supplies Co., Inc.
- Status: New | Category: Bill Needs Paid | Assignee: — | Updated: 2026-09-18T12:22
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: account_ref:provided(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43542 — Ordered: "1000 Pack Secure Seal White..."
- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:20
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: order_amazon:111-6425162-2592263; item_no:202029700(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43552 — Invoice 341571 from AM Shipping Supplies Co., Inc.
- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:19
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: account_ref:provided(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43565 — Ordered: 2 "Ribbon Cutting Machine,..." and 6 more items
- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:14
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: order_amazon:114-5077128-2677066; order_amazon:114-5944516-3478625; order_id:43564; item_no:202029700(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43569 — Shipped: "12" X 15" (500 Count) Self..." and 1 more item
- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:14
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>30h; status:new; unassigned+open; has-entities)
- Entities: order_amazon:114-5944516-3478625; item_no:200962600(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43572 — Re: Invoice 341571 from AM Shipping Supplies Co., Inc.
- Status: New | Category: Bill Needs Paid | Assignee: — | Updated: 2026-09-18T12:13
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>31h; status:new; unassigned+open; has-entities)
- Entities: account_ref:specialists(medium); account_ref:provided(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
### #43586 — Shipped: 2 "Ribbon Cutting Machine,..." and 4 more items
- Status: New | Category: Receipts | Assignee: — | Updated: 2026-09-18T12:02
- Spam: `automated-operational` score=0 — automated operational notification
- Triage score: **78.0** (stale>31h; status:new; unassigned+open; has-entities)
- Entities: order_amazon:114-5077128-2677066; item_no:200962600(medium)
- Suggested action: Auto-categorize/route to Receipts/Bill Needs Paid and resolve if no action needed.; High attention: needs Dispatch containment review now.; Run `entities nmi` to attach private NMI brief note.
## Recommended next steps
1. Run `spam-check --apply` then `close-if-spam --apply` on the cached set to clear obvious promotional spam (#43676).
2. For automated operational tickets already categorized (Receipts/Bill Needs Paid), consider a rule to auto-set status to Resolved with an internal note.
3. For #43675 (Cyberstorm tracking inquiry), queue an NMI private note via `entities nmi --issue-id 43675 --queue-private-note`.
4. For high-score tickets (e.g., #43690, #43711), perform containment pass: check escalation vectors, set priority/dates, create subtask if specific person needed.
5. Extend the same search to `quality-tracker`, `attendance-schedule`, and `meetings` once configs are added.
_Report generated by maintainer using existing redmine_tickets.py / redmine_triage.py / redmine_ticket_entities.py (read-only, no Redmine writes)._
docs/2026-07-14_localsearch_before_sanity.md
+218
View File
@@ -0,0 +1,218 @@
# Local `search` in `redmine_tickets.py` (pre-sanity snapshot)
**Date:** 2026-07-14
**Status:** documentation of current behavior *before* any search sanity/cleanup pass
**Script:** `/home/jason/projects/redmine/redmine_tickets.py`
**Typical cache:** `/home/jason/projects/redmine/cache/redmine_tickets`
This note captures what local `search` does and does **not** do as of 2026-07-14, after remote sync filters and run-log work landed. It is intentionally frank: docs had been thin; this is the baseline.
---
## Mental model
```text
sync = talk to Redmine → update local JSON cache
search = filter/list what is already cached
```
`search` is a **cache browser / structured filterer**, not:
- a live Redmine issue query tool
- full-text corpus search
- the semantic Redmine search path (that lives elsewhere)
If a ticket is not already under the cache directory for the selected project, **`search` cannot see it.**
---
## Where data comes from
Cached issues live as one JSON file per ticket:
```text
<cache-dir>/<project>/<issue-id>.json
```
Example for the durable customer-service cache:
```text
/home/jason/projects/redmine/cache/redmine_tickets/customer-service/*.json
/home/jason/projects/redmine/cache/redmine_tickets/manifest.json
```
`search` loads via `TicketCache.iter_cached_issues()` (reads every `*.json` in the project subdir). It does **not** call Redmine and does **not** require `REDMINE_API_KEY`.
---
## CLI surface (current)
Parent options still apply:
- `--base-url` (unused by search, but accepted on the parent parser)
- `--project`
- `--cache-dir`
Subcommand:
```bash
./redmine_tickets.py \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
search [filters...] [output...]
```
### Filters (all local, AND’d)
| Flag | Matches on | Notes |
|------|------------|--------|
| `--updated-since` | issue `updated_on` ≥ value | via `parse_dt` (absolute date/datetime) |
| `--updated-before` | issue `updated_on` ≤ value | same |
| `--status` | status **name** | case-insensitive **exact** match (`Closed`, `New`, …) — **not** `status_id` |
| `--category` | category **name** | case-insensitive **exact** match |
Implementation: `filter_issues()` in `redmine_tickets.py`.
### Order + limit
| Flag | Behavior |
|------|----------|
| `--order newest-updated-first` | **default** |
| `--order oldest-updated-first` | |
| `--order id-asc` / `id-desc` | |
| `--order as-is` | filesystem glob order from `iter_cached_issues` |
| `--max-tickets N` | after filter + sort, keep first N |
Implementation: `select_issues()`.
### Output
| Mode | What you get |
|------|----------------|
| default table | `#id`, `updated_on`, status name, category name, subject (via `print_issue_rows`) |
| `--json` | full cached issue payloads as one JSON array |
| `--jsonl` | one full issue JSON object per line (good for piping) |
Human table is summary-only. Full subject/body/journals/helpdesk are only in `--json` / `--jsonl`.
---
## What it allows (summary)
1. Scope one project + one cache root.
2. Filter cached issues by `updated_on` window, status **name**, category **name**.
3. Sort and cap results.
4. Emit table, JSON, or JSONL.
5. Run offline against cache (no Redmine, no API key).
---
## What it does **not** allow (current gaps)
This is the important half of the snapshot:
- **No free-text search** (subject / description / journals / from-address)
- **No sender / email / contact filters**
- **No tracker, assignee, priority, tags**
- **No `status_id` / `open` / `closed` shorthand** on search (only status **name**)
- **No remote/API filters** on `search`
(`--remote-updated-on`, `--remote-updated-since`, `--remote-status-id` are **sync-only**)
- **No relative time helpers** like `30d` / `2 days ago` on local search flags
(those helpers exist for **remote sync** convenience flags; local `--updated-since` wants a real parseable date/datetime)
- **No boolean logic** (AND/OR/NOT nesting)
- **No query language** comparable to Redmine UI filters or `rg`
- **No ranking / relevance**
- **No “search then sync missing ones”**
- **No pagination UI** beyond `--max-tickets`
- **No writes** (search never mutates cache or Redmine)
### Practical content search today
Pipe full payloads and use external tools:
```bash
./redmine_tickets.py \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
search --jsonl | rg -i 'gaylord|rate sheet'
```
Or inspect per-ticket files under the project cache dir directly.
---
## Examples that work
```bash
cd /home/jason/projects/redmine
# Everything currently cached for customer-service (table)
./redmine_tickets.py \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
search
# Cached tickets updated since a date, newest first, cap 50
./redmine_tickets.py \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
search \
--updated-since 2026-06-14 \
--max-tickets 50
# Only closed, as full JSONL for downstream tools
./redmine_tickets.py \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
search \
--status Closed \
--jsonl
```
---
## Related commands people confuse with “search”
| Command | Relation to local search |
|---------|--------------------------|
| `sync` | fills the cache that `search` reads; supports **server-side** `status_id` + `updated_on` |
| `spam-check` | walks **cache**, scores spam; not general search |
| `known-spam-scan` | profile match over **cache** |
| `decompose` | one-ticket analysis, cache-first |
| semantic index / external search | separate stack; not this subcommand |
---
## Remote vs local filter naming (context for sanity later)
As of the 2026-07-14 sync work:
| Side | Flags (examples) | Used by |
|------|------------------|---------|
| **Remote (API)** | `--status-id` / `--remote-status-id`, `--remote-updated-on`, `--remote-updated-since`, `--remote-updated-before` | `sync` list call only |
| **Local (post)** | `--updated-since`, `--updated-before`, `--status`, `--category` | `sync` post-filter **and** `search` |
Help text for the shared local flags still says things like “applied after API list,” which is accurate for `sync` but slightly misleading when those same flags appear under `search` (where there is no API list). A future sanity pass may want search-specific wording.
---
## Implementation pointers
| Piece | Location |
|-------|----------|
| Filter logic | `filter_issues()` |
| Sort / max | `select_issues()` |
| Cache read | `TicketCache.iter_cached_issues()` |
| Table output | `print_issue_rows()` |
| CLI wiring | `build_parser()` → `search` subparser; `main()` branch `args.command == "search"` |
| Shared filter args | `add_common_filters()` |
---
## Bottom line (baseline for sanity)
**Does:** local structured filter/list over already-synced ticket JSON — date window, status name, category name, sort, limit, table/json/jsonl.
**Doesn’t:** live Redmine query, free-text, contact/sender, remote filters, relative-date helpers on local flags, or advanced query syntax.
Any “make search sane” work after this file should treat the above as the pre-change contract and call out intentional expansions (e.g. `--from`, `--subject-contains`, `--text`, relative dates on local filters) separately.
docs/cli_power_user_client.md
+685
View File
@@ -0,0 +1,685 @@
# Power-user Redmine client — usage (`redmine_tickets.py`)
Scriptable local client for **humans and agents**: durable ticket cache, property search, Markdown view, and interactive walk.
| Env | Redmine | Cache |
|-----|---------|--------|
| **Prod** | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` |
| **Dev / mutation testing** | LAN (e.g. `192.168.50.170` / `retired.int.ldrprep.com`) | `/home/jason/projects/redmine/cache-dev/redmine_tickets` |
**Safety:** treat prod cache as read/sync. Test any **writes** on LAN first. Mutations impersonate `rebot` when implemented.
Related:
- Roadmap index: [raw-notes/features-for-redmine_tickets-sync.md](raw-notes/features-for-redmine_tickets-sync.md)
- Pre-MVP search snapshot: [2026-07-14_localsearch_before_sanity.md](2026-07-14_localsearch_before_sanity.md)
- Sync plan: [ticket_sync_tool_plan.md](ticket_sync_tool_plan.md)
- Cron scripts: `cron-scripts/sync-live-customer-service.sh`, `cron-scripts/sync-dev-customer-service.sh`
---
## Setup
```bash
cd /home/jason/projects/redmine
export REDMINE_API_KEY=… # required for sync / live API only
CACHE=/home/jason/projects/redmine/cache/redmine_tickets
PROJECT=customer-service
```
Shorthand used below:
```bash
RT="./redmine_tickets.py --project $PROJECT --cache-dir $CACHE"
```
---
## Sync (fill / refresh cache)
Remote filters hit Redmine; local cache is updated only for tickets whose `updated_on` changed.
```bash
# Initial / catch-up window
$RT sync --remote-status-id '*' --remote-updated-since 30d --run-log
# Frequent refresh (preferred after warm cache)
$RT sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet
```
| Flag | Meaning |
|------|---------|
| `--remote-status-id` / `--status-id` | Server: `*`, `open`, `closed`, or numeric |
| `--remote-updated-since` | Server: `30d`, `2h`, `YYYY-MM-DD`, … → `updated_on >= …` |
| `--remote-updated-on` | Server: raw Redmine operator string |
| `--run-log` | Ring-buffer stats → `$CACHE/run-stats.json` |
| `--run-log-keep N` | Keep last N runs (default 3; cron uses 20) |
| `--quiet` | Less progress noise |
**Do not** use local `--updated-since` as a substitute for remote filters on large projects (local only post-filters stubs already listed).
Cron note: live scripts currently still use `30d` on a short interval (~3–4s after warm-up). **Tighten later** to `2h`/`6h`.
---
## Search (local cache only)
No API key required. Only sees tickets already on disk.
## Search table columns
Default table (tab-separated):
```text
#id updated_on status assignee contact category control_tags subject
```
Contact comes from `helpdesk_ticket.contact.email` (or name/company/from_address).
### Classic filters
```bash
$RT search --status New --max-tickets 50
$RT search --updated-since 2026-07-01 --order newest-updated-first
```
### Property queries (`--where`)
Engine: `redmine_ticket_query.py`. Repeatable `--where` = **AND**.
```bash
$RT search --where 'subject.includes("account")' --max-tickets 20
$RT search --where 'assigned_to IS NULL' --max-tickets 10
$RT search \
--where 'helpdesk_ticket.contact.email == helpdesk_ticket.contact.company' \
--ids-only
$RT search \
--where '(subject.includes("verification code") OR subject.includes("magic link")) AND assigned_to IS NULL'
```
| Construct | Example |
|-----------|---------|
| Equality | `status.name == "New"` |
| Inequality | `assigned_to != "Jason T."` |
| Null | `assigned_to IS NULL` / `IS NOT NULL` |
| Substring | `subject.includes("rate sheet")` |
| Relative time | `updated_on.since("5 days")` |
| Path == path | `helpdesk_ticket.contact.email == helpdesk_ticket.contact.company` |
| Boolean | `AND` `OR` `&&` `||` + `()` |
Nested dicts compared to strings use `.name` (or email/id) when present.
### Output modes
| Flag | Output |
|------|--------|
| (default) | Table: id, updated_on, status, category, subject |
| `--json` | Full issue JSON array |
| `--jsonl` | One full issue JSON per line |
| `--ids-only` | Issue ids only (best for scripts) |
Match count goes to **stderr** on table mode so stdout stays pipe-clean for `--ids-only` / jsonl.
### Automation pattern
```bash
$RT search --where 'status.name == "New"' --where 'assigned_to IS NULL' --ids-only \
| while read -r id; do
$RT view "$id" --print > "/tmp/ticket-$id.md"
done
```
---
## Saved filters (`saved-filters/`)
Put one expression per file (comments with `#` allowed):
```text
saved-filters/unassigned.expr
saved-filters/helpdeskEmailMatchCompany.expr
saved-filters/triage-needed-5d.expr
```
Reference them from CLI:
```bash
$RT search --where '@unassigned' --max-tickets 20
$RT search --where 'alias.helpdeskEmailMatchCompany' --ids-only
$RT walk --where '@triage-needed-5d' --max-tickets 10 --pager cat
```
Bare names that match a file also work: `--where unassigned`.
### Short configs / wrappers
```bash
./bin/rtl search --where 'assigned_to IS NULL' --max-tickets 10 # prod CS
./bin/rtd note --issue-id 39305 --body 'LAN' --apply # dev CS
./redmine_tickets.py --config configs/live.json ...
```
`--allow-prod-writes` is a **parent** flag (before the subcommand):
```bash
./bin/rtl --allow-prod-writes draft apply <draft_id>
```
---
## Named queue folders (`materialize-queues`)
Config: [queues.example.json](../queues.example.json)
```bash
$RT materialize-queues --config queues.example.json
# writes: $CACHE/queues/<name>/*.json → symlinks into $CACHE/$PROJECT/<id>.json
```
Agents that only understand files can list `$CACHE/queues/unassigned/`.
Dry-run:
```bash
$RT materialize-queues --config queues.example.json --dry-run
```
Refresh after cron sync if you want queues kept current (optional second cron step).
---
## View (single ticket → Markdown)
Cache-first. Does not call Redmine unless you add that later.
```bash
$RT view 41934 --print
$RT view --issue-id 41934 --pager auto # glow → bat → batcat → cat
$RT view 41934 --write-md --pager cat
```
| Flag | Meaning |
|------|---------|
| `--print` | Markdown to stdout (no pager) |
| `--pager auto\|glow\|bat\|batcat\|cat` | How to display when not `--print` |
| `--write-md` | Force write under markdown cache |
| `--force-md` | Regen even if `updated_on` stamp matches |
Markdown cache path:
```text
$CACHE/$PROJECT/markdown/<id>.md
$CACHE/$PROJECT/markdown/<id>.md.updated_on
```
Regenerates when ticket `updated_on` changes (stamp file).
---
## Walk (interactive feed)
```bash
# From property search
$RT walk --where 'assigned_to IS NULL' --max-tickets 10 --pager cat
# Explicit ids
$RT walk --ids 41934,41899 --pager cat
```
Controls: `n` / Enter = next, `p` = previous, `q` = quit, `r` = redraw.
Same filter flags as search (`--where`, `--status`, `--max-tickets`, …). `--ids` overrides search selection.
---
## Internal note vs customer email (critical)
These are **different Redmine/Helpdesk operations**. The CLI uses **separate commands** so they are hard to confuse.
| | `note` | `customer-email` |
|--|--------|------------------|
| **What it is** | Internal journal note on the issue | Helpdesk “send note” / customer reply |
| **Emails customer?** | **No** | **Yes** (when applied) |
| **API** | `PUT /issues/:id.json` with `notes` (+ optional `private_notes`) | `POST /helpdesk/email_note.json` with `message.content` |
| **Private?** | Optional `--private` | **Never** (private cannot email) |
| **Confirm flag** | None needed | **Required:** `--i-understand-this-emails-the-customer` |
| **Default** | Queue local draft | Queue local draft |
| **Apply now** | `--apply` | `--apply` (+ confirm flag already required) |
| **Prod writes** | Need `--allow-prod-writes` | Need `--allow-prod-writes` |
| **Impersonation** | `--impersonate-user rebot` (default) on apply | same |
### Queue an internal note (safe default — no email)
```bash
$RT note --issue-id 41934 --body 'Staff-only follow-up' --private
# later:
$RT draft apply <draft-id> # still does NOT email
```
### Apply internal note immediately (LAN/test preferred)
```bash
$RT --base-url http://192.168.50.170 note --issue-id 123 \
--body 'LAN test note' --apply
```
### Queue a customer email (explicit, scary name + confirm flag)
```bash
$RT customer-email --issue-id 41934 \
--body 'Thanks for writing — we received your request.' \
--i-understand-this-emails-the-customer
# later:
$RT draft apply <draft-id> --i-understand-this-emails-the-customer
```
### Send customer email immediately
```bash
$RT --base-url http://192.168.50.170 customer-email --issue-id 123 \
--body 'LAN test customer reply' \
--i-understand-this-emails-the-customer \
--apply
```
### Production writes
Writes to hosts outside the LAN allow-list (e.g. `https://rm.ldrprep.com`) are **refused** unless you pass `--allow-prod-writes`. Prefer testing on LAN first.
### Draft list
```bash
$RT draft list
# columns include EMAILS_CUSTOMER vs no-email
```
**Do not** use `note` when you mean to email a customer. **Do not** use `customer-email` for staff chatter.
---
## set-fields (status / assignee / priority / category / due date)
Does **not** email the customer. Default queues a draft; `--apply` writes (LAN-safe by default).
```bash
# By numeric id
$RT set-fields --issue-id 41934 --status-id 5 --assigned-to-id 0
# By human name (uses meta cache; run meta-refresh once)
$RT meta-refresh --force
$RT set-fields --issue-id 41934 --status Closed --assigned-to unassigned \
--notes 'Unassign + close' --private
```
| Flag | Meaning |
|------|---------|
| `--status` / `--status-id` | name or numeric status |
| `--assigned-to` / `--assigned-to-id` | name/login or id; `unassigned` / `0` clears |
| `--priority` / `--priority-id` | name or id |
| `--category` / `--category-id` | name or id; `none` / `0` clears |
| `--due-date` | `YYYY-MM-DD` |
| `--subject` | replace subject |
| `--notes` / `--private` | optional internal journal note with the update |
| `--refresh-meta` | force name→id cache refresh |
Meta cache: `$CACHE/meta/<project>/`
---
## Contacts (local mirror + light mutations)
Separate from full `redmine_contacts.py` CRM CLI on purpose (different cache + surface), but **workflows can hand off**.
```bash
# Mirror / search / show
$RT contacts-sync --max-contacts 100
$RT contacts-search --query bavaria --max-contacts 20
$RT contact-show --contact-id 1890
# Ticket → contact bridge (starts on ticket, ends on contact tools)
$RT ticket-contact --issue-id 41934
$RT ticket-contact --issue-id 41934 --show --fetch
# Contact mutation (draft/apply, same safety as tickets; does NOT email)
$RT contact-set --contact-id 4515 --add-tag maybe-customer
$RT --allow-prod-writes draft apply <id> # if draft was for prod
# LAN apply:
$RT --base-url http://192.168.50.170 contact-set --contact-id 2284 \
--add-tag maybe-customer --apply --refresh-cache
```
| Piece | Path |
|-------|------|
| Local mirror | `$CACHE/contacts/<id>.json` |
| Full CRM (batch/loops/custom fields) | `redmine_contacts.py` |
| Ticket workbench contacts | `contacts-*`, `contact-set`, `ticket-contact` |
Store: `$CACHE/contacts/<id>.json` + `manifest.json`.
---
## Local control tags (not Redmine tags)
```bash
$RT control-tag add --tag bavaria --issue-id 41934 --property subject --annotation '…'
$RT control-tag list
$RT search --control-tag bavaria
$RT control-tag suggest --tag account --where 'subject.includes("account")' --max-tickets 20
$RT control-tag suggest --tag account --where 'subject.includes("account")' --apply
$RT control-tag wipe --i-understand-this-wipes-local-control-tags
```
Store: `$CACHE/control-tags/tags.json`
---
## Attachments
```bash
$RT attachments list --issue-id 39305 --refresh
$RT attachments download --issue-id 39305 --all
$RT attachments upload --issue-id 39305 --file ./doc.pdf # dry-run
$RT --base-url http://192.168.50.170 attachments upload \
--issue-id 39305 --file ./doc.pdf --apply --notes 'attached scan'
```
Default download dir: `$CACHE/$PROJECT/attachments/<issue_id>/`.
Does **not** email the customer. Prod write needs `--allow-prod-writes`.
---
## Entities (tracking / order patterns)
```bash
$RT entities show --issue-id 41394
$RT entities scan --where 'subject.includes("1Z")' --max-tickets 20
$RT entities scan --type tracking_ups --ids-only
```
Heuristic only — not NMI-backed. Also appears in `view` Markdown.
---
## Browse (interactive loop: list → pager → actions)
```bash
# Default: fzf (preview pane) → glow/bat pager → action menu → back to fzf
./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --open-only
# Explicit pager / no preview
./bin/rtl browse --where '@unassigned' --pager glow
./bin/rtl browse --status New --no-preview
# One-shot (pick once, page, exit — no menu/loop)
./bin/rtl browse --where 'subject.includes("invoice")' --once
```
### Loop keys
| Step | What happens |
|------|----------------|
| **List** | fzf list (Esc quits). Header lists actions. Preview shows ticket snippet. |
| **Enter** | Open ticket Markdown in glow/bat/batcat (`--pager`) |
| **After pager** | Action menu for that ticket |
### Action menu
| Key | Action |
|-----|--------|
| Enter / `v` | View again |
| `n` | Queue **internal note** (no email) |
| `p` | Queue **private** internal note |
| `f` | Queue **field update** (status/assignee by name) |
| `t` | Add **local control-tag** |
| `c` | Show ticket→contact handoff |
| `e` | Extract entities (patterns) |
| `m` | **NMI brief** → confirm → queue as **private** note draft |
| `i` | Print issue id |
| `l` | Back to list |
| `q` | Quit browse |
Mutations from browse **queue drafts only** (loud `draft_id=` banner). Apply later with `draft apply`.
Customer email is **not** in the menu (use CLI deliberately).
```bash
# Notes from file (+ optional Textile via pandoc)
./bin/rtl note --issue-id 41934 --body-file ./reply.md --private
./bin/rtl note --issue-id 41934 --body-file ./reply.md --to-textile --private
# Open tickets only (search/browse)
./bin/rtl search --open-only --where 'assigned_to IS NULL' --max-tickets 20
```
Non-interactive (no TTY): prints list lines and exits.
---
## OCR rename (scanned attachments → better subject)
```bash
# One ticket
./bin/rtl ocr-rename --issue-id 12345
# Scan for generic "Scanned document…" tickets
./bin/rtl ocr-rename --where '@scanned-document' --only-generic-subjects --max-tickets 15
# Apply subject on LAN immediately (note still queued private by default)
./bin/rtd ocr-rename --issue-id 12345 --apply-subject
```
Uses `pdftotext`, else `pdftoppm`+`tesseract`. Queues subject `fields` draft + private OCR note (disable with `--no-queue-subject` / `--no-queue-note`).
---
## Categories
```bash
./bin/rtl categories list
./bin/rtl categories list --refresh
```
---
## Entities + NMI
```bash
./bin/rtl entities show --issue-id 41394
./bin/rtl entities nmi --issue-id 41394
# Env: NMI_CLI, NMI_ENV_FILE, NMI_MODE — see docs/raw-notes/features/nmi-bridge.md
# Browse action `m` queues NMI brief as private note draft
```
---
## Local drafts (queue + apply)
```bash
$RT draft list
$RT draft cancel <id>
$RT draft cancel --all-pending
$RT draft apply <id> # internal notes
$RT draft apply <id> --i-understand-this-emails-the-customer # customer_email only
```
Stored under `$CACHE/drafts/*.json`.
---
## Other subcommands (ops)
| Command | Role |
|---------|------|
| `spam-check` / `close-if-spam` | Deterministic spam tooling (LAN for apply) |
| `known-spam-scan` / `known-spam-close` | Profile-based spam |
| `tag-maybe-customer` | Contact tagging from inquiry evidence |
| `decompose` | Dry-run vault-oriented plan for one ticket |
---
## Modules
| File | Role |
|------|------|
| `redmine_tickets.py` | CLI entry: sync, search, view, walk, spam, … |
| `redmine_ticket_query.py` | Property expression compiler |
| `redmine_ticket_view.py` | Markdown render + pager |
| `redmine_contacts.py` | Shared Redmine client / contacts |
Tests: `tests/test_redmine_ticket_query_and_view.py`, `tests/test_redmine_tickets_remote_filters.py`.
---
## What is **not** here yet
- Queued notes / helpdesk replies / field mutations (P2 remaining)
- Free-text semantic search (Qdrant / semantic_index — separate track)
- SQLite projection layer (planned architecture, not shipped)
**Shipped for queues/filters:** saved-filters + `materialize-queues` (see above).
---
## Quick “is the client alive?” check
```bash
$RT search --max-tickets 3
$RT search --where 'assigned_to IS NULL' --max-tickets 3 --ids-only
ID=$($RT search --max-tickets 1 --ids-only | head -1)
$RT view "$ID" --print | head -30
```
---
## TUI (curses keyboard-driven batch processor) [NEW]
Fast single-screen TUI for the exact workflow you described: queue a template of changes, scroll/cycle tickets, paint the template onto many tickets with one key, then one button to write everything.
```bash
./bin/rtl tui --where 'assigned_to IS NULL' --max-tickets 60 --open-only
./bin/rtl tui --control-tag triage-needed --max-tickets 30
```
**Workflow in the TUI:**
1. **Set template once** (optional fields; menus or type+filter):
- `c` : Category picker (from meta cache; type to filter)
- `a` : Assignee picker (unassigned=0)
- `s` : Status picker
- `p` : Priority (type a number, e.g. 2; hotkeys 1/2/3 also work)
- `P` : Project ID (for moving the ticket)
- `t` : Local control tag (string)
- `C` : Add contact tag (repeatable; creates if new)
- `n` : Private note (applies to painted tickets)
- `e` / `m` : Toggle entity-extraction / NMI-brief flags (become control tags + notes)
2. **Navigate the list**:
- `j` / `k` / arrows : move
- `/` : filter the visible list (live)
- `r` : redraw
3. **Paint / apply**:
- `SPACE` or `Enter` on a ticket: **apply the current template** to it (records per-ticket pending: fields, note, tags, toggles). Shows marker [FNTEM] etc.
4. **Commit**:
- `A` : Queue drafts for all painted tickets (safe default).
- (If you passed `--apply` and have `--allow-prod-writes` on LAN/prod, `S` for direct apply — use with care.)
Other keys: `x` clear pending on current, `R` force meta refresh, `q` quit (prompts if pending), `?` help.
**What gets written on commit:**
- Field updates (status/assignee/priority/category/project) via KIND_FIELDS draft (or direct).
- Private note (per ticket or from template).
- Local control tags (via control-tag add).
- Contact tags (via contact_fields drafts for the helpdesk contact).
- Special toggles → control tags ("entity-extract") + placeholder NMI note if needed.
All the safety rules (drafts vs apply, prod gate, impersonate rebot) still apply. The TUI just removes the per-ticket menu tax.
**Tips:**
- Run `meta-refresh --force` once for full picker lists.
- Combine with saved filters: `./bin/rtl tui --where '@unassigned'`
- The TUI loads from the same cache/search logic as browse/walk.
## TUI Commit behavior update (auto-apply on A)
As of latest change:
- Pressing **A** in the TUI now:
1. Shows a **single** "Are you SURE?" confirmation for the whole batch.
2. Queues the drafts for every painted ticket.
3. Immediately applies those drafts (using the normal apply_draft_to_redmine + mark_applied path).
This removes the need to manually run `draft apply <id>` for each ticket.
`S` remains available for the old direct-apply (bypass draft) path if you ever want it.
All normal safety (LAN/prod gate, --allow-prod-writes, impersonate rebot, no customer emails from this path) remains in force.
See the module docstring in redmine_ticket_tui.py or run the TUI and press `?` for current keys.
## NMI Enrichment Improvements (entities nmi)
The `entities nmi` command (and the NMI path used by browse 'm' and TUI) now:
- Automatically generates rich deep links when it sees item numbers, tracking, request ids, order ids, accounts:
- Item Manager + Item Statement for item_no
- Box History (boxFinder) for tracking numbers
- Order summary and package/request history pages
- User Info for account numbers
(See also the reference note: [[INBOX/WIP/Linking to Entities Within NMI]] )
- When an item_no is resolved, it calls the new `getItemHistory` API endpoint via `nmi item history --item-no=... --snapshot`:
- Returns merged timeline of product_history (moves, condition changes), service_fees (billings), and comments.
- The private note / cs-brief output now includes a "=== NMI Deep Links ===" section and a "=== Item History (getItemHistory) ===" section (truncated for notes; full in --format json).
Usage:
```
rtl entities nmi --issue-id 123456 --format note
rtl entities nmi --issue-id 123456 --format json # full history payload
rtl entities nmi --issue-id 123456 --no-history
```
The same enrichment logic feeds the TUI and browse NMI quick action.
---
## Post-apply cache refresh (anti-staleness)
After any mutation that hits live Redmine (TUI `A`, `draft apply`, `set-fields --apply`, direct notes, etc.) the affected ticket's JSON is immediately re-fetched and overwritten in the local cache.
This means:
- Next `rtl browse`, `rtl tui`, or search sees the new status / assignee / journals / updated_on right away.
- No need to wait for the next full sync for the tickets you just touched.
In the TUI:
- `r` → local redraw (fast)
- `R` → force live refresh of the **currently highlighted** ticket (pulls fresh from Redmine + updates cache + view)
This directly supports the rule: "update the existing cached file when the ticket has changed" and "agents update the ticket when their work is done."
### TUI "S" key (direct apply)
`S` is the **direct/bypass** path (still requires write permission).
- It skips the draft queue for field changes and does an immediate `update_issue`.
- We now also call the same live cache refresh that `A` uses, so the TUI and later `browse` see the change.
- Behavior difference from `A`:
- No "are you sure?" batch confirmation.
- No draft records created for the main fields.
- Still gated by `--allow-prod-writes` (or `--apply`).
Current recommended path for most batch work remains **A** (one confirm, goes through drafts for auditability, then auto-applies).
### TUI safety / UX improvements (2026-07)
- If you start without `--allow-prod-writes`, a clear **READ-ONLY** banner appears near the top.
- Bottom status/help line is now wrapped to avoid `_curses.error` on narrow/short terminals (common when ssh'ing from Windows Terminal, Tabby, or split panes).
- After confirming `A`, the screen immediately shows "COMMIT + APPLY IN PROGRESS" feedback so it doesn't look hung while the writes happen.
- `S` now shows "fields updated DIRECTLY" + cache sync messages when it actually performs a live write.
docs/contact_loop_filter_contract.md
+183
View File
@@ -0,0 +1,183 @@
# Contact Loop Filter Contract
## Purpose
This document defines the extensible contact-tagging loop contract for `redmine_contacts.py loop-run`.
The loop runner is intentionally boring: it selects contacts, fetches recent Redmine helpdesk context, calls filters, and handles generic queue-state tags. Rich business logic belongs in FILTER-2 scripts/programs.
## Architecture
```mermaid
flowchart TD
A[loop-run profile + CLI filters] --> B[Select contacts]
B --> C{For each contact}
C --> D[Run FILTER-1]
D -->|false| E[Runner adds DTAG]
D -->|integer X| F[Runner fetches X recent tickets]
F --> G[Optional junk/text sanitizer]
G --> H[Run FILTER-2 command]
H -->|false / no-action| E
H -->|true / handled| I[Runner removes RTAGS]
E --> J[Next contact]
I --> J
```
## Runtime profile
Profiles are JSON files. The initial version deliberately supports a small surface that can grow without forcing all policy logic into the main CLI.
Example:
```json
{
"name": "noop-ai-checked",
"default_tag": "ai-checked",
"remove_tags": ["needs-review"],
"filter_1": {
"type": "builtin",
"name": "fixed-ticket-limit",
"limit": 5
},
"ticket_fetch": {
"fetch_full_issues": false,
"message_limit": 25,
"sanitize_text": true
},
"filter_2": {
"type": "command",
"argv": ["python3", "filters/noop_filter2.py"],
"timeout_seconds": 30,
"allow_mutation": false
}
}
```
Required fields:
- `default_tag`: tag added by the runner when FILTER-1 or FILTER-2 returns false.
- `filter_1`: contact-level gate.
- `filter_2`: ticket/contact worker.
Optional fields:
- `name`: human label for reports.
- `remove_tags`: tags removed by the runner when FILTER-2 returns true.
- `ticket_fetch.default_limit`: fallback ticket count if FILTER-1 omits a limit.
- `ticket_fetch.fetch_full_issues`: if true, fetch `issue-full` payloads for each recent issue.
- `ticket_fetch.message_limit`: message limit for full issue payloads.
- `ticket_fetch.sanitize_text`: compact obvious junk/noisy text before passing to FILTER-2.
## FILTER-1 contract
FILTER-1 receives a contact object and decides whether to continue.
Supported v0 modes:
- builtin `fixed-ticket-limit`: returns a configured integer.
- command filters can be added using the same stdin/stdout JSON pattern as FILTER-2.
Result meanings:
```json
{"result": false, "reason": "skip this contact"}
```
Runner behavior: add `default_tag`, do not fetch tickets, move to next contact.
```json
{"result": 5, "reason": "inspect recent tickets"}
```
Runner behavior: fetch 5 recent helpdesk issues for this contact and call FILTER-2.
## FILTER-2 contract
FILTER-2 is the extensibility point for meaningful business workflows. It may classify contacts, update tickets, update contacts, create/link external records, or do nothing.
The runner passes one JSON bundle on stdin:
```json
{
"profile": {
"name": "noop-ai-checked",
"default_tag": "ai-checked",
"remove_tags": ["needs-review"]
},
"contact": {"id": 123},
"issues": [
{
"issue_summary": {"issue_id": 456, "subject": "Example"},
"full_issue": null
}
],
"runtime": {
"base_url": "http://192.168.50.170",
"project": "customer-service",
"dry_run": true,
"helper_command": "/home/jason/projects/redmine/redmine_contacts.py"
}
}
```
FILTER-2 writes JSON to stdout:
```json
{"result": false, "summary": "not enough evidence"}
```
Runner behavior: add `default_tag`.
```json
{
"result": true,
"summary": "classified as customer",
"mutations": ["contact:123 add customer, needs-enriched, needs-nmi-sync"]
}
```
Runner behavior: remove `remove_tags` from the contact. FILTER-2 is responsible for any domain-specific mutations it claims to perform.
## Redmine access for FILTER-2
FILTER-2 programs get standard environment variables:
- `REDMINE_BASE_URL`
- `REDMINE_PROJECT`
- `REDMINE_API_KEY` (inherited only if present; never printed by the runner)
- `REDMINE_CONTACTS_HELPER`
- `REDMINE_LOOP_DRY_RUN` (`1` unless `loop-run --apply` is used)
Command FILTER paths are executed with the profile directory as the working directory unless `filter_2.cwd` overrides it. Keep profile-local scripts under `filters/` when possible.
Recommended mutation path for v0 workers is to call the helper CLI, for example:
```bash
$REDMINE_CONTACTS_HELPER update 123 --add-tag customer --apply
```
In dry-run mode, FILTER-2 must not mutate Redmine. The runner sets `REDMINE_LOOP_DRY_RUN=1` so scripts can enforce this consistently.
## Runner-owned semantics
The runner always owns generic queue state:
- FILTER-1 false -> add `default_tag`.
- FILTER-2 false -> add `default_tag`.
- FILTER-2 true -> remove `remove_tags`.
- FILTER crash/non-JSON -> fail the run; do not mutate that contact as a fallback.
This keeps specialized FILTER-2 programs powerful without making failure handling spooky.
## Initial tracer goal
The first implementation should prove the conveyor belt, not intelligence:
- select a filtered contact batch,
- run builtin FILTER-1 with a fixed ticket limit,
- fetch recent helpdesk issues,
- call a no-op FILTER-2 command,
- dry-run the default tag update,
- optionally apply it with `--apply`.
Once that is reliable, richer FILTER-2 programs can classify customers, automailers, vendors, enrichment targets, and company/person relationships.
docs/contact_triage_loop_draft_2026-07-09.md
+57
View File
@@ -0,0 +1,57 @@
# Contact Triage Loop – Draft-Stage Contact-First Approach (2026-07-09)
**Status**: Draft / exploratory. Intended as a lightweight, iterative starting point that can be tested manually in small batches before any automation is written.
**Context**: This note captures the refined direction discussed on 2026-07-09 for cleaning the customer-service project (currently >1700 open tickets). The previous keyword-on-subject approach is deprioritized. The stronger signal is **contact-first review** using recent activity history.
## Core Principle (Contact-First)
- Group decisions by **contact** (author/email) rather than individual tickets.
- For each contact, examine their **most recent 3–5 tickets/messages**.
- Decide whether that contact (and most of their open tickets) can be safely closed as spam, stale, promotional, or low-value.
- This directly targets the observed pattern that many spam/low-value tickets come from the same small set of repeat contacts.
## Draft Workflow (Repeatable in Small Batches)
1. **Discover candidate contacts**
Find authors with multiple open tickets whose most recent activity is old or low-value.
Starting signals:
- Authors with 5+ open tickets
- Authors whose most recent ticket is >60–90 days old
- Contacts whose tickets are mostly short/generic with zero follow-up
2. **Per-contact review (the decision step)**
For each contact, review their last 3–5 tickets and ask:
- Is this contact mainly sending spam, sales pitches, or unrelated noise?
- Has there been any meaningful reply or resolution path in the last 60–90 days?
- Are the tickets duplicates, test messages, or one-off low-value imports?
- Do they match existing `spam` or `promotional` tag patterns?
3. **Close decision rules**
Close most or all of a contact’s open tickets if any of the following are true:
- Recent pattern is clearly spam/stale/no-value
- No meaningful activity or resolution path in 90+ days
- Tickets are duplicates or low-value imports
**Do not close** if the contact shows any recent legitimate customer issue or money-related matter.
4. **Execution & logging**
- Use existing Redmine MCP tools or the semantic search CLI on mcpness for discovery.
- Apply `spam`/`promotional` tags where appropriate for future filtering.
- Record a short note on the contact or in a running log (which contacts were reviewed and outcome).
5. **Guardrails**
- Never close tickets involving money, formal complaints, or recent customer replies.
- When in doubt, skip the contact.
- After bulk cleanup, maintain a monthly review of “Open tickets from contacts with no activity in 60+ days.”
## Why This Stays Draft-Stage for Now
- Allows manual testing on the first 10–20 worst contacts to see what the real data looks like.
- Validates the contact-history signal before investing in looping scripts or automation.
- Directly supports shrinking the 1700+ open ticket count with low risk.
## Next Natural Steps (for the other thread)
- Run the first discovery pass on the worst offending contacts using current tools.
- Refine the decision questions or tagging rules based on what the contacts actually show.
- Decide whether to move any part of this into a lightweight script once the manual pattern is validated.
This note is a clean hand-off for the active thread working on the contact triage loop project.
docs/contact_triage_loop_raw_notes_2026-07-04.md
+154
View File
@@ -0,0 +1,154 @@
eh, I think you misunderstood me about the filters. I see this:
` batch-update Apply one payload to a filtered contact batch.`
Unless I'm misunderstanding how to use that. it sounds like it's not what I wanted/
I was hoping to be able to do this:
1. In batches of perhaps 10-20, pull a list of contacts that have not been updated for a while, and who either:
- also have specific tags
- also lack specific tags
2. For each of those contacts, do the following:
- check if there's anything about the contact itself that informs me I shouldn't need to look at anything else about it, and I can quickly tag them as something like "spam". I expect most of the time, I _won't_ be able to do that.
- Fetch the 3-5 most recent tickets from the contact, and look at them, using some framework to score any tags that should apply
- Examples:
- **customer**:
- somebody on our side emailed back +1
- seems like they're talking about packages or items in our warehouse +1
- **promo** + **spam**: contact seems like a business sending us advertisements
- from address is something like "no-reply", "noreply", "deals", or "promotions" +1
- it mentions a deal, discount, or coupon +1
- If the scores for any particular set set of tags >= a threshhold, use add_tag to add those tags to the contact
- otherwise, add a tag like "ai-checked"
- The last step caused the updated date of the contact to increase
3. Due to the very last step above (which bumps the updated date), if I wait and loop back through Step 1 again a few minutes or hours later, I won't retrieve the same contacts I already tagged.
- In this way, a looping/repeating task will eventually cycle through all contacts.
- Eventually, the ones it has already tagged will have been updated long enough ago, that they will get pulled again
## To be clear:
I am describing a broad pattern of workflow here, not specifics. I am just explaining this because:
a) The way you implemented the changes seems counterintuitive for how I want to use this, but I could of course be misunderstanding.
b) I think the basic concept of this loop should be added to the project documentation, maybe even with a diagram
## Further down the road:
- [ ] I will want to develop tags that indicates a specific kind of evaluation needs performed with a contact, just as summarizing their most recent communications with us.
- [ ] When looping through contacts, I can add that tag to contacts whose comms need to be summarized
- [ ] Then, another agent or script can fetch only contacts tagged to indicate they need summarized
- [ ] That script/agent fetches their recent communications, does it's job, and removes the tag, indicating that job is finished
- [ ] Other likely tags I will eventually implement:
- [ ] basic info needs corrected:
- Typical auto-import problems exist in the contact data, such as:
- company field contains their email address
- username portion of email address is in first_name, but their actual name is in the footer of the email or in the "from" part
- basic contact info is actually in the ticket, but not in the correct contact fields:
- first name, last name
- additional email addresses
- phone number(s)
- job title
- name of their business
- We know the name of the company they work for, but no contact.is_company=true exists with that name
- Create it
- Tag it as needing enrichment
- Update the contact(s) who work for that company, setting company = that company
- [ ] **NMI mapping**:
- [ ] Contact seems to be a customer, but their "Account Number" field is blank, so we're not sure which NMI account is their's
- [ ] Try to figure it out by comparing stuff like:
- [ ] email address
- [ ] name of company
- [ ] tracking numbers or order ids mentioned
- [ ] ^^ above, but for other contacts who work at the same company
- [ ] Update linkages all around:
- [ ] Update that company (and it's employees) contacts in redmine so "Account Number" matches `user_account.user_account_id` in NMI
- [ ] Set the redmine.contact_id's into a mapping table in NMI that link which contacts are for which account
- [ ] Update obsidian profile about the contact/person to include both pieces of info
- [ ] **entity extraction and enrichment**:
- members of our team responded to somebody, but we don't have (much) info about them in either redmine contacts or in obsidian
- This tells us the person is probably an active employee, so we:
- search for 5-10 other tickets they worked on lately, and use that to infer a profile of what their job is or fill in context about what they've worked on lately
- **phone number linkage to accounts**:
- [ ] transcribe/ingest tickets that are call transcripts, adding summary to contact file automatically
- [ ] add "note/call" to contact automatically
- [ ] identify if follow-up action is needed; if it is, create ticket, assigned to relevant person
- [ ] **bobcat caller id**: recognized contact calls us...
- [ ] **route call based on context, not just the number**:
- [ ] we're super busy, short staffed, and person likes to talk alot: "sorry, we're not available"
- [ ] Bob usually talks to this person, but Bob is marked "do not disturb": "voice mail / not available"
- [ ] **show on-screen bobcat overlay caller-id**
- [ ] **Needing enrichment** (likely more than one tag to represent the kind of enrichment):
- [ ] **social media**:
- [ ] try to identify their social media profiles (linkedin, facebook, twitter, etc) using name, email, etc.
- [ ] possible search known social media groups for people with same/similar name
- [ ] identify product website(s)=
- [ ] **potential suppliers**:
- [ ] search places like alibaba for products we've seen them sell, and try to narrow down who their suppliers are
- [ ] pull checkin pictures of shipping labels from them, and cross-reference with other info to locate possible suppliers
- [ ] **special handling**:
- [ ] **SLA/VIP**: this contact needs very fast or special responses for some reason
- [ ] possibly tied to dedicated agent who does something special with their emails, like:
- [ ] **trace package**: pull data we'd need to trace the workflows about it:
- [ ] if it looks like it has a tracking number, kick off the workflow to pull that up in NMI, get pictures, check who worked on it, tracking status, etc.
- [ ] Possibly fetch surveillance from synology autmatically:
- [ ] For each major timestamp involved ( box.checkin_date, box.presort_date, individual_product.checkin_date ) pull 2-3 minutes of footage before/after from cameras known to overlook the 2-3 nearest locations in the warehouse
- [ ] split the video feed into a series of still images 30-seconds apart
- [ ] have a VLM explain what it sees in the images, even checking for specific things:
- [ ] presence of truck at loading dock
- [ ] person seen walking away from area carrying similar locking box/item
- [ ] **import order**: email is an invoice, PO, or similar listing what a person is sending to us, perhaps the reason, and when it should arrive.
- [ ] _they're a vendor_: log that supplies were being delivered and what we spent on them
- [ ] _prep customer_: make a prep inbound order if it doesn't exist already
- [ ] **vendor receipt or invoice**:
- [ ] automatically log/input that into the expense tracker if it's not there already
- [ ] check if there is a purchase order for it.
- [ ] if there is, do something like mark it en-route, received, etc.
- [ ] create ticket, assigning somebody to put it away
- [ ] assign an AI-INFO-BLOCK to the manager, asking them to find out who ordered it, what for, and why they didn't make a PO
- [ ] **employee and it's about scheduling**
- [ ] check the attendance-schedule project to see if they already reported it in there
- [ ] if not, add it
- [ ] assign an AI-INFO-BLOCK asking them to explain why they could not do that themselves
- [ ] **check for time-off blockers**:
- [ ] somebody else already has time off within 1-2 days
- [ ] customer has complained about delays in the past week
- [ ] same person asked off on the same weekday/pattern in the past 3-4 weeks, and no good reason
- [ ] overall billings/productivity have been low for over a week
- [ ] **if blocked**: warn them they can't have the day off unless they can prove it's manditory/unavoidable
- [ ] if they say it is, AI-INFO-BLOCK until they provide proof or admit they don't have it
- [ ] **very simple question**
- [ ] **sales 20-questions**: they sent an email that amounts to playing 20-questions about capacity (wanting to start project)
- [ ] **AI-agent uses obsidian and other data**:
- [ ] generate really slick looking brochure quickly, using obsidian-backed data
- [ ] loop human in to approve it
- [ ] update/expand based on human input, then send it over to client
- [ ] save it for later, so next time, it can just be sent right away
- [ ] **spammer-be-gone**:
- [ ] Identify habitual or serious spammers, time wasters, and promoters emailing us
- [ ] use gmail MCP to add filters to contact@ldrprep.com so that person's mail doesn't get imported into redmine anymore
- [ ] **high-risk contact**:
- [ ] there is some special risk or liability related to talking to that contact. For examples:
- [ ] litigious person or debt collector
- [ ] disgruntled former employee
- [ ] stalker spouse of current employee
- [ ] handle ticket(s) in a manner people who shouldn't talk to them can't see it or can't respond to it
- [ ] phone calls from them do not ring to the general line anyone can answer
- [ ] **lead follow-up**:
- [ ] contacts marked as wanting to potentially buy something from us or start services, but we've heard crickets for 7+ days
- [ ] encourage somebody to call/email them, maybe even suggest what to say/send
## Please ensure this info is saved with the project and easy to locate
For right now, much of this will not be immediately practical to implement, but I think storing it here is both important context and saves time.
docs/contact_triage_loop_workflow.md
+211
View File
@@ -0,0 +1,211 @@
# Contact Triage Loop Workflow (Vision + Operating Pattern)
## Why this exists
This document captures Jason's intended **repeatable contact-review loop** so work can proceed in deterministic batches without reprocessing the same contacts immediately.
Captured from uploaded project note on **2026-07-04** (saved into repo context to avoid losing the design intent).
Raw source capture:
- `/home/jason/projects/redmine/docs/contact_triage_loop_raw_notes_2026-07-04.md`
---
## Core loop pattern
This is a **multi-pass routing system**, not a final one-shot classifier.
1. Select a small batch (10-20 contacts) where `updated_on` is old enough and optional tag constraints match.
2. Sort **newest stale first** (recently stale / last-updated descending), so the loop works through the freshest cooled-down contacts first.
3. For each contact:
- quick contact-only triage
- inspect recent helpdesk tickets/messages when the current pass requires it
- route by adding/removing tags that represent either classifications or workflow state
4. Apply tag updates:
- if the pass identifies a route, add that route/classification tag
- otherwise add a status marker like `ai-checked`
5. Because contact metadata was updated, `updated_on` advances.
6. On future runs, recently processed contacts fall out of the immediate stale batch; later passes can pick them up again when stale or when a designated tag routes them to another worker.
7. Later filters/workers may deliberately remove status tags such as `ai-checked` after consuming them, allowing the contact to re-enter another pass of the conveyor belt.
This forms a self-advancing conveyor belt for incremental cleanup/enrichment. Tags are not all semantic classifications: some are routing/status markers. For example, `ai-checked` means this pass touched the contact; it does not mean the contact's final identity is "AI checked." Likewise, `spam`/`promotions` can be step 1 in a longer handling pipeline, not the final disposition.
Operational objective: reducing clutter in the customer-service/sales channel is valid even when a sender is technically a vendor, billing system, automailer, or promotional sender, because Redmine customer-service is not the only channel available to them.
---
## Existing tooling alignment (current)
`redmine_contacts.py` already supports key building blocks:
- batch candidate selection:
- `batch-update --max-contacts`
- date/tag filters (`--updated-before`, `--with-tag`, `--without-tag`, `--no-tags`)
- deterministic ordering (`--order newest-updated-first` etc)
- safe mutation workflow:
- dry-run by default
- write only with `--apply`
What is still missing for full loop automation:
- richer FILTER-2 workers that classify contacts and mutate tickets/contacts when needed
- scoring engines that map evidence to tags with thresholds
- downstream queue workers for enrichment/NMI/company/person linking
Implemented tracer pieces:
- `redmine_contacts.py loop-run`
- profile contract in `docs/contact_loop_filter_contract.md`
- sample tracer profile: `profiles/contact-loop/noop-ai-checked.json`
- sample no-op FILTER-2 worker: `profiles/contact-loop/filters/noop_filter2.py`
The intended architecture is now: **runner owns conveyor-belt mechanics; FILTER-2 owns domain-specific ticket/contact mutation logic.**
---
## Suggested scoring framework (v0)
### customer
Signals:
- internal/team response exists (`+1`)
- discussion indicates warehouse packages/items (`+1`)
### promo + spam
Signals:
- sender hints (`no-reply`, `noreply`, `deals`, `promotions`) (`+1`)
- message mentions discount/coupon/deal language (`+1`)
Threshold model (initial):
- tag set applied when score >= `2`
- if no tag set crosses threshold: add `ai-checked`
---
## Diagrams
### Diagram 1 — Conveyor-belt loop
```mermaid
flowchart TD
A[Candidate pool\nRedmine contacts] --> B[Select stale contacts\nupdated_before + tag filters\nmax 10-20]
B --> C[Sort newest stale first\nupdated_on descending]
C --> D{For each contact}
D --> E[Run current pass\nFILTER-1 / worker gate]
E -->|pass can route immediately| F[Add route/status/classification tags]
E -->|needs evidence| G[Fetch recent tickets/messages]
G --> H[Manual review or FILTER-2 worker]
H --> I{Route decided?}
I -->|yes| F
I -->|no| J[Add status marker\nai-checked]
F --> K[Update contact/tickets as allowed]
J --> K
K --> L[updated_on advances\ncontact leaves immediate batch]
L --> M[Later passes use staleness + tags\nto route next work]
M --> N[Later worker may remove status tag\ne.g. ai-checked]
N --> B
```
### Diagram 2 — Tag/status re-entry lifecycle
```mermaid
stateDiagram-v2
[*] --> Eligible: updated_on older than cutoff\nand tag filters match
Eligible --> InCurrentPass: newest-stale-first selection
InCurrentPass --> Routed: worker adds route tag\nspam/promotions/etc.
InCurrentPass --> Checked: no route yet\nadd ai-checked
Routed --> CoolingDown: tag update bumps updated_on
Checked --> CoolingDown: status tag update bumps updated_on
CoolingDown --> HeldOut: too fresh for current cutoff
HeldOut --> Eligible: time passes / becomes stale again
Checked --> DownstreamQueued: later worker selects ai-checked\nor another status tag
DownstreamQueued --> ReEntered: worker removes ai-checked\nafter consuming status marker
ReEntered --> Eligible: tag filters match next pass
```
---
## Future queue/tag orchestration pattern
The long-term architecture should treat tags as explicit work queues, for example:
- `needs-summary` -> summary worker processes recent communications, then removes tag
- `needs-enrichment:*` -> enrichment workers (social/entity/profile data)
- `needs-nmi-mapping` -> account-linkage worker for NMI mapping tasks
- `special-handling:*` -> SLA/VIP/high-risk handling workflows
This lets one loop classify and route, while specialized workers execute downstream tasks.
---
## Proposed implementation phases
### Phase 1 (implemented tracer)
- Add `loop-run` command that performs a **per-contact review loop** in dry-run mode:
- select candidates
- run FILTER-1
- fetch recent tickets/messages
- call a profile-defined FILTER-2 command
- print proposed runner-owned tag updates
### Phase 2
- Replace no-op FILTER-2 with a classifier/mutator worker.
- Use FILTER-2 to update contacts/tickets directly through helper CLI calls.
- Keep runner-owned fallback `ai-checked` and RTAG removal semantics.
### Phase 3
- Add queue tags + worker handoff contracts (`needs-summary`, `needs-nmi-mapping`, etc.).
### Phase 4
- Integrate external systems (NMI mapping tables, obsidian profile updates, comm summaries, etc.) with strict scoped guards.
---
## Practical run pattern (current CLI)
Dry-run stale-first loop tracer:
```bash
./redmine_contacts.py \
--base-url http://192.168.50.170 \
--project customer-service \
loop-run \
--profile profiles/contact-loop/noop-ai-checked.json \
--source cache \
--updated-before 2026-06-01 \
--without-tag ai-checked \
--max-contacts 10 \
--order newest-updated-first
```
Apply runner-owned fallback/default tags after reviewing output:
```bash
./redmine_contacts.py \
--base-url http://192.168.50.170 \
--project customer-service \
loop-run \
--profile profiles/contact-loop/noop-ai-checked.json \
--source cache \
--updated-before 2026-06-01 \
--without-tag ai-checked \
--max-contacts 10 \
--order newest-updated-first \
--apply
```
Older dry-run batch-update skeleton:
```bash
./redmine_contacts.py \
--base-url http://192.168.50.170 \
--project customer-service \
batch-update \
--source cache \
--updated-before 2026-06-01 \
--with-tag customer \
--max-contacts 20 \
--order newest-updated-first \
--set background=ai-checked
```
> Replace payload fields/tags only after reviewing dry-run output.
docs/decomposition_pilot_contact_update_proposals_2026-07-09.json
+53
View File
@@ -0,0 +1,53 @@
{
"generated_at": "2026-07-09",
"status": "proposal-only-not-applied",
"notes": "Generated by ticket decomposition pilot. Do not apply blindly; first decide Redmine custom field/background convention for Onyx article links.",
"proposals": [
{
"redmine_issue_id": 41424,
"contact_id": 4470,
"email": "support@repackify.com",
"entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/repackify.md",
"onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md",
"add_tags": [
"maybe-customer",
"partnership-opportunity",
"buy-from-us",
"gaylords"
],
"confidence": "medium",
"rationale": "Inbound request to buy used Gaylord boxes from LDR; not generic spam, but not confirmed customer."
},
{
"redmine_issue_id": 41448,
"contact_id": 3919,
"email": "sales@u2wear.com",
"entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/u2wear.md",
"onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/u2wear.md",
"add_tags": [
"Customer",
"returns-management",
"fulfillment",
"elora",
"u2wear"
],
"confidence": "high",
"rationale": "Jason confirmed sales/info@u2wear.com are customers; ticket references a previously discussed fulfillment and returns workflow."
},
{
"redmine_issue_id": 41504,
"contact_id": 4476,
"email": "justine.micks@snugglepuppy.com",
"entity_note": "/home/jason/lilaBuild/vaults/obsidian-ldr/entities/organizations/snuggle-puppy.md",
"onyx_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/snuggle-puppy.md",
"add_tags": [
"maybe-customer",
"returns-management",
"fba-returns",
"pricing"
],
"confidence": "high",
"rationale": "Prospect/customer inquiry about warehousing and FBA returns; exposed returns-pricing/rate-sheet collateral gap."
}
]
}
docs/karpathywiki_plugin_inspection.md
+290
View File
@@ -0,0 +1,290 @@
# KarpathyWiki Obsidian Plugin Inspection
**Date:** 2026-07-09
**Plugin:** `karpathywiki` / `green-dalii/obsidian-llm-wiki`
**Repo inspected:** `https://github.com/green-dalii/obsidian-llm-wiki`
**Fresh clone verification:** commit `0979d50`
**License:** Apache-2.0
**Version observed:** 1.24.0
## Why inspected
Jason observed that the Obsidian Karpathy LLM Wiki plugin is doing much of what the Redmine ticket decomposition process should do: entity/concept extraction, knowledge graph page generation, links, indexes, and conversational query. The question was whether we can trigger it headlessly/CLI or mirror its logic.
## Repository / structure findings
The plugin is a compact TypeScript Obsidian plugin.
Important files:
```text
src/main.ts
src/wiki/wiki-engine.ts
src/wiki/source-analyzer.ts
src/wiki/page-factory.ts
src/wiki/prompts/ingestion.ts
src/wiki/prompts/generation.ts
src/wiki/prompts/merge.ts
src/wiki/prompts/lint.ts
src/wiki/query-engine/
src/core/
```
Package metadata:
```json
{
"name": "karpathywiki",
"version": "1.24.0",
"main": "main.js"
}
```
Fresh clone verification confirmed `package.json` has no `bin` entry:
```text
PACKAGE {'name': 'karpathywiki', 'version': '1.24.0', 'main': 'main.js', 'bin': None, 'license': 'Apache-2.0'}
```
There is no package `bin` entry and no obvious standalone CLI exposed by the project.
## Obsidian command entry points
`src/main.ts` registers Obsidian commands:
```text
ingest-source
ingest-folder
ingest-multiple-files
query-wiki
lint-wiki
regenerate-index
cancel-ingestion
ingest-active-file
view-ingestion-history
recreate-welcome-note
```
These are Obsidian command palette commands, not shell commands.
If Obsidian is running, it may be possible to trigger commands through Obsidian/Advanced URI style automation, but that would still depend on Obsidian being open and the plugin being loaded. It is not a true headless pipeline.
## Core engine dependency
The core class is `WikiEngine` in `src/wiki/wiki-engine.ts`.
It is separate from the plugin UI, but its constructor requires Obsidian types and APIs:
```ts
constructor(
app: App,
settings: LLMWikiSettings,
getLLMClient: () => LLMClient | null,
schemaManager: SchemaManager,
onFileWrite?: (path: string) => void,
onProgress?: (message: string) => void,
onDone?: (report: IngestReport) => void
)
```
It uses:
- `app.vault.read`
- `app.vault.getMarkdownFiles`
- `app.vault.process`
- `app.vault.create`
- `app.vault.getAbstractFileByPath`
- `app.metadataCache.getFileCache`
- `TFile`, `TFolder`, `Notice`, `normalizePath`
Therefore, the engine cannot be called directly from Node/Python CLI without either:
1. running inside Obsidian, or
2. writing an adapter/shim that emulates enough of Obsidian's `App/Vault/metadataCache/TFile` behavior.
## Portable logic worth mirroring
The useful logic is not locked to Obsidian conceptually. It can be mirrored or adapted:
### Source analysis
`src/wiki/source-analyzer.ts`:
- reads source content
- checks blank/duplicate content
- prompts LLM for structured JSON
- extracts entities and concepts
- includes aliases
- collects verbatim quotes / provenance
- identifies contradictions, related pages, key points
- normalizes irregular LLM JSON
The core ingestion prompt is in `src/wiki/prompts/ingestion.ts`.
Notable extraction output shape:
```json
{
"source_title": "...",
"summary": "...",
"entities": [
{
"name": "...",
"type": "person|organization|project|product|event|place|other",
"aliases": [],
"summary": "...",
"mentions_in_source": [],
"mentions_with_provenance": [],
"related_entities": [],
"related_concepts": []
}
],
"concepts": [],
"contradictions": [],
"related_pages": [],
"key_points": []
}
```
### Page generation
`src/wiki/page-factory.ts`:
- slugifies entities/concepts
- checks existing pages
- resolves duplicates via slug/alias/LLM semantic match
- generates entity/concept pages
- merges into existing pages
- protects reviewed pages
- corrects polluted wikilinks
- injects provenance/mentions
### Prompts
`src/wiki/prompts/generation.ts` includes high-value generation prompts for:
- entity pages
- concept pages
- source summary pages
- preserving reviewed pages
### Query / graph retrieval
The README claims zero-embedding graph retrieval via Personalized PageRank over the wiki link graph. Source references in `WikiEngine` show:
```ts
buildGraphFromContent(...)
getOrBuildGraph(...)
```
and query pipeline files exist under:
```text
src/wiki/query-engine/pipeline/
```
This is useful design inspiration, but not required for the first Redmine decomposition tool because gbrain already supplies search/indexing and the vault graph itself supplies wikilinks.
## Recommendation
Do **not** depend on Obsidian running for Redmine ticket decomposition.
Instead:
1. Mirror the plugin's source-analysis and page-generation logic in our own CLI-oriented tool.
2. Reuse its prompt structure and output schema ideas.
3. Write directly to the filesystem vault, but outside the plugin-owned `wiki/` tree by default.
4. Use gbrain for indexing/search after files are written.
5. Keep Redmine contact updates as a separate explicit apply phase.
6. Preserve compatibility with Obsidian/KarpathyWiki by using normal Markdown, wikilinks, and frontmatter fields the plugin understands where practical.
Compatibility rule: the plugin owns `wiki/entities`, `wiki/concepts`, `wiki/sources`, `wiki/index.md`, and `wiki/log.md`. The headless decomposer should default to LDR-owned paths like `Timeline/`, `entities/organizations/`, and `Processes/Redmine/`, linking to `wiki/...` pages when useful but not editing them.
This gives us the same broad effect without requiring a GUI Obsidian session and without competing with the existing plugin state.
## Possible implementation paths
### Path A — Thin TypeScript headless port
Build a Node/TypeScript CLI in `/home/jason/projects/redmine` or a sibling project that imports/copied logic from the plugin and supplies a fake Obsidian adapter.
Pros:
- Closer to upstream code.
- Easier to vendor future prompt improvements.
- Can reuse zod/AI SDK provider setup.
Cons:
- Need to emulate enough of Obsidian `Vault`, `TFile`, metadata cache, and settings.
- Adds Node/TypeScript toolchain complexity to Redmine workflow.
### Path B — Python mirror using existing Redmine tooling
Implement the same conceptual pipeline in Python alongside `redmine_tickets.py`:
- source normalization
- LLM JSON extraction
- entity/concept dedup via filesystem search + slug rules
- page generation templates
- timeline/entity writers
- gbrain sync hooks
Pros:
- Fits current Redmine tooling.
- Easier to integrate with ticket cache and contact updates.
- No Obsidian runtime needed.
Cons:
- We do not directly reuse TypeScript implementation.
- Need to maintain our own prompt/schema logic.
### Path C — Obsidian-driven automation
Use Obsidian command automation / Advanced URI to trigger plugin commands.
Pros:
- Minimal code if it works.
Cons:
- Requires Obsidian running.
- Harder to make deterministic/headless/cron-safe.
- Harder to feed Redmine tickets and capture structured contact-update proposals.
## Best fit for LDR Redmine decomposition
Path B is the best first implementation path.
Reasons:
- Redmine tickets are already fetched/cached in Python.
- We need Redmine contact updates and safety gates.
- We need Onyx URLs and gbrain sync hooks.
- We need headless/cron/agent execution.
- We already proved manual note generation format in the decomposition pilot.
Use the plugin as a reference for:
- `src/wiki/prompts/ingestion.ts` — source analysis prompt and entity/concept output shape
- `src/wiki/prompts/generation.ts` — entity/concept/source page generation prompts
- `src/wiki/prompts/merge.ts` — classify-then-route merge strategy for updating existing pages
- `src/wiki/prompts/lint.ts` — wiki quality checks and contradiction review ideas
- `src/wiki/source-analyzer.ts` — JSON normalization, duplicate/blank source handling, provenance mentions
- `src/wiki/page-factory.ts` — slug/alias resolution, cross-type collision handling, merge-vs-append routing
- `src/wiki/query-engine/` — graph retrieval ideas for later, not first pass
Do not try to invoke it via Obsidian as the primary production pipeline; use the source as a design reference for a headless Python mirror.
## Immediate next action
Update `ticket_decomposition_knowledge_ingestion_plan.md` so Phase 2 says:
- use KarpathyWiki plugin prompt/schema as reference
- implement a Python headless mirror first
- consider TypeScript adapter later only if Python mirror becomes too costly
Then implement a `decompose --dry-run` engine using the pilot tickets as fixtures.
docs/known_spam_fast_close_plan.md
+623
View File
@@ -0,0 +1,623 @@
# Known-Spam Fast Close Implementation Plan
**Date:** 2026-07-09
**Project:** `/home/jason/projects/redmine`
**Primary tool:** `redmine_tickets.py`
**Target Redmine project:** `customer-service`
**Production base URL:** `https://rm.ldrprep.com`
**Agent write attribution:** `rebot` / Harry Botter
## Goal
Build a reliable two-prong production spam-cleanup workflow:
1. **Slow/reviewed path** — the current classifier-driven `sync → spam-check → close-if-spam` flow, used for discovering and validating new spam patterns.
2. **Fast known-profile path** — a verified-profile bulk closer that quickly locates all open `customer-service` tickets matching known spam sender/profile rules and closes them in one bounded, idempotent production operation.
The immediate business goal is to close known promotional/spam clutter quickly without risking customer tickets.
---
## Current Reality
The slow path already works in production:
- `redmine_tickets.py sync` caches enriched tickets using `include=journals,helpdesk`.
- `spam-check` categorizes tickets into classes such as:
- `spam-promotional`
- `automated-operational`
- `unknown-review`
- `close-if-spam` closes reviewed `spam-promotional` tickets using `--impersonate-user rebot`.
- Writes are verified by API as `status=Closed` and close-note author `Harry B.`.
- Idempotency is recorded via local action markers.
- Customer-contact guard is active: before spam-close, the tool fetches the Helpdesk contact and skips if the contact has tag `Customer`.
Validated production cleanup so far:
- First production pass closed 20 reviewed promotional-junk tickets.
- Current 200-ticket cache has a reviewed 27-ticket candidate set pending apply.
- Customer pre-tagging has been performed for confirmed customers including Cyberstorm, Esire-family contacts, Shinkiya, Bohoflow, BavariaTrachten, Solride, and u2wear.
Known safety rule:
> Absence of `Customer` does **not** imply spam. Presence of `Customer` blocks automated spam closure.
---
## Desired Two-Prong Architecture
```text
┌────────────────────────────┐
│ redmine_tickets.py substrate│
│ sync/cache/helpdesk/actions │
└──────────────┬─────────────┘
│
┌───────────────────────────┴───────────────────────────┐
│ │
Slow / reviewed path Fast / known profile path
discover new patterns close already-verified spam
│ │
sync → spam-check → review → close-if-spam known-spam-profile scan → dry-run → apply
│ │
produces new safe profiles uses frozen allowlisted rules
```
### Slow path responsibilities
Use for:
- New sender types.
- Ambiguous business/vendor/customer cases.
- Classifier development.
- False-positive discovery.
- Contact/customer enrichment.
### Fast path responsibilities
Use for:
- Previously reviewed spam senders/domains/campaigns.
- High-volume obvious junk such as known promo newsletters and sales blasts.
- Rapid production cleanup once a profile has been approved.
The fast path must not rediscover intent. It should only act on a curated profile registry.
---
## Fast Known-Spam Profile Design
### Risk Model Update — Closed Is Recoverable
Jason clarified an important Redmine/Helpdesk behavior:
- Closed tickets are not deleted.
- If the other party sends a new message to a closed ticket, Redmine reopens it and status changes to `New Message`.
That means accidental closure is usually recoverable if the other party still cares enough to reply. The larger risk is **re-closing a ticket after the sender already reopened it**, because that means the system is ignoring renewed human intent.
Policy consequence:
1. If a ticket is clearly `spam-promotional` and stale, especially `updated_on` older than 30 days, it is reasonable to close automatically once other safety gates pass.
2. If a ticket has evidence that it was previously closed and later reopened by a new Helpdesk message, do **not** close it automatically, even if it matches spam.
3. Instead, escalate it for human attention: bump priority one level if possible and optionally set a due date 1–3 days out.
### Terminology
A **known spam profile** is a conservative, reviewed rule bundle that identifies tickets we are comfortable closing automatically, subject to safety gates.
Example profiles:
```yaml
profiles:
globalindustrial-promos:
senders:
- reply@globalindustrial.com
subject_patterns:
- "(?i)\\b(clearance|sale|save|% off|bestsellers|fans|blowers|cooling|storage|power tools|kee safety)\\b"
category: spam-promotional
confidence: high
notes: "Repeated promotional vendor emails; reviewed in LAN + prod broom passes."
vistaprint-promos:
senders:
- noreply@mail.vistaprint.com
subject_patterns:
- "(?i)\\b(sale|save|% off|email exclusive|ends|qualifying orders|favorite sale|customer faves)\\b"
category: spam-promotional
confidence: high
qnap-newsletters:
senders:
- enews-noreply@qnap.com
subject_patterns:
- "(?i)\\b(qnap|photodirector|nas|security advisory|product of the year|jc-star|file backup)\\b"
category: spam-promotional
confidence: high
business-journals-promos:
senders:
- replycincinnati@engaged.bizjournals.com
- replycincinnati@partner.bizjournals.com
- acollett@bizjournals.com
subject_patterns:
- "(?i)\\b(events|nominations|business summit|brand next to key business lists|chambercomp)\\b"
category: spam-promotional
confidence: high
```
### Safety gates always applied before close
Fast path must still apply these universal gates:
1. Ticket must be open.
2. Ticket must be in `customer-service` unless explicitly overridden.
3. Ticket Helpdesk contact must be fetched.
4. Contact with tag `Customer` must be skipped.
5. Optional: contact with tag `maybe-customer` should be skipped by default unless `--allow-maybe-customer` is explicitly passed.
6. Ticket must not already have local `known-spam-close` / `close-if-spam` marker.
7. If the ticket was previously closed and then reopened by a new Helpdesk message, skip spam-close and escalate instead.
8. Apply writes must impersonate `rebot`.
9. Apply must produce a verification report.
### Reopened-ticket escalation behavior
Add a helper such as `ticket_was_reopened_after_close(issue)` that inspects journals/details for evidence of a transition from Closed/Resolved into `New Message` or another open status after a Helpdesk reply.
When true, do not close as spam. Instead, planned behavior should be:
```json
{
"priority_id": "one level higher if known/mappable",
"due_date": "today + 1 to 3 days",
"notes": "Matched spam profile, but ticket appears to have been reopened after closure. Escalating for human review instead of auto-closing."
}
```
This should be its own idempotent action marker, e.g. `spam-reopen-escalation`, so the ticket is not bumped repeatedly.
---
## Files to Modify
- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Modify: `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- Create: `/home/jason/projects/redmine/profiles/known-spam/customer-service.json`
- Create: `/home/jason/projects/redmine/docs/rebot_legacy_spam_rules.md`
- Create/Update: `/home/jason/projects/redmine/docs/known_spam_fast_close_plan.md`
- Update: `/home/jason/projects/redmine/HANDOFF.md`
- Update: `/home/jason/projects/redmine/docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- Optional later: `/home/jason/projects/redmine/docs/known_spam_profile_registry.md`
---
## Proposed CLI
### Dry-run known-spam scan
```bash
cd /home/jason/projects/redmine
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
known-spam-scan \
--profile profiles/known-spam/customer-service.yaml \
--status-id open \
--max-tickets 2000
```
Expected output:
```text
Known-spam scan complete: seen=1732 matched=214 skipped_customer=3 skipped_maybe_customer=5 planned=206 apply=False
profile=globalindustrial-promos issue=... sender=... subject=...
...
Dry run only. Re-run known-spam-close --apply using this cache/profile after review.
```
### Apply known-spam close
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
known-spam-close \
--profile profiles/known-spam/customer-service.yaml \
--close-status-id 5 \
--impersonate-user rebot \
--apply
```
Expected output:
```text
Known-spam close complete: planned=206 closed=206 skipped_customer=0 skipped_existing_marker=0 apply=True
```
### Verify apply
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_<timestamp> \
verify-known-spam-close
```
Expected output:
```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```
---
## Implementation Tasks
### Task 1: Add profile loader
**Objective:** Load known-spam YAML profiles safely.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
- Create: `profiles/known-spam/customer-service.yaml`
**Implementation notes:**
- Use stdlib if PyYAML is not guaranteed? Existing environment may support YAML via project deps, but avoid installing packages.
- Safer first version: store profile as JSON instead of YAML if dependency uncertainty matters.
- Recommended file if avoiding deps:
```text
profiles/known-spam/customer-service.json
```
**Test cases:**
- valid profile loads
- missing required fields fails
- invalid regex fails with helpful error
**Acceptance:**
```bash
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```
---
### Task 2: Implement known-profile matcher
**Objective:** Determine whether a cached ticket matches a known spam profile.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Rules:**
A ticket matches when:
- sender matches one of profile senders/domains, and
- subject/body pattern matches if provided, and
- category is `spam-promotional`, and
- it does not match profile exclusions.
**Test examples:**
- `reply@globalindustrial.com` + `Clearance Sale: Top Steel Storage And Power Tools` matches `globalindustrial-promos`.
- `reply@globalindustrial.com` + a real invoice/order/customer thread does **not** match if exclusion pattern present.
- `enews-noreply@qnap.com` + `QNAP Security Advisory` matches only if we decide QNAP security advisories are clutter; this profile should be explicit so no one later acts surprised.
---
### Task 3: Add customer/maybe-customer universal skip gates
**Objective:** Ensure known-spam fast path has the same safety guard as slow path.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Tests:**
- contact tagged `Customer` is skipped even if ticket matches known profile.
- contact tagged `maybe-customer` is skipped by default.
- `--allow-maybe-customer` permits maybe-customer if explicitly requested.
- absence of tags does not block known spam profile matching.
**Acceptance:**
Fast path must never bypass customer safety.
---
### Task 3B: Detect reopened-after-close tickets and escalate instead of closing
**Objective:** Prevent the cleaner from repeatedly closing a ticket after the other party has reopened it with a new message.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Implementation notes:**
- Add `ticket_was_reopened_after_close(issue)`.
- Inspect `issue['journals']` and their `details` for status changes.
- Treat evidence of Closed/Resolved → New Message/open status as reopened-after-close.
- If exact status IDs are easier than names, map known status names/IDs from existing Redmine data before hardcoding.
- Do not rely on current status alone; the point is historical reopening.
**Tests:**
- Journal with status change Closed → New Message returns true.
- Journal with only New → Closed returns false.
- Known-spam close skips reopened-after-close tickets.
- Apply path records `spam-reopen-escalation` instead of `known-spam-close`.
**Escalation behavior:**
- Dry-run reports the escalation plan.
- Apply should update priority/due date only when explicit `--apply-escalations` or equivalent is provided; do not mix escalation writes into spam-close apply without a clear flag.
---
### Task 4: Add `known-spam-scan` command
**Objective:** Sync/search open tickets and report matches without closing.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
- Fetch open `customer-service` tickets with `include=journals,helpdesk`.
- Cache ticket payloads.
- Evaluate known spam profiles.
- Fetch contacts for matched tickets only.
- Report:
- matched profile
- issue ID
- sender
- subject
- contact ID
- age / `updated_on`
- skipped reason if skipped
- Mark tickets older than 30 days and matching a verified `spam-promotional` profile as strong fast-close candidates.
- Newer tickets may still be candidates, but the report should separate them from stale 30+ day candidates.
- Record local action marker:
- `known-spam-scan = matched|matched-stale|skipped-customer|skipped-maybe-customer|skipped-reopened|not-matched`
**Dry-run only:** This command should never write Redmine.
---
### Task 5: Add `known-spam-close` command
**Objective:** Close the previously scanned/planned known-spam tickets.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
- Reads cache + scan markers.
- Only closes tickets with marker `known-spam-scan = matched`.
- Re-checks contact tags before close. Do not rely only on stale scan state.
- Updates ticket with:
```json
{
"status_id": 5,
"notes": "Closed as known promotional spam by redmine_tickets.py known-spam-close. Profile: <profile-name>."
}
```
- Writes as `rebot`.
- Records:
- `known-spam-close = closed`
- profile name
- close payload
- verification pending
**Tests:**
- apply closes only matched markers
- second apply closes zero
- customer tag recheck blocks closure even if scan marker was matched
- impersonation defaults to `rebot`
---
### Task 6: Add verification command/report
**Objective:** Verify production writes after apply.
**Files:**
- Modify: `redmine_tickets.py`
- Test: `tests/test_redmine_tickets.py`
**Behavior:**
For every `known-spam-close = closed` marker:
- fetch issue with `include=journals,helpdesk`
- verify `status.name == Closed`
- verify close note exists
- verify note author is `Harry B.`
- print failures clearly
**Output:**
```text
Verify known-spam close: checked=206 closed=206 author_harry_b=206 failures=0
```
---
### Task 7: Seed first known-spam profile registry from verified production data
**Objective:** Convert already-validated spam senders into conservative profiles.
**Initial profile groups:**
- Global Industrial promos
- Vistaprint promos
- QNAP newsletters/promos
- Business Journals / BizJournals promo/events
- TikTok Shop seller promos
- Popmenu restaurant promos
- Samsung promo newsletters
- Cintas mop/product promos
- Wings and Rings restaurant promos
- SellerEssentials promos
**Do not include yet without explicit decision:**
- `sales@supermarketdistribution.com` / stock/MOQ updates — may be vendor-ish; review first.
- QNAP security advisories — likely clutter, but security language can look operational; make it explicit in profile if included.
- Any sender with `Customer` or `maybe-customer` contacts.
---
### Task 8: Production rollout procedure
**Objective:** Define exact operator sequence for today.
**Step A: Commit or checkpoint local code**
Before production bulk close, ensure current code/tests/docs are checkpointed in git or at least `git diff` saved.
```bash
cd /home/jason/projects/redmine
git status --short
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```
**Step B: First fast-path dry run**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_known_spam_$(date +%Y%m%dT%H%M%SZ) \
known-spam-scan \
--profile profiles/known-spam/customer-service.json \
--status-id open \
--max-tickets 2000
```
**Step C: Human review**
Review summary counts and sample candidate rows. If any candidate looks like customer/prospect/operational context, do not apply; add exclusion/profile refinement.
**Step D: Apply**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir <reviewed-cache> \
known-spam-close \
--profile profiles/known-spam/customer-service.json \
--close-status-id 5 \
--impersonate-user rebot \
--apply
```
**Step E: Verify**
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir <reviewed-cache> \
verify-known-spam-close
```
**Step F: Handoff update**
Update:
- `HANDOFF.md`
- `docs/ticket_spam_cleanup_handoff_2026-07-09.md`
- optionally `docs/known_spam_profile_registry.md`
---
## Acceptance Criteria
The implementation is acceptable when:
1. Slow path still works and tests pass.
2. Fast path can dry-run all open production customer-service tickets without writes.
3. Fast path can report exact match profile per candidate.
4. Fast path skips contacts tagged `Customer`.
5. Fast path skips contacts tagged `maybe-customer` by default.
6. Fast path writes only when `--apply` is passed.
7. Fast path writes as `rebot`.
8. Second apply is idempotent.
9. Verification command proves status closed + Harry B. note author.
10. Profile registry is conservative and versioned in the repo.
---
## Short-Term Recommendation for Today
Before building the full fast-profile command, there is still an immediate safe action available:
1. Apply the currently reviewed 27-ticket production close set from:
```text
.cache/redmine_tickets_prod_200_promotional_eval_1783612885
```
2. Verify closed / Harry B. / idempotency.
3. Then implement the fast known-spam profile path above.
This keeps the broom moving while the turbo-broom gets brakes installed. Very advanced broom science.
---
## Legacy Rebot Rule Extraction Status
Completed source inspection of old Rebot spam/routing logic and captured it in:
```text
docs/rebot_legacy_spam_rules.md
profiles/known-spam/customer-service.json
```
Validated profile seed:
```text
JSON_OK
profiles 7
globalindustrial-promos spam-promotional 1 sender regex / 1 subject regex
vistaprint-promos spam-promotional 1 / 1
qnap-newsletters spam-promotional 1 / 1
business-journals-promos spam-promotional 3 / 1
legacy-rebot-promotional-senders spam-promotional 16 / 0
legacy-rebot-operational-clutter automated-operational 4 / 10
legacy-rebot-receipts-routing-reference routing-reference 4 / 1
```
Important split:
- `legacy-rebot-promotional-senders` can be considered for fast close after dry-run review.
- `legacy-rebot-operational-clutter` preserves old Rebot close rules but should remain report-only until Jason explicitly approves operational clutter closing.
- `legacy-rebot-receipts-routing-reference` should not be closed as spam; it is future routing/reference material.
docs/raw-notes/Untitled.md
+33
View File
@@ -0,0 +1,33 @@
@everyone I think we should do some tune-ups and checks on the "advanced" Redmine CLI located in ~/projects/redmine/ that is pointed to by ~/projects/redmine/bin/rtl
We kind of had a history with this tool:
- Initially, it kicked ass and was very fast and powerful, but it lacked a nice TUI
- We attempted to build a TUI using curses, and found that had severe limitations on the UI; the functionality was powerful, but the UI/UX sucked ass.
- We tried rebuilding a TUI using more modern libraries and apps as reference points, and somehow totally lost the plot. Not only did we end up with incomplete features agents swore they had completed, we ended up regressing the codebase terribly, to the point even the original CLI features became unreliable, didn't do what the docs/specs said they did, and all kinds of problems. Nobody involved was able to identify a straight path to fixing that.
- We only recently seem to have restored the basic CLI functionality of the tool. From what I understand, the majority of the problem(s) turned out to be:
a) Unreachable code in CLI argument parsing
b) Something to do with order of arguments supplied before or after others would cause silent failures / weird behavior
c) Unverified, but we've seen evidence there is a bug that can leave the CACHE in an unusable state if the script is terminated before it finishes a sync, and the code is seems to be unable to recover from or _recognize_ that problem. On subsequent runs, it will act like there are no problems, it just doesn't give correct output if this happens.
### As far as I know, at this point, the following things works correctly:
- `jason@onyx > ./bin/rtl sync --remote-status-id '*' --remote-updated-since 30d --run-log --run-log-keep 20` run every 5-10 minutes to keep the local cache updated
- the `walk` verb, except it does not have a help menu for what commands it supports (n, p, q, r)
- view verb, but unclear if it displays local cache or fetches live
- `materialize-queues` seems to work
- `contacts-sync`
- Need to make sure this _does not_ delete old contacts if run, for instance, with something like `--nax0contacts 50`
-
### Minor issues
- `draft` verb:
- doesn't have mechanism to _show_ a drafted change
- should have simple way to clear old drafts or something
-
### Definitely not working:
- triage verb
-
docs/raw-notes/features-for-redmine_tickets-sync.md
+332
View File
@@ -0,0 +1,332 @@
# Features / Roadmap: power-user Redmine client (`redmine_tickets` plane)
> [!todo] Keep this file as the **index**: status, priority, and short summaries.
> Long implementation detail lives in `docs/raw-notes/features/*` and is linked from here.
> Update checkboxes when work lands; do not re-expand full specs into this file.
**Last status refresh:** 2026-07-15 (post browse-loop + attachments + entities + contacts bridge)
---
## Product intent (what this is)
A **scriptable power-user Redmine client** for humans and agents:
- durable local ticket (+ contact) cache as the **ops workbench**
- property-based local search, saved filters, filesystem queues
- CLI / TUI-lite read path (Markdown + `glow`/`bat`/`fzf`) then **queued** writes
- place to invent/test filters, tools, and agent loops without living in the web UI
**Not** the system of record (Redmine is).
**Not** the vault/wiki (curated memory).
**Not** Qdrant (optional later semantic layer).
First project focus: **`customer-service`**.
| Env | Redmine | Cache | Wrapper |
|-----|---------|--------|---------|
| Prod | `https://rm.ldrprep.com` | `/home/jason/projects/redmine/cache/redmine_tickets` | `bin/rtl` |
| Dev / mutation testing | `http://192.168.50.170` | `/home/jason/projects/redmine/cache-dev/redmine_tickets` | `bin/rtd` |
**Safety defaults**
- Writes/applies prefer LAN; prod needs parent `--allow-prod-writes`
- `note` never emails; `customer-email` always emails (ugly confirm flag)
- Browse/actions **queue drafts** by default (apply later)
- Dual tools OK for now: tickets workbench vs full `redmine_contacts.py` CRM CLI, with bridges
---
## Priority stack (current)
| Priority | Area | Status |
|----------|------|--------|
| P0 | Durable incremental ticket sync + remote filters | **Done** |
| P0 | Cron for CS refresh (+ queue rematerialize) | **Running** (6h window) |
| P1 | Property-based local search (`--where`) | **Done** |
| P1 | Markdown view + walk | **Done** |
| P1 | Browse loop (fzf → pager → action menu) | **MVP done** (polish open) |
| P2 | Saved filters + symlink queues | **Done** |
| P2 | Queued notes / customer-email / set-fields | **Done** |
| P2 | Config wrappers (`--config`, `rtl`/`rtd`) | **Minimal done** |
| P3 | Local control tags + suggest + JSONL export | **Done** (LLM refine open) |
| P3 | Contacts mirror + contact-set + ticket-contact | **Done** (cache unify optional) |
| P4 | Attachments list/download/upload | **Done** |
| P4 | Entity extract (patterns) + NMI bridge CLI | **MVP done** |
| Later | Full curses TUI / nested config DSL / master layout | Deferred |
| Adjacent | Semantic index / vault decompose / OCR rename skill | Separate tracks |
---
## Capability map (what you can run today)
| Need | Command(s) |
|------|------------|
| Refresh tickets | `sync` · cron: `cron-scripts/sync-live-customer-service.sh` |
| Search cache | `search --where '…'` · `@saved` · `--control-tag` · `--has-attachments` |
| Read ticket | `view ID` · `walk` · **`browse`** (loop) |
| Internal note | `note` (draft → `draft apply`) |
| Customer email | `customer-email` + confirm flag |
| Fields | `set-fields` (`--status Closed`, etc.) · `meta-refresh` |
| Local labels | `control-tag add|suggest|export|…` |
| Contacts | `contacts-sync` · `contacts-search` · `contact-show` · `contact-set` · `ticket-contact --fetch` |
| Files | `attachments list|download|upload` |
| Tracking/order hints | `entities show|scan` · `entities nmi` · browse `m` |
| OCR scan rename | `ocr-rename --issue-id` / `--where '@scanned-document'` |
| Categories | `categories list [--refresh]` |
| Queues on disk | `materialize-queues --config queues.example.json` |
Cookbook: **`docs/cli_power_user_client.md`**
---
## P0 — Sync foundation
**Detail:** [[features/sync-foundation]]
- [x] Stable cache path (not per-run timestamped dirs)
- [x] Incremental full-fetch skip when `updated_on` unchanged
- [x] Server-side `updated_on` + `status_id` (remote flags)
- [x] Local post-filters retained
- [x] Sync progress + `--quiet` for cron
- [x] Cache never auto-deletes old ticket JSON
- [x] Run-stats ring buffer (`--run-log`)
- [x] Cron: live + dev CS scripts; **6h** remote window; rematerialize queues after sync
- [x] Optional contacts cron: `cron-scripts/sync-live-contacts.sh`
- [ ] Confirm long-term cadence (currently ~every few minutes) is still desired
- [x] Full fetch includes `attachments` metadata (not just journals/helpdesk)
```bash
./bin/rtl sync --remote-status-id '*' --remote-updated-since 2h --run-log --quiet
```
---
## P1 — Local search
**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_query.py`
**Historical thin search:** [[../2026-07-14_localsearch_before_sanity]]
- [x] JSON path filters + `includes()` / `since()` / AND/OR / parens
- [x] Nested helpdesk/contact paths
- [x] `--ids-only`, ordering, max caps
- [x] Table columns: assignee, contact, control tags
- [x] `--has-attachments` (cache metadata)
- [x] **No default status filter** — closed tickets appear if cached
- [ ] Optional default “open-ish” profile (if desired later)
---
## P1 — Read path + browse loop
**Detail:** [[features/local-search-and-cli-client]] · `redmine_ticket_view.py` · `redmine_ticket_browse.py`
- [x] Ticket → Markdown (`view`) + MD cache under `markdown/`
- [x] Pagers: glow / bat / batcat / cat
- [x] `walk` (n/p/q)
- [x] **`browse` loop:** fzf (or numbered) → pager → action menu → list
- Actions: view again, queue note/private note, queue fields, control-tag, contact handoff, entities, print id
- Mutations **queue only**
- `--once`, `--pager`, filters same family as search
- [ ] Browse polish (user-noted “minor things”) — collect below under **Open polish**
- [ ] Full curses/TUI (linear-tui style) still **later**, not blocking ops
```bash
./bin/rtl browse --where 'assigned_to IS NULL' --max-tickets 80 --pager glow
```
---
## P2 — Filters, queues, mutations
**Detail:** [[features/local-search-and-cli-client]] · drafts in `redmine_ticket_drafts.py`
- [x] `saved-filters/*.expr` + `@name`
- [x] `materialize-queues` symlink folders
- [x] `note` vs `customer-email` (hard separation; Helpdesk `email_note` path)
- [x] Draft list/cancel/apply; draft binds `base_url`; obvious `draft_id=`
- [x] `set-fields` + name→id (`meta-refresh`)
- [x] Parent `--allow-prod-writes` (not subparser-shadowed)
- [x] Minimal `--config` + `bin/rtl` / `bin/rtd` / `bin/qaa`
- [ ] Nested config defaults (`search.max-tickets` etc.) — [[features/config-files]]
- [ ] Note body from file + optional pandoc→Textile (steering list)
```bash
./bin/rtl note --issue-id 41934 --body 'staff context' --private
./bin/rtl customer-email --issue-id 41934 --body '…' --i-understand-this-emails-the-customer
./bin/rtl --allow-prod-writes draft apply <id>
```
---
## P3 — Control tags
**Detail:** [[features/control-tags]] · export: `redmine_ticket_control_export.py`
- [x] Local wipeable store under `control-tags/tags.json`
- [x] add / remove / list / show / for-issue / wipe
- [x] suggest (`--where` rules, optional `--apply` local)
- [x] `export` → JSONL for training / refine
- [ ] LLM refine loop (consume export, propose tags, human correct/incorrect)
---
## P3 — Contacts
**Detail:** [[features/contacts-sync-integration]] · `redmine_ticket_contacts.py`
**Full CRM still:** `redmine_contacts.py` (separate cache/CLI by design for now)
- [x] Durable mirror: `<cache>/contacts/`
- [x] `contacts-sync` / `contacts-search` / `contact-show`
- [x] `contact-set` draft/apply (tags, company, names, `--set`)
- [x] `ticket-contact` bridge; **`--fetch` implies show+refresh** (bugfix)
- [x] Contact column in ticket search
- [ ] Optional unify on-disk layout with `redmine_contacts.py` (only if dual-cache pain grows)
---
## P4 — Attachments
**Detail:** [[features/attachments]] · `redmine_ticket_attachments.py`
- [x] `attachments list` (cache-first, `--refresh`)
- [x] `attachments download` (one / `--all`)
- [x] `attachments upload --apply` (POST `/uploads.json` + issue `uploads`; no email)
- [x] OCR rename workflow CLI (`ocr-rename`) — [[features/ocr-rename]]
---
## P4 — Entities + NMI
**Detail:** [[features/entity-type-mapper]] · `redmine_ticket_entities.py` · `redmine_ticket_nmi.py`
- [x] Pattern extract: UPS/USPS, order/SKU/account + LDR `item_no`
- [x] Shown in Markdown view
- [x] `entities show|scan`
- [x] `entities nmi` bridge to `mrscClient-php bin/nmi` (cs-brief / resolve)
- [x] Noise filters on medium-confidence `order_id` hits
- [ ] Richer NMI membership index (known order/item windows) if CLI alone isn’t enough
- [x] One-key NMI from browse menu (`m` → private note draft)
---
## Open polish / small bugs (browse + ops)
- [x] Browse: fzf header/keybinding discoverability
- [x] Browse: optional preview window in fzf (`--no-preview` to disable)
- [x] Browse: louder draft queue banner (`announce_draft`)
- [x] Browse: `m` → entities nmi brief → private note draft
- [x] Entity noise filters + LDR 9-digit `item_no`
- [x] Note `--body-file` (already) + `--to-textile` (pandoc)
- [x] `search`/`browse`/`walk` `--open-only`
- [x] OCR attachment → rename (`ocr-rename`) MVP
- [ ] OCR cron pack for HR “Scanned document” inboxes
- [ ] categories create/update (list only for now)
---
## Steering / next decisions (pick a lane)
### A — Daily-driver polish — **done for now**
Reopen if browse UX still nags.
### B — CS frontage / NMI / OCR — **MVP+ done**
Optional: OCR cron for HR; NMI membership index.
### C — CRM / spam tooling **← good next lane**
1. Deeper `contact-set` (custom fields) or keep routing to `redmine_contacts.py`
2. Control-tag LLM refine from JSONL export
3. Issue categories create/update for triage taxonomy
### D — Bigger product bets (defer unless intentional)
- Full curses TUI
- Unified contact cache
- Composable remote/local filter DSL
- Redmine project/version/custom-field admin
---
## Explicitly out of this index (or later)
| Item | Notes |
|------|--------|
| Semantic index v2 | `semantic_index/` — feed from this cache |
| Vault decompose | separate track |
| Redmine admin: versions / custom field *definitions* / project modules | not ticket ops |
| Master global JSON + symlink org | nice-to-have |
---
## Related docs
- `docs/cli_power_user_client.md` — **usage cookbook**
- `docs/ticket_sync_tool_plan.md` — original sync plan
- `docs/2026-07-14_localsearch_before_sanity.md` — historical
- `docs/ticket_spam_cleanup_handoff_2026-07-09.md` — spam consumers
- `docs/raw-notes/features/*.md` — detail files
- `configs/live.json`, `configs/dev.json` · `bin/rtl` · `bin/rtd`
### Feature detail files
| File | Topic |
|------|--------|
| [[features/sync-foundation]] | Sync / cache / remote filters |
| [[features/local-search-and-cli-client]] | Search, view, walk, queues, writes |
| [[features/control-tags]] | Local control tags |
| [[features/contacts-sync-integration]] | Contacts mirror + bridge |
| [[features/attachments]] | File attachments |
| [[features/entity-type-mapper]] | Entities + NMI |
| [[features/nmi-bridge]] | NMI env + browse `m` |
| [[features/ocr-rename]] | OCR subject rename |
| [[features/browse-loop]] | fzf browse loop |
| [[features/config-files]] | Config / aliases |
---
## Maintenance rule
1. Put full detail in `docs/raw-notes/features/<name>.md`
2. Leave status + short bullets + link here
3. Prefer updating the feature file over re-pasting long prose into this index
4. When fixing a bug from **Bug reports**, check the box and leave the minimal repro snippet
---
## Bug reports and QOL (resolved log)
### --allow-prod-writes unrecognized / ignored
- [x] Parent flag only; subparser copies removed (child defaults overwrote parent)
### Draft id not obvious
- [x] `draft_id=` / `apply_hint=` key=value output
### Drafts ignore create-time base-url
- [x] Apply prefers draft `base_url`; conflict needs `--allow-base-url-override`
### Configs / short aliases
- [x] Minimal `--config` + `bin/rtl|rtd|qaa`
### `ticket-contact --fetch` alone did nothing
- [x] `--fetch` implies show + live refresh into contacts cache
---
## Modules (code map)
| Module | Role |
|--------|------|
| `redmine_tickets.py` | CLI entry + sync + routing |
| `redmine_ticket_query.py` | Local AST filters |
| `redmine_ticket_view.py` | Markdown render / pager |
| `redmine_ticket_browse.py` | fzf loop + action menu |
| `redmine_ticket_filters.py` | Saved filters / queues |
| `redmine_ticket_drafts.py` | Draft queue + write safety hosts |
| `redmine_ticket_meta.py` | Name→id meta cache |
| `redmine_ticket_control_tags.py` | Local tags store |
| `redmine_ticket_control_export.py` | Tag JSONL export |
| `redmine_ticket_contacts.py` | Contact mirror |
| `redmine_ticket_attachments.py` | Upload/download helpers |
| `redmine_ticket_entities.py` | Pattern extract |
| `redmine_ticket_nmi.py` | NMI CLI bridge |
| `redmine_ticket_ocr.py` | Attachment OCR + subject suggest |
| `redmine_contacts.py` | Full CRM CLI (sibling tool) |
docs/raw-notes/features/attachments.md
+28
View File
@@ -0,0 +1,28 @@
# Feature detail: Ticket attachments
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** MVP 2026-07-15
**Module:** `redmine_ticket_attachments.py`
## Commands
```bash
./bin/rtl attachments list --issue-id 39305 --refresh
./bin/rtl attachments download --issue-id 39305 --attachment-id 55917
./bin/rtl attachments download --issue-id 39305 --all
./bin/rtd attachments upload --issue-id 39305 --file ./scan.pdf # dry-run
./bin/rtd attachments upload --issue-id 39305 --file ./scan.pdf --apply --notes 'scan attached'
```
| Op | Emails customer? | Notes |
|----|------------------|--------|
| list | no | cache-first; `--refresh` pulls `include=attachments` |
| download | no | needs API key; default dir `<cache>/<project>/attachments/<issue_id>/` |
| upload | **no** | Redmine `POST /uploads.json` + issue `uploads` token; prod needs `--allow-prod-writes` |
Sync now also requests `attachments` on full issue fetch so metadata lands in cache over time.
## Still open
- OCR rename workflow (see [[../ocr-ticket-attachment-to-rewrite-update-ticket]])
- Draft-queue binary uploads (paths) if desired
docs/raw-notes/features/browse-loop.md
+44
View File
@@ -0,0 +1,44 @@
# Feature detail: Interactive browse loop
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** MVP+ polish 2026-07-15
**Module:** `redmine_ticket_browse.py`
## Flow
1. Filtered ticket list via **fzf** (preview pane) or numbered pick
2. **Enter** → Markdown pager (`glow` / `bat` / `batcat` / `cat`)
3. After pager → **action menu**
4. Back to list until Esc/q
## Actions (queue-safe)
| Key | Behavior |
|-----|----------|
| v / Enter | View again |
| n / p | Queue internal / private note |
| f | Queue field update (names) |
| t | Local control-tag |
| c | ticket-contact handoff |
| e | Pattern entities |
| **m** | NMI cs-brief → confirm → **private** note draft |
| i | Print id |
| l / q | List / quit |
## Flags
- `--where`, `--status`, `--control-tag`, `--has-attachments`, **`--open-only`**
- `--max-tickets`, `--order`, `--pager`, `--once`, **`--no-preview`**
## Done polish
- [x] fzf header lists actions
- [x] fzf preview pane
- [x] loud draft banner
- [x] NMI menu action
## Non-goals
- Full curses app (separate spike)
- Customer-email from browse (CLI-only by design)
- Applying prod writes from browse without draft step
docs/raw-notes/features/config-files.md
+41
View File
@@ -0,0 +1,41 @@
# Config files / short aliases (minimal)
**Parent:** [[../features-for-redmine_tickets-sync]]
**Status:** minimal MVP (global defaults + bin wrappers). Nested per-command defaults deferred.
## Runtime config
```bash
./redmine_tickets.py --config configs/live.json search --max-tickets 10
./redmine_tickets.py --config configs/dev.json note --issue-id 39305 --body 'x' --apply
```
JSON (always works) or YAML if PyYAML is installed.
Supported top-level keys (also under `redmine_tickets:`):
- `base-url` / `base_url`
- `project`
- `cache-dir` / `cache_dir`
- `allow-prod-writes` (bool)
CLI flags already present win over config.
## Short wrappers (`bin/`)
| Command | Maps to |
|---------|---------|
| `bin/rtl` | live/prod CS (`configs/live.json`) |
| `bin/rtd` | LAN/dev CS (`configs/dev.json`) |
| `bin/qaa` | alias → `rtl` (muscle-memory example) |
```bash
./bin/rtl search --where 'assigned_to IS NULL' --max-tickets 5
./bin/rtl --allow-prod-writes draft apply <id>
./bin/rtd note --issue-id 39305 --body 'LAN note' --apply
```
## Not yet
- Nested `search.max-tickets` / command-specific defaults from YAML namespaces
- Auto-discovery of many named profiles beyond live/dev
docs/raw-notes/features/contacts-handling.sh
+5
View File
@@ -0,0 +1,5 @@
## Contact Triage / Fixing
- [ ] Properly link "call transcript" and "voicemail" tickets to the correct person by searching the contact database with the phone number involved
- [ ] Ingest call transcripts into our timeline
- [ ] Fix contact company is set to their email address, when the name of their business is easily found through context in tickets
docs/raw-notes/features/contacts-sync-integration.md
+38
View File
@@ -0,0 +1,38 @@
# Feature detail: Contacts sync + mutation (tie-in)
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** Sync/search/show + draftable `contact-set` + `ticket-contact` bridge (2026-07-15)
**Module:** `redmine_ticket_contacts.py`
**Still separate from:** full `redmine_contacts.py` CRM CLI (batch/loops/custom-field tooling)
## Why not one tool
- Tickets workbench wants co-located per-id JSON + draft/apply safety next to tickets.
- `redmine_contacts.py` already owns heavy CRM workflows and a different cache layout.
- Do not rewrite/delete the CRM CLI; add bridges so a ticket-started workflow can end on contacts.
## Implemented
```bash
./bin/rtl contacts-sync --max-contacts 100
./bin/rtl contacts-search --query ada
./bin/rtl contact-show --contact-id 14 --fetch
# Bridge ticket → contact id
./bin/rtl ticket-contact --issue-id 41934
./bin/rtl ticket-contact --issue-id 41934 --show --fetch
# CRM mutation (draft by default; same prod/LAN gates as ticket writes)
./bin/rtl contact-set --contact-id 4515 --add-tag maybe-customer
./bin/rtd contact-set --contact-id 2284 --add-tag maybe-customer --apply --refresh-cache
```
- Durable path: `<cache-dir>/contacts/<id>.json` + `manifest.json`
- Draft kind: `contact_fields` (apply via `draft apply`)
- Handoff hints print `redmine_contacts.py show …` for full CRM when needed
## Still open
- Richer custom-field helpers (use `redmine_contacts.py` or extend `--set`)
- Optional cron for contacts next to ticket sync
- Shared library / single on-disk layout if dual-cache friction becomes painful
docs/raw-notes/features/control-tags.md
+31
View File
@@ -0,0 +1,31 @@
# Feature detail: Local control tags (filter-assist labeling)
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** MVP implemented 2026-07-15 (`control-tag` CLI + `search --control-tag`)
**Module:** `redmine_ticket_control_tags.py`
**Critical distinction:** these tags are **local-only**. They are **not** Redmine issue tags, **not** contact CRM tags.
## Storage
`<cache-dir>/control-tags/tags.json` — wipe without touching Redmine or ticket JSON.
## CLI
```bash
./redmine_tickets.py ... control-tag add --tag bavaria --issue-id 41934 \
--property subject --property helpdesk_ticket.contact.company \
--annotation 'BavariaTrachten shape'
./redmine_tickets.py ... control-tag list
./redmine_tickets.py ... control-tag show --tag bavaria
./redmine_tickets.py ... control-tag for-issue --issue-id 41934
./redmine_tickets.py ... control-tag remove --tag bavaria --issue-id 41934
./redmine_tickets.py ... control-tag wipe --i-understand-this-wipes-local-control-tags
./redmine_tickets.py ... search --control-tag bavaria
```
## Still open
- Suggest/refine loop (LLM or rules + human correct/incorrect)
- Training-export format for classifier experiments
docs/raw-notes/features/entity-type-mapper.md
+25
View File
@@ -0,0 +1,25 @@
# Feature detail: External entity type mapper / identifier
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** pattern MVP + NMI CLI bridge + browse `m` (2026-07-15)
**Modules:** `redmine_ticket_entities.py`, `redmine_ticket_nmi.py`
**NMI env docs:** [[nmi-bridge]]
## Commands
```bash
./bin/rtl entities show --issue-id 41394
./bin/rtl entities scan --where 'subject.includes("1Z")' --max-tickets 20
./bin/rtl entities nmi --issue-id 41394
```
## Done
- Pattern types: UPS/USPS, FedEx candidates, Amazon order, order_id, item_or_sku, **item_no** (LDR 9-digit)
- Markdown view section: **Detected entities**
- NMI bridge + browse **`m`** → private note draft
## Open
- [ ] Optional membership index of recent NMI order/item ids
- [ ] Postal addresses / phones / social (much later)
docs/raw-notes/features/local-search-and-cli-client.md
+39
View File
@@ -0,0 +1,39 @@
# Feature detail: Local search + CLI Redmine client
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** MVP+ (2026-07-14 … 2026-07-15) — ops-usable
**Cookbook:** `docs/cli_power_user_client.md`
## Modules
| Module | Role |
|--------|------|
| `redmine_ticket_query.py` | Property AST (`--where`) |
| `redmine_ticket_view.py` | Markdown + pager |
| `redmine_ticket_browse.py` | fzf → pager → action menu loop |
| `redmine_ticket_filters.py` | Saved filters + symlink queues |
| `redmine_ticket_drafts.py` | Draft queue + host safety |
| `redmine_ticket_meta.py` | Name→id for set-fields |
| `redmine_ticket_ocr.py` | Attachment text extract + subject suggest |
## Done
- [x] Local search / `@saved` filters / queues
- [x] view / walk (`--open-only` on search/browse/walk)
- [x] note vs customer-email vs set-fields (draft/apply)
- [x] `--body-file` + `--to-textile` (pandoc)
- [x] browse **loop** with action menu (queue-only mutations + NMI `m`)
- [x] configs: `bin/rtl`, `bin/rtd`, `--config`
- [x] `ocr-rename` + `@scanned-document`
- [x] `categories list`
## Open
- [ ] Nested config defaults
- [ ] Full curses TUI (optional later)
- [ ] categories create/update
## Non-goals
- Replacing Redmine web UI for everything
- Silent customer email from browse
docs/raw-notes/features/nmi-bridge.md
+41
View File
@@ -0,0 +1,41 @@
# NMI bridge env + usage
**Parent:** [[entity-type-mapper]] · module `redmine_ticket_nmi.py`
## Environment
| Variable | Default | Purpose |
|----------|---------|---------|
| `NMI_CLI` | `/home/jason/projects/mrscClient-php/bin/nmi` | Path to NMI PHP CLI |
| `NMI_ENV_FILE` | `/home/jason/projects/mrscClient-php/examples/modern/.env` | Env file for credentials/endpoint |
| `NMI_MODE` | `dev` | CLI `--mode` (dev/prod as supported by nmi) |
Requires `php` on PATH.
## CLI
```bash
# Pattern extract only
./bin/rtl entities show --issue-id 41394
./bin/rtl entities scan --type tracking_ups --max-tickets 20
# Resolve / cs-brief via NMI CLI
./bin/rtl entities nmi --issue-id 41394
./bin/rtl entities nmi --issue-id 41394 --nmi-mode dev
./bin/rtl entities nmi --issue-id 41394 --as-private-note-draft # if wired; else use browse `m`
```
## Browse
After opening a ticket in `browse`, press **`m`**:
1. Runs NMI brief from extracted entities
2. Shows brief text
3. Confirms → queues **private** internal note draft
Never emails the customer.
## Safety
- NMI is read-side ops context, not customer mail
- Private note drafts still need `draft apply` (+ prod flags) to land in Redmine
docs/raw-notes/features/ocr-rename.md
+46
View File
@@ -0,0 +1,46 @@
# Feature detail: OCR attachment → subject rename
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** MVP 2026-07-15
**Module:** `redmine_ticket_ocr.py` · CLI: `ocr-rename`
## Intent
Scanned docs land in Helpdesk as tickets titled “Scanned document…”.
Ops need: open attachment → understand contents → rename subject (+ optional notes).
## Commands
```bash
# One ticket: download OCR-able attachments, propose subject, queue private note (+ subject draft)
./bin/rtl ocr-rename --issue-id 12345
# Scan cache for generic scan subjects
./bin/rtl ocr-rename --where '@scanned-document' --only-generic-subjects --max-tickets 20
# Apply subject now (LAN/prod safety gates); still queues private OCR note by default
./bin/rtd ocr-rename --issue-id 12345 --apply-subject
# Extract only (no drafts)
./bin/rtl ocr-rename --issue-id 12345 --no-queue-subject --no-queue-note --json
```
## Pipeline
1. Load ticket + attachments (`include=attachments` refresh if needed)
2. Download PDF/image candidates (multi-URL fallback)
3. Extract text: `pdftotext` → else `pdftoppm` + `tesseract`
4. Heuristic `suggest_subject_from_text` (doc type, person/company, date)
5. Queue drafts:
- `fields` subject change (unless `--no-queue-subject`)
- private `internal_note` with OCR snippet (unless `--no-queue-note`)
## Saved filter
`saved-filters/scanned-document.expr` → `@scanned-document`
## Still open
- [ ] Stronger LLM subject summarization (optional; keep heuristics default)
- [ ] Cron wrapper that only processes new generic-scan tickets
- [ ] HR-project default project config example
docs/raw-notes/features/p5-exploration.md
+3
View File
@@ -0,0 +1,3 @@
- [ ] Download voicemails and call recordings from voip.ms automatically
- [ ] helper PHP client library https://git.int.ldrprep.com/dusk/php-voip.ms
docs/raw-notes/features/sync-foundation.md
+91
View File
@@ -0,0 +1,91 @@
# Feature detail: Ticket sync foundation
**Parent roadmap:** [[../features-for-redmine_tickets-sync]]
**Status:** largely implemented (2026-07-14); cron wiring still open
**Primary script:** `redmine_tickets.py`
**Target project (first):** `customer-service`
## Intent
Durable, incremental local mirror of Redmine issues so humans and agents can work offline-first, cron-refresh frequently, and not re-pull full payloads for unchanged tickets.
## Requirements
- Sync tickets updated within a window (e.g. past 30 days) into a **stable** cache path (not timestamped per run).
- Skip full fetch when cached `updated_on` still matches the list stub.
- Support frequent refresh (every few minutes / couple times per hour) using a tight remote window after the initial warm-up.
- Never delete old ticket JSON unless explicitly asked.
- Optional small run-stats ring buffer for cron observability.
## Cache locations
| Environment | Path |
|-------------|------|
| Production (`https://rm.ldrprep.com`) | `/home/jason/projects/redmine/cache/redmine_tickets` |
| Dev / LAN (`http://192.168.50.170`) | `/home/jason/projects/redmine/cache-dev/redmine_tickets` |
Layout:
```text
<cache-dir>/
manifest.json
run-stats.json # optional, with --run-log
<project>/
<issue-id>.json
```
## Server-side vs local filters (sync)
| Flag | Side | Meaning |
|------|------|---------|
| `--status-id` / `--remote-status-id` | **remote** | Redmine API `status_id` (`*`, `open`, `closed`, numeric) |
| `--remote-updated-on` | **remote** | Raw Redmine syntax: `>=2026-06-14`, `>=2014-01-02T08:12:32Z`, `><a\|b` |
| `--remote-updated-since` | **remote** | Convenience → `>=...` (YYYY-MM-DD, ISO, `30d`, `2 days ago`, …) |
| `--remote-updated-before` | **remote** | Convenience → `<=...`; with since → `><since\|before` |
| `--updated-since` / `--updated-before` / `--status` / `--category` | **local** | Post-filters after API list only |
Implementation notes:
- List stubs intentionally omit heavy `include=journals,helpdesk`; full include is only on per-ticket fetch when `should_fetch` is true.
- Progress on by default; `--quiet` for cron.
- Unit tests: `tests/test_redmine_tickets_remote_filters.py`
- Run log: `--run-log` → default `<cache-dir>/run-stats.json`, keep last N (`--run-log-keep`, default 3).
## Recommended commands
```bash
cd /home/jason/projects/redmine
# Initial 30-day window (server-side)
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
sync \
--remote-status-id '*' \
--remote-updated-since 30d \
--run-log
# Frequent refresh
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir /home/jason/projects/redmine/cache/redmine_tickets \
sync \
--remote-status-id '*' \
--remote-updated-since 2h \
--run-log \
--quiet
```
Raw Redmine form:
```bash
./redmine_tickets.py ... sync --remote-updated-on '>=2026-06-14' --status-id '*'
```
## Open
- [x] Scripts under `cron-scripts/` installed + crontab running (live + dev)
- [ ] **ADJUST LATER (do not forget):** cron still uses `--remote-updated-since 30d` on a few-minute cadence. Fine while validating (~3–4s/run after warm cache) but should tighten to `2h`/`6h` (or marker-based since-last-success) so list pages stay cheap long-term. `run-log-keep 20` is already set.
- [ ] Optional: master storage + symlink organization of JSON (see parent roadmap)
docs/raw-notes/ocr-ticket-attachment-to-rewrite-update-ticket.md
+31
View File
@@ -0,0 +1,31 @@
I want a skill (or set of skills) that accomplishes the following:
## Context
We have found that using networked printers like HPs to scan various documents have them emailed to inboxes wired up to the Redmine Helpdesk plugin a very useful way to avoid losing paperwork and speed up finding it later. That has been instrumental on several occassions, where we've had the printer in my office configured to email scanned files to hr@ldrprep.com, where they get imported as tickets with the scanned document as a file attachment.
Even with that, to make it truly useful, somebody still has to typically do this:
1. Log into redmine and go to the hr project's issues.
2. Identify scanned document emails (usually the subject like says something like "Scanned document...")
3. Open the attachment to see what it is.
4. Edit the ticket and change the subject line so if we search later on, we'll most likely be able to find the ticket with the attached file.
5. Optionally, transcribe key points or info from the document and add them to the ticket.
## Goal: automate this process
- [ ] **Convert target ticket**: When passed a redmine issue id and asked to peform this action, the following should happen:
- Fetch the ticket and check if it has any PDF files attached to it.
- Download the PDFs and use OCR or similar technology to extract the main topics and entities mentioned within the document(s).
- Update the ticket by changing it's subject line to reflect what the attached documents are. If they're mainly about one particular person or company, the subject line should include those.
- [ ] **Scan project**: When asked, search/scan a specified Redmine project to locate tickets where this hasn't been done yet, and then do it for all of the tickets found.
- [ ] **Cron-usable skill**: Once this is done, I want to have it as a skill I could easily add to an agentic cronjob, what will enable it to scan/check a specified list of Redmine projects whenever it runs, and then automatically perform this action.
## Testing parameters
- The test instance of redmine located at `192.168.50.170` can be used to test functionality. It supports the same authentication info as the live redmine.
- The email aspect of this is not important at all. It would be fine to simulate it as follows:
- Create a new ticket simulating what an ideal target would look like
- Upload a pdf to it
- Then, run the relevant workflow against the test ticket to check results
docs/rebot_legacy_spam_rules.md
+154
View File
@@ -0,0 +1,154 @@
# Rebot Legacy Known-Spam Rules
**Source inspected:**
- `/home/jason/projects/rebot/rebot/RebotSlick.php`
- `/home/jason/projects/rebot/rebot/RebotSlick.php.save`
- `/home/jason/projects/rebot/rebot/RebotSlick.php.save.1`
- `/home/jason/projects/rebot/rebot/Classify/Receipts.php`
## Summary
The old Rebot spam cleanup used `RebotSlick::CleanSpam()` which iterated open customer-service issues and called `KnownSpam($issue)`. When `KnownSpam()` returned true, Rebot updated the issue to Closed and set category `Spam`.
Current source behavior:
```php
if ( $this->KnownSpam($issue)) {
$this->client->issue->update($issue['id'], [
"status_id" => $this->statusByName['Closed'],
"category_id" => $this->categoriesByName['Spam']
]);
}
```
The saved older copies also used `KnownSpam()` and closed tickets, sometimes with note `Rebot identified as spam`.
## Exact KnownSpam source logic
### Direct sender equality
```php
if ($from_address == 'noreply@rush-delivery.com') return true;
```
### Amazon package special case
```php
if ($from_address == 'no-reply@amazon.com') {
if (strpos($subject, "Your Amazon package") !== false) return true;
}
```
### Current `badEmails` regex list
```php
"/^.*@indeedemail.com/i"
"/^no-reply@indeed.com/i"
"/^.*@.*roku.com/i"
"/^no.reply@masterclass.com/i"
"/^donotreply@walmart.com/i"
"/^.*@uber.com/i"
"/^no-reply@inhome.walmart.com/i"
"/^.*wingsandrings.com/i"
"/mailnull@amazon.com/i"
"/^.*@tryfinally.co/i"
"/^.*crackerbarrel.com/i"
"/^.*shop.tiktok.com/i"
"/^noreply_samsung@sea.samsung.com/i"
"/^noreply@email.crunchbase.com/i"
"/^reply@globalindustrial.com/i"
"/^.*webstaurantstore.com/i"
"/^info@emailbiz.dukeenergy.com/i"
"/^support@helium10.com/i"
"/^noreply@popmenu.com/i"
"/^info@go.cbts.com/i"
```
### Older saved `badEmails` list
The `.save` and `.save.1` variants had the earlier subset:
```php
"/^.*@indeedemail.com/i"
"/^no-reply@indeed.com/i"
"/^.*@.*roku.com/i"
"/^no.reply@masterclass.com/i"
"/^donotreply@walmart.com/i"
"/^.*@uber.com/i"
"/^no-reply@inhome.walmart.com/i"
"/^.*@wingsandrings.com/i"
"/mailnull@amazon.com/i"
"/^.*@tryfinally.co/i"
"/^.*crackerbarrel.com/i"
"/^.*shop.tiktok.com/i"
```
The current source adds Samsung, Crunchbase, Global Industrial, Webstaurant Store, Duke Energy newsletters, Helium10, Popmenu, and CBTS.
### `badSubjects` regex list
```php
"/Your Amazon.com order.*has shipped/i"
"/^Your Duke energy statement is ready/i"
"/^Your Amazon.com order.*/i"
"/^UPS Billing: Payment Received.*/i"
"/^UPS Billing:.*new invoice.*/i"
"/^.applied on indeed.*/i"
"/^See results for the assessments you sent to*/i"
"/^Reminder: Leave feedback on your interview with.*/i"
"/^Pekin AutoPay Payment.*/i"
"/^Notification of Active Listings Report availability/i"
```
## Important interpretation for modern tooling
The old Rebot called all of the above `spam`, but several of these are better understood today as **operational clutter** rather than promotional spam:
- Amazon order/shipping/package notifications
- UPS billing/payment/invoice notices
- Duke Energy statements/newsletters
- Pekin AutoPay payments
- Indeed applicant/assessment/interview mail
- notification/report availability emails
For the new fast path, preserve the rules but do **not** treat every legacy rule as `spam-promotional`. Recommended split:
1. `legacy-rebot-known-spam-promotional`
- promotional/vendor/newsletter/sales senders
- safe for fast close if contact is not `Customer` / `maybe-customer`
2. `legacy-rebot-operational-clutter`
- Amazon order/shipping, UPS billing, Duke/Pekin billing, Indeed workflow mail
- can be fast-closed only if Jason explicitly approves closing operational clutter as a separate lane
3. `legacy-rebot-receipts-routing`
- from `Classify/Receipts.php`; these were routed to Receipts / Accounting / IT, not generic spam close
## Receipts classifier reference
`rebot/Classify/Receipts.php` used `GenericMail` to classify:
### Email regexes
```php
/culliganbillingoh@culliganwaterco.com/i
/accounting@amshippingsupplies.com/i
/bookkeeperasl@gmail.com/i
/bounces-noreply@mandrillapp.com/i
```
### Subject regexes
```php
/You have a new bill from altafiber/i
```
### Actions
- Default: status `Resolved`, category `Receipts`
- AMShipping invoices: category `Bill Needs Paid`, assigned Accounting
- JoeLevine/bookkeeper: category `Bill Needs Paid`, assigned Accounting
- Mandrill bounces: status `Resolved`, category `IT Issue`, assigned Technical Services
These should not automatically enter the promotional spam fast-close registry, but they are useful future operational-clutter/routing profiles.
docs/redmine_contacts_duplicate_merge_feasibility.md
+69
View File
@@ -0,0 +1,69 @@
# Redmine Contacts duplicate-merge feasibility (API vs script)
## Scope
Assess whether duplicate contact merge (web flow: `/contacts/<contact_id>/duplicates?project_id=<project_slug>` with `duplicate_id=<target_id>`) can be safely automated from `redmine_contacts.py` or shell tooling.
## Findings from local plugin source
### 1) Merge flow exists as a controller action, not a JSON API endpoint
- Controller: `plugins/redmine_contacts/app/controllers/contacts_duplicates_controller.rb`
- Routes: `plugins/redmine_contacts/config/routes.rb`
- `match 'contacts/:contact_id/duplicates' => 'contacts_duplicates#index', :via => [:get, :post]`
- `match 'contacts_duplicates/:action' => 'contacts_duplicates', :via => [:get, :post]`
`ContactsDuplicatesController` does **not** declare `accept_api_auth` and does **not** implement `format.api` responses.
### 2) Merge behavior is server-side and destructive
Inside `merge` action:
- combines notes, deals, related_deals, issues, projects from source contact into duplicate target,
- unions email/phone/tag sets,
- saves target,
- destroys source contact.
This matches the intended business behavior (source contact removed, linked records moved).
### 3) Web form posts to merge action with `duplicate_id`
- View: `plugins/redmine_contacts/app/views/contacts_duplicates/index.html.erb`
- Uses Rails `form_tag` to `contacts_duplicates#merge` with `project_id`, `contact_id`, and posted `duplicate_id`.
## Practical feasibility
## A) Native Redmine REST API with API key
**Not currently feasible** for this merge operation.
Reason: no API auth route/serializer for duplicate merge in this plugin path.
## B) Scripted browser-equivalent flow (`curl`/requests with session cookies)
**Feasible with caution**.
Required mechanics:
1. Authenticate as a normal Redmine user (session/cookie auth).
2. GET duplicate page for the source contact to obtain valid session + CSRF authenticity token.
3. POST to merge action with:
- `project_id`
- `contact_id` (source)
- `duplicate_id` (target)
- authenticity token
4. Verify redirect/result contact and audit that source contact no longer exists.
## Risk notes
- This is a destructive operation (deletes source contact).
- Needs stronger guardrails than normal update operations:
- explicit `--apply` style confirmation,
- dry-run mode that only validates both contacts and project scope,
- optional `--test-instance-only` safety default,
- write operation log with source/target IDs and timestamp.
## Recommendation for `redmine_contacts.py`
1. **Do not** add duplicate merge via API-key-only path unless plugin gains a dedicated JSON API action.
2. If we implement now, add a **separate command** using session-auth web flow (not mixed into normal contact update path), e.g.:
- `merge-duplicate --source-contact <id> --target-contact <id> --project <slug> [--apply]`
3. Keep it disabled-by-default for production base URL, with explicit override flag.
## Suggested next technical step
Prototype a standalone script against `http://192.168.50.170` that:
- logs in,
- resolves CSRF token,
- performs merge POST,
- verifies resulting contact IDs/issue linkage,
- emits structured JSON result for later CLI integration.
docs/redmine_contacts_search.md
+145
View File
@@ -0,0 +1,145 @@
# Contact search (`redmine_contacts.py`)
`./redmine_contacts.py search --help` prints three short examples. This page is the situation list. Gaps are in [redmine_contacts_search_proposals.md](redmine_contacts_search_proposals.md).
This CLI was not changed to write this page. Commands below match the current `search` / `show` / `helpdesk-issues` behavior.
## What `search` actually does
1. Load the contact cache, or reload it with `--refresh`.
2. Apply exact filters: `--with-tag`, `--without-tag`, `--no-tags`, `--created-after`, `--created-before`, `--updated-after`, `--updated-before`.
3. Fuzzy-score the remaining contacts against the required positional query.
4. Print text lines, capped by `--limit` (default 20).
It does not search tickets. It does not print JSON. A normal search still requires `REDMINE_API_KEY` even when it only reads the cache. `--offline-cache` is the exception.
Default host is the LAN copy, `http://192.168.50.170`. Production:
```bash
cd /home/jason/projects/redmine
./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod"
```
Cache file: `.cache/redmine_contacts/<host>_<project>.json`, relative to the current directory. A cache built for one host is rejected for another. This is not `bin/rtl contacts-search`.
A hit is a substring of a scored field after lowercasing, or a fuzzy score of at least `0.68`. Scored fields: name, email, phone, company, job title, background, website, skype, tags, address, and custom-field values. A phone query is also matched on digits only, so `847-313-1342` hits `8473131342`.
Output shape: `id: name | company | email | phone | score=1.00 via email`. Use `show <id>` for the record.
`--refresh` pages through every project contact before scoring. Without it, a tag or field written since the last fetch is invisible.
## Supported
### 1. Find a person by name, email, or phone
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com search "Sahar Daod"
./redmine_contacts.py --base-url https://rm.ldrprep.com search sahardaod4@groovinfulfillment.com
./redmine_contacts.py --base-url https://rm.ldrprep.com search 847-313-1342
```
Then the record, including `tag_list`:
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com show 4589
```
`show` prints JSON. `--source api` skips a stale cache. `--source cache` does not call Redmine.
### 2. Find people at a company, or everyone on one email domain
Company and email are scored fields, not dedicated filters. A domain that appears in the address scores `1.00`.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com search gatsbylady.com --refresh
./redmine_contacts.py --base-url https://rm.ldrprep.com search "Cyberstorm" --min-score 0.9
```
Raise `--min-score` when a short company name also fuzzy-matches unrelated contacts. There is no `--company` or `--email-domain` flag. An exact company filter is proposal 1.
### 3. Exact tag
`--with-tag` is exact and case-insensitive. The positional query is still required and still fuzzy, so repeat the tag as the query. Otherwise a contact with the tag can be filtered in and then dropped because the name did not fuzzy-match.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search promotional-spam --with-tag promotional-spam --refresh --limit 50
```
Offline, against the cache already on disk for that host:
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search promotional-spam --with-tag promotional-spam --offline-cache
```
Repeat `--with-tag` to require every tag. Repeat `--without-tag` to exclude any of those tags.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search customer --with-tag customer --without-tag promotional-spam --limit 20
```
`--no-tags` cannot be combined with `--with-tag`. Listing every untagged contact is not this command. See proposal 2.
### 4. A tag, narrowed by when the contact was created or updated
Date flags are exact. The query is still required, so this finds "this text, inside this window," not "everyone touched this week."
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search customer --with-tag customer --updated-after 2026-09-01 --refresh
```
Dates are `YYYY-MM-DD` or a full ISO datetime. A date-only `--updated-before` includes that whole UTC day. Listing by date with no text query is proposal 2.
### 5. A known account number or other custom-field value
Custom-field values are scored if they are already on the cached contact. Account Number is custom field 70.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com search 7385Y6 --refresh
```
This finds a value you already know. It cannot list customers whose Account Number is empty. That is proposal 3.
### 6. Their tickets, once you have the contact id
`search` stops at the contact. These follow-ups are live reads, not the contact cache.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com helpdesk-issues 4589 --limit 20
./redmine_contacts.py --base-url https://rm.ldrprep.com helpdesk-timeline 4589 --limit 20
```
Both print JSON from `/helpdesk_search/contacts/:id/...`.
Cached Customer Service tickets, any status:
```bash
bin/rtl search --order newest-updated-first --max-tickets 20 \
--where 'helpdesk_ticket.contact_id == 4589 AND project.name == "Customer Service"'
```
Open New only:
```bash
bin/rtl search --status New --open-only \
--where 'helpdesk_ticket.contact_id == 4589'
```
`bin/rtl search` cannot filter on `tag_list`. The ticket's contact stub is `id`, `name`, `email`, `company`. This returns nothing and does not error:
```bash
bin/rtl search --where 'helpdesk_ticket.contact.tag_list.includes("promotional-spam")'
```
One command that means "New tickets from contacts tagged promotional-spam" is proposal 4. `bin/rtl contacts-search --tag` is a different cache and can lag a live tag.
## Not a search
`fetch --with-tag customer` writes a cache and prints a count. It does not list contacts.
`batch-update` can select by the same tag and date filters, but it requires an update payload and exists to write. Do not use it as a lister.
`loop-run` is a triage worker, not a search.
docs/redmine_contacts_search_proposals.md
+99
View File
@@ -0,0 +1,99 @@
# Contact search proposals
Situations `redmine_contacts.py search` does not cover, as of the current CLI. Do not treat these commands as existing. The working commands are in [redmine_contacts_search.md](redmine_contacts_search.md).
None of these require changing ticket search. They belong on the contacts CLI, with a dry-run default and no write path.
## 1. Exact company or email domain
**Need:** "everyone at gatsbylady.com" or "company is UPS" without fuzzy misses and without a short string scoring unrelated names.
**Today:** company and email are haystack fields. `search gatsbylady.com` works when that string is inside an email. `search UPS` also fuzzy-matches other fields above `0.68`. There is no `--company` or `--domain`.
**Proposal:**
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search --domain gatsbylady.com --refresh
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search --company UPS --company-exact --refresh
```
`--domain` matches the part after `@`, case-insensitive, across `emails[]`. `--company-exact` matches `company` after trim and case-fold, not a substring of the whole record. If either flag is set, the positional query becomes optional. Print the same text line `search` prints now, plus `--json` from proposal 5.
## 2. List by filter, with no fuzzy query
**Need:** every contact with no tags, or every contact updated since a date, or every `customer` tag, without inventing a query that happens to score.
**Today:** `search` always requires `query`, then drops anyone under `--min-score`. `--no-tags` and the date flags only narrow that fuzzy set. `fetch --no-tags` caches matches and prints a count, not rows.
**Proposal:** a `list` subcommand that reuses `filter_contacts` and does not call `search_contacts`.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
list --no-tags --updated-after 2026-09-01 --refresh --limit 50
./redmine_contacts.py --base-url https://rm.ldrprep.com \
list --with-tag promotional-spam --refresh
```
Require at least one filter, same guard as `batch-update`, so `list` cannot dump the whole CRM by accident. Default `--limit 50`. Order by `updated_on` descending. Text lines should include id, name, email, and `tag_list`, because a bare fuzzy score is meaningless here.
Do not overload `search` by making `query` optional. Operators already have to know that `search` is fuzzy.
## 3. Tagged customers with an empty Account Number
**Need:** billed or tagged customers who have no NMI account id in custom field 70 (`Account Number`), so the jumpstart can see who is unmatched.
**Today:** a known account number can be fuzzy-searched if it is already stored. There is no empty-field filter. `show` is one id at a time.
**Proposal:**
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
list --with-tag customer --missing-custom-field account-number --refresh
```
Resolve `account-number` with the existing custom-field slug (`slugify_custom_field_name`). Missing means the field is absent, null, or whitespace. Do not treat `"0"` as empty. Read-only. This is the list command from proposal 2 plus one predicate, not a new writer.
## 4. Open tickets from contacts with a tag
**Need:** New Customer Service tickets whose sender is tagged `promotional-spam`, without hand-joining ids.
**Today:** ticket JSON does not store `tag_list`. `bin/rtl search --where 'helpdesk_ticket.contact.tag_list.includes(...)'` matches nothing. The working path is tag search, then `helpdesk_ticket.contact_id == N` per id.
**Proposal:** a read-only join, not a ticket-cache field.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
tagged-issues --with-tag promotional-spam --issue-status New --open-only --refresh
```
Steps:
1. Resolve contact ids with the list filter from proposal 2, live or `--refresh`.
2. For each id, call the existing `helpdesk-issues` endpoint, or scan the ticket cache for `helpdesk_ticket.contact_id`.
3. Keep issues that are New, open, and in `customer-service`.
4. Print `issue_id contact_id email subject`, and stop after `--limit` issues (default 20).
Do not close anything. Do not add `tag_list` to the ticket cache in this change. A stale contacts cache must be called out in the header line (`cache_refreshed=yes|no`). Refuse to run if the contact cache host does not match `--base-url`, which the current cache reader already does.
## 5. Script output: ids and JSON
**Need:** feed contact ids into `bin/rtl` or a later join without scraping `score=1.00 via email`.
**Today:** `search` prints prose. `show` and `helpdesk-issues` print JSON for one id. There is no `--ids-only` on this CLI. `bin/rtl contacts-search --ids-only` is the other cache.
**Proposal:** `--json` and `--ids-only` on `search` and on `list`.
```bash
./redmine_contacts.py --base-url https://rm.ldrprep.com \
search promotional-spam --with-tag promotional-spam --refresh --ids-only
```
`--json` prints one object per line: `id`, `name`, `email`, `tag_list`, and `score` when the command was fuzzy. `--ids-only` prints one integer per line. Neither flag changes which rows match.
## Out of scope
Duplicate detection (same email or phone on two contacts) needs a separate scan and a human merge step. Do not fold it into `search`. The existing duplicate-merge note stays the place for that design.
Do not make `batch-update` printable as a search. It requires a mutation payload on purpose.
docs/ticket_decomposition_knowledge_ingestion_plan.md
+788
View File
@@ -0,0 +1,788 @@
# Ticket Decomposition & Knowledge Ingestion Implementation Plan
**Date:** 2026-07-09
**Project repo:** `/home/jason/projects/redmine`
**Primary vault:** `/home/jason/lilaBuild/vaults/obsidian-ldr`
**Primary Redmine project to start:** `customer-service`
**Production Redmine:** `https://rm.ldrprep.com`
**Onyx vault URL prefix:** `https://onyx.int.ldrprep.com/projects/obsidian-ldr/`
## Goal
Build a ticket decomposition process that converts meaningful Redmine tickets into durable organizational memory: timeline event notes, entity/contact notes, Redmine contact enrichment, cross-links, tags, and gbrain-ingestable vault content.
The immediate wedge is Redmine tickets, especially `customer-service`, but the final interface should support:
```text
decompose <target>
```
where `<target>` may be:
- Redmine issue ID / Redmine project query
- email text
- PDF/document file
- pasted text
- cached ticket JSON
## Scope
This plan is about **knowledge ingestion and decomposition**, not spam closure. It should build on the ticket sync substrate, not replace it.
Existing substrate to reuse:
- `/home/jason/projects/redmine/redmine_tickets.py`
- `/home/jason/projects/redmine/redmine_contacts.py`
- Redmine issue API with `include=journals,helpdesk`
- Obsidian LDR vault
- Onyx browser links
- gbrain source/index tooling
---
## Ticket Buckets Worth Decomposing
A ticket should be eligible for decomposition if it appears to fit one or more of these buckets:
### 1. Definite customer ticket with meaningful event
Customer ticket where:
- a decision was made
- a complaint was involved
- a new project started
- a workflow/process implication was exposed
- an account/service/pricing change happened
Examples:
- customer complaint about missing photos, missing package, wrong condition, wrong item
- pricing/service inquiry that caused a meaningful answer or decision
- customer onboarding/project kickoff
- customer-specific exception or policy decision
### 2. Durable purchase receipt
Receipt about a durable item or business asset worth remembering.
Examples:
- equipment
- tools
- technology/hardware
- warehouse/office asset
- anything likely to matter for warranty, accounting, maintenance, or future context
### 3. Regular utility / recurring service receipt or invoice
Receipt/invoice/payment confirmation about a regular service.
Examples:
- utilities
- insurance
- phone/VoIP
- shipping/billing services
- cloud/SaaS subscriptions
- recurring vendors
This may not need rich event notes every time, but should enrich vendor/entity records and optionally create monthly financial/operations timeline entries.
### 4. Partnership / inbound opportunity / buy-from-us request
Not a customer and not a sales person trying to sell us something, but someone proposing a relationship or wanting to buy something **from us**.
Examples:
- buy Gaylords
- buy liquidation
- advertise/affiliate/blog partnership
- ask about partnership or marketplace cooperation
- use LDR as a source/supplier
These should become opportunity/entity notes, not spam.
### 5. Any meaningful operational event
Catch-all for tickets that refer to a meaningful event:
- policy change
- operational incident
- vendor problem
- staff/customer decision
- process drift or known-failure-method recurrence
- new obligation or future follow-up
---
## Non-Goals / Safety Boundaries
1. Do not ingest every Redmine ticket into the vault as a polished note. Use cache/archive for raw data; decompose only meaningful tickets.
2. Do not overwrite existing entity pages destructively.
3. Do not invent facts. Use evidence blocks and confidence levels.
4. Do not send customer-visible Helpdesk replies.
5. Do not update Redmine contacts until the proposed changes are explicit and reviewable.
6. Do not let spam-cleanup closures trigger decomposition unless the ticket also has meaningful non-spam content.
---
## Vault Information Architecture
### Timeline root
Create a new timeline tree:
```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Timeline/
```
Monthly folders:
```text
Timeline/2026-07/
Timeline/2026-08/
```
Monthly index:
```text
Timeline/2026-07/index.md
```
Event notes:
```text
Timeline/2026-07/2026-07-09_customer-name-short-subject.md
```
Filename rules:
- kebab-case subject slug
- no spaces
- prefix `YYYY-MM-DD_`
- keep short but human-readable
Example:
```text
Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry.md
Timeline/2026-07/2026-07-09_u2wear-returns-provider-followup.md
Timeline/2026-07/2026-07-09_cyberstorm-order-return-confirmation.md
```
### Monthly index format
`Timeline/YYYY-MM/index.md` should contain:
```markdown
# YYYY-MM Timeline Index
## Key events
- [[2026-07-09_repackify-gaylord-inquiry]] — Partnership / buy-from-us inquiry. Redmine #41424.
- [[2026-07-09_u2wear-returns-provider-followup]] — Existing/prospective customer follow-up. Redmine #41448.
## Decisions
- ...
## Complaints / service failures
- ...
## Purchases / receipts
- ...
## Partnerships / opportunities
- ...
```
### Entity notes
Use the existing vault `entities/` tree as the default entity home.
Proposed paths:
```text
entities/people/<person-slug>.md
entities/organizations/<organization-slug>.md
entities/concepts/<concept-slug>.md
entities/places/<place-slug>.md
```
If the repo prefers flat `entities/<slug>.md`, the first implementation can create flat notes and later migrate. But the decomposition code should centralize path decisions in one function, e.g. `entity_note_path(entity)`.
### Entity note frontmatter
Minimum frontmatter for Redmine contacts:
```yaml
---
type: organization
name: RePackify
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
tags:
- redmine-contact
- maybe-customer
- partnership-opportunity
aliases:
- RePackify
---
```
For a person:
```yaml
---
type: person
name: Sarah Johnson
organization: [[repackify]]
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
onyx_url: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/people/sarah-johnson.md
tags:
- redmine-contact
- contact-person
---
```
### Entity body format
```markdown
# Entity Name
## Summary
Brief factual summary with confidence level.
## Redmine links
- Contact: https://rm.ldrprep.com/contacts/4470
- Related tickets:
- https://rm.ldrprep.com/issues/41424
## Timeline
- [[Timeline/2026-07/2026-07-09_repackify-gaylord-inquiry]]
## Known facts
- Fact with source link.
## Open questions
- Question that needs human/agent follow-up.
```
---
## Event Note Format
Each event note should be evidence-first and dialectic.
```markdown
---
type: timeline-event
date: 2026-07-09
source_type: redmine-ticket
redmine_issue_id: 41424
redmine_url: https://rm.ldrprep.com/issues/41424
helpdesk_contact_id: 4470
entities:
- [[repackify]]
tags:
- timeline
- redmine
- partnership-opportunity
- buy-from-us
confidence: medium
---
# 2026-07-09 — RePackify Gaylord Inquiry
## One-line summary
RePackify asked whether LDR has used Gaylord boxes available to purchase.
## Source evidence
- Redmine: https://rm.ldrprep.com/issues/41424
- From: support@repackify.com
- Subject: Gaylord inquiry-RePackify
- Key excerpt:
> We’re interested in purchasing used Gaylord boxes and wanted to see if you have any available.
## Classification
- Bucket: partnership / buy-from-us request
- Customer status: not confirmed customer
- Suggested contact tag: maybe-customer or partnership-opportunity
## Dialectic analysis
### Claim
This is not spam; it is a potential inbound business/opportunity ticket.
### Supporting evidence
- The sender wants to buy something from LDR, not sell LDR a generic service.
- The request references a concrete asset class: used Gaylord boxes.
### Counterpoints / uncertainty
- It may be a one-off opportunistic buyer, not a durable customer relationship.
- Needs human judgment on whether LDR wants to sell Gaylords.
### Decision / outcome
- No final business decision captured yet.
- Needs follow-up if LDR wants a liquidation/Gaylord sale process.
## Entities to update
- [[repackify]]
## Redmine contact enrichment proposal
```json
{
"contact_id": 4470,
"add_tags": ["maybe-customer", "partnership-opportunity", "buy-from-us"],
"obsidian_article_url": "https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md"
}
```
## Follow-up candidates
- Decide whether LDR sells used Gaylords.
- If yes, define a standard response/process.
```
---
## Dialectic Reasoning Requirements
Each decomposition should include:
1. **Claim** — what seems to be true or meaningful.
2. **Evidence** — exact ticket excerpts, sender, date, Redmine link.
3. **Counterpoints** — why the interpretation might be wrong.
4. **Decision/outcome** — what was decided or what remains undecided.
5. **Recommended graph updates** — timeline note, entity notes, contact updates, tags.
6. **Confidence** — low/medium/high.
This prevents the vault from becoming a pile of confident hallucinated summaries. A small mercy in a cruel world.
---
## Redmine Contact Cross-Linking
### Vault → Redmine
Entity frontmatter should include:
```yaml
redmine_contact_id: 4470
redmine_contact_url: https://rm.ldrprep.com/contacts/4470
```
### Redmine → Vault
The Redmine contact should be updated with an Onyx URL, not an `obsidian://` link.
Preferred URL format:
```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/<vault-relative-path>.md
```
Example:
```text
https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```
Implementation detail:
- If a Redmine contact custom field exists for `Obsidian Article`, use `redmine_contacts.py --set-custom-field`.
- If no suitable custom field exists, append a clearly delimited line to contact background/notes, e.g.:
```text
Obsidian article: https://onyx.int.ldrprep.com/projects/obsidian-ldr/entities/organizations/repackify.md
```
Do not guess custom field IDs. Discover available custom fields from `get_contact()` payloads first.
---
## `decompose <target>` Architecture
### KarpathyWiki plugin reference
Jason identified the Obsidian `karpathywiki` plugin as a close match for the desired behavior. Inspection notes are saved at:
```text
/home/jason/projects/redmine/docs/karpathywiki_plugin_inspection.md
```
Finding: the plugin does not expose a true standalone CLI. It registers Obsidian commands and its `WikiEngine` depends on Obsidian `App/Vault/TFile/metadataCache` APIs. However, its logic is highly relevant and portable as a design reference:
- source analysis prompt / JSON schema
- entity vs concept decision tree
- alias handling
- duplicate/semantic entity resolution
- entity/concept/source page generation prompts
- contradiction and related-page fields
- graph-based query/retrieval ideas
Recommended implementation path for Redmine decomposition: build a Python headless mirror integrated with `redmine_tickets.py`, using KarpathyWiki's prompts/schema as reference. Consider a TypeScript adapter later only if maintaining a Python mirror becomes too costly.
### Compatibility contract with the Obsidian KarpathyWiki plugin
The LDR vault already contains a plugin-managed wiki tree:
```text
wiki/entities/
wiki/concepts/
wiki/sources/
wiki/index.md
wiki/log.md
```
The headless Redmine decomposer must be a **companion**, not a competing owner of that tree.
Default rules:
1. Do **not** write into `wiki/` by default.
2. Do **not** edit `wiki/index.md` or `wiki/log.md`; those are plugin-owned.
3. Do **not** overwrite or reformat plugin-generated `wiki/entities/*`, `wiki/concepts/*`, or `wiki/sources/*` pages.
4. Write Redmine-derived timeline/event artifacts into LDR-owned paths:
- `Timeline/YYYY-MM/...`
- `entities/organizations/...`
- `entities/people/...`
- `Processes/Redmine/...`
5. Use Obsidian-compatible wikilinks so plugin-generated pages and headless-generated pages can link to each other.
6. If linking to plugin pages, use explicit paths such as `[[wiki/concepts/returns-management|returns management]]` rather than ambiguous bare names.
7. Use plugin-compatible frontmatter concepts where practical (`type`, `created`, `updated`, `tags`, `sources`, `confidence`, `contradictions`) but allow LDR-specific types such as `timeline-event` outside `wiki/`.
8. If we ever want the plugin itself to ingest Redmine-derived material, create a deliberate handoff/export file outside `wiki/` first and let the plugin ingest it through Obsidian; do not silently write into plugin-owned `sources/`.
This keeps Obsidian’s KarpathyWiki plugin useful in the desktop vault while allowing Redmine decomposition to run headlessly/cron/agent-side without fighting plugin state.
Concrete upstream files to mirror first:
- `src/wiki/prompts/ingestion.ts` — extraction prompt / JSON schema
- `src/wiki/prompts/generation.ts` — page generation prompts
- `src/wiki/prompts/merge.ts` — merge/update triage prompts
- `src/wiki/source-analyzer.ts` — source analysis normalization/provenance behavior
- `src/wiki/page-factory.ts` — slug/alias/page merge behavior
The first Python implementation should not attempt full parity. It should clone only the useful minimum:
1. analyze source into JSON entities/concepts/key points/contradictions;
2. resolve entity slugs against existing vault files;
3. generate or update entity/event notes;
4. produce a reviewable contact-update proposal;
5. leave graph retrieval to Obsidian/gbrain for now.
### Input abstraction
Create a common normalized input model:
```python
@dataclass
class DecompositionSource:
source_type: str # redmine-ticket, email, pdf, text, file
source_id: str | None
title: str
body: str
date: str | None
sender: str | None
redmine_issue_id: int | None
redmine_contact_id: int | None
attachments: list[Path]
metadata: dict[str, Any]
```
Adapters:
- `from_redmine_issue(issue_json)`
- `from_email_text(raw_text)`
- `from_pdf(path)`
- `from_plain_text(text)`
- `from_file(path)`
### Decomposition output model
```python
@dataclass
class DecompositionPlan:
source: DecompositionSource
buckets: list[str]
timeline_events: list[TimelineEvent]
entities: list[EntityUpdate]
redmine_contact_updates: list[ContactUpdate]
tags: list[str]
gbrain_actions: list[GBrainAction]
confidence: str
review_required: bool
```
### Apply phases
1. `decompose --dry-run` — produce JSON/Markdown plan only.
2. `decompose --write-vault` — write timeline/entity notes.
3. `decompose --update-redmine-contact` — update Redmine contact with Onyx article link/tags.
4. `decompose --gbrain-sync` — run/queue gbrain sync/index.
Each phase should be separately idempotent.
---
## CLI Proposal
### Decompose a Redmine ticket
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
decompose \
--issue-id 41424 \
--dry-run
```
### Decompose from cached tickets matching a bucket
```bash
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_prod_meaningful_20260709 \
--project customer-service \
decompose-cached \
--bucket partnership-opportunity \
--dry-run
```
### Decompose arbitrary text
```bash
./decompose.py text \
--title "Supplier proposal" \
--input /tmp/message.txt \
--dry-run
```
### Write vault notes after review
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
decompose \
--issue-id 41424 \
--write-vault
```
---
## gbrain Integration
Initial approach should be simple and explicit:
1. Write/update Obsidian notes.
2. Run or queue gbrain sync for the LDR vault source.
3. Verify that the new notes are searchable.
Commands need source discovery first:
```bash
gbrain sources list --json
gbrain stats
```
Then likely:
```bash
gbrain sync --source obsidian-ldr --no-embed --json
gbrain embed --stale
gbrain extract --stale
gbrain search "RePackify Gaylord inquiry"
```
Do not assume the exact source ID is `obsidian-ldr` until verified by `gbrain sources list --json`.
---
## Implementation Phases
### Phase 0 — Immediate schema/docs checkpoint
**Goal:** Preserve this architecture and make future implementation straightforward.
Tasks:
1. Save this plan.
2. Link it from Redmine `HANDOFF.md`.
3. Optionally create an Obsidian process note at:
```text
/home/jason/lilaBuild/vaults/obsidian-ldr/Processes/Redmine/ticket-decomposition-and-knowledge-ingestion.md
```
Definition of done:
- Plan exists.
- Handoff links it.
- Next worker can start implementation without transcript context.
### Phase 1 — Manual pilot decomposition for 3 tickets
**Goal:** Prove note structure and cross-linking manually before automating.
Suggested pilot tickets:
- RePackify Gaylord inquiry (`#41424`)
- u2wear returns-provider follow-up (`#41448`)
- Snuggle Puppy / 3PL & FBA Returns (`#41504`)
Tasks:
1. Fetch each ticket with `include=journals,helpdesk`.
2. Write one timeline event note per ticket.
3. Create/update entity notes.
4. Prepare Redmine contact update payloads, dry-run first.
5. Review Onyx links.
6. Optionally run gbrain sync/search smoke test.
Definition of done:
- 3 timeline notes exist.
- Monthly index links them.
- Entity notes exist or are updated.
- Redmine contact update payloads are reviewed.
- No customer-visible Redmine replies are sent.
### Phase 2 — Implement decomposition dry-run engine
**Goal:** Generate decomposition plans from cached Redmine tickets without writing vault/Redmine.
**Status — 2026-07-10:** First deterministic slice implemented in `redmine_tickets.py decompose`. It supports `--issue-id`, `--input-json`, and `--format json|markdown`. It produces dry-run plans for the three pilot fixtures and respects the KarpathyWiki compatibility boundary by planning writes outside `wiki/`. A KarpathyWiki-style structured extraction layer now cleans Helpdesk/forwarded-message noise and emits `analysis.summary`, `analysis.entities`, `analysis.concepts`, `analysis.key_points`, `analysis.contradictions`, `analysis.related_pages`, and `analysis.clean_text`. Tests pass (`23` tests). Current limitation: extraction is still deterministic/rule-backed rather than LLM-backed; next slice should add prompt-driven JSON extraction for broader ticket types before vault-write mode.
Files:
- Modify: `/home/jason/projects/redmine/redmine_tickets.py`
- Add tests: `/home/jason/projects/redmine/tests/test_ticket_decomposition.py` or extend `test_redmine_tickets.py`
Tasks:
1. Add `DecompositionSource` and `DecompositionPlan` models.
2. Add `source_from_redmine_issue(issue)`.
3. Add bucket classifier:
- customer-decision
- customer-complaint
- customer-new-project
- durable-purchase-receipt
- recurring-utility-receipt
- partnership-buy-from-us
- meaningful-event
4. Add dialectic plan generator.
5. Add JSON/Markdown dry-run output.
6. Add KarpathyWiki-inspired source cleanup and LLM JSON extraction for higher-quality summaries/entities.
Definition of done:
- Unit tests cover each bucket.
- Dry-run output includes source evidence, entities, proposed notes, and proposed contact updates.
- Output quality is comparable to the hand-written pilot notes for the first three fixtures.
### Phase 3 — Implement vault writer
**Goal:** Write idempotent timeline/entity notes.
Tasks:
1. Add path builder for `Timeline/YYYY-MM/YYYY-MM-DD_slug.md`.
2. Add monthly index updater.
3. Add entity path resolver.
4. Add frontmatter merge/update helper.
5. Add idempotency marker using source Redmine issue ID in frontmatter.
Definition of done:
- Re-running same decomposition does not duplicate notes or index bullets.
- Existing entity notes are updated, not overwritten.
### Phase 4 — Implement Redmine contact updater
**Goal:** Update Redmine contacts with Onyx article links and useful tags.
Tasks:
1. Discover contact custom fields.
2. Decide field for Obsidian/Onyx article URL.
3. Add dry-run contact update report.
4. Add `--apply-contact-updates` using `rebot` only when approved.
5. Verify contact after update.
Definition of done:
- Contact has vault article URL.
- Vault article has `redmine_contact_id`.
- Writes are impersonated appropriately.
### Phase 5 — gbrain sync/index integration
**Goal:** Make decomposed notes searchable in gbrain.
Tasks:
1. Discover LDR vault gbrain source ID.
2. Add optional command wrapper or instruction to run sync/embed/extract.
3. Verify search for a newly created entity/event.
Definition of done:
- gbrain search finds the decomposed event/entity.
### Phase 6 — Generalize `decompose <target>`
**Goal:** Support non-ticket inputs with the same decomposition outputs.
Tasks:
1. Add plain-text adapter.
2. Add email adapter.
3. Add PDF adapter using existing OCR/document pipeline if needed.
4. Keep output model the same.
Definition of done:
- Same event/entity/timeline/contact note logic works for non-ticket sources.
---
## Recommended First Execution Step
Do **Phase 1 manual pilot** before writing more automation.
Use the 3 known “meaningful but not spam” tickets from the recent spam-cleanup work:
```text
#41424 RePackify Gaylord inquiry
#41448 u2wear returns-provider follow-up
#41504 Snuggle Puppy / 3PL & FBA Returns
```
That gives us immediate signal on:
- note format quality
- entity matching
- Onyx link usefulness
- Redmine contact update shape
- gbrain ingestion friction
Only after that should we automate the decomposition engine. Otherwise we risk building a very elegant machine for generating annoying notes. We have enough of those already.
docs/ticket_spam_cleanup_handoff_2026-07-09.md
+824
View File
@@ -0,0 +1,824 @@
# Ticket-Centric Spam Cleanup Handoff — 2026-07-09
## Purpose
This handoff captures the current state of the new ticket-centric Redmine maintenance substrate and the spam cleanup workflow for the `customer-service` project.
The immediate business goal is to reduce visible spam clutter in `customer-service` without disrupting existing NMI production scripts or doing unsafe bulk Redmine writes.
## Important Safety / Operating Rules
1. **Do not modify NMI scripts right now.** Existing NMI cron scripts are reference material only.
2. **Dry-run first.** Redmine mutations require explicit `--apply`.
3. **Reads do not impersonate.** Ordinary sync/search/dry-run planning should use normal API access.
4. **Writes impersonate `rebot`.** Spam cleanup / ticket maintenance writes should use `X-Redmine-Switch-User: rebot` (Harry Botter), because Redmine activity logs should show automated cleanup under that account, not Jason.
5. **Treat impersonation like sudo.** Use it only where the write is supposed to happen under that agent account.
6. **LAN first.** The validated target so far is the LAN Redmine copy at `http://192.168.50.170`, project `customer-service`.
7. **Confirm status IDs before applying.** Existing evidence suggests `status_id=5` is closed, but verify before first real apply if uncertain.
## Relevant Files
- Tool: `/home/jason/projects/redmine/redmine_tickets.py`
- Tests: `/home/jason/projects/redmine/tests/test_redmine_tickets.py`
- Plan: `/home/jason/projects/redmine/docs/ticket_sync_tool_plan.md`
- General project handoff: `/home/jason/projects/redmine/HANDOFF.md`
- Helpdesk API include docs: `/home/jason/projects/redmine/docs/redmine_issue_api_helpdesk_include.md`
- Existing contact tool: `/home/jason/projects/redmine/redmine_contacts.py`
## Why `redmine_tickets.py` Exists
The earlier `redmine_contacts.py` contact conveyor is useful for contact enrichment and contact-level routing, but it is not the best first tool for clearing visible spam from `customer-service` because the UI pain is ticket-level: the top/open tickets themselves are clutter.
`redmine_tickets.py` is the new ticket-centric substrate. It syncs enriched issue payloads, searches cached tickets, and runs idempotent ticket actions.
## Modern Helpdesk Access
Do **not** recreate RebotSlick `PullHelpdeskInfo` behavior. The old separate helpdesk pull/cache pattern is obsolete because the local Redmine Helpdesk plugin now supports:
```text
GET /issues/:id.json?include=journals,helpdesk
```
This returns normal Redmine issue data plus embedded `helpdesk_ticket` metadata, including contact/email/source fields.
## Current Architecture
### Ticket payload cache
Per-ticket JSON files are stored under a configurable cache directory, e.g.:
```text
.cache/redmine_tickets_lan/customer-service/41788.json
.cache/redmine_tickets_lan/customer-service/41789.json
```
### Ticket/action state
State is currently manifest-backed:
```text
.cache/redmine_tickets_lan/manifest.json
```
But the code now separates payload cache from state storage:
- `TicketCache` handles ticket JSON cache + delegates state operations.
- `ManifestTicketStateStore` is the current state backend.
- The abstraction is intentionally small so SQLite can replace the manifest later for thousands of tickets.
Required state-store methods:
```python
get_ticket_state(issue_id)
upsert_ticket_state(issue_id, fields)
has_action(issue_id, action_name)
record_action(issue_id, action_name, result, metadata=None)
```
## Implemented Commands
### Sync enriched tickets
```bash
cd /home/jason/projects/redmine
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan \
sync \
--status-id open \
--max-tickets 10 \
--order newest-updated-first
```
Behavior:
- fetches issue stubs/pages from Redmine
- fetches full issue payloads with `include=journals,helpdesk`
- writes only missing/updated tickets to cache
- second identical run skips unchanged tickets
Verified output:
```text
Sync complete: seen=10 fetched=10 skipped_cached=0 written=10
Sync complete: seen=10 fetched=0 skipped_cached=10 written=0
```
### Search cached tickets
```bash
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_lan \
--project customer-service \
search \
--max-tickets 10 \
--order newest-updated-first
```
Validated that this matches the UI top-open-ticket pattern and includes spam-looking tickets such as:
```text
#41788 Up to 20% Off Select Fans & Blowers
#41789 Up to 20% Off Select Fans & Blowers
```
### Spam check cached tickets
```bash
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_lan \
--project customer-service \
spam-check
```
Validated result:
```text
#41788 SPAM score=2 from=reply@globalindustrial.com Up to 20% Off Select Fans & Blowers
#41789 SPAM score=2 from=reply@globalindustrial.com Up to 20% Off Select Fans & Blowers
```
Record local idempotency only (no Redmine mutation):
```bash
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_lan \
--project customer-service \
spam-check \
--apply
```
After this, a second `spam-check` checks 0 tickets because `spam-check` action markers are recorded.
### Plan closure for spam tickets
Dry-run:
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10
```
Validated output:
```text
#41788 close_payload={"notes": "Closed as likely spam by redmine_tickets.py close-if-spam.", "status_id": 5} contact_id=621 contact_spam=4/10 likely_spammer=True Up to 20% Off Select Fans & Blowers
#41789 close_payload={"notes": "Closed as likely spam by redmine_tickets.py close-if-spam.", "status_id": 5} contact_id=621 contact_spam=4/10 likely_spammer=True Up to 20% Off Select Fans & Blowers
Close-if-spam complete: planned=2 apply=False
Dry run only. Re-run with --apply to close Redmine issues and record idempotency markers.
```
No Redmine mutation has been applied yet.
## Actual Closure Command (Pending Approval)
After confirming `status_id=5` is the correct closed status on the LAN copy, this is the command that would close the two planned tickets and record idempotency markers:
```bash
cd /home/jason/projects/redmine
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10 \
--impersonate-user rebot \
--apply
```
Expected write behavior:
- sends `PUT /issues/:id.json`
- payload:
```json
{
"issue": {
"status_id": 5,
"notes": "Closed as likely spam by redmine_tickets.py close-if-spam."
}
}
```
- includes header:
```text
X-Redmine-Switch-User: rebot
```
- records local actions:
- `close-if-spam = closed`
- `contact-spam-review = likely-spammer` or `single-ticket-spam`
## Tests / Verification
Current test command:
```bash
python3 -m unittest discover -s tests -p 'test_redmine_tickets.py' -v
```
Last verified:
```text
Ran 11 tests
OK
```
Compile command:
```bash
python3 -m py_compile redmine_tickets.py tests/test_redmine_tickets.py redmine_contacts.py
```
Last verified: OK.
## 25-Ticket LAN Evaluation — 2026-07-09
Jason confirmed `status_id=5` is Closed. A fresh throwaway cache was used so old action markers would not hide results:
```text
.cache/redmine_tickets_lan_25_eval
```
Commands run:
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan_25_eval \
sync \
--status-id open \
--max-tickets 25 \
--order newest-updated-first
./redmine_tickets.py \
--cache-dir .cache/redmine_tickets_lan_25_eval \
--project customer-service \
spam-check \
--apply
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan_25_eval \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10
```
Verified output summary:
```text
Sync complete: seen=25 fetched=25 skipped_cached=0 written=25
Spam check complete: checked=25 apply=True
Close-if-spam complete: planned=5 apply=False
SUMMARY cached_tickets=25
SUMMARY spam=5 not_spam=20
```
Closure candidates produced by dry-run:
```text
#41753 Automated unfulfillable FBA inventory removal notification — from fba-noreply@amazon.com — contact_id=51 — contact_spam=10/10 — likely_spammer=True
#41760 Celebrate America's 250th: Save Up To 40% Off — from reply@globalindustrial.com — contact_id=621 — contact_spam=4/10 — likely_spammer=True
#41761 Celebrate America's 250th: Save Up To 40% Off — from reply@globalindustrial.com — contact_id=621 — contact_spam=4/10 — likely_spammer=True
#41767 — — from enews-noreply@qnap.com — contact_id=3162 — contact_spam=10/10 — likely_spammer=True
#41776 Your favorite sale ENDS THURSDAY ⏳ — from noreply@mail.vistaprint.com — contact_id=3795 — contact_spam=10/10 — likely_spammer=True
```
Important finding from first run: the scorer conflated **spam/promotional clutter** with **automated operational notifications**. Example: `#41753` from `fba-noreply@amazon.com` is likely an automated Amazon/FBA operational notification, not normal spam.
No Redmine close mutation was applied during this 25-ticket evaluation.
## Refined Promotional-Junk Evaluation — 2026-07-09
The scorer was refined to target Jason's immediate goal: quickly and reliably close **obvious unsolicited promotional/sales junk**, such as Walmart/Uline/GlobalIndustrial/Vistaprint/QNAP-style promotional mail and lead-gen/sales outreach from people trying to sell LDR something without an established relationship.
False-positive protections added:
- `automated-operational`: Amazon/FBA/order/API/payment-style notifications are excluded from the default auto-close class.
- Customer/vendor operational threads are protected from staff-reply contamination: LDR staff-authored journal notes are no longer scored as spam evidence.
- Standalone `SEO` in customer operational context is not treated as a sales trigger.
Regression tests now cover:
- promotional sale email => `spam-promotional`
- cleaning-service sales pitch => `spam-promotional`
- Amazon/FBA notification => `automated-operational`, not spam
- real FBA returns inquiry with LDR pricing reply => not spam
- customer operational context mentioning SEO => not spam
Fresh final cache:
```text
.cache/redmine_tickets_lan_25_promotional_eval_final_1783600119
```
Final verified summary:
```text
Sync complete: seen=25 fetched=25 skipped_cached=0 written=25
Close-if-spam complete: planned=6 apply=False
SUMMARY categories={'unknown-review': 17, 'automated-operational': 2, 'spam-promotional': 6}
```
Current safe close candidates from the final dry-run:
```text
#41767 — from enews-noreply@qnap.com — QNAP newsletter/promo
#41776 Your favorite sale ENDS THURSDAY ⏳ — from noreply@mail.vistaprint.com
#41788 Up to 20% Off Select Fans & Blowers — from reply@globalindustrial.com
#41789 Up to 20% Off Select Fans & Blowers — from reply@globalindustrial.com
#41804 Do you serve e-commerce brands? — from d.elliott@fulfillmentviewpro.com — 3PL lead-gen/sales pitch
#41834 Japan’s “JC-STAR Level 1” — from enews-noreply@qnap.com — QNAP promotional/newsletter item
```
No Redmine close mutation was applied during this final evaluation.
### LAN apply verification — 2026-07-09
Jason approved moving toward closing the obvious junk. The following command was run against the final 25-ticket cache:
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan_25_promotional_eval_final_1783600119 \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10 \
--impersonate-user rebot \
--apply
```
Applied closures:
```text
#41767
#41776
#41788
#41789
#41804
#41834
```
Idempotency check:
```text
Close-if-spam complete: planned=0 apply=True
```
API verification after apply:
```text
41767 status=Closed close_note_author=Harry B. close_note_found=True
41776 status=Closed close_note_author=Harry B. close_note_found=True
41788 status=Closed close_note_author=Harry B. close_note_found=True
41789 status=Closed close_note_author=Harry B. close_note_found=True
41804 status=Closed close_note_author=Harry B. close_note_found=True
41834 status=Closed close_note_author=Harry B. close_note_found=True
```
## Current Limitations / Known Next Improvements
1. `manifest.json` is acceptable for the current tracer, but SQLite should be considered once caches/actions cover thousands of tickets.
2. Spam scoring is still intentionally simple and reviewable. It now distinguishes `spam-promotional`, `automated-operational`, and `unknown-review`; future classes can be added as patterns emerge.
3. Contact-level spammer handling is only advisory right now. The tool reports `likely_spammer=True`, but does not yet tag the contact.
4. First real close apply for each newly expanded batch should happen on LAN only, then immediately re-sync/verify status and activity author.
5. The production Redmine path should remain out of scope until LAN write behavior is proven and reviewed.
## 50-Ticket Refined Promotional-Junk Dry Run — 2026-07-09
Fresh cache:
```text
.cache/redmine_tickets_lan_50_promotional_eval_refined_1783600628
```
Verified summary:
```text
Sync complete: seen=50 fetched=50 skipped_cached=0 written=50
Close-if-spam complete: planned=9 apply=False
SUMMARY categories={'unknown-review': 32, 'spam-promotional': 9, 'automated-operational': 9}
```
Promotional close candidates:
```text
#41685 Samsung promo — noreply_samsung@sea.samsung.com — "Jason, we’ve missed you. We just can't stop innovating."
#41687 QNAP promo — enews-noreply@qnap.com — "Qsync Now Supports File Backup on Mac"
#41698 Vistaprint promo — noreply@mail.vistaprint.com — "Life’s best moments, treasured forever"
#41705 Popmenu/restaurant promo — noreply@popmenu.com — "Take a break from cooking!"
#41722 3PL lead-gen/sales pitch — hello@offer.search3pl.com — "Live fulfilment leads waiting for you to connect"
#41738 Vistaprint promo — noreply@mail.vistaprint.com — "Up to 15% off top business products that’ll draw them in"
#41746 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com — "Hi LDR, here are our upcoming events and nominations"
#41760 Global Industrial sale — reply@globalindustrial.com — "Celebrate America's 250th: Save Up To 40% Off"
#41761 Global Industrial sale — reply@globalindustrial.com — "Celebrate America's 250th: Save Up To 40% Off"
```
Operational/system notifications excluded from auto-close:
```text
#41694 USPS eReceipt
#41704 2024 UNPAID INVOICES $3,413.55
#41714 Undelivered Mail Returned to Sender
#41716 Critical security alert
#41726 UPS Invoice Now Available
#41753 Automated unfulfillable FBA inventory removal notification
#41765 Amazon Listings API changes
#41772 Cintas Payment Status
#41823 Amazon order confirmation
```
No Redmine mutation was applied during this 50-ticket dry run.
### 50-ticket LAN apply verification — 2026-07-09
Jason approved applying the 9-candidate promotional set. The following command was run against the exact reviewed cache:
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan_50_promotional_eval_refined_1783600628 \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10 \
--impersonate-user rebot \
--apply
```
Applied closures:
```text
#41685
#41687
#41698
#41705
#41722
#41738
#41746
#41760
#41761
```
Idempotency check:
```text
Close-if-spam complete: planned=0 apply=True
```
API verification after apply:
```text
41685 status=Closed close_note_author=Harry B. close_note_found=True
41687 status=Closed close_note_author=Harry B. close_note_found=True
41698 status=Closed close_note_author=Harry B. close_note_found=True
41705 status=Closed close_note_author=Harry B. close_note_found=True
41722 status=Closed close_note_author=Harry B. close_note_found=True
41738 status=Closed close_note_author=Harry B. close_note_found=True
41746 status=Closed close_note_author=Harry B. close_note_found=True
41760 status=Closed close_note_author=Harry B. close_note_found=True
41761 status=Closed close_note_author=Harry B. close_note_found=True
```
## Recommended Next Session Steps
1. If Jason approves the current 27-candidate production 200-ticket set, apply closures on production using `--impersonate-user rebot` and the exact cache `.cache/redmine_tickets_prod_200_promotional_eval_1783612885`.
2. Optionally apply the 3 `maybe-customer` contact tags from the same cache.
3. Re-sync/inspect via Redmine API/UI and verify closed status, close-note author `Harry B.`, second-apply idempotency, and local action markers.
4. Continue with another bounded production dry-run batch.
## Customer identification update — 2026-07-09
Jason confirmed the following contacts/patterns are customers and should be tagged `Customer`:
```text
koki-takeda@shinkiya-inc.com
hello@bohoflow.co
hi5@bavariatrachten.com
maia@solride.org
mike@solride.org
info@u2wear.com
sales@u2wear.com
any Cyberstorm email
Esire-family emails
```
Important correction: do **not** treat incidental substrings such as `desiree` in Indeed/applicant email aliases as Esire-family customer matches.
Production contact tagging was applied with `impersonate_user='rebot'`:
```text
MATCHED_COUNT 31
UPDATED_COUNT 25
SKIPPED_ALREADY_CUSTOMER 6
```
Two accidental applicant matches were corrected immediately:
```text
4253 conversation-desireethrasher-customerreturnsinspect-0n3@indeedemail.com: removed Customer, left applicant
3934 conversation-desireethrasher-customerreturnsinspect-9nv@indeedemail.com: removed Customer, left applicant
```
After customer tagging, re-running the production 200 close plan still produced 27 candidates and kept `#41238` removed from the plan.
## Customer-tag safety gate — 2026-07-09
New hard rule: before closing any ticket as spam, `redmine_tickets.py` fetches the Helpdesk contact and checks `tag_list`. If the contact has tag `customer`, the ticket is **not** closed as spam. Absence of the `customer` tag does **not** imply spam.
Jason also clarified the closure risk model:
- Closed tickets are not deleted.
- If the other party replies to a closed Helpdesk ticket, Redmine reopens it as `New Message`.
- Therefore, stale 30+ day tickets that match a verified `spam-promotional` profile are reasonable fast-close candidates once contact safety gates pass.
- However, if a ticket was previously closed and then reopened because the contact messaged again, do **not** auto-close it again. Escalate instead: bump priority and optionally set a due date 1–3 days out.
A separate action now exists to tag contacts needing manual/customer review:
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir <cache> \
tag-maybe-customer [--apply]
```
This adds `maybe-customer` to contacts with inbound/established-business inquiry evidence, using `--impersonate-user rebot` for writes.
Tests now cover 20 cases, including the customer-tag skip and `maybe-customer` tag action.
## Production 200-Ticket Dry Run With Customer Gate — 2026-07-09
Production cache:
```text
.cache/redmine_tickets_prod_200_promotional_eval_1783612885
```
Re-running the close plan after adding the contact `customer` safety gate produced:
```text
Close-if-spam complete: planned=27 apply=False
```
The customer-tag gate removed `#41238` from the close plan, which was the suspicious `manager@sonisha.com` / `Account Queries` candidate.
Current 27 close candidates:
```text
#41098
#41114
#41121
#41122
#41127
#41128
#41153
#41164
#41167
#41168
#41201
#41222
#41226
#41227
#41242
#41248
#41269
#41293
#41297
#41315
#41331
#41340
#41341
#41347
#41368
#41373
#41850
```
Maybe-customer dry-run produced 3 contact tag candidates:
```text
contact_id=4470 source_issue_id=41424 tag_list=["maybe-customer"]
contact_id=3919 source_issue_id=41448 tag_list=["maybe-customer"]
contact_id=4476 source_issue_id=41504 tag_list=["maybe-customer"]
```
No production mutation was applied after adding this guard.
## Production 100-Ticket Promotional-Junk Dry Run — 2026-07-09
Production URL:
```text
https://rm.ldrprep.com
```
Fresh cache:
```text
.cache/redmine_tickets_prod_100_promotional_eval_1783609390
```
Verified summary:
```text
Sync complete: seen=100 fetched=100 skipped_cached=0 written=100
Close-if-spam complete: planned=20 apply=False
SUMMARY categories={'unknown-review': 59, 'automated-operational': 21, 'spam-promotional': 20}
```
Promotional close candidates:
```text
#41397 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com
#41430 Vistaprint sale — noreply@mail.vistaprint.com
#41431 Business Summit promo — replycincinnati@partner.bizjournals.com
#41657 Global Industrial Kee Safety promo — reply@globalindustrial.com
#41658 Global Industrial Kee Safety promo — reply@globalindustrial.com
#41685 Samsung promo — noreply_samsung@sea.samsung.com
#41687 QNAP promo — enews-noreply@qnap.com
#41698 Vistaprint promo — noreply@mail.vistaprint.com
#41705 Popmenu restaurant promo — noreply@popmenu.com
#41722 3PL lead-gen sales pitch — hello@offer.search3pl.com
#41738 Vistaprint promo — noreply@mail.vistaprint.com
#41746 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com
#41760 Global Industrial sale — reply@globalindustrial.com
#41761 Global Industrial sale — reply@globalindustrial.com
#41767 QNAP newsletter/promo — enews-noreply@qnap.com
#41776 Vistaprint sale — noreply@mail.vistaprint.com
#41788 Global Industrial sale — reply@globalindustrial.com
#41789 Global Industrial sale — reply@globalindustrial.com
#41804 3PL lead-gen sales pitch — d.elliott@fulfillmentviewpro.com
#41834 QNAP promotional/newsletter item — enews-noreply@qnap.com
```
No production mutation was applied during this dry run.
### Production apply verification — 2026-07-09
Jason approved applying the 20-candidate promotional set. The following command was run against the exact reviewed production cache:
```bash
./redmine_tickets.py \
--base-url https://rm.ldrprep.com \
--project customer-service \
--cache-dir .cache/redmine_tickets_prod_100_promotional_eval_1783609390 \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10 \
--impersonate-user rebot \
--apply
```
Applied closures:
```text
#41397
#41430
#41431
#41657
#41658
#41685
#41687
#41698
#41705
#41722
#41738
#41746
#41760
#41761
#41767
#41776
#41788
#41789
#41804
#41834
```
Idempotency check:
```text
Close-if-spam complete: planned=0 apply=True
```
API verification after apply:
```text
41397 status=Closed close_note_author=Harry B. close_note_found=True
41430 status=Closed close_note_author=Harry B. close_note_found=True
41431 status=Closed close_note_author=Harry B. close_note_found=True
41657 status=Closed close_note_author=Harry B. close_note_found=True
41658 status=Closed close_note_author=Harry B. close_note_found=True
41685 status=Closed close_note_author=Harry B. close_note_found=True
41687 status=Closed close_note_author=Harry B. close_note_found=True
41698 status=Closed close_note_author=Harry B. close_note_found=True
41705 status=Closed close_note_author=Harry B. close_note_found=True
41722 status=Closed close_note_author=Harry B. close_note_found=True
41738 status=Closed close_note_author=Harry B. close_note_found=True
41746 status=Closed close_note_author=Harry B. close_note_found=True
41760 status=Closed close_note_author=Harry B. close_note_found=True
41761 status=Closed close_note_author=Harry B. close_note_found=True
41767 status=Closed close_note_author=Harry B. close_note_found=True
41776 status=Closed close_note_author=Harry B. close_note_found=True
41788 status=Closed close_note_author=Harry B. close_note_found=True
41789 status=Closed close_note_author=Harry B. close_note_found=True
41804 status=Closed close_note_author=Harry B. close_note_found=True
41834 status=Closed close_note_author=Harry B. close_note_found=True
```
## 100-Ticket Refined Promotional-Junk Dry Run — 2026-07-09
Fresh cache:
```text
.cache/redmine_tickets_lan_100_promotional_eval_refined_1783601115
```
Verified summary:
```text
Sync complete: seen=100 fetched=100 skipped_cached=0 written=100
Close-if-spam complete: planned=9 apply=False
SUMMARY categories={'unknown-review': 68, 'spam-promotional': 9, 'automated-operational': 23}
```
Promotional close candidates:
```text
#41227 Global Industrial clearance — reply@globalindustrial.com — "Clearance: Bestsellers At Best Prices"
#41297 Vistaprint sale — noreply@mail.vistaprint.com — "Up to 30% off select customer faves, like mugs, postcards & more"
#41340 Global Industrial AC/fans — reply@globalindustrial.com — "Advanced AC, Wall Fans, And Cooling Units"
#41373 TikTok Shop promo — taylor@shop.tiktok.com — "The simplest way to start selling this June"
#41397 Business Journals events/nominations promo — replycincinnati@engaged.bizjournals.com — "Hi LDR, here are our upcoming events and nominations"
#41430 Vistaprint sale — noreply@mail.vistaprint.com — "Email exclusive! Up to 40% off select top products ENDS TOMORROW"
#41431 Business Summit promo — replycincinnati@partner.bizjournals.com — "Regional Business Summit Returns August 5"
#41657 Global Industrial Kee Safety promo — reply@globalindustrial.com — "'Kee Safety' Modular & Durable Fall Prevention Equipment"
#41658 Global Industrial Kee Safety promo — reply@globalindustrial.com — "'Kee Safety' Modular & Durable Fall Prevention Equipment"
```
False-positive protections added/verified during this 100-ticket run:
- VoIP voicemail / call transcription tickets are `automated-operational`, not promotional spam.
- Inbound buying/relationship inquiries like RePackify Gaylord purchase interest and u2wear returns-provider follow-up remain `unknown-review`, not auto-close.
- Receipts, bounces, security alerts, invoices, Amazon/FBA/order notices remain `automated-operational`, not auto-close.
No Redmine mutation was applied during this 100-ticket dry run.
### 100-ticket LAN apply verification — 2026-07-09
Jason approved applying the 9-candidate promotional set. The following command was run against the exact reviewed cache:
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
--cache-dir .cache/redmine_tickets_lan_100_promotional_eval_refined_1783601115 \
close-if-spam \
--close-status-id 5 \
--contact-ticket-limit 10 \
--impersonate-user rebot \
--apply
```
Applied closures:
```text
#41227
#41297
#41340
#41373
#41397
#41430
#41431
#41657
#41658
```
Idempotency check:
```text
Close-if-spam complete: planned=0 apply=True
```
API verification after apply:
```text
41227 status=Closed close_note_author=Harry B. close_note_found=True
41297 status=Closed close_note_author=Harry B. close_note_found=True
41340 status=Closed close_note_author=Harry B. close_note_found=True
41373 status=Closed close_note_author=Harry B. close_note_found=True
41397 status=Closed close_note_author=Harry B. close_note_found=True
41430 status=Closed close_note_author=Harry B. close_note_found=True
41431 status=Closed close_note_author=Harry B. close_note_found=True
41657 status=Closed close_note_author=Harry B. close_note_found=True
41658 status=Closed close_note_author=Harry B. close_note_found=True
```
docs/ticket_sync_tool_plan.md
+191
View File
@@ -0,0 +1,191 @@
# Ticket Sync Tool — Architecture & Implementation Plan
**Status**: Phase 1–3 tracer implemented and LAN-smoke-tested (2026-07-09)
**Goal**: Build a reliable, incremental ticket sync + query foundation (`redmine_tickets.py`) that replaces the obsolete RebotSlick `PullHelpdeskInfo` pattern with the modern `include=helpdesk` capability. This becomes the core data layer for all future agentic ticket workflows (spam cleanup, enrichment, dropped-ball detection, etc.).
**Primary target**: `customer-service` project (currently >1700 tickets). Tool must work for any project.
**Key constraints**
- Do **not** modify any NMI components.
- Reuse existing `RedmineClient` and helpdesk methods from `redmine_contacts.py`.
- Leverage the `include=helpdesk` (and `include=journals,helpdesk`) patch already deployed.
- Cache must be idempotent and cheap to re-run.
- Tool must support “do nothing but sync what’s new” when run with no action.
---
## 1. Core Requirements (from user spec)
1. **Incremental sync**
- Recursively fetch tickets with optional criteria (`--updated-since`, `--category`, `--status`, `--project`, `--with-tag`, etc.).
- Store full enriched payload (issue + journals + helpdesk_ticket + messages) as local JSON.
- Only re-fetch when the ticket’s `updated_on` is newer than the cached file mtime (or use a small manifest).
2. **No-op default behavior**
- Running with no filters/action does a safe, incremental sync of everything (or a sensible default window).
3. **Filter/search mode**
- When filters are supplied but no action is chosen, behave as a powerful search/report tool (list matching tickets, optionally write JSONL).
4. **Idempotent action layer (future)**
- Higher-level workers (spam-check, close, enrich, etc.) must be able to mark a ticket as “already processed” so repeated runs skip it.
- Suggested mechanism: sidecar manifest or lightweight SQLite table (`ticket_id`, `action_name`, `last_run`, `result`).
5. **Helpdesk enrichment**
- Always request `include=journals,helpdesk` (or configurable) so every cached ticket already contains the rich contact/source metadata.
---
## 2. Proposed Architecture
### 2.1 Tool location & reuse
- New script: `/home/jason/projects/redmine/redmine_tickets.py`
- Reuses the existing `RedmineClient` class (especially `issue_with_helpdesk`, `get_issue`, list methods).
- Same argument patterns, dry-run safety, `--apply`, error handling, and completion scripts as `redmine_contacts.py`.
### 2.2 Cache layout (local filesystem)
Recommended structure (under a configurable `--cache-dir`, default `/home/jason/projects/redmine/cache/tickets/` or `/tmp/redmine-tickets-cache/`):
```
cache/tickets/
├── manifest.json # { "ticket_id": { "updated_on": "...", "last_synced": "...", "actions": {...} } }
├── customer-service/
│ ├── 12345.json # full enriched payload
│ ├── 12346.json
│ └── ...
└── other-project/
└── ...
```
- One JSON file per ticket (stable, human-inspectable, easy to diff).
- `manifest.json` tracks `updated_on` + per-action idempotency state.
- Simple mtime + `updated_on` comparison is sufficient for v1; manifest makes it robust.
- Implementation note: ticket payload storage and ticket/action state storage are separated in code. `ManifestTicketStateStore` is the current implementation, but `TicketCache` only depends on a small state-store method set (`get_ticket_state`, `upsert_ticket_state`, `has_action`, `record_action`). If the manifest becomes cumbersome with thousands of tickets, replace it with SQLite or another backend without rewriting sync/search/action logic.
### 2.3 CLI structure (v1 scope)
```bash
./redmine_tickets.py \
--base-url http://192.168.50.170 \
--project customer-service \
[sync|search] \
[--updated-since YYYY-MM-DD] \
[--updated-before YYYY-MM-DD] \
[--status open|closed|*] \
[--category "Spam"] \
[--max-tickets N] \
[--order newest-updated-first] \
[--cache-dir ...] \
[--dry-run] [--apply] \
[--json] [--jsonl]
```
**Commands**
- `sync` (default if omitted): Perform incremental fetch + cache update.
- `search`: Same selection logic, but only output matching tickets (no cache write unless `--write-cache`).
**Safety**
- Dry-run by default for any write.
- `--apply` required to persist cache or perform mutations.
### 2.4 Idempotency & action hooks (v1 + v2)
- Manifest stores per-ticket action state:
```json
"actions": {
"spam-check": { "last_run": "2026-07-09", "result": "not-spam" },
"close-if-spam": { "last_run": "...", "result": "skipped" }
}
```
- Future workers read the manifest before acting and update it after.
---
## 3. Implementation Phases
### Phase 1 — Core sync + cache (implemented tracer)
- Implement `RedmineClient` enhancements if needed (list issues with `include=journals,helpdesk`, pagination).
- Basic `sync` command that:
- Accepts date/status/project filters.
- Fetches pages of issues.
- For each issue, writes `cache/<project>/<id>.json` only if newer than manifest entry.
- Updates manifest.
- `--max-tickets` safety limit.
- Dry-run shows what would be fetched/updated.
**Verified 2026-07-09**: `redmine_tickets.py sync` pulled 10 open `customer-service` tickets from LAN Redmine (`192.168.50.170`), wrote enriched per-ticket JSON, and a second identical run fetched 0 / skipped 10 unchanged cached tickets.
### Phase 2 — Search / filter-only mode (implemented tracer)
- `search` subcommand that reuses the same selection engine.
- Output options: human table, `--json`, `--jsonl` (for downstream agents).
- Ability to pipe results into later tools.
**Verified 2026-07-09**: cached search over the 10-ticket LAN cache listed the same newest open tickets seen in the UI, including promo candidates like `#41788` and `#41789`.
### Phase 3 — Idempotency manifest + action skeleton (implemented tracer)
- Add manifest read/write helpers.
- Skeleton `--action` interface (initially just records “action X run on ticket Y”).
- Example action: `spam-check` (calls the logic we started in `spam_promo_scorer.py` but at ticket level).
**Verified 2026-07-09**: `spam-check` identified `#41788` and `#41789` as SPAM based on promotional subject language, recorded local manifest idempotency with `--apply`, and a second `spam-check` run checked 0 tickets. `close-if-spam` dry-run planned closures for those two tickets and reviewed sender `contact_id=621`, finding 4/10 recent tickets scored as spam (`likely_spammer=True`). No Redmine close mutation has been applied yet.
### Phase 4 — Spam cleanup worker (next)
- Concrete `spam-check` + `close-if-spam` actions.
- Rules based on sender patterns, subject language, lack of human replies, existing contact tags, etc.
- Produces a report + optional `--apply` to close tickets and add `spam` tag to contact.
Preferred spam policy:
1. When a ticket is high-confidence spam, close that ticket.
2. Use `helpdesk_ticket.contact.id` to fetch/analyze the last 5–10 tickets from the same contact.
3. Decide whether the sender/contact appears to be a recurring spammer or whether only the specific ticket/email is spam.
4. Only mark/tag the contact as a spammer when the contact-level lookback supports that conclusion.
5. Record both `close-if-spam` and `contact-spam-review` idempotency markers so repeated runs do not re-close/re-review the same work.
6. Treat Redmine impersonation like `sudo`: do not use it for ordinary reads or dry-run planning, but all spam-cleanup write calls should impersonate `rebot` / “Harry Botter” so Redmine activity logs clearly attribute automated cleanup work.
---
## 4. Non-Goals (for this phase)
- Do not replace or modify any NMI cron scripts.
- Do not build full agent orchestration yet (this is the data layer).
- Do not implement complex NLP classifiers in v1 (keep scoring simple and reviewable).
- Cache does not need to be a database (filesystem + manifest is sufficient and inspectable).
---
## 5. Success Criteria
- Running the tool twice in a row with the same filters produces zero API calls on the second run (everything is cached and up-to-date).
- A filter-only search for “tickets updated in last 30 days with no human reply” returns results in < 5 seconds after initial sync.
- The same tool can be pointed at any project and produce usable enriched ticket JSON.
- Higher-level workers can be written as thin Python scripts that consume the cache/manifest without needing their own Redmine client logic.
---
## 6. Next Immediate Action
The implementation tracer is complete through dry-run `close-if-spam`.
Next safe execution sequence:
1. Confirm LAN issue `status_id=5` is the intended closed status.
2. Run `close-if-spam --apply` against the LAN copy only, impersonating `rebot` (the CLI default).
3. Re-sync or inspect the affected LAN tickets.
4. Verify:
- `#41788` and `#41789` are closed,
- Redmine activity author is `rebot` / Harry Botter,
- local state records `close-if-spam` and `contact-spam-review`,
- a second `close-if-spam --apply` does nothing.
5. Only then expand the batch size beyond the initial 10-ticket cache.
Detailed handoff: `docs/ticket_spam_cleanup_handoff_2026-07-09.md`.
---
**References**
- `docs/redmine_issue_api_helpdesk_include.md` (the key enabler)
- `redmine_contacts.py` (client patterns, filtering, dry-run UX)
- `docs/contact_triage_loop_*` (overall workflow philosophy)
- RebotSlick `RebotSlick.php` (historical cache idea — modernized)
This plan deliberately keeps scope tight while giving us a clean, extensible foundation for the spam-reduction work that is currently blocking the customer-service UI.
docs/visual/helpdesk-ticket-rendering.md
+11
View File
@@ -0,0 +1,11 @@
![[message (1).eml]]## Helpdesk ticket rendering notes
**Ticket 41753** is used here as an example.
![rendered helpdesk ticket 41753 with circled areas for annotation](./rendered-helpdesk-ticket-41753.png)
## Ticket https://rm.ldrprep.com/issues/43025
The resulting subject is just "Let's", and it seems like this is caused by UTF-8 characters. The actual email text is as follows:
```text
Subject: =?UTF-8?B?TGV04oCZcw==?= update your password
```
docs/visual/original-helpdesk-ticket-41753.eml
+948
View File
@@ -0,0 +1,948 @@
Delivered-To: contact@ldrprep.com
Received: by 2002:a05:6918:5408:b0:476:29de:ede8 with SMTP id s8csp5332841ysl;
Mon, 6 Jul 2026 09:49:14 -0700 (PDT)
X-Received: by 2002:a05:622a:4015:b0:51b:feee:b08f with SMTP id d75a77b69052e-51c73f516b5mr19056341cf.1.1783356554502;
Mon, 06 Jul 2026 09:49:14 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1783356554; cv=none;
d=google.com; s=arc-20260327;
b=jXuQyGgK9DIPjBYAbzAJ20wdcyzgO5pw27qvr4DQZcBYquyPpTbDvGYJNhRRYZeuce
vq4r5c/lCmDLRVaYF23nVaq+mcj9xttCKoWMphIJB2nW9EwsDSiPBKRCNBkeOGxxoG6F
uFFkrLi952oy+B7oc/pW1PeXINIoMqa4+6Ggp0aTfrCj6rYVopN8AmpR1KF/gsKxoDy6
G5q4nbmOH/QD9yNTHDgCRQKRbzFg8Qh1cXZJa0UQATPL+C91e/BbQyK4/TK4AGJrbvwx
9AxR1asvqr6TLk2WkyWEbDa7sr38cK8YDBtEyL3U/4kJcJJ4kIQ25snOwPRE0T9tZv65
LW1g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327;
h=feedback-id:bounces-to:mime-version:subject:message-id:to:reply-to
:from:date:dkim-signature:dkim-signature;
bh=R3n7y8rm5NjqL3jRteMIf7Tu+z0CzAY1J5Yk3jgKsq8=;
fh=EfWMslCv/1V8JdxM9Auar9N+FfH8Z6xRox18ZgotYNQ=;
b=gTQr3GOnpaPlTobB0OGtJ0D9XzXAIZOw6fckc5GbfhAx0YEUPO+WkAQA7Ikxh22emG
weiP4UEax/mOo0n+RryW3tH6lItdnKadqTYPhORRc++dHSeVpE32C/cPjuNgXnOwCtI5
mb+9qJr/pTzyr0kBJTSvB8xw4KaD5zpktbnJ7FMml9wySGxhhNPs3brN5MW9oWgHuiTk
xPooIa/Dt3iNOBtmLlRNEyTtcU2OAnZ6AsafDvkkY79S/Oxv+rZP8h+DdKp8xbtEtFy9
sLaNrAwyK47pT17yFm3f+AGOk4fG7KZNuJQqxj8mid3nwsN5GW9He7xoQUYDOxUo0ECX
XRUA==;
dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@amazon.com header.s=pwdll6lpa66mxupg2aa3uosx2zizclvc header.b="06YN//A6";
dkim=pass header.i=@amazonses.com header.s=224i4yxa5dv7c2xz3womw6peuasteono header.b=D4Muravu;
spf=pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) smtp.mailfrom=0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com
Return-Path: <0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com>
Received: from c143-195.smtp-out.amazonses.com (c143-195.smtp-out.amazonses.com. [76.223.143.195])
by mx.google.com with ESMTPS id d75a77b69052e-51c41b26f47si99447201cf.88.2026.07.06.09.49.14
for <contact@ldrprep.com>
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Mon, 06 Jul 2026 09:49:14 -0700 (PDT)
Received-SPF: pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) client-ip=76.223.143.195;
Authentication-Results: mx.google.com;
dkim=pass header.i=@amazon.com header.s=pwdll6lpa66mxupg2aa3uosx2zizclvc header.b="06YN//A6";
dkim=pass header.i=@amazonses.com header.s=224i4yxa5dv7c2xz3womw6peuasteono header.b=D4Muravu;
spf=pass (google.com: domain of 0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com designates 76.223.143.195 as permitted sender) smtp.mailfrom=0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
s=pwdll6lpa66mxupg2aa3uosx2zizclvc; d=amazon.com; t=1783356554;
h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type;
bh=+oIkeaNXB+80Md33rZGcSEaAg+nS01oDB42PknjIXIk=;
b=06YN//A69cb10ZreMsauMMCcF/nQHOCjBwAbnJFQAC6+ijGUmCmpkGXHtRnTidQ0
mQdNdy2Sx2E438pdWtouKSi5xuhUN+vNqit+YwMVd6a7IC0YuyLTAtRbBQinoYAr6Tl
UJKXEIVYvmanrUjMjMuGo2kf7pz9EpdosXCngYs7e3swBVESKnT8D109Ia+Kmpk7leD
J+VmxC3sBtQVzEvY5unzq027vIZdQZ0K2GAdpyAHWy0+BCcvesB9lowLj5nea/L8HOd
cJsPffRGUm2G6wQNQm6fo2QybowSdPCA18k/VCT+Iz2l2oyjzFkFmX1ycoKM7O/dXwT
iFfC9dkUGA==
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1783356554;
h=Date:From:Reply-To:To:Message-ID:Subject:MIME-Version:Content-Type:Feedback-ID;
bh=+oIkeaNXB+80Md33rZGcSEaAg+nS01oDB42PknjIXIk=;
b=D4MuravuFghRiehrC6vr6KgCKcqFQl99m0hY50EceMrsHg0fm+0TeZof3DJINLbn
HlVVLcr30EjkLgVPGuN4/zf5jlTqmFNQQLrMNITJguluM4N712Ih3dsGN31M06m1rBw
G5a4Wy34Ss11oIvUH7Bba+sanCQb8m2vsSO5m8as=
Date: Mon, 6 Jul 2026 16:49:14 +0000
From: Fulfillment by Amazon Operations <fba-noreply@amazon.com>
Reply-To: Fulfillment by Amazon Operations <fba-noreply@amazon.com>
To: contact@ldrprep.com
Message-ID: <0100019f38557b1a-ce103552-25f4-4d40-a76a-d8f8cf98fa1b-000000@email.amazonses.com>
Subject: Automated unfulfillable FBA inventory removal notification
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_196911_132258034.1783356554001"
Bounces-to: RTE+NE-null-de34cA07089053HVB1V8PA3SYZ@sellernotifications.amazon.com
X-Space-Message-ID: 387030323632580
X-Marketplace-ID: ATVPDKIKX0DER
X-Space-Notification-Type: PRODUCT_REMOVAL_ORDER_CREATED_AUTO
Feedback-ID: ::1.us-east-1.apt6UHTqs3880kGtYdHRf9uSLRw5TOiel8eOJZxLSu8=:AmazonSES
X-SES-Outgoing: 2026.07.06-76.223.143.195
------=_Part_196911_132258034.1783356554001
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.=
w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns=3D"http://www.w3.org/1999/xhtml" xmlns:v=3D"urn:schemas-microso=
ft-com:vml" xmlns:o=3D"urn:schemas-microsoft-com:office:office">
<!-- Email Header -->
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8">=
=09
<!-- Included header file, can be cloned and/or replaced. -->
<!--[if gte mso 15]>
<xml>
=09<o:OfficeDocumentSettings>
=09<o:AllowPNG/>
=09<o:PixelsPerInch>96</o:PixelsPerInch>
=09</o:OfficeDocumentSettings>
</xml>
<![endif]-->
<meta charset=3D"UTF-8">
<!--[if !mso]><!-->
=09=09<meta http-equiv=3D"X-UA-Compatible" content=3D"IE=3Dedge">
=09<!--<![endif]-->
<meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-scale=3D1"=
>
<title></title>
<link href=3D"http://fonts.googleapis.com/css?family=3DLato:400,700" rel=3D=
"stylesheet" type=3D"text/css">
<style type=3D"text/css">
=09
=09.nbus-survey{color: #FFF;font-family:'Lato', Helvetica, Arial, sans-seri=
f;font-size:13px;line-height:150%;text-align:center}
=09.nbus-survey:visited{color: #FFF;font-family:'Lato', Helvetica, Arial, s=
ans-serif;font-size:13px;line-height:150%;text-align:center}
=09.nbus-survey:hover{color: #FFF;font-family:'Lato', Helvetica, Arial, san=
s-serif;font-size:13px;line-height:150%;text-align:center}
=09.nbus-survey:focus{color: #FFF;font-family:'Lato', Helvetica, Arial, san=
s-serif;font-size:13px;line-height:150%;text-align:center}
=09.nbus-survey:active{color: #FFF;font-family:'Lato', Helvetica, Arial, sa=
ns-serif;font-size:13px;line-height:150%;text-align:center}
=09
=09one-column{border-spacing:0px;background-color:#FFFFFF;border:0px;paddin=
g:0px;width:100%;column-count:1;}
=09endrImageBlock{padding:0px;border-spacing:0px;min-width:100%;border-coll=
apse:collapse;width:100%;border:0px;}
=09endrImageBlockInner{padding:0px;}
=09endrImageContentContainer{adding:0px;border-spacing:0px;min-width:100%;b=
order-collapse:collapse;width:100%;border:0px;}
=09endrTextContentContainer{min-width:100%;width:100%;border-collapse:colla=
pse;background-color:#FFFFFF;border:0px;padding:0px;border-spacing:0px;}
=09endrTextBlock{min-width:100%;border-collapse:collapse;background-color:#=
FFFFFF;width:100%padding:0px;border-spacing:0px;border:0px;}
=09preview-text{display:none;font-size:1px;line-height:1px;max-height:0px;m=
ax-width:0px;opacity:0;overflow:hidden;mso-hide:all;font-family: sans-serif=
;}
=09
=09p{
=09text-align:left;
=09margin-top:10px;
=09margin-bottom:10px;
=09margin-right:0;
=09margin-left:0;
=09padding-top:0;
=09padding-bottom:0;
=09padding-right:0;
=09padding-left:0;
=09line-height:185%;
=09}
=09table{
=09border-collapse:collapse;
=09}
=09h1,h2,h3,h4,h5,h6{
=09display:block;
=09margin:0;
=09padding:0;
=09}
=09img,a img{
=09border:0;
=09height:auto;
=09outline:none;
=09text-decoration:none;
=09}
=09pre{
=09height:100%;
=09margin:0px;
=09padding:0px;
=09width:100%;
=09font-family:'Lato',Helvetica, Arial, sans-serif;
=09min-width:100%;
=09white-space: pre-wrap; /* Since CSS 2.1 */
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
=09}
=09body,#bodyTable,#bodyCell{
=09height:100%;
=09margin:0px;
=09padding:0px;
=09width:100%;
=09background-color:#e4e3e4;
=09color:#999999
=09font-family:'Lato', Helvetica, Arial, sans-serif;
=09min-width:100%;
=09}
=09#outlook a{
=09padding:0;
=09}
=09img{
=09-ms-interpolation-mode:bicubic;
=09}
=09table{
=09mso-table-lspace:0pt;
=09mso-table-rspace:0pt;
=09}
=09.ReadMsgBody{
=09width:100%;
=09}
=09.ExternalClass{
=09width:100%;
=09}
=09p,a,li,td,blockquote{
=09mso-line-height-rule:exactly;
=09}
=09a[href^=3Dtel],a[href^=3Dsms]{
=09color:inherit;
=09cursor:default;
=09text-decoration:none;
=09}
=09p,a,li,td,body,table,blockquote{
=09-ms-text-size-adjust:100%;
=09-webkit-text-size-adjust:100%;
=09}
=09.ExternalClass,.ExternalClass p,.ExternalClass td,.ExternalClass div,.Ex=
ternalClass span,.ExternalClass font{
=09line-height:100%;
=09}
=09a[x-apple-data-detectors]{
=09color:inherit !important;
=09text-decoration:none !important;
=09font-size:inherit !important;
=09font-family:inherit !important;
=09font-weight:inherit !important;
=09line-height:inherit !important;
=09}
=09.templateContainer{
=09max-width:600px !important;
=09}
=09.endrImage{
=09vertical-align:bottom;
=09}
=09.endrTextContent{
=09word-break:break-word;
=09padding-top:15px;
=09padding-bottom:10px;
=09padding-right:18px;
=09padding-left:18px;
=09text-align:left;
=09}
=09.endrTextContent img{
=09height:auto !important;
=09}
=09.endrDividerBlock{
=09table-layout:fixed !important;
=09}
=09body { margin:0 !important; }
=09div[style*=3D"margin: 16px 0"] { margin:0 !important; }
=09body,#bodyTable{
=09background-color:#e4e3e4;
=09color:#999999;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09}
=09
=09.templateBlocks{
=09background-color:#FFFFFF;
=09border-top-width:0;
=09border-bottom-width:0;
=09padding-top:0;
=09padding-bottom:0;
=09font-size:15px;
=09line-height:185%;
=09text-align:left;
=09background-color:#FFFFFF;
=09}
=09
=09.templateQuoteBlocks{
=09background-color:#F04D44;
=09}
=09
=09#bodyCell{
=09border-top:0;
=09}
=09h1{
=09color:#455c64;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:30px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:120%;
=09letter-spacing:normal;
=09padding-top:2px;
=09padding-bottom:2px;
=09}
=09a{
=09color:#e74c3c;
=09font-weight:normal;
=09text-decoration:underline;
=09}
=09h2{
=09color:#848484;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:15px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:145%;
=09letter-spacing:1px;
=09padding-top:5px;
=09padding-bottom:4px;
=09}
=09h3{
=09color:#455c64;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:20px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:140%;
=09letter-spacing:normal;
=09text-align:left;
=09padding-top:2px;
=09padding-bottom:2px;
=09}
=09h4{
=09color:#666666;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:16px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:125%;
=09letter-spacing:normal;
=09text-align:left;
=09padding-top:11px;
=09padding-bottom:4px;
=09}
=09h5{
=09color:#ffffff;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:20px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:135%;
=09letter-spacing:normal;
=09text-align:left;
=09padding-top:11px;
=09padding-right:20px;
=09padding-bottom:8px;
=09padding-left:20px;
=09}
=09h6{
=09color:#ffffff;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:26px;
=09font-style:normal;
=09font-weight:normal;
=09line-height:135%;
=09letter-spacing:normal;
=09text-align:right;
=09padding-top:11px;
=09padding-right:20px;
=09padding-bottom:8px;
=09padding-left:20px;
=09}
=09#templatePreheader{
=09border-top:0;
=09border-bottom:0;
=09padding-top:4px;
=09padding-bottom:12px;
=09}
=09#templatePreheader .endrTextContent,#templatePreheader .endrTextContent =
p{
=09color:#fbfbfb;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:12px;
=09line-height:150%;
=09text-align:center;
=09}
=09#templatePreheader .endrTextContent a,#templatePreheader .endrTextConten=
t p a{
=09color:#fbfbfb;
=09font-weight:normal;
=09text-decoration:underline;
=09}
=09#templateHeader{
=09background-color:#303942;
=09border-top:0px solid #e4e3e4;
=09border-bottom:0;
=09padding-top:0px;
=09padding-bottom:0px;
=09}
=09#templateHeader .endrTextContent,#templateHeader .endrTextContent p{
=09color:#ffffff;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:13px;
=09line-height:100%;
=09text-align:right;
=09}
=09#templateHeader .endrTextContent,#templateHeader .endrTextContent h1{
=09color:#ffffff;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:20px;
=09line-height:100%;
=09text-align:right;
=09}
=09#templateHeader .endrTextContent a,#templateHeader .endrTextContent p a{
=09color:#ffffff;
=09font-weight:normal;
=09text-decoration:none;
=09}
=09#templateSeparator{
=09padding-top:8px;
=09padding-bottom:8px;
=09}
=09.templateLowerBody{
=09background-color:#455C64;
=09border-bottom:0;
=09padding-top:1px;
=09padding-bottom:1px;
=09}
=09.templateLowerBody .endrTextContent,.templateLowerBody .endrTextContent =
p{
=09color:#ffffff;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:13px;
=09line-height:150%;
=09text-align:left;
=09}
=09.templateLowerBody .endrTextContent a,.templateLowerBody .endrTextConten=
t p a{
=09color:#ffffff;
=09font-weight:normal;
=09text-decoration:underline;
=09}
=09.templateLowerBody .endrTextContent h1 {
=09color:#ffffff;
=09font-weight:700;
=09font-size:18px;
=09}
=09.templateSocial{
=09background-color:#e4e3e4;
=09padding-top:13px;
=09padding-bottom:3px;
=09}
=09#templateFooter{
=09border-top:0;
=09border-bottom:0;
=09padding-top:5px;
=09padding-bottom:5px;
=09}
=09#templateFooter .endrTextContent,#templateFooter .endrTextContent p{
=09color:#fbfbfb;
=09font-family: 'Lato', Helvetica, Arial, sans-serif;
=09font-size:12px;
=09line-height:150%;
=09text-align:center;
=09}
=09#templateFooter .endrTextContent a,#templateFooter .endrTextContent p a{
=09color:#fbfbfb;
=09font-weight:normal;
=09text-decoration:underline;
=09}
=09
=09@media only screen and (min-width:768px){
=09.templateContainer{
=09width:600px !important;
=09}
=09}=09
=09
=09@media only screen and (max-width: 480px){
=09
=09.templateHeader{
=09=09display: none;
=09}
=09=09
=09.bigimage .endrImageContent{
=09padding-top:0px !important;
=09}
=09.templateContainer{
=09width:100% !important;
=09max-width:600px;
=09}=09@media only screen and (max-width: 480px){
=09body,table,td,p,a,li,blockquote{
=09-webkit-text-size-adjust:none !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09body{
=09width:100% !important;
=09min-width:100% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#bodyCell{
=09padding-top:10px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.columnWrapper{
=09max-width:100% !important;
=09width:100% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImage{
=09width:100% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrCaptionTopContent,.endrCaptionBottomContent,.endrTextContentContain=
er,.endrBoxedTextContentContainer,.endrImageGroupContentContainer,.endrCapt=
ionLeftTextContentContainer,.endrCaptionRightTextContentContainer,.endrCapt=
ionLeftImageContentContainer,.endrCaptionRightImageContentContainer,.endrIm=
ageCardLeftTextContentContainer,.endrImageCardRightTextContentContainer{
=09max-width:100% !important;
=09width:100% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrBoxedTextContentContainer{
=09min-width:100% !important;
=09}
=09} @media only screen and (max-width: 480px){
=09.column{
=09width:100% !important;
=09max-width:100% !important;
=09}
=09} @media only screen and (max-width: 480px){
=09.endrImageGroupContent{
=09padding:9px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrCaptionLeftContentOuter .endrTextContent,.endrCaptionRightContentOu=
ter .endrTextContent{
=09padding-top:9px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImageCardTopImageContent,.endrCaptionBlockInner .endrCaptionTopCont=
ent:last-child .endrTextContent{
=09padding-top:18px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImageCardBottomImageContent{
=09padding-bottom:9px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImageGroupBlockInner{
=09padding-top:0 !important;
=09padding-bottom:0 !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImageGroupBlockOuter{
=09padding-top:9px !important;
=09padding-bottom:9px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrTextContent,.endrBoxedTextContentColumn{
=09padding-right:18px !important;
=09padding-left:18px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.endrImageCardLeftImageContent,.endrImageCardRightImageContent{
=09padding-right:18px !important;
=09padding-bottom:0 !important;
=09padding-left:18px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.mcpreview-image-uploader{
=09display:none !important;
=09width:100% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09h1{
=09font-size:22px !important;
=09line-height:125% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09h2{
=09font-size:20px !important;
=09line-height:125% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09h3{
=09font-size:18px !important;
=09line-height:125% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09h4{
=09font-size:16px !important;
=09line-height:150% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09
=09.endrBoxedTextContentContainer .endrTextContent,.endrBoxedTextContentCon=
tainer .endrTextContent p{
=09font-size:14px !important;
=09line-height:150% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#templatePreheader{
=09display:block !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#templatePreheader .endrTextContent,#templatePreheader .endrTextContent =
p{
=09font-size:12px !important;
=09line-height:150% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#templateHeader .endrTextContent,#templateHeader .endrTextContent p{
=09font-size:16px !important;
=09line-height:100% !important;
=09text-align:center !important;
=09}
=09#templateHeader .endrTextContent, #templateHeader .endrTextContent h1{
=09font-size:20px !important;
=09line-height:100% !important;
=09padding-bottom:10px !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#templateUpperBody .endrTextContent,#templateUpperBody .endrTextContent =
p{
=09font-size:16px !important;
=09line-height:150% !important;
=09}
=09
=09}=09@media only screen and (max-width: 480px){
=09#templateColumns .columnContainer .endrTextContent,#templateColumns .col=
umnContainer .endrTextContent p{
=09font-size:16px !important;
=09line-height:150% !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09.templateLowerBody .endrTextContent,.templateLowerBody .endrTextContent =
p{
=09font-size:16px !important;
=09line-height:150% !important;
=09text-align:center !important;
=09}
=09}=09@media only screen and (max-width: 480px){
=09#templateFooter .endrTextContent,#templateFooter .endrTextContent p{
=09font-size:12px !important;
=09line-height:150% !important;
=09}
=09}
</style>
<!--[if mso]>
<style type=3D"text/css">
body, table, td {font-family: Arial, Helvetica, sans-serif !important;}
h1 {font-family: Arial, Helvetica, sans-serif !important;}
h2 {font-family: Arial, Helvetica, sans-serif !important;}
h3 {font-family: Arial, Helvetica, sans-serif !important;}
h4 {font-family: Arial, Helvetica, sans-serif !important;}
h5 {font-family: Arial, Helvetica, sans-serif !important;}
h6 {font-family: Arial, Helvetica, sans-serif !important;}
h7 {font-family: Arial, Helvetica, sans-serif !important;}
p {font-family: Arial, Helvetica, sans-serif !important;}
</style>
<![endif]-->
<!--[if gt mso 15]>
<style type=3D"text/css" media=3D"all">
/* Outlook 2016 Height Fix */
table, tr, td {border-collapse: collapse;}
tr {border-collapse: collapse; }
body {background-color:#ffffff;}
</style>
<![endif]-->
<!-- Title - Optional content - The title tag shows in email notifications =
on mobile devices -->
<title>
</title>=20
</head>
<!-- Email Body -->
<body style=3D"body">
<center class=3D"wrapper" style=3D"width:100%;table-layout:fixed;background=
-color:#e4e3e4;">
<div class=3D"webkit" style=3D"max-width:600px;margin:0 auto;">
<table align=3D"center" border=3D"0" cellpadding=3D"0" cellspacing=3D=
"0" height=3D"100%" width=3D"100%" id=3D"bodyTable" style=3D"border-collaps=
e:collapse;height:100%;margin-top:0;margin-bottom:0;margin-right:0;margin-l=
eft:0;padding-top:0;padding-bottom:0;padding-right:0;padding-left:0;width:1=
00%;background-color:#e4e3e4;color:#5a5a5a;font-family:'Lato', Helvetica, A=
rial, sans-serif;">
<tbody><tr>
<td align=3D"center" valign=3D"top" id=3D"bodyCell" style=3D"he=
ight:100%;margin-top:0;margin-bottom:0;margin-right:0;margin-left:0;width:1=
00%;padding-top:10px;padding-bottom:10px;border-top-width:0;">
<!-- BEGIN TEMPLATE // -->
<!--[if (gte mso 9)|(IE)]>
<table align=3D"center" border=3D"0" cellspacing=3D"0" cellpadding=3D"0" wi=
dth=3D"600" style=3D"width:600px;border-collapse:collapse;" >
=09<tr>
=09<td align=3D"center" valign=3D"top" width=3D"600" style=3D"width:600px;"=
>
=09=09<![endif]-->
=09=09<table border=3D"0" cellpadding=3D"0" cellspacing=3D"0" class=3D"temp=
lateContainer" style=3D"width:100%;max-width:600px;border-collapse:collapse=
;" data-space-header>
=09=09<tbody><tr>
=09=09<td>
<!-- Previewtext - Optional content - This text will appear in the inbox pr=
eview, but not the email body. div class =3D "preview-text" -->
<div style=3D"display:none;font-size:1px;line-height:1px;max-height:0px;max=
-width:0px;opacity:0;overflow:hidden;mso-hide:all;font-family: 'Amazon Embe=
r';">
</div>
<!-- Included header file(graphics) -->
<!-- BLOCK Logo Center -->
<table class=3D"one-column" border=3D"0" cellpadding=3D"0" cellspacing=3D"=
0" width=3D"100%" style=3D"border-spacing:0;" data-space-sc-header>
=09=09=09=09=09<tbody><tr valign=3D"top" style=3D"border-top-width:0;border=
-bottom-width:0;font-size:14px;line-height:185%;text-align:left;">
=09=09=09=09=09=09<td valign=3D"top" class=3D"templateLowerBody" style=3D"b=
ackground-color:#2a323a;">
=09<table border=3D"0" cellpadding=3D"0" cellspacin=
g=3D"0" width=3D"100%" class=3D"endrTextBlock" style=3D"min-width:100%;bord=
er-collapse:collapse;background-color:#2a323a;" bgcolor=3D"#2a323a">
=09=09=09=09=09=09=09=09<tbody class=3D"endrTextBlockOuter">
=09=09=09=09=09=09=09=09=09<tr>
=09=09=09=09=09=09=09=09=09=09<td valign=3D"top" class=3D"endrTextBlockInne=
r">
=09=09=09=09=09=09=09=09=09=09=09<table align=3D"left" border=3D"0" cellpad=
ding=3D"0" cellspacing=3D"0" width=3D"100%" class=3D"endrTextContentContain=
er" style=3D"min-width:100%;border-collapse:collapse;" bgcolor=3D"#2a323a">
=09=09=09=09=09=09=09=09=09=09=09<tbody>
=09=09=09=09=09=09=09=09=09=09=09<tr>
<td class=3D"templateHeader" valign=3D"top" style=3D"padding: 20p=
x 0; padding-left:40px">
<img align=3D"center" alt=3D"" src=3D"https://m.media-amazon.com/=
images/G/01/SPACE/logo-selling_coach.png" width=3D"200" style=3D"max-width:=
200px;padding-bottom:0;display:inline !important;vertical-align:bottom;bord=
er-width:0;height:auto;outline-style:none;text-decoration:none;-ms-interpol=
ation-mode:bicubic;">
</td>
=09=09=09=09=09</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
=09=09=09=09=09=09</td>
=09=09=09=09=09</tr>
=09=09=09=09=09</tbody></table>
<!-- ENDR Header -->
</td>
</tr>
</tbody></table>
<table border=3D"0" cellpadding=3D"0" cellspacing=3D"0" class=3D"templateCo=
ntainer" width=3D"100%" style=3D"width:100%;max-width:600px;border-collapse=
:collapse;background-color:#ffffff;" bgcolor=3D"#ffffff">
<tbody><tr>
<td>
<!-- General Block of text - modify/replace with your content -->
<table class=3D"one-column">
<tbody><tr valign=3D"top" class=3D"templateBlocks">
=09<td valign=3D"top">
=09=09<table class=3D"endrTextBlock">
=09=09=09<tbody class=3D"endrTextBlockOuter">
=09=09=09<tr>
=09=09=09=09<td valign=3D"top" class=3D"endrTextBlockInner">
=09=09=09=09<table align=3D"left" class=3D"endrTextContentContainer">
=09=09=09=09=09<tbody>
=09=09=09=09=09<tr>
<td valign=3D"top" class=3D"endrTextContent" align=3D"center">
<div style=3D"text-align:left;margin-top:10px;margin-bottom:10px;margin-rig=
ht:0;margin-left:0;padding-top:0;padding-bottom:0;padding-right:0;padding-l=
eft:0;line-height:185%;">
=09Hello peleya,<br><br>The following automated unfulfillable FBA removal o=
rders have been created based on your automated settings.
=09<br>
=09=09<table>
=09=09=09=09<tbody><tr>
=09=09<td style=3D"text-align:left">Removal order ID kFAkH567aC</td>
=09=09</tr>
=09=09=09</tbody></table>
=09<br>
=09You can view the removals on <a href=3D"https://sellercentral.amazon.com=
/nms/sellermobile/redirect/3dde1230-92a2-374e-bda6-f268b74e11a3?nt=3DPRODUC=
T_REMOVAL_ORDER_CREATED_AUTO&amp;sk=3DJXuLdbiY_zOmIxz0DWMUglEOnStr_axYYt2V8=
M2pcfUvK_eGXXajeG-bzHxL6vSHE9LAQhUA6k95HvXgik59KQ&amp;n=3D1&amp;u=3DaHR0cHM=
6Ly9zZWxsZXJjZW50cmFsLmFtYXpvbi5jb20vZ3Avb3JkZXJzLXYyL2xpc3QvLi4vLi4vLi4vcm=
Vwb3J0Y2VudHJhbC9SRU1PVkFMX09SREVSX0RFVEFJTC8w">Removal Order Details</a> i=
n your seller account by using the Search tool.<br><br>The next removal ord=
er will be created as scheduled on July 20, 2026 in accordance with your se=
ttings if you have unfulfillable inventory at that time.<br><br>To make cha=
nges to the shipping address, frequency of your automated unfulfillable rem=
ovals, or other settings, go to <a href=3D"https://sellercentral.amazon.com=
/nms/sellermobile/redirect/f612c1d5-63e4-3fd4-ae67-e8598384b896?nt=3DPRODUC=
T_REMOVAL_ORDER_CREATED_AUTO&amp;sk=3DZBwUT-MEmjUHrMUqzMkgwTqOj8GU5wicW-4_3=
wvmNJfjKjY2SD2A7pmHFY6AEaQ-dCj5KzjuyUtTYzaZkr1S9Q&amp;n=3D1&amp;u=3DaHR0cHM=
6Ly9zZWxsZXJjZW50cmFsLmFtYXpvbi5jb20vZ3Avb3JkZXJzLXYyL2xpc3QvLi4vLi4vLi4vZm=
JhL3NldHRpbmdzL2luZGV4Lmh0bWwvcmVmPXh4X2ZiYXNldHRpbmdzX2RuYXZfeHgjLw">Autom=
ated unfulfillable removal settings</a> in your seller account.
=09<br><br>
=09Thank you. <br><br>
The Fulfillment by Amazon team <br><br>
Please note: This email was sent from a notification-only address that cann=
ot accept incoming email. Please do not reply to this email.
</div>
</td>
=09=09=09=09=09</tr>
=09=09=09=09=09</tbody>
=09=09=09=09</table>
=09=09=09=09</td>
=09=09=09</tr>
=09=09=09</tbody>
=09=09</table>
=09</td>
</tr>
</tbody></table>
<!-- Footer -->
<!-- Included footer file, can be cloned and/or replaced. -->
=09=09=09=09=09
=09=09=09=09=09<table border=3D"0" cellpadding=3D"0" cellspacing=3D"0" clas=
s=3D"templateContainer" width=3D"100%" style=3D"width:100%;max-width:600px;=
border-collapse:collapse;" data-space-footer data-space-sc-footer>
=09=09=09=09=09<tbody><tr>
=09=09=09=09=09<td>
=09=09=09=09
=09=09=09=09=09<!-- BLOCK Footer About Us -->
=09=09=09=09=09<table class=3D"one-column" border=3D"0" cellpadding=3D"0" c=
ellspacing=3D"0" width=3D"100%" style=3D"border-spacing:0;" dir=3D"auto">
=09=09=09=09=09<tbody><tr valign=3D"top" style=3D"border-top-width:0;border=
-bottom-width:0;font-size:14px;line-height:185%;text-align:left;">
=09=09=09=09=09=09<td valign=3D"top" class=3D"templateLowerBody" style=3D"b=
ackground-color:#2a323a;">
=09<table border=3D"0" cellpadding=3D"0" cellspacin=
g=3D"0" width=3D"100%" class=3D"endrTextBlock" style=3D"min-width:100%;bord=
er-collapse:collapse;background-color:#2a323a;" bgcolor=3D"#2a323a">
=09=09=09=09=09=09=09=09<tbody class=3D"endrTextBlockOuter">
=09=09=09=09=09=09=09=09=09<tr>
=09=09=09=09=09=09=09=09=09=09<td valign=3D"top" class=3D"endrTextBlockInne=
r">
=09=09=09=09=09=09=09=09=09=09=09<center style=3D"margin-top: 20px;">
=09<a style=3D"padding: 7px 24px; text-decoration: =
none; background: #fff; border: 1px solid #EC937C; border-radius: 5px; colo=
r: #7F1809; font-family: 'Amazon Ember'; font-size: 13px; font-weight: 600;=
display: inline-block; margin-bottom: 10px" href=3D"https://sellercentral.a=
mazon.com/notifications/feedback?deliveryId=3D387030323632580&amp;communica=
tionName=3DPRODUCT_REMOVAL_ORDER_CREATED_AUTO&amp;deliveryChannel=3DEMAIL">
<img src=3D"https://m.media-amazon.com/images/G/01/spac=
e/icon.png" style=3D"margin-right: 10px;vertical-align: middle;" width=3D"2=
0" height=3D"20">Report an issue with this email
</a>
</center><p style=3D"text-align:center !important;m=
argin-top:10px;margin-bottom:10px;margin-right:10px;margin-left:15px;paddin=
g-top:0;padding-bottom:0;padding-right:0;padding-left:0;color:#ffffff;font-=
family:'Amazon Ember';font-size:13px;line-height:150%;">If you have any que=
stions visit: <a href=3D"https://sellercentral.amazon.com/nms/sellermobile/=
redirect/4e2a53a7-bed2-3e52-a967-b56eaa782802?nt=3DPRODUCT_REMOVAL_ORDER_CR=
EATED_AUTO&amp;sk=3DNIjLb1NYcbLiLnOYE5cew8DSC1z0NHVj5wxU-_YkUXvhCAnENnYDu5_=
S0ejHUU0Lah0T--j33o_cMnp5yXQoFw&amp;n=3D1&amp;u=3DaHR0cHM6Ly9zZWxsZXJjZW50c=
mFsLmFtYXpvbi5jb20" metric=3D"help" style=3D"text-decoration: none; color: =
#ffffff;">Seller Central</a><br><br>
=09=09=09=09=09=09=09=09=09=09=09 To change your email preferences visit=
: <a href=3D"https://sellercentral.amazon.com/notifications/preferences/ref=
=3Did_notifpref_dnav_xx_" metric=3D"optout" style=3D"text-decoration: none;=
color: #ffffff;">Notification Preferences</a><br><br>
=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09=09Copyright 2026 Amazo=
n, Inc, or its affiliates. All rights reserved.<br>=20
=09=09=09=09=09=09=09=09=09=09=09=09Amazon.com, 410 Terry Avenue North, Sea=
ttle, WA 98109-5210<br></p><table align=3D"left" border=3D"0" cellpadding=
=3D"0" cellspacing=3D"0" width=3D"100%" class=3D"endrTextContentContainer" =
style=3D"min-width:100%;border-collapse:collapse;" bgcolor=3D"#2a323a">
=09=09=09=09=09=09=09=09=09=09=09<tbody>
=09=09=09=09=09=09=09=09=09=09=09<tr>
=20
=20
=20
=09=09=09=09=09=09=09=09=09=09=09
=09=09=09=09=09</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
=09=09=09=09=09=09</td>
=09=09=09=09=09</tr>
=09=09=09=09=09</tbody></table>
=09=09=09=09=09<!-- BLOCK Footer About Us -->
=09=09=09=09=09</td>
=09=09=09=09=09</tr>
=09=09=09=09=09</tbody></table>
=09=09=09=09=09
=09=09=09=09=09<table border=3D"0" cellpadding=3D"0" cellspacing=3D"0" clas=
s=3D"templateContainer" style=3D"width:100%;max-width:600px;border-collapse=
:collapse;">
=09=09=09=09=09<tbody><tr>
=09=09=09=09=09<td>
=09=09=09=09
=09=09=09=09=09</td>
=09=09=09=09=09</tr>
=09=09=09=09=09</tbody></table>
=09=09=09=09=09<!--[if (gte mso 9)|(IE)]>
=09=09=09=09</td>
=09=09=09=09</tr>
=09=09=09</table>
=09=09=09<![endif]-->
=09=09=09<!-- // END TEMPLATE -->
=09=09=09</td>
=09=09</tr>
=09=09</tbody></table><!-- Footer -->
</td></tr></tbody></table></div></center><img src=3D"https://sellercentral.=
amazon.com/nms/img/1880a936-aac9-354d-9307-a42c98511ed9?sk=3D0zFq3o6e7YSXAf=
Y3L78KAvv8dglf5D-YNaNNqF2EVAWuYpaBKA65EbkSubw2VKlbXpxJHAPRK_wmJO25bt-jBw&am=
p;n=3D1">
<div id=3D"spc-watermark">
<p style=3D"font-size: 10px !important;padding-bottom: 10px !important;di=
splay: table !important;margin: 5px auto 10px !important;text-align: center=
!important;color: #a2a2a2 !important;font-family: 'Amazon Ember' !importan=
t;font-weight: 400 !important;">SPC-USAmazon-387030323632580</p>
</div></body></html>
------=_Part_196911_132258034.1783356554001--
docs/visual/rendered-helpdesk-ticket-41753.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 391 KiB

docs/visual/webview-helpdesk-ticket-41753.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 229 KiB

fix_globals.py
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env python3
"""Fix global flag placement for subcommands like tui."""
with open("redmine_tickets.py") as f:
txt = f.read()
# 1. Add --allow-prod-writes to the parent value options
old_parent = '_PARENT_VALUE_OPTIONS = frozenset({"--base-url", "--project", "--cache-dir", "--config"})'
new_parent = '_PARENT_VALUE_OPTIONS = frozenset({"--base-url", "--project", "--cache-dir", "--config", "--allow-prod-writes"})'
if old_parent in txt:
txt = txt.replace(old_parent, new_parent)
print("1. Updated _PARENT_VALUE_OPTIONS to include --allow-prod-writes")
else:
print("1. PARENT set change skipped (not exact match)")
with open("redmine_tickets.py", "w") as f:
f.write(txt)
print("Saved parent options update")
fix_postparse.py
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env python3
with open("redmine_tickets.py") as f:
txt = f.read()
old = ''' normalized_argv = inject_default_sync_command(raw_argv)
parser = build_parser()
args = parser.parse_args(normalized_argv)
cache = TicketCache(args.cache_dir, args.project)
'''
new = ''' normalized_argv = inject_default_sync_command(raw_argv)
parser = build_parser()
args = parser.parse_args(normalized_argv)
# Post-process common globals so subparsers (tui/browse/...) that re-declare them
# (to support "cmd --allow-prod-writes ...") don't leave them as None.
if getattr(args, "cache_dir", None) in (None, ""):
from pathlib import Path as _P
args.cache_dir = _P(".cache/redmine_tickets")
if getattr(args, "project", None) in (None, ""):
args.project = "customer-service"
if getattr(args, "base_url", None) in (None, ""):
args.base_url = os.getenv("REDMINE_BASE_URL", "https://rm.ldrprep.com")
cache = TicketCache(args.cache_dir, args.project)
'''
if old in txt:
txt = txt.replace(old, new)
with open("redmine_tickets.py", "w") as f:
f.write(txt)
print("Inserted post-parse normalizer for common globals")
else:
print("Anchor not found for post-parse fix")
fix_syntax.py
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env python3
with open("redmine_tickets.py") as f:
txt = f.read()
# Fix the missing ) after no-preview arg
old = ''' help="Disable fzf right-hand preview pane.",
if add_tui_parser is not None:'''
new = ''' help="Disable fzf right-hand preview pane.",
)
if add_tui_parser is not None:'''
if old in txt:
txt = txt.replace(old, new, 1)
print("Replaced missing ) block")
else:
print("Old not found, trying looser fixes")
# Ensure ) before the if
txt = txt.replace('pane.",\n if add', 'pane.",\n )\n\n if add')
with open("redmine_tickets.py", "w") as f:
f.write(txt)
print("Saved")
fix_tui_registration.py
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
import re
with open("redmine_tickets.py") as f:
txt = f.read()
# Clean any previous broken insertions around the area
txt = re.sub(
r'(\s*)\s*if add_tui_parser is not None:\s*add_tui_parser\(subparsers\)\s*\n\s*\)',
r'\1)',
txt
)
marker = ' help="Disable fzf right-hand preview pane.",\n )\n\n ocr_parser = subparsers.add_parser('
insert = ''' help="Disable fzf right-hand preview pane.",
)
if add_tui_parser is not None:
add_tui_parser(subparsers)
ocr_parser = subparsers.add_parser('''
if marker in txt:
txt = txt.replace(marker, insert, 1)
print("Inserted cleanly via marker")
else:
# Find by context and insert before ocr
ocr_pos = txt.find('ocr_parser = subparsers.add_parser(\n "ocr-rename"')
no_prev_pos = txt.find('--no-preview', 0)
if ocr_pos > 0 and no_prev_pos > 0 and ocr_pos > no_prev_pos:
# insert just before ocr
insert_block = '''
if add_tui_parser is not None:
add_tui_parser(subparsers)
'''
txt = txt[:ocr_pos] + insert_block + txt[ocr_pos:]
print("Inserted before ocr via fallback")
else:
print("Could not find good spot")
with open("redmine_tickets.py", "w") as f:
f.write(txt)
print("Done fixing registration")
icons/box-icon.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 158 KiB

icons/box-isometric-pallet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

icons/box-motion-pallet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 172 KiB

icons/box-shield-pallet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 141 KiB

icons/box-shield-stacked.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

icons/redmine-semantic-64px-conduit.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 127 KiB

icons/redmine-semantic-64px-emblem.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 141 KiB

icons/redmine-semantic-64px-magnifier.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

icons/redmine-semantic-conduit-badass.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 240 KiB

icons/redmine-semantic-emblem-badass.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 371 KiB

icons/redmine-semantic-fast-professional-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 155 KiB

icons/redmine-semantic-fast-professional-2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 87 KiB

icons/redmine-semantic-fast-professional-3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 131 KiB

icons/redmine-semantic-hybrid-badass.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 291 KiB

icons/request-icon.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 120 KiB

icons/request-manifest-tablet.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 140 KiB

icons/request-route-buildings.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

icons/request-shield-grouped.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 125 KiB

icons/request-shield-manifest.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 124 KiB

queues.example.json
+22
View File
@@ -0,0 +1,22 @@
{
"queues": [
{
"name": "unassigned",
"where": ["@unassigned"],
"max_tickets": 100,
"order": "newest-updated-first"
},
{
"name": "triage-needed-5d",
"where": ["@triage-needed-5d"],
"max_tickets": 100,
"order": "newest-updated-first"
},
{
"name": "helpdesk-email-eq-company",
"where": ["@helpdeskEmailMatchCompany"],
"max_tickets": 200,
"order": "newest-updated-first"
}
]
}
redMCP/README.md
+34
View File
@@ -220,6 +220,22 @@ The stdio server supports both MCP framing styles used by clients in the wild:
Responses mirror the detected input framing mode for compatibility.
### JSON-RPC response normalization (2026-07-10)
When running under Hermes's current Python MCP client, JSON-RPC responses with
`result` as a top-level JSON array (for example `[]`) can trigger repeated
Pydantic validation errors in `mcp.client.stdio`, followed by reconnect loops.
To prevent that log spam and reconnect churn, redMCP now normalizes list-style
JSON-RPC results at the dispatcher boundary:
- associative arrays (objects) pass through unchanged
- list arrays are wrapped as `{ "items": [...] }`
- empty list results are wrapped as `{ "items": [] }`
This keeps response payloads object-shaped for broad client compatibility while
preserving the underlying data.
For local testing, run the Streamable HTTP server:
```sh
@@ -336,6 +352,24 @@ customer-visible mail require an explicit tool call such as
`redmine_send_helpdesk_response` or `redmine_update_issue` with
`send_helpdesk_email=true`.
All MCP tools also accept optional `impersonate_user` to run the call as a
specific Redmine login when the configured API account has permission to switch
users. When omitted, calls run as the configured API user.
```json
{
"name": "redmine_update_issue",
"arguments": {
"impersonate_user": "jason",
"issue_id": 39858,
"notes": "Replying as Jason via MCP"
}
}
```
This applies to the entire tool call, including multi-step actions such as
`redmine_update_issue` with `send_helpdesk_email=true`.
Run the local no-network query normalizer checks with:
```sh
redMCP/app/McpDispatcher.php
+77 -59
View File
@@ -10,6 +10,11 @@ use Throwable;
final class McpDispatcher
{
private const PROJECT_ID_DESCRIPTION = 'Redmine project identifier or numeric id. If unsure, call redmine_find_project first and use project_id_to_use.';
private const IMPERSONATE_USER_ARG = 'impersonate_user';
private const IMPERSONATE_USER_SCHEMA = [
'type' => 'string',
'description' => 'Optional Redmine login to impersonate for this tool call. Requires switch-user permission on the authenticated API account.',
];
private const ISSUE_FIELD_ARGUMENT_KEYS = [
'project_id',
@@ -64,8 +69,9 @@ final class McpDispatcher
try {
$result = $this->dispatch($method, $params);
$normalizedResult = $this->normalizeJsonRpcResult($result);
$this->logCall($context, $method, $params, true, $started);
return ['jsonrpc' => '2.0', 'id' => $id, 'result' => $result];
return ['jsonrpc' => '2.0', 'id' => $id, 'result' => $normalizedResult];
} catch (Throwable $exception) {
$this->logCall($context, $method, $params, false, $started, $exception->getMessage());
return [
@@ -79,6 +85,27 @@ final class McpDispatcher
}
}
/**
* @param array<mixed> $result
*
* @return array<string,mixed>
*/
private function normalizeJsonRpcResult(array $result): array
{
// JSON-RPC 2.0 allows any JSON value, but the Python MCP client used by
// Hermes currently validates response.result as an object/dict.
// Normalize list-style results so stdio parsing does not fail.
if ($result === []) {
return ['items' => []];
}
if (array_is_list($result)) {
return ['items' => $result];
}
return $result;
}
/**
* @param array<string,mixed> $params
*
@@ -348,6 +375,8 @@ final class McpDispatcher
*/
private function tool(string $name, string $description, array $properties, array $required = []): array
{
$properties[self::IMPERSONATE_USER_ARG] = self::IMPERSONATE_USER_SCHEMA;
return [
'name' => $name,
'description' => $description,
@@ -369,109 +398,84 @@ final class McpDispatcher
{
$name = (string) ($params['name'] ?? '');
$arguments = is_array($params['arguments'] ?? null) ? $params['arguments'] : [];
$impersonateUser = $this->optionalStringArg($arguments, self::IMPERSONATE_USER_ARG);
unset($arguments[self::IMPERSONATE_USER_ARG]);
$result = $this->redmine->withImpersonation($impersonateUser, function () use ($name, $arguments) {
switch ($name) {
case 'redmine_list_projects':
$result = $this->redmine->listProjects(ListQueryNormalizer::listParams($arguments));
break;
return $this->redmine->listProjects(ListQueryNormalizer::listParams($arguments));
case 'redmine_find_project':
$result = $this->findProject($this->stringArg($arguments, 'query'), $this->intArg($arguments, 'limit', 10));
break;
return $this->findProject($this->stringArg($arguments, 'query'), $this->intArg($arguments, 'limit', 10));
case 'redmine_get_project':
$result = $this->redmine->project($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_get_project'), $this->objectArg($arguments, 'params'));
break;
return $this->redmine->project($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_get_project'), $this->objectArg($arguments, 'params'));
case 'redmine_list_project_memberships':
$result = $this->redmine->projectMemberships($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_memberships'), ListQueryNormalizer::listParams($arguments));
break;
return $this->redmine->projectMemberships($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_memberships'), ListQueryNormalizer::listParams($arguments));
case 'redmine_list_users':
$result = $this->redmine->listUsers(ListQueryNormalizer::userParams($arguments));
break;
return $this->redmine->listUsers(ListQueryNormalizer::userParams($arguments));
case 'redmine_get_user':
$result = $this->redmine->user($this->intArg($arguments, 'user_id'), $this->objectArg($arguments, 'params'));
break;
return $this->redmine->user($this->intArg($arguments, 'user_id'), $this->objectArg($arguments, 'params'));
case 'redmine_list_issues':
$result = $this->redmine->filterIssues(ListQueryNormalizer::issueFilters($this->resolvedProjectArgument($arguments, 'redmine_list_issues')));
break;
return $this->redmine->filterIssues(ListQueryNormalizer::issueFilters($this->resolvedProjectArgument($arguments, 'redmine_list_issues')));
case 'redmine_search':
$result = $this->redmine->search($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search')));
break;
return $this->redmine->search($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search')));
case 'redmine_search_issues':
$result = $this->redmine->searchIssues($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search_issues')));
break;
return $this->redmine->searchIssues($this->stringArg($arguments, 'query'), ListQueryNormalizer::searchParams($this->resolvedProjectArgument($arguments, 'redmine_search_issues')));
case 'redmine_get_issue':
$result = $this->redmine->issue($this->intArg($arguments, 'issue_id'), $this->stringListArg($arguments, 'include', ['journals', 'attachments']));
break;
return $this->redmine->issue($this->intArg($arguments, 'issue_id'), $this->stringListArg($arguments, 'include', ['journals', 'attachments']));
case 'redmine_list_issue_relations':
$result = $this->redmine->listIssueRelations($this->intArg($arguments, 'issue_id'));
break;
return $this->redmine->listIssueRelations($this->intArg($arguments, 'issue_id'));
case 'redmine_get_issue_relation':
$result = $this->redmine->issueRelation($this->intArg($arguments, 'relation_id'));
break;
return $this->redmine->issueRelation($this->intArg($arguments, 'relation_id'));
case 'redmine_create_issue_relation':
$result = $this->redmine->createIssueRelation($this->intArg($arguments, 'issue_id'), $this->objectArg($arguments, 'fields'));
break;
return $this->redmine->createIssueRelation($this->intArg($arguments, 'issue_id'), $this->objectArg($arguments, 'fields'));
case 'redmine_remove_issue_relation':
$result = ['ok' => $this->redmine->removeIssueRelation($this->intArg($arguments, 'relation_id'))];
break;
return ['ok' => $this->redmine->removeIssueRelation($this->intArg($arguments, 'relation_id'))];
case 'redmine_list_issue_children':
$filters = ListQueryNormalizer::issueFilters($arguments);
unset($filters['parent_id']);
$result = $this->redmine->listIssueChildren($this->intArg($arguments, 'issue_id'), $filters);
break;
return $this->redmine->listIssueChildren($this->intArg($arguments, 'issue_id'), $filters);
case 'redmine_set_issue_parent':
$result = ['ok' => $this->redmine->setIssueParent($this->intArg($arguments, 'issue_id'), $this->intArg($arguments, 'parent_issue_id'))];
break;
return ['ok' => $this->redmine->setIssueParent($this->intArg($arguments, 'issue_id'), $this->intArg($arguments, 'parent_issue_id'))];
case 'redmine_clear_issue_parent':
$result = ['ok' => $this->redmine->clearIssueParent($this->intArg($arguments, 'issue_id'))];
break;
return ['ok' => $this->redmine->clearIssueParent($this->intArg($arguments, 'issue_id'))];
case 'redmine_issue_with_helpdesk':
$result = $this->redmine->issueWithHelpdesk(
return $this->redmine->issueWithHelpdesk(
$this->intArg($arguments, 'issue_id'),
$this->intArg($arguments, 'message_limit', 100),
$this->stringListArg($arguments, 'include', ['journals', 'attachments'])
);
break;
case 'redmine_create_issue':
$result = $this->redmine->createIssue($this->issueFieldsArg($arguments, 'redmine_create_issue'));
break;
return $this->redmine->createIssue($this->issueFieldsArg($arguments, 'redmine_create_issue'));
case 'redmine_update_issue':
$result = ['ok' => $this->redmine->updateIssue($this->intArg($arguments, 'issue_id'), $this->issueFieldsArg($arguments, 'redmine_update_issue'), $this->objectArg($arguments, 'options'))];
break;
return ['ok' => $this->redmine->updateIssue($this->intArg($arguments, 'issue_id'), $this->issueFieldsArg($arguments, 'redmine_update_issue'), $this->objectArg($arguments, 'options'))];
case 'redmine_list_project_issue_categories':
$result = $this->redmine->listProjectIssueCategories($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_issue_categories'));
break;
return $this->redmine->listProjectIssueCategories($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_list_project_issue_categories'));
case 'redmine_get_issue_category':
$result = $this->redmine->issueCategory($this->intArg($arguments, 'category_id'));
break;
return $this->redmine->issueCategory($this->intArg($arguments, 'category_id'));
case 'redmine_create_issue_category':
$result = $this->redmine->createIssueCategory($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_create_issue_category'), $this->objectArg($arguments, 'fields'));
break;
return $this->redmine->createIssueCategory($this->resolvedProjectIdArg($arguments, 'project_id', 'redmine_create_issue_category'), $this->objectArg($arguments, 'fields'));
case 'redmine_update_issue_category':
$result = $this->redmine->updateIssueCategory($this->intArg($arguments, 'category_id'), $this->objectArg($arguments, 'fields'));
break;
return $this->redmine->updateIssueCategory($this->intArg($arguments, 'category_id'), $this->objectArg($arguments, 'fields'));
case 'redmine_get_attachment':
$result = $this->redmine->attachment($this->intArg($arguments, 'attachment_id'));
break;
return $this->redmine->attachment($this->intArg($arguments, 'attachment_id'));
case 'redmine_upload_attachment':
$result = $this->redmine->uploadAttachment($arguments);
break;
return $this->redmine->uploadAttachment($arguments);
case 'redmine_download_attachment':
$result = $this->redmine->downloadAttachment(
return $this->redmine->downloadAttachment(
$this->intArg($arguments, 'attachment_id'),
$this->stringArg($arguments, 'path'),
$this->boolArg($arguments, 'include_base64', false),
$this->intArg($arguments, 'max_base64_bytes', 262144)
);
break;
case 'redmine_update_attachment':
$result = $this->redmine->updateAttachment($this->intArg($arguments, 'attachment_id'), $this->objectArg($arguments, 'fields'));
break;
return $this->redmine->updateAttachment($this->intArg($arguments, 'attachment_id'), $this->objectArg($arguments, 'fields'));
case 'redmine_send_helpdesk_response':
$result = $this->redmine->sendHelpdeskIssueResponse($this->intArg($arguments, 'issue_id'), $this->stringArg($arguments, 'content'), $this->objectArg($arguments, 'options'));
break;
return $this->redmine->sendHelpdeskIssueResponse($this->intArg($arguments, 'issue_id'), $this->stringArg($arguments, 'content'), $this->objectArg($arguments, 'options'));
default:
throw new RuntimeException('Unknown tool: ' . $name);
}
});
$prepared = $this->redactSensitive($result);
if ($this->sanitizeToolText) {
@@ -721,6 +725,20 @@ final class McpDispatcher
return $value;
}
/**
* @param array<string,mixed> $arguments
*/
private function optionalStringArg(array $arguments, string $key): ?string
{
if (!array_key_exists($key, $arguments)) {
return null;
}
$value = trim((string) $arguments[$key]);
return $value === '' ? null : $value;
}
/**
* @param array<string,mixed> $arguments
*/
redMCP/app/RedmineClient.php
+25
View File
@@ -22,6 +22,31 @@ final class RedmineClient
$this->client = $client;
}
/**
* Run a callback while impersonating an existing Redmine username.
*
* Redmine impersonation only applies when the authenticated API user has
* permission to switch users.
*
* @param callable $callback
*
* @return mixed
*/
public function withImpersonation(?string $username, callable $callback)
{
$normalized = is_string($username) ? trim($username) : '';
if ($normalized === '') {
return $callback();
}
$this->client->startImpersonateUser($normalized);
try {
return $callback();
} finally {
$this->client->stopImpersonateUser();
}
}
public static function fromCredentials(string $url, string $apiKeyOrUsername, ?string $password = null): self
{
return new self(new NativeCurlClient($url, $apiKeyOrUsername, $password));
redMCP/bin/test-mcp-http-handler.php
+1 -1
View File
@@ -54,7 +54,7 @@ try {
assertContains('text/event-stream', $sse['headers'], 'SSE POST returns event stream content type', $assertions);
assertContains('X-Accel-Buffering: no', $sse['headers'], 'SSE POST disables proxy buffering', $assertions);
assertContains("event: message\n", $sse['body'], 'SSE POST emits a message event', $assertions);
assertContains('data: {"jsonrpc":"2.0","id":1,"result":[]}', $sse['body'], 'SSE POST emits JSON-RPC response data', $assertions);
assertContains('data: {"jsonrpc":"2.0","id":1,"result":{"items":[]}}', $sse['body'], 'SSE POST emits JSON-RPC response data', $assertions);
$json = httpRequest(
'POST',
redMCP/bin/test-redmine-structure.php
+58 -2
View File
@@ -19,6 +19,9 @@ final class RecordingClient implements Client, HttpClient
/** @var array<int,array{method:string,path:string,content_type:string,content:string}> */
public array $requests = [];
/** @var array<int,string> */
public array $impersonationEvents = [];
/** @var array<int,Response> */
private array $responses = [];
@@ -54,8 +57,15 @@ final class RecordingClient implements Client, HttpClient
throw new RuntimeException('Unexpected vendor API call for ' . $name);
}
public function startImpersonateUser(string $username): void {}
public function stopImpersonateUser(): void {}
public function startImpersonateUser(string $username): void
{
$this->impersonationEvents[] = 'start:' . $username;
}
public function stopImpersonateUser(): void
{
$this->impersonationEvents[] = 'stop';
}
public function requestGet(string $path): bool { return false; }
public function requestPost(string $path, string $body): bool { return false; }
public function requestPut(string $path, string $body): bool { return false; }
@@ -82,6 +92,8 @@ final class RedmineStructureTest
$this->testMcpGetProjectShowsHelpfulErrorForAmbiguousHumanName();
$this->testMcpSearchSanitizesNoisyTextFields();
$this->testMcpSearchCanDisableTextSanitization();
$this->testMcpToolCallSupportsImpersonationAndOmitsArgumentFromPayload();
$this->testMcpToolCallStopsImpersonationOnFailure();
$this->testCreateRelationDefaultsToRelatesAndRequiresTarget();
$this->testAttachmentUploadSupportsPathAndBase64();
$this->testAttachmentUploadAcceptsPdfDataUrl();
@@ -347,6 +359,49 @@ final class RedmineStructureTest
);
}
private function testMcpToolCallSupportsImpersonationAndOmitsArgumentFromPayload(): void
{
$http = new RecordingClient();
$http->queueJson(['issue' => ['id' => 90210]], 201);
$dispatcher = new McpDispatcher(new RedmineClient($http));
$result = $this->callToolJson($dispatcher, 'redmine_create_issue', [
'impersonate_user' => 'jason',
'project_id' => 'quality-tracker',
'subject' => 'Impersonation create smoke',
'description' => 'created via MCP test',
]);
$payload = $this->json($http->requests[0]['content']);
$this->assertSame(90210, $result['id'], 'impersonated create issue returns created issue payload');
$this->assertSame(['start:jason', 'stop'], $http->impersonationEvents, 'impersonation wraps full tool call');
$this->assertSame(false, isset($payload['issue']['impersonate_user']), 'impersonate_user is not forwarded as issue field');
}
private function testMcpToolCallStopsImpersonationOnFailure(): void
{
$http = new RecordingClient();
$http->queueJson(['errors' => ['forbidden']], 500);
$dispatcher = new McpDispatcher(new RedmineClient($http));
$response = $dispatcher->handleMessage([
'jsonrpc' => '2.0',
'id' => 1,
'method' => 'tools/call',
'params' => [
'name' => 'redmine_create_issue',
'arguments' => [
'impersonate_user' => 'jason',
'project_id' => 'quality-tracker',
'subject' => 'Will fail intentionally',
],
],
]);
$this->assertSame(true, is_array($response) && isset($response['error']), 'failed tool call returns MCP error response');
$this->assertSame(['start:jason', 'stop'], $http->impersonationEvents, 'impersonation is cleared when tool call fails');
}
private function testAttachmentUploadSupportsPathAndBase64(): void
{
$path = sys_get_temp_dir() . '/redmcp-upload-test.txt';
@@ -507,6 +562,7 @@ final class RedmineStructureTest
}
$projectDescription = (string) $createIssueTool['inputSchema']['properties']['project_id']['description'];
$this->assertStringContains('redmine_find_project', $projectDescription, 'project_id schema points agents to project resolver');
$this->assertSame(true, isset($createIssueTool['inputSchema']['properties']['impersonate_user']), 'tool schema advertises impersonate_user');
}
/**
redmine_contacts.py
+1064 -30
View File
File diff suppressed because it is too large Load Diff
redmine_contacts_completion.sh
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env bash
# Bash completion for redmine_contacts.py
# Usage:
# source /path/to/redmine_contacts_completion.sh
# Or copy into /etc/bash_completion.d/ (or ~/.local/share/bash-completion/completions/)
_redmine_contacts_completion() {
local cur prev cmd i
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
local global_opts="--base-url --project --cache -h --help"
local commands="fetch search show update batch-update loop-run helpdesk-ticket issue-full helpdesk-issues helpdesk-messages helpdesk-timeline"
# Determine active subcommand (first non-option token after argv0)
cmd=""
for ((i=1; i<${#COMP_WORDS[@]}; i++)); do
if [[ "${COMP_WORDS[i]}" != -* ]]; then
cmd="${COMP_WORDS[i]}"
break
fi
done
# Value completion for option arguments
case "${prev}" in
--cache|--profile)
COMPREPLY=( $(compgen -f -- "${cur}") )
return 0
;;
--cache-mode)
COMPREPLY=( $(compgen -W "full filtered" -- "${cur}") )
return 0
;;
--source)
if [[ "${cmd}" == "show" ]]; then
COMPREPLY=( $(compgen -W "auto cache api" -- "${cur}") )
else
COMPREPLY=( $(compgen -W "cache api" -- "${cur}") )
fi
return 0
;;
--contact-source)
COMPREPLY=( $(compgen -W "api cache auto" -- "${cur}") )
return 0
;;
--order)
COMPREPLY=( $(compgen -W "as-is oldest-updated-first newest-updated-first oldest-created-first newest-created-first id-asc id-desc" -- "${cur}") )
return 0
;;
--set)
COMPREPLY=( $(compgen -W "first_name= middle_name= last_name= company= website= skype_name= birthday= job_title= background= phone= email= tag_list=" -- "${cur}") )
return 0
;;
--set-custom-field)
COMPREPLY=( $(compgen -W "account-number= instagram= facebook= youtube= linkedin= whatsapp= x-twitter=" -- "${cur}") )
return 0
;;
esac
# If no command selected yet, complete command names + global options
if [[ -z "${cmd}" ]]; then
COMPREPLY=( $(compgen -W "${commands} ${global_opts}" -- "${cur}") )
return 0
fi
local common_filter_opts="--created-after --created-before --updated-after --updated-before --with-tag --without-tag --no-tags"
case "${cmd}" in
fetch)
COMPREPLY=( $(compgen -W "${global_opts} --limit --cache-mode ${common_filter_opts}" -- "${cur}") )
;;
search)
COMPREPLY=( $(compgen -W "${global_opts} --refresh --offline-cache --limit --min-score ${common_filter_opts}" -- "${cur}") )
;;
show)
COMPREPLY=( $(compgen -W "${global_opts} --source --refresh-cache" -- "${cur}") )
;;
update)
COMPREPLY=( $(compgen -W "${global_opts} --set --add-tag --remove-tag --set-custom-field --contact-source --apply" -- "${cur}") )
;;
batch-update)
COMPREPLY=( $(compgen -W "${global_opts} --max-contacts --order --source --refresh-cache ${common_filter_opts} --set --add-tag --remove-tag --set-custom-field --apply" -- "${cur}") )
;;
loop-run)
COMPREPLY=( $(compgen -W "${global_opts} --profile --max-contacts --order --source --refresh-cache ${common_filter_opts} --apply" -- "${cur}") )
;;
helpdesk-ticket)
COMPREPLY=( $(compgen -W "${global_opts}" -- "${cur}") )
;;
issue-full)
COMPREPLY=( $(compgen -W "${global_opts} --message-limit" -- "${cur}") )
;;
helpdesk-issues|helpdesk-messages|helpdesk-timeline)
COMPREPLY=( $(compgen -W "${global_opts} --limit" -- "${cur}") )
;;
esac
return 0
}
complete -F _redmine_contacts_completion redmine_contacts.py
complete -F _redmine_contacts_completion redmine_contacts
redmine_crm/__init__.py
View File
redmine_crm/crm_client.py
+230
View File
@@ -0,0 +1,230 @@
"""CRM upsert client supporting idempotency by Redmine ID and email fallback."""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Any, Callable, Dict, List, Optional
logger = logging.getLogger(__name__)
class CrmError(Exception):
"""Base exception for CRM errors."""
def __init__(
self,
message: str,
*,
redmine_id: Optional[int] = None,
crm_id: Optional[str] = None,
) -> None:
super().__init__(message)
self.redmine_id = redmine_id
self.crm_id = crm_id
@dataclass(frozen=True, slots=True)
class UpsertResult:
crm_id: str
created: bool
updated: bool
skipped_unchanged: bool
redmine_contact_id: int
class CrmClient:
"""Generic CRM upsert wrapper.
Provides primitives for local testing; real CRM adapters should subclass
and implement _find_by_redmine_id, _find_by_email, _create, and _update.
"""
# Shared simulated CRM store for the in-memory adapter.
_store: Dict[str, Dict[str, Any]] = {}
def __init__(
self,
*,
api_key: Optional[str] = None,
base_url: Optional[str] = None,
find_by_redmine_id: Optional[Callable[[int], Optional[Dict[str, Any]]]] = None,
find_by_email: Optional[Callable[[str], Optional[Dict[str, Any]]]] = None,
create: Optional[Callable[[Dict[str, Any]], str]] = None,
update: Optional[Callable[[str, Dict[str, Any]], None]] = None,
) -> None:
self.api_key = api_key
self.base_url = base_url
self._find_by_redmine_id_fn = find_by_redmine_id
self._find_by_email_fn = find_by_email
self._create_fn = create
self._update_fn = update
# --- Extensibility hooks (override in subclasses) -----------------------
def _find_by_redmine_id(self, redmine_id: int) -> Optional[Dict[str, Any]]:
if self._find_by_redmine_id_fn:
return self._find_by_redmine_id_fn(redmine_id)
for record in self._store.values():
if record.get("redmine_contact_id") == redmine_id:
return record
return None
def _find_by_email(self, email: str) -> Optional[Dict[str, Any]]:
if self._find_by_email_fn:
return self._find_by_email_fn(email)
normalized = email.lower().strip()
for record in self._store.values():
for entry in record.get("emails", []):
if entry.get("value", "").lower().strip() == normalized:
return record
return None
def _create(self, record: Dict[str, Any]) -> str:
if self._create_fn:
return self._create_fn(record)
crm_id = f"crm::{record['redmine_contact_id']}"
self._store[crm_id] = record.copy()
return crm_id
def _update(self, crm_id: str, record: Dict[str, Any]) -> None:
if self._update_fn:
self._update_fn(crm_id, record)
return
self._store[crm_id] = record.copy()
# --- Public API ---------------------------------------------------------
def _normalize_email(self, email: Optional[str]) -> Optional[str]:
if not email:
return None
return email.lower().strip()
def _has_changed(
self,
existing: Dict[str, Any],
candidate: Dict[str, Any],
) -> bool:
"""Return True if meaningful Redmine data changed."""
if not existing:
return True
candidate_ts = candidate.get("last_redmine_update")
existing_ts = existing.get("last_redmine_update")
if candidate_ts and existing_ts:
return candidate_ts != existing_ts
# Fallback to structural comparison for fields we care about.
keys = {
"full_name",
"company",
"title",
"emails",
"phones",
"urls",
"address",
"notes",
"tags",
"birthday",
}
for key in keys:
if existing.get(key) != candidate.get(key):
return True
return False
def upsert(self, record: Dict[str, Any]) -> UpsertResult:
"""Idempotently upsert a mapped Redmine contact into the CRM."""
redmine_id = record.get("redmine_contact_id")
if redmine_id is None:
raise CrmError("Record missing redmine_contact_id")
existing = self._find_by_redmine_id(redmine_id)
fallback_email: Optional[str] = None
if not existing and record.get("emails"):
fallback_email = self._normalize_email(record["emails"][0]["value"])
if fallback_email:
existing = self._find_by_email(fallback_email)
if existing:
crm_id = existing["crm_id"]
if not crm_id:
raise CrmError(
"Existing CRM record missing crm_id",
redmine_id=redmine_id,
)
if not self._has_changed(existing, record):
logger.info(
"CRM record %s unchanged for Redmine contact %s; skipping",
crm_id,
redmine_id,
)
return UpsertResult(
crm_id=crm_id,
created=False,
updated=False,
skipped_unchanged=True,
redmine_contact_id=redmine_id,
)
record["crm_id"] = crm_id
if fallback_email and not existing.get("redmine_contact_id"):
# If matched by email, attach Redmine ID to close the loop.
logger.info(
"Matched Redmine contact %s by email %s; updating CRM record %s",
redmine_id,
fallback_email,
crm_id,
)
self._update(crm_id, record)
return UpsertResult(
crm_id=crm_id,
created=False,
updated=True,
skipped_unchanged=False,
redmine_contact_id=redmine_id,
)
crm_id = self._create(record)
record["crm_id"] = crm_id
self._store[crm_id] = record.copy()
return UpsertResult(
crm_id=crm_id,
created=True,
updated=False,
skipped_unchanged=False,
redmine_contact_id=redmine_id,
)
def sync_batch(
self,
records: List[Dict[str, Any]],
) -> Dict[str, Any]:
"""Sync a batch and return aggregated stats."""
stats = {
"processed": 0,
"created": 0,
"updated": 0,
"skipped": 0,
"errors": 0,
"error_records": [],
}
for record in records:
stats["processed"] += 1
try:
result = self.upsert(record)
if result.created:
stats["created"] += 1
elif result.updated:
stats["updated"] += 1
else:
stats["skipped"] += 1
except CrmError as exc:
stats["errors"] += 1
stats["error_records"].append(
{
"redmine_contact_id": exc.redmine_id,
"crm_id": exc.crm_id,
"reason": str(exc),
}
)
logger.warning("CRM upsert failed: %s", exc)
logger.info("CRM sync batch stats: %s", stats)
return stats
redmine_crm/mapper.py
+310
View File
@@ -0,0 +1,310 @@
"""Map Redmine contact payloads to the target CRM schema and enrich missing data."""
from __future__ import annotations
import logging
import re
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Tuple
logger = logging.getLogger(__name__)
_EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
_PHONE_RE = re.compile(r"[\d+\-().\s]{7,}")
def _coalesce(*values: Any) -> Any:
"""Return the first non-empty value."""
for value in values:
if value:
return value
return None
def _to_iso(ts: Optional[str]) -> Optional[str]:
"""Normalize Redmine timestamps to ISO-8601 in UTC."""
if not ts:
return None
try:
# Redmine uses ISO-ish strings; this accepts several common formats.
parsed = datetime.fromisoformat(ts.replace("Z", "+00:00"))
parsed = parsed.astimezone(timezone.utc)
return parsed.isoformat()
except ValueError as exc:
logger.debug("Could not parse timestamp %r: %s", ts, exc)
return ts
def _normalize_name(
first: Optional[str],
last: Optional[str],
middle: Optional[str],
) -> Tuple[Optional[str], Optional[str], Optional[str]]:
first = (first or "").strip() or None
middle = (middle or "").strip() or None
last = (last or "").strip() or None
return first, middle, last
def _build_full_name(
first: Optional[str],
last: Optional[str],
middle: Optional[str],
company: Optional[str],
is_company: bool,
) -> str:
if is_company and company:
return company
parts = [p for p in (first, middle, last) if p]
if parts:
return " ".join(parts)
if company:
return company
return "Unknown"
def _classify_phone(raw: str) -> str:
raw = re.sub(r"[^\d+]", "", raw)
# Heuristic: mobile numbers are often longer and start with a country code or specific prefixes.
if raw.startswith("+") and len(raw) >= 11:
return "mobile"
# North American mobile indicators are unreliable; default to work.
return "work"
def _normalize_phone(raw: Optional[str]) -> Optional[str]:
if not raw:
return None
digits = re.sub(r"[^\d+]", "", raw)
return digits if digits else None
def _extract_emails_from_text(text: Optional[str]) -> List[str]:
if not text:
return []
return list(dict.fromkeys(_EMAIL_RE.findall(text)))
def _company_from_email(email: Optional[str]) -> Optional[str]:
"""Heuristic: derive a company display name from an email domain."""
if not email or "@" not in email:
return None
domain = email.split("@")[1].strip().lower()
if not domain:
return None
# Strip common TLDs and subdomain noise.
name = domain.split(":")[0]
tokens = name.split(".")
if tokens[0] in ("www", "mail", "smtp", "contact", "info"):
tokens = tokens[1:]
if len(tokens) >= 2:
tokens = tokens[:-1]
core = tokens[0] if tokens else domain.split(".")[0]
title = core.replace("-", " ").title()
if title.lower() in ("gmail", "yahoo", "hotmail", "outlook", "aol"):
return None
return f"{title} Inc"
def _parse_address(addr: Optional[Dict[str, Any]]) -> Dict[str, Optional[str]]:
if not addr:
return {
"street": None,
"city": None,
"region": None,
"country": None,
"postal_code": None,
}
return {
"street": _coalesce(addr.get("street")),
"city": _coalesce(addr.get("city")),
"region": _coalesce(addr.get("state")),
"country": _coalesce(addr.get("country")),
"postal_code": _coalesce(addr.get("zip")),
}
def _normalize_tags(tag_list: Any) -> List[str]:
tags: List[str] = []
if isinstance(tag_list, str):
tag_list = [t.strip() for t in tag_list.split(",")]
if isinstance(tag_list, list):
for tag in tag_list:
if tag:
normalized = str(tag).strip().lower()
if normalized and normalized not in tags:
tags.append(normalized)
# Always mark origin.
if "from-redmine" not in tags:
tags.append("from-redmine")
return sorted(tags)
def _extract_custom(
custom_fields: Optional[List[Dict[str, Any]]],
keys: List[str],
) -> Optional[str]:
"""Return the value of the first custom field matching one of keys (case-insensitive)."""
if not custom_fields:
return None
key_set = {k.lower() for k in keys}
for field in custom_fields:
name = (field.get("name") or "").strip().lower()
if name in key_set:
value = field.get("value")
if value:
return str(value).strip()
return None
def _merge_notes(
background: Optional[str],
custom_fields: Optional[List[Dict[str, Any]]],
contact_url: str,
) -> str:
parts = []
if background:
parts.append(str(background).strip())
if custom_fields:
for field in custom_fields:
name = field.get("name")
value = field.get("value")
if name and value:
# Skip fields already promoted to typed CRM fields (e.g. LinkedIn).
if name.lower() in {"linkedin"}:
continue
parts.append(f"{name}: {value}")
parts.append(f"Redmine source: {contact_url}")
return "\n\n".join(parts)
class EnrichmentResult:
"""Result of enrichment: data plus metadata."""
def __init__(self) -> None:
self.company_source: Optional[str] = None
self.email_source: Optional[str] = None
self.confidence_issues: List[str] = []
def enrich(
mapped: Dict[str, Any],
raw: Dict[str, Any],
*,
enable_domain_lookup: bool = True,
confidence_threshold: float = 0.7,
) -> Tuple[Dict[str, Any], EnrichmentResult]:
"""Apply enrichment rules to a mapped CRM record."""
result = EnrichmentResult()
# Email fallback: scan background and custom fields.
if not mapped.get("emails"):
candidates = _extract_emails_from_text(raw.get("background"))
if candidates:
mapped["emails"] = [{"value": candidates[0], "type": "work"}]
result.email_source = "background"
if not mapped.get("emails"):
result.confidence_issues.append("No email found; record may not be upsertable")
# Company enrichment.
if not mapped.get("company") and mapped.get("emails"):
email = mapped["emails"][0]["value"]
if enable_domain_lookup:
company = _company_from_email(email)
if company:
mapped["company"] = company
result.company_source = "domain_heuristic"
if not mapped.get("company"):
result.confidence_issues.append("Company unknown")
if mapped.get("company") and not mapped.get("emails"):
# Company but no email: confidence low.
result.confidence_issues.append("Company available but no email")
# Phone normalization.
phones = mapped.get("phones") or []
normalized_phones: List[Dict[str, str]] = []
seen: set[str] = set()
for phone in phones:
value = _normalize_phone(phone.get("value"))
if value and value not in seen:
seen.add(value)
normalized_phones.append({
"value": value,
"type": phone.get("type", "work"),
})
mapped["phones"] = normalized_phones
# Confidence gate: not a hard failure, but logged.
if result.confidence_issues:
logger.info(
"Enrichment confidence issues for Redmine contact %s: %s",
mapped.get("redmine_contact_id"),
"; ".join(result.confidence_issues),
)
mapped["enriched_at"] = datetime.now(timezone.utc).isoformat()
return mapped, result
def map_contact(raw: Dict[str, Any], *, base_url: str) -> Dict[str, Any]:
"""Map a Redmine contact payload to the normalized CRM schema."""
contact_id = raw.get("id")
first_name, middle_name, last_name = _normalize_name(
raw.get("first_name"),
raw.get("last_name"),
raw.get("middle_name"),
)
company = _coalesce(raw.get("company"))
is_company = bool(raw.get("is_company"))
full_name = _build_full_name(
first_name, last_name, middle_name, company, is_company
)
emails: List[Dict[str, str]] = []
primary_email = _coalesce(raw.get("email"))
if primary_email:
emails.append({"value": primary_email, "type": "work"})
phones: List[Dict[str, str]] = []
raw_phone = _coalesce(raw.get("phone"))
if raw_phone:
phones.append({"value": raw_phone, "type": _classify_phone(raw_phone)})
urls: List[Dict[str, str]] = []
website = _coalesce(raw.get("website"))
if website:
urls.append({"value": website, "type": "work"})
linkedin = _extract_custom(raw.get("custom_fields"), ["linkedin", "linked_in"])
if linkedin:
urls.append({"value": linkedin, "type": "linkedin"})
contact_url = f"{base_url.rstrip('/')}/contacts/{contact_id}"
return {
"crm_id": None,
"redmine_contact_id": contact_id,
"full_name": full_name,
"first_name": first_name,
"middle_name": middle_name,
"last_name": last_name,
"company": company,
"title": _coalesce(raw.get("job_title")),
"emails": emails,
"phones": phones,
"urls": urls,
"website": website,
"birthday": _to_iso(raw.get("birthday")),
"address": _parse_address(raw.get("address_attributes")),
"notes": _merge_notes(raw.get("background"), raw.get("custom_fields"), contact_url),
"tags": _normalize_tags(raw.get("tag_list")),
"source": "redmine",
"synced_at": datetime.now(timezone.utc).isoformat(),
"enriched_at": None,
"last_redmine_update": _to_iso(raw.get("updated_at")),
"is_company": is_company,
"project_id": raw.get("project_id"),
"assigned_to_id": raw.get("assigned_to_id"),
}
redmine_crm/redmine_client.py
+225
View File
@@ -0,0 +1,225 @@
"""Redmine Contacts REST API client with retry, auth, and logging."""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass
from http import HTTPStatus
from typing import Any, Dict, List, Optional
import requests
logger = logging.getLogger(__name__)
class RedmineError(Exception):
"""Base exception for Redmine client errors."""
def __init__(self, message: str, *, redmine_id: Optional[int] = None) -> None:
super().__init__(message)
self.redmine_id = redmine_id
class RedmineAuthError(RedmineError):
"""Permanent auth failure (401, 403)."""
class RedmineTransientError(RedmineError):
"""Retryable error (network, 5xx, rate limit)."""
class RedmineNotFoundError(RedmineError):
"""Resource not found (404)."""
@dataclass(frozen=True, slots=True)
class RetryConfig:
max_attempts: int = 5
base_delay_seconds: float = 1.0
max_delay_seconds: float = 30.0
class RedmineClient:
"""Thin, retry-aware wrapper around Redmine's REST API."""
def __init__(
self,
base_url: str,
api_key: str,
*,
project_id: Optional[int] = None,
timeout: float = 30.0,
retry: Optional[RetryConfig] = None,
) -> None:
self.base_url = base_url.rstrip("/")
self.api_key = api_key
self.project_id = project_id
self.timeout = timeout
self.retry = retry or RetryConfig()
self.session = requests.Session()
self.session.headers.update({
"X-Redmine-API-Key": self.api_key,
"Content-Type": "application/json",
"Accept": "application/json",
})
def _request(
self,
method: str,
path: str,
*,
params: Optional[Dict[str, Any]] = None,
json_body: Optional[Dict[str, Any]] = None,
redmine_id: Optional[int] = None,
) -> Dict[str, Any]:
url = f"{self.base_url}{path}"
last_error: Optional[Exception] = None
for attempt in range(1, self.retry.max_attempts + 1):
transient_error: Optional[RedmineTransientError] = None
try:
response = self.session.request(
method,
url,
params=params,
json=json_body,
timeout=self.timeout,
)
except requests.RequestException as exc:
last_error = exc
logger.warning(
"Redmine network error on attempt %d/%d: %s",
attempt,
self.retry.max_attempts,
exc,
)
if attempt == self.retry.max_attempts:
raise RedmineTransientError(
f"Network error after {self.retry.max_attempts} attempts: {exc}",
redmine_id=redmine_id,
) from exc
else:
logger.debug(
"Redmine request %s %s attempt %d/%d status=%d",
method,
url,
attempt,
self.retry.max_attempts,
response.status_code,
)
if response.status_code == HTTPStatus.OK.value:
if not response.content:
return {}
return response.json()
if response.status_code in (
HTTPStatus.UNAUTHORIZED.value,
HTTPStatus.FORBIDDEN.value,
):
raise RedmineAuthError(
f"Authentication failed ({response.status_code})",
redmine_id=redmine_id,
)
if response.status_code == HTTPStatus.NOT_FOUND.value:
raise RedmineNotFoundError(
f"Not found ({response.status_code})",
redmine_id=redmine_id,
)
if response.status_code == HTTPStatus.TOO_MANY_REQUESTS.value:
transient_error = RedmineTransientError(
"Rate limited (429)", redmine_id=redmine_id
)
elif response.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR.value:
transient_error = RedmineTransientError(
f"Server error ({response.status_code})",
redmine_id=redmine_id,
)
else:
# 4xx other than auth/404 are permanent.
raise RedmineError(
f"Client error ({response.status_code}): {response.text}",
redmine_id=redmine_id,
)
if transient_error is not None:
if attempt == self.retry.max_attempts:
raise transient_error
logger.warning(
"Redmine transient error on attempt %d/%d: %s",
attempt,
self.retry.max_attempts,
transient_error,
)
if attempt < self.retry.max_attempts:
delay = min(
self.retry.base_delay_seconds * (2 ** (attempt - 1)),
self.retry.max_delay_seconds,
)
logger.info("Retrying Redmine request in %.1f seconds", delay)
time.sleep(delay)
# Should never be reached because the loop raises.
raise RedmineTransientError("Unexpected loop exit", redmine_id=redmine_id) from last_error
def fetch_contacts(
self,
*,
updated_since: Optional[str] = None,
limit: int = 100,
offset: int = 0,
status_filter: Optional[str] = "active",
) -> List[Dict[str, Any]]:
params: Dict[str, Any] = {"limit": limit, "offset": offset}
if self.project_id is not None:
params["project_id"] = self.project_id
if updated_since:
params["updated_on"] = f">={updated_since}"
if status_filter:
params["status"] = status_filter
data = self._request("GET", "/contacts.json", params=params)
contacts = data.get("contacts", [])
logger.info(
"Fetched %d contacts from Redmine (limit=%d, offset=%d)",
len(contacts),
limit,
offset,
)
return contacts
def fetch_contact(self, contact_id: int) -> Dict[str, Any]:
data = self._request(
"GET",
f"/contacts/{contact_id}.json",
redmine_id=contact_id,
)
return data.get("contact", data)
def iter_all_contacts(
self,
*,
updated_since: Optional[str] = None,
batch_size: int = 100,
) -> List[Dict[str, Any]]:
"""Iterate through paginated contacts and collect all results."""
all_contacts: List[Dict[str, Any]] = []
offset = 0
while True:
batch = self.fetch_contacts(
updated_since=updated_since,
limit=batch_size,
offset=offset,
)
if not batch:
break
all_contacts.extend(batch)
if len(batch) < batch_size:
break
offset += batch_size
return all_contacts
redmine_crm/sync.py
+65
View File
@@ -0,0 +1,65 @@
"""High-level Redmine -> CRM sync orchestrator."""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
from redmine_crm.crm_client import CrmClient
from redmine_crm.mapper import enrich, map_contact
from redmine_crm.redmine_client import RedmineClient
logger = logging.getLogger(__name__)
class SyncState:
"""Minimal sync state tracker."""
def __init__(self, storage: Optional[Dict[str, Any]] = None) -> None:
self._data = storage if storage is not None else {}
def get_last_sync(self) -> Optional[str]:
return self._data.get("last_sync_timestamp")
def update(self, timestamp: str, stats: Dict[str, Any]) -> None:
self._data["last_sync_timestamp"] = timestamp
self._data["last_run_stats"] = stats
def run_sync(
redmine: RedmineClient,
crm: CrmClient,
sync_state: SyncState,
*,
base_url: str,
batch_size: int = 100,
) -> Dict[str, Any]:
"""Run the full ETL pipeline: Extract, Transform, Load."""
last_sync = sync_state.get_last_sync()
logger.info("Starting Redmine -> CRM sync; last_sync=%s", last_sync)
# Extract
contacts = redmine.iter_all_contacts(
updated_since=last_sync,
batch_size=batch_size,
)
logger.info("Extracted %d Redmine contacts", len(contacts))
# Transform
records: List[Dict[str, Any]] = []
for contact in contacts:
try:
mapped = map_contact(contact, base_url=base_url)
enriched, _enrich_meta = enrich(mapped, contact)
records.append(enriched)
except Exception as exc:
logger.exception("Failed to map Redmine contact %s: %s", contact.get("id"), exc)
# Load
stats = crm.sync_batch(records)
logger.info("CRM upsert stats: %s", stats)
import datetime
sync_state.update(datetime.datetime.now(datetime.timezone.utc).isoformat(), stats)
return stats
redmine_ticket_attachments.py
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""Attachment helpers for the power-user Redmine ticket client.
List (cache-first), download to local dir, upload via Redmine REST:
POST /uploads.json?filename=… (raw body)
PUT /issues/:id.json with issue.uploads token
"""
from __future__ import annotations
import json
import mimetypes
import re
from pathlib import Path
from typing import Any
from urllib import error as urllib_error
from urllib import parse as urllib_parse
from urllib import request as urllib_request
def safe_filename(name: str) -> str:
name = (name or "attachment.bin").strip().replace("\\", "_").replace("/", "_")
name = re.sub(r"[^\w.\- ()\[\]+@]+", "_", name)
return name or "attachment.bin"
def attachments_from_issue(issue: dict[str, Any]) -> list[dict[str, Any]]:
atts = issue.get("attachments")
if isinstance(atts, list):
return [a for a in atts if isinstance(a, dict)]
return []
def attachment_dir(cache_dir: Path, project: str, issue_id: int) -> Path:
path = Path(cache_dir) / project / "attachments" / str(int(issue_id))
path.mkdir(parents=True, exist_ok=True)
return path
def pick_attachment(attachments: list[dict[str, Any]], attachment_id: int | None) -> dict[str, Any]:
if not attachments:
raise LookupError("Issue has no attachments.")
if attachment_id is None:
if len(attachments) == 1:
return attachments[0]
raise LookupError(
"Multiple attachments; pass --attachment-id. ids="
+ ",".join(str(a.get("id")) for a in attachments)
)
for att in attachments:
if int(att.get("id") or 0) == int(attachment_id):
return att
raise LookupError(f"Attachment id {attachment_id} not found on issue.")
def download_url_for_attachment(att: dict[str, Any], base_url: str) -> str:
urls = download_url_candidates(att, base_url)
return urls[0]
def download_url_candidates(att: dict[str, Any], base_url: str) -> list[str]:
"""Ordered download URL candidates (host rewrite + path variants)."""
base = base_url.rstrip("/")
aid = att.get("id")
filename = str(att.get("filename") or "").strip()
candidates: list[str] = []
def add(url: str) -> None:
if url and url not in candidates:
candidates.append(url)
for key in ("content_url", "contentUrl"):
raw = att.get(key)
if not raw:
continue
raw = str(raw)
add(raw)
# rewrite host to active base_url (prod content_url on LAN client, etc.)
try:
from urllib.parse import urlparse, urlunparse
parsed = urlparse(raw)
base_p = urlparse(base if "://" in base else f"http://{base}")
rewritten = urlunparse(
(
base_p.scheme or parsed.scheme,
base_p.netloc or parsed.netloc,
parsed.path,
"",
parsed.query,
"",
)
)
add(rewritten)
except Exception:
pass
if aid is not None:
add(f"{base}/attachments/download/{int(aid)}")
if filename:
from urllib.parse import quote
add(f"{base}/attachments/download/{int(aid)}/{quote(filename)}")
if not candidates:
raise LookupError("Attachment missing id and content_url.")
return candidates
def download_attachment_bytes(
*,
url: str,
api_key: str,
impersonate_user: str | None = None,
timeout: int = 120,
fallback_urls: list[str] | None = None,
) -> bytes:
headers = {
"X-Redmine-API-Key": api_key,
"Accept": "*/*",
}
if impersonate_user:
headers["X-Redmine-Switch-User"] = impersonate_user
urls = [url] + [u for u in (fallback_urls or []) if u != url]
last_err: Exception | None = None
for u in urls:
req = urllib_request.Request(u, headers=headers, method="GET")
try:
with urllib_request.urlopen(req, timeout=timeout) as resp:
return resp.read()
except urllib_error.HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")[:200]
last_err = RuntimeError(f"HTTP {exc.code} downloading attachment ({u}): {detail}")
continue
except urllib_error.URLError as exc:
last_err = RuntimeError(f"Could not download attachment ({u}): {exc.reason}")
continue
raise last_err or RuntimeError("Attachment download failed")
def upload_file_token(
*,
base_url: str,
api_key: str,
path: Path,
filename: str | None = None,
impersonate_user: str | None = None,
timeout: int = 120,
) -> dict[str, Any]:
"""POST raw file to /uploads.json?filename=… Returns upload object with token."""
path = Path(path)
if not path.is_file():
raise FileNotFoundError(f"File not found: {path}")
name = filename or path.name
data = path.read_bytes()
q = urllib_parse.urlencode({"filename": name})
url = f"{base_url.rstrip('/')}/uploads.json?{q}"
headers = {
"X-Redmine-API-Key": api_key,
"Content-Type": "application/octet-stream",
"Accept": "application/json",
}
if impersonate_user:
headers["X-Redmine-Switch-User"] = impersonate_user
req = urllib_request.Request(url, data=data, headers=headers, method="POST")
try:
with urllib_request.urlopen(req, timeout=timeout) as resp:
raw = resp.read()
except urllib_error.HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")[:500]
raise RuntimeError(f"HTTP {exc.code} uploading file: {detail}") from exc
except urllib_error.URLError as exc:
raise RuntimeError(f"Could not upload file: {exc.reason}") from exc
if not raw:
raise RuntimeError("Empty response from /uploads.json")
payload = json.loads(raw.decode("utf-8"))
upload = payload.get("upload") if isinstance(payload, dict) else None
if not isinstance(upload, dict) or not upload.get("token"):
raise RuntimeError(f"Unexpected upload response: {payload!r}")
content_type = mimetypes.guess_type(name)[0] or "application/octet-stream"
upload.setdefault("filename", name)
upload.setdefault("content_type", content_type)
return upload
redmine_ticket_browse.py
+312
View File
@@ -0,0 +1,312 @@
#!/usr/bin/env python3
"""Interactive browse loop: fzf → pager → action menu → back to fzf.
Flow:
1. Present filtered tickets in fzf (or numbered list fallback)
2. On select: open ticket Markdown in glow/bat/less
3. After pager exits: action menu for that ticket
4. Return to list until user quits (Esc/q)
Actions queue local drafts by default (same safety model as CLI); optional
apply is left to `draft apply` so browse never silently emails/writes prod.
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Any, Callable
def nested_name(issue: dict[str, Any], key: str) -> str:
obj = issue.get(key)
if isinstance(obj, dict):
return str(obj.get("name") or "")
return ""
def issue_line(issue: dict[str, Any]) -> str:
iid = issue.get("id")
status = nested_name(issue, "status") or ""
assignee = nested_name(issue, "assigned_to") or ""
subj = str(issue.get("subject") or "").replace("\n", " ").replace("\t", " ")[:90]
tags = issue.get("_control_tags")
tag_s = ",".join(tags) if isinstance(tags, list) and tags else ""
return f"{iid}\t{status}\t{assignee}\t{tag_s}\t{subj}"
def build_lines(issues: list[dict[str, Any]]) -> list[str]:
return [issue_line(i) for i in issues if i.get("id") is not None]
def parse_id_from_line(line: str) -> int | None:
line = (line or "").strip()
if not line:
return None
head = line.split("\t", 1)[0].strip()
try:
return int(head)
except ValueError:
return None
# preview helper written at runtime under /tmp by pick_with_fzf
def pick_with_fzf(
lines: list[str],
*,
query: str | None = None,
cache_dir: Path | None = None,
project: str | None = None,
preview: bool = True,
) -> int | None:
fzf = shutil.which("fzf")
if not fzf:
return None
header = (
"Enter=open · Esc=quit · after open: n/p note · f fields · t tag · "
"c contact · e entities · m NMI brief · i id · l list · q quit"
)
cmd = [
fzf,
"--height=90%",
"--reverse",
"--prompt=ticket> ",
f"--header={header}",
"--info=inline",
"--bind=ctrl-c:abort",
]
preview_script: Path | None = None
if preview and cache_dir is not None and project:
# Write a tiny preview helper so quoting stays sane
preview_script = Path(tempfile.gettempdir()) / "rtl_browse_preview.py"
preview_script.write_text(
"#!/usr/bin/env python3\n"
"import json, pathlib, sys\n"
f"cache = pathlib.Path({str(Path(cache_dir))!r})\n"
f"proj = {project!r}\n"
"line = sys.argv[1] if len(sys.argv) > 1 else ''\n"
"iid = line.split('\\t')[0].strip()\n"
"md = cache / proj / 'markdown' / f'{iid}.md'\n"
"jp = cache / proj / f'{iid}.json'\n"
"print(f'# ticket {iid}')\n"
"print()\n"
"if md.is_file():\n"
" print(md.read_text(encoding='utf-8')[:3500])\n"
"elif jp.is_file():\n"
" d = json.loads(jp.read_text(encoding='utf-8'))\n"
" print(d.get('subject') or '')\n"
" print()\n"
" print(str(d.get('description') or '')[:2500])\n"
"else:\n"
" print('(no preview)')\n",
encoding="utf-8",
)
cmd.extend(
[
"--preview",
f"python3 {preview_script} {{}}",
"--preview-window=right,50%,wrap",
]
)
if query:
cmd.extend(["--query", query])
try:
proc = subprocess.run(
cmd,
input="\n".join(lines) + "\n",
text=True,
capture_output=True,
)
finally:
# leave preview script; cheap and reusable
pass
if proc.returncode != 0 or not proc.stdout.strip():
return None
return parse_id_from_line(proc.stdout.strip().splitlines()[-1])
def pick_numbered(lines: list[str]) -> int | None:
for i, line in enumerate(lines, 1):
print(f"{i:3d}. {line}")
if not sys.stdin.isatty():
return None
raw = input("Pick # (or q to quit list): ").strip()
if raw.lower() in {"q", "quit", ""}:
return None
try:
idx = int(raw)
except ValueError:
print(f"Invalid selection: {raw!r}", file=sys.stderr)
return None
if idx < 1 or idx > len(lines):
print("Out of range.", file=sys.stderr)
return None
return parse_id_from_line(lines[idx - 1])
def page_ticket_markdown(
issue: dict[str, Any],
*,
cache_dir: Path,
project: str,
pager: str = "auto",
) -> None:
from redmine_ticket_view import display_markdown_file, render_ticket_markdown, write_ticket_markdown
try:
path = write_ticket_markdown(issue, Path(cache_dir), project)
display_markdown_file(path, pager=pager)
except Exception:
md = render_ticket_markdown(issue)
with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8") as fh:
fh.write(md)
tmp = Path(fh.name)
try:
display_markdown_file(tmp, pager=pager)
finally:
try:
tmp.unlink(missing_ok=True)
except OSError:
pass
def prompt_multiline(label: str) -> str | None:
print(f"{label} (end with a single '.' line, or empty first line to cancel):")
lines: list[str] = []
while True:
try:
line = input()
except EOFError:
break
if line == ".":
break
if not lines and line.strip() == "":
return None
lines.append(line)
text = "\n".join(lines).strip()
return text or None
def announce_draft(path: Path) -> None:
"""Louder draft feedback for browse UX."""
draft_id = path.stem
print()
print("=" * 48)
print(f" DRAFT QUEUED draft_id={draft_id}")
print(f" path={path}")
print(" apply later: draft apply " + draft_id)
print(" (browse never auto-applies / never emails customers)")
print("=" * 48)
print()
def action_menu(
issue_id: int,
issue: dict[str, Any],
*,
handlers: dict[str, Callable[[], None]],
) -> str:
"""Show action menu. Returns 'list' | 'quit' | 'view' | 'menu'."""
subj = str(issue.get("subject") or "")[:70]
print()
print(f"── ticket #{issue_id} ── {subj}")
print(" [Enter/v] view again")
print(" [n] queue internal note (no email)")
print(" [p] queue private internal note")
print(" [f] queue field update (status/assignee name)")
print(" [t] local control-tag add")
print(" [c] show ticket-contact handoff")
print(" [e] extract entities (patterns)")
print(" [m] NMI brief → queue as PRIVATE note draft")
print(" [i] print issue id only")
print(" [l] back to list")
print(" [q] quit browse")
raw = input("action> ").strip().lower()
if raw in {"", "v", "view"}:
return "view"
if raw in {"l", "list", "b", "back"}:
return "list"
if raw in {"q", "quit"}:
return "quit"
if raw in handlers:
try:
handlers[raw]()
except Exception as exc: # noqa: BLE001 — surface in TUI
print(f"action failed: {exc}", file=sys.stderr)
input("press Enter to continue…")
return "menu"
print(f"Unknown action: {raw!r}", file=sys.stderr)
return "menu"
def run_browse_loop(
issues: list[dict[str, Any]],
*,
cache_dir: Path,
project: str,
pager: str = "auto",
once: bool = False,
preview: bool = True,
action_factory: Callable[[int, dict[str, Any]], dict[str, Callable[[], None]]] | None = None,
) -> int:
"""Main browse loop. Returns process exit code."""
if not issues:
print("No matching tickets.", file=sys.stderr)
return 0
lines = build_lines(issues)
by_id = {int(i["id"]): i for i in issues if i.get("id") is not None}
use_fzf = bool(shutil.which("fzf") and sys.stdin.isatty())
if not sys.stdin.isatty():
for line in lines:
print(line)
return 0
while True:
if use_fzf:
selected = pick_with_fzf(
lines,
cache_dir=Path(cache_dir),
project=project,
preview=preview,
)
else:
selected = pick_numbered(lines)
if selected is None:
print("browse done", file=sys.stderr)
return 0
issue = by_id.get(selected)
if issue is None:
print(f"#{selected} not in current result set", file=sys.stderr)
if once:
return 1
continue
page_ticket_markdown(issue, cache_dir=cache_dir, project=project, pager=pager)
if once:
return 0
while True:
handlers: dict[str, Callable[[], None]] = {}
if action_factory:
handlers = action_factory(selected, issue)
choice = action_menu(selected, issue, handlers=handlers)
if choice == "view":
page_ticket_markdown(issue, cache_dir=cache_dir, project=project, pager=pager)
continue
if choice == "list":
break
if choice == "quit":
print("browse done", file=sys.stderr)
return 0
continue
redmine_ticket_contacts.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python3
"""Durable local contact cache for the power-user Redmine client.
Reuses RedmineClient from redmine_contacts.py. Contacts live under:
<cache-dir>/contacts/<id>.json
<cache-dir>/contacts/manifest.json
This is intentionally thinner than redmine_contacts.py's full CRM CLI —
enough for sync + local search + show next to tickets.
"""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Iterable
def now_iso() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
class ContactCache:
def __init__(self, cache_dir: Path):
self.cache_dir = Path(cache_dir)
self.contacts_dir = self.cache_dir / "contacts"
self.contacts_dir.mkdir(parents=True, exist_ok=True)
self.manifest_path = self.contacts_dir / "manifest.json"
def contact_path(self, contact_id: int | str) -> Path:
return self.contacts_dir / f"{int(contact_id)}.json"
def load_manifest(self) -> dict[str, Any]:
if not self.manifest_path.is_file():
return {"version": 1, "contacts": {}, "updated_at": None}
try:
data = json.loads(self.manifest_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {"version": 1, "contacts": {}, "updated_at": None}
if not isinstance(data, dict):
return {"version": 1, "contacts": {}, "updated_at": None}
data.setdefault("version", 1)
data.setdefault("contacts", {})
return data
def save_manifest(self, data: dict[str, Any]) -> None:
data = dict(data)
data["version"] = 1
data["updated_at"] = now_iso()
tmp = self.manifest_path.with_suffix(".tmp")
tmp.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
tmp.replace(self.manifest_path)
def should_fetch(self, stub: dict[str, Any]) -> bool:
cid = stub.get("id")
if cid is None:
return False
path = self.contact_path(int(cid))
if not path.is_file():
return True
manifest = self.load_manifest()
entry = (manifest.get("contacts") or {}).get(str(int(cid))) or {}
return str(entry.get("updated_on") or "") != str(stub.get("updated_on") or "")
def write_contact(self, contact: dict[str, Any]) -> Path:
cid = contact.get("id")
if cid is None:
raise ValueError("contact missing id")
path = self.contact_path(int(cid))
path.write_text(json.dumps(contact, indent=2, sort_keys=True) + "\n", encoding="utf-8")
manifest = self.load_manifest()
contacts = manifest.setdefault("contacts", {})
contacts[str(int(cid))] = {
"id": int(cid),
"updated_on": contact.get("updated_on"),
"last_synced": now_iso(),
"email": primary_email(contact),
"name": contact_display_name(contact),
}
self.save_manifest(manifest)
return path
def load_contact(self, contact_id: int) -> dict[str, Any]:
path = self.contact_path(contact_id)
if not path.is_file():
raise FileNotFoundError(f"No cached contact #{contact_id} at {path}")
return json.loads(path.read_text(encoding="utf-8"))
def iter_contacts(self) -> Iterable[dict[str, Any]]:
for path in sorted(self.contacts_dir.glob("*.json")):
if path.name == "manifest.json":
continue
try:
yield json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
continue
def primary_email(contact: dict[str, Any]) -> str:
for key in ("email", "primary_email"):
val = contact.get(key)
if val:
return str(val)
emails = contact.get("emails") or contact.get("address")
if isinstance(emails, list) and emails:
first = emails[0]
if isinstance(first, dict):
return str(first.get("address") or first.get("email") or "")
return str(first)
return ""
def contact_display_name(contact: dict[str, Any]) -> str:
if contact.get("name"):
return str(contact["name"])
first = contact.get("first_name") or contact.get("firstname") or ""
last = contact.get("last_name") or contact.get("lastname") or ""
full = f"{first} {last}".strip()
return full or primary_email(contact) or str(contact.get("id") or "")
def contact_id_from_issue(issue: dict[str, Any]) -> int | None:
"""Extract helpdesk contact id from a cached/enriched issue payload."""
helpdesk = issue.get("helpdesk_ticket")
if not isinstance(helpdesk, dict):
return None
if helpdesk.get("contact_id") is not None:
try:
return int(helpdesk["contact_id"])
except (TypeError, ValueError):
pass
contact = helpdesk.get("contact")
if isinstance(contact, dict) and contact.get("id") is not None:
try:
return int(contact["id"])
except (TypeError, ValueError):
return None
return None
def contact_text_blob(contact: dict[str, Any]) -> str:
parts = [
contact_display_name(contact),
primary_email(contact),
str(contact.get("company") or ""),
str(contact.get("tag_list") or contact.get("tags") or ""),
]
return " ".join(parts).lower()
def filter_contacts(
contacts: list[dict[str, Any]],
*,
query: str | None = None,
email: str | None = None,
company: str | None = None,
tag: str | None = None,
) -> list[dict[str, Any]]:
out = contacts
if email:
e = email.lower()
out = [c for c in out if e in primary_email(c).lower()]
if company:
co = company.lower()
out = [c for c in out if co in str(c.get("company") or "").lower()]
if tag:
t = tag.lower()
def has_tag(c: dict[str, Any]) -> bool:
raw = c.get("tag_list") or c.get("tags") or []
if isinstance(raw, str):
tags = [x.strip().lower() for x in raw.split(",")]
elif isinstance(raw, list):
tags = []
for item in raw:
if isinstance(item, dict):
tags.append(str(item.get("name") or "").lower())
else:
tags.append(str(item).lower())
else:
tags = []
return t in tags
out = [c for c in out if has_tag(c)]
if query:
q = query.lower()
out = [c for c in out if q in contact_text_blob(c)]
return out
def sync_contacts(
client: Any,
cache: ContactCache,
*,
page_size: int = 100,
max_contacts: int | None = None,
progress: bool = True,
dry_run: bool = False,
) -> dict[str, int]:
"""Fetch project contacts and write changed ones into the local cache."""
if progress:
print("Fetching contact list…", flush=True)
stubs = client.fetch_contacts(limit=page_size)
if max_contacts is not None:
stubs = stubs[: int(max_contacts)]
result = {"seen": len(stubs), "fetched": 0, "skipped_cached": 0, "written": 0}
for index, stub in enumerate(stubs, start=1):
cid = stub.get("id")
if cid is None:
continue
if not cache.should_fetch(stub):
result["skipped_cached"] += 1
if progress and (index == 1 or index % 50 == 0 or index == len(stubs)):
print(f" [{index}/{len(stubs)}] #{cid} skip-cached", flush=True)
continue
result["fetched"] += 1
if dry_run:
if progress:
print(f" [{index}/{len(stubs)}] #{cid} would-fetch", flush=True)
continue
# Prefer full contact payload when list stub is thin
try:
full = client.get_contact(int(cid))
except Exception:
full = stub
cache.write_contact(full)
result["written"] += 1
if progress and (index == 1 or index % 25 == 0 or index == len(stubs)):
print(f" [{index}/{len(stubs)}] #{cid} written", flush=True)
if progress:
print(
f"contacts sync done: seen={result['seen']} fetched={result['fetched']} "
f"skipped={result['skipped_cached']} written={result['written']}",
flush=True,
)
return result
redmine_ticket_control_export.py
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""Training/export helpers for local control tags."""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any, Iterable
def export_control_tag_jsonl(
*,
cache_dir: Path,
tags_payload: dict[str, Any],
issues_by_id: dict[int, dict[str, Any]],
tag: str | None = None,
out_path: Path,
) -> int:
"""Write one JSONL record per tagged ticket for later LLM refine / training.
Each line: {tag, issue_id, subject, status, assignee, contact_email,
properties, annotation, text_snippet}
"""
out_path = Path(out_path)
out_path.parent.mkdir(parents=True, exist_ok=True)
tags = tags_payload.get("tags") if isinstance(tags_payload, dict) else {}
if not isinstance(tags, dict):
tags = {}
names = [tag] if tag else sorted(tags.keys())
count = 0
with out_path.open("w", encoding="utf-8") as fh:
for name in names:
bucket = tags.get(name) or {}
tickets = bucket.get("tickets") if isinstance(bucket, dict) else {}
if not isinstance(tickets, dict):
continue
for iid_s, meta in tickets.items():
try:
iid = int(iid_s)
except (TypeError, ValueError):
continue
issue = issues_by_id.get(iid) or {}
helpdesk_raw = issue.get("helpdesk_ticket")
helpdesk: dict[str, Any] = helpdesk_raw if isinstance(helpdesk_raw, dict) else {}
contact_raw = helpdesk.get("contact")
contact: dict[str, Any] = contact_raw if isinstance(contact_raw, dict) else {}
status_raw = issue.get("status")
status_obj: dict[str, Any] = status_raw if isinstance(status_raw, dict) else {}
assignee_raw = issue.get("assigned_to")
assignee_obj: dict[str, Any] = assignee_raw if isinstance(assignee_raw, dict) else {}
meta_d = meta if isinstance(meta, dict) else {}
rec = {
"tag": name,
"issue_id": iid,
"subject": issue.get("subject"),
"status": status_obj.get("name"),
"assignee": assignee_obj.get("name"),
"contact_email": contact.get("email") or helpdesk.get("from_address"),
"properties": meta_d.get("properties") or [],
"annotation": meta_d.get("annotation") or "",
"text_snippet": (
f"{issue.get('subject') or ''}\n{(issue.get('description') or '')[:500]}"
).strip(),
}
fh.write(json.dumps(rec, sort_keys=True) + "\n")
count += 1
return count
redmine_ticket_control_tags.py
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""Local-only control tags for the power-user Redmine client.
These are NOT Redmine issue tags and NOT CRM/contact tags.
Stored under <cache-dir>/control-tags/ so they can be wiped without
touching Redmine or ticket JSON.
"""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
def now_iso() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
def control_tags_root(cache_dir: Path) -> Path:
path = Path(cache_dir) / "control-tags"
path.mkdir(parents=True, exist_ok=True)
return path
def store_path(cache_dir: Path) -> Path:
return control_tags_root(cache_dir) / "tags.json"
def load_store(cache_dir: Path) -> dict[str, Any]:
path = store_path(cache_dir)
if not path.is_file():
return {"version": 1, "tags": {}}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {"version": 1, "tags": {}}
if not isinstance(data, dict):
return {"version": 1, "tags": {}}
tags = data.get("tags")
if not isinstance(tags, dict):
data["tags"] = {}
data.setdefault("version", 1)
return data
def save_store(cache_dir: Path, data: dict[str, Any]) -> Path:
path = store_path(cache_dir)
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
payload = {
"version": int(data.get("version") or 1),
"updated_at": now_iso(),
"tags": data.get("tags") or {},
}
tmp.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
tmp.replace(path)
return path
def normalize_tag(name: str) -> str:
tag = (name or "").strip().lower()
if not tag:
raise ValueError("Control tag name is required.")
if any(ch.isspace() for ch in tag):
raise ValueError(f"Control tag must not contain spaces: {name!r} (use hyphens).")
return tag
def add_tag(
cache_dir: Path,
*,
tag: str,
issue_id: int,
properties: list[str] | None = None,
annotation: str | None = None,
) -> dict[str, Any]:
tag = normalize_tag(tag)
data = load_store(cache_dir)
tags = data.setdefault("tags", {})
entry = tags.setdefault(tag, {"tickets": {}})
tickets = entry.setdefault("tickets", {})
key = str(int(issue_id))
existing = tickets.get(key) if isinstance(tickets.get(key), dict) else {}
props = list(properties or existing.get("properties") or [])
# de-dupe preserve order
seen: set[str] = set()
clean_props: list[str] = []
for p in props:
p = str(p).strip()
if p and p not in seen:
seen.add(p)
clean_props.append(p)
tickets[key] = {
"issue_id": int(issue_id),
"properties": clean_props,
"annotation": annotation if annotation is not None else existing.get("annotation") or "",
"tagged_at": existing.get("tagged_at") or now_iso(),
"updated_at": now_iso(),
}
save_store(cache_dir, data)
return tickets[key]
def remove_tag(cache_dir: Path, *, tag: str, issue_id: int) -> bool:
tag = normalize_tag(tag)
data = load_store(cache_dir)
tags = data.get("tags") or {}
entry = tags.get(tag)
if not isinstance(entry, dict):
return False
tickets = entry.get("tickets") or {}
key = str(int(issue_id))
if key not in tickets:
return False
del tickets[key]
if not tickets:
# keep empty tag shell for discovery, or drop — drop for cleanliness
del tags[tag]
save_store(cache_dir, data)
return True
def list_tags(cache_dir: Path) -> list[dict[str, Any]]:
data = load_store(cache_dir)
out: list[dict[str, Any]] = []
for name, entry in sorted((data.get("tags") or {}).items()):
tickets = (entry or {}).get("tickets") or {}
out.append(
{
"tag": name,
"ticket_count": len(tickets) if isinstance(tickets, dict) else 0,
"issue_ids": sorted(int(k) for k in tickets.keys()) if isinstance(tickets, dict) else [],
}
)
return out
def tickets_for_tag(cache_dir: Path, tag: str) -> list[int]:
tag = normalize_tag(tag)
data = load_store(cache_dir)
entry = (data.get("tags") or {}).get(tag) or {}
tickets = entry.get("tickets") or {}
if not isinstance(tickets, dict):
return []
return sorted(int(k) for k in tickets.keys())
def tags_for_issue(cache_dir: Path, issue_id: int) -> list[str]:
key = str(int(issue_id))
data = load_store(cache_dir)
found: list[str] = []
for name, entry in (data.get("tags") or {}).items():
tickets = (entry or {}).get("tickets") or {}
if isinstance(tickets, dict) and key in tickets:
found.append(name)
return sorted(found)
def issue_tag_detail(cache_dir: Path, tag: str, issue_id: int) -> dict[str, Any] | None:
tag = normalize_tag(tag)
data = load_store(cache_dir)
entry = (data.get("tags") or {}).get(tag) or {}
tickets = entry.get("tickets") or {}
if not isinstance(tickets, dict):
return None
detail = tickets.get(str(int(issue_id)))
return detail if isinstance(detail, dict) else None
def wipe_all(cache_dir: Path) -> Path:
"""Delete the entire control-tag store (does not touch ticket cache)."""
path = store_path(cache_dir)
if path.is_file():
path.unlink()
return path
def filter_issues_by_control_tag(
issues: list[dict[str, Any]],
cache_dir: Path,
tag: str,
) -> list[dict[str, Any]]:
allowed = set(tickets_for_tag(cache_dir, tag))
return [i for i in issues if i.get("id") is not None and int(i["id"]) in allowed]
def annotate_issues_with_control_tags(
issues: list[dict[str, Any]],
cache_dir: Path,
) -> list[dict[str, Any]]:
"""Attach _control_tags list onto issue dicts (local only, not persisted to ticket JSON)."""
data = load_store(cache_dir)
by_issue: dict[int, list[str]] = {}
for name, entry in (data.get("tags") or {}).items():
tickets = (entry or {}).get("tickets") or {}
if not isinstance(tickets, dict):
continue
for key in tickets:
try:
iid = int(key)
except (TypeError, ValueError):
continue
by_issue.setdefault(iid, []).append(name)
for issue in issues:
iid = issue.get("id")
if iid is None:
issue["_control_tags"] = []
continue
issue["_control_tags"] = sorted(by_issue.get(int(iid), []))
return issues
redmine_ticket_drafts.py
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env python3
"""Local draft mutation queue for internal notes vs customer emails.
CRITICAL DISTINCTION (do not collapse these):
- internal_note → Redmine journal note only. NEVER emails the customer.
Optional private_notes. PUT /issues/:id.json
- customer_email → Helpdesk "Send note" path. ALWAYS emails the customer
(when applied). POST /helpdesk/email_note
Cannot be private.
Drafts are stored under <cache-dir>/drafts/*.json until applied or cancelled.
"""
from __future__ import annotations
import json
import re
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
KIND_INTERNAL_NOTE = "internal_note"
KIND_CUSTOMER_EMAIL = "customer_email"
KIND_FIELDS = "fields"
KIND_CONTACT_FIELDS = "contact_fields"
# Legacy aliases from earlier draft CLI
LEGACY_KIND_MAP = {
"note": KIND_INTERNAL_NOTE,
"helpdesk_reply": KIND_CUSTOMER_EMAIL,
}
# Hosts allowed for write/apply without --allow-prod-writes
DEFAULT_SAFE_WRITE_HOST_PATTERNS = (
r"^localhost$",
r"^127\.0\.0\.1$",
r"^192\.168\.\d+\.\d+$",
r"^10\.\d+\.\d+\.\d+$",
r"^retired\.int\.ldrprep\.com$",
r"^.*\.170$", # common LAN Redmine
)
def now_iso() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
def normalize_kind(kind: str) -> str:
k = (kind or "").strip()
k = LEGACY_KIND_MAP.get(k, k)
if k not in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL, KIND_FIELDS, KIND_CONTACT_FIELDS}:
raise ValueError(
f"Unsupported draft kind {kind!r}. "
f"Use {KIND_INTERNAL_NOTE!r}, {KIND_CUSTOMER_EMAIL!r}, "
f"{KIND_FIELDS!r}, or {KIND_CONTACT_FIELDS!r}."
)
return k
def kind_label(kind: str) -> str:
kind = normalize_kind(kind)
if kind == KIND_INTERNAL_NOTE:
return "INTERNAL NOTE (does NOT email customer)"
if kind == KIND_CUSTOMER_EMAIL:
return "CUSTOMER EMAIL (WILL email customer via Helpdesk)"
if kind == KIND_FIELDS:
return "FIELD UPDATE (does NOT email customer; changes issue fields)"
if kind == KIND_CONTACT_FIELDS:
return "CONTACT UPDATE (does NOT email; changes CRM contact fields/tags)"
return kind
def drafts_dir(cache_dir: Path) -> Path:
path = Path(cache_dir) / "drafts"
path.mkdir(parents=True, exist_ok=True)
return path
def assert_kind_flags(kind: str, *, private: bool) -> str:
kind = normalize_kind(kind)
if kind == KIND_CUSTOMER_EMAIL and private:
raise ValueError(
"customer_email drafts cannot be private. "
"Private notes never email; customer emails always email. "
"Use kind=internal_note with --private for staff-only notes."
)
return kind
def add_draft(
cache_dir: Path,
*,
issue_id: int | None = None,
contact_id: int | None = None,
kind: str,
body: str = "",
private: bool = False,
fields: dict[str, Any] | None = None,
project: str | None = None,
base_url: str | None = None,
) -> Path:
kind = assert_kind_flags(kind, private=private)
body = (body or "").strip()
if kind in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL} and not body:
raise ValueError("Draft body is required for notes/emails.")
if kind == KIND_FIELDS and not fields:
raise ValueError("Field-update drafts require a non-empty fields payload.")
if kind == KIND_CONTACT_FIELDS:
if contact_id is None:
raise ValueError("contact_fields drafts require contact_id.")
if not fields:
raise ValueError("contact_fields drafts require a non-empty fields payload.")
if kind in {KIND_INTERNAL_NOTE, KIND_CUSTOMER_EMAIL, KIND_FIELDS} and issue_id is None:
raise ValueError(f"{kind} drafts require issue_id.")
draft_id = uuid.uuid4().hex[:12]
payload = {
"id": draft_id,
"created_at": now_iso(),
"status": "pending",
"project": project,
"base_url": base_url,
"issue_id": int(issue_id) if issue_id is not None else None,
"contact_id": int(contact_id) if contact_id is not None else None,
"kind": kind,
"kind_label": kind_label(kind),
"body": body,
"private": bool(private) if kind == KIND_INTERNAL_NOTE else False,
"fields": fields or {},
"applied_at": None,
"error": None,
"emails_customer": kind == KIND_CUSTOMER_EMAIL,
}
path = drafts_dir(cache_dir) / f"{draft_id}.json"
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return path
def load_draft(cache_dir: Path, draft_id: str) -> tuple[Path, dict[str, Any]]:
path = drafts_dir(cache_dir) / f"{draft_id}.json"
if not path.is_file():
matches = list(drafts_dir(cache_dir).glob(f"{draft_id}*.json"))
if len(matches) != 1:
raise FileNotFoundError(f"Draft not found or ambiguous: {draft_id}")
path = matches[0]
data = json.loads(path.read_text(encoding="utf-8"))
data["kind"] = normalize_kind(str(data.get("kind") or ""))
return path, data
def list_drafts(cache_dir: Path, status: str | None = "pending") -> list[dict[str, Any]]:
items: list[dict[str, Any]] = []
root = drafts_dir(cache_dir)
for path in sorted(root.glob("*.json")):
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
continue
if status and data.get("status") != status:
continue
try:
data["kind"] = normalize_kind(str(data.get("kind") or ""))
except ValueError:
pass
data["kind_label"] = kind_label(str(data.get("kind") or "internal_note")) if data.get("kind") else ""
data["_path"] = str(path)
items.append(data)
return items
def cancel_draft(cache_dir: Path, draft_id: str) -> bool:
try:
path, data = load_draft(cache_dir, draft_id)
except FileNotFoundError:
return False
data["status"] = "cancelled"
data["cancelled_at"] = now_iso()
path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return True
def cancel_all_pending(cache_dir: Path) -> int:
count = 0
for draft in list_drafts(cache_dir, status="pending"):
if cancel_draft(cache_dir, str(draft["id"])):
count += 1
return count
def mark_draft_applied(cache_dir: Path, draft_id: str, *, result: dict[str, Any] | None = None) -> None:
path, data = load_draft(cache_dir, draft_id)
data["status"] = "applied"
data["applied_at"] = now_iso()
data["error"] = None
data["apply_result"] = result or {}
path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def mark_draft_error(cache_dir: Path, draft_id: str, error: str) -> None:
path, data = load_draft(cache_dir, draft_id)
data["status"] = "error"
data["error"] = error
data["error_at"] = now_iso()
path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def host_is_safe_for_writes(base_url: str, patterns: tuple[str, ...] = DEFAULT_SAFE_WRITE_HOST_PATTERNS) -> bool:
host = (urlparse(base_url).hostname or "").lower()
if not host:
return False
return any(re.search(pat, host) for pat in patterns)
def assert_write_target_allowed(base_url: str, *, allow_prod_writes: bool) -> None:
if allow_prod_writes:
return
if host_is_safe_for_writes(base_url):
return
raise RuntimeError(
f"Refusing writes to {base_url!r}. "
"Default allow-list is LAN/test hosts only. "
"Re-run against the test Redmine, or pass --allow-prod-writes if you truly intend production."
)
redmine_ticket_entities.py
+173
View File
@@ -0,0 +1,173 @@
#!/usr/bin/env python3
"""Extract common CS entity identifiers from ticket text (pattern-based MVP).
Does not call NMI. Purpose: surface tracking numbers / order-like IDs / SKUs
so humans and agents can jump systems without copy-paste gymnastics.
Patterns are conservative heuristics — false positives are possible; treat hits
as hints, not ground truth.
"""
from __future__ import annotations
import re
from typing import Any, Iterable
# USPS: 20–22 digit, or 420… + 26 digits, or 13-char ending US
_USPS = re.compile(
r"\b("
r"9[0-9]{19,21}"
r"|420\d{5}\d{20,26}"
r"|[A-Z]{2}\d{9}US"
r")\b",
re.I,
)
_UPS = re.compile(r"\b(1Z[0-9A-Z]{16})\b", re.I)
_FEDEX = re.compile(r"\b(\d{12}|\d{15})\b")
_AMZ_ORDER = re.compile(r"\b(\d{3}-\d{7}-\d{7})\b")
# Prefer digit-heavy order captures; reject pure alpha HTML fragments
_ORDER_LABEL = re.compile(
r"(?:order\s*(?:id|#|number|no\.?)|po\s*(?:#|number)?)\s*[:=\s#]*([A-Z0-9][A-Z0-9\-_/]{3,32})",
re.I,
)
_ITEM_LABEL = re.compile(
r"(?:item\s*(?:#|no\.?|number)|sku|merchant\s*sku)\s*[:=\s#]*([A-Z0-9][A-Z0-9\-_/]{2,40})",
re.I,
)
_ACCOUNT = re.compile(r"\baccount\s*[#:]?\s*([A-Z0-9]{4,20})\b", re.I)
# LDR unit item numbers: 9 digits starting with 2
_ITEM_NO_BARE = re.compile(r"\b(2\d{8})\b")
_ORDER_STOP = {
"number",
"id",
"no",
"none",
"null",
"true",
"false",
"class",
"style",
"mode",
"text",
"title",
"guide",
"html",
"body",
"rted",
"rtal",
"licy",
"lation-mode",
"rtant",
"nsive",
"nsive-text",
"nsive-title",
"nsive-h2",
"ints",
}
def _text_from_issue(issue: dict[str, Any]) -> str:
chunks: list[str] = []
for key in ("subject", "description"):
val = issue.get(key)
if val:
chunks.append(str(val))
helpdesk = issue.get("helpdesk_ticket")
if isinstance(helpdesk, dict):
for key in ("from_address", "to_address", "message_id"):
if helpdesk.get(key):
chunks.append(str(helpdesk[key]))
contact = helpdesk.get("contact")
if isinstance(contact, dict):
for key in ("email", "company", "name", "first_name", "last_name"):
if contact.get(key):
chunks.append(str(contact[key]))
journals = issue.get("journals")
if isinstance(journals, list):
for j in journals[-15:]:
if isinstance(j, dict) and j.get("notes"):
chunks.append(str(j["notes"]))
return "\n".join(chunks)
def extract_entities(text: str) -> list[dict[str, str]]:
"""Return unique {type, value, confidence} hits from free text."""
hits: list[dict[str, str]] = []
seen: set[tuple[str, str]] = set()
def add(kind: str, value: str, confidence: str = "high") -> None:
value = value.strip()
if not value:
return
key = (kind, value.upper())
if key in seen:
return
seen.add(key)
hits.append({"type": kind, "value": value, "confidence": confidence})
for m in _UPS.finditer(text):
add("tracking_ups", m.group(1).upper())
for m in _USPS.finditer(text):
add("tracking_usps", m.group(1).upper())
for m in _AMZ_ORDER.finditer(text):
add("order_amazon", m.group(1))
for m in _ORDER_LABEL.finditer(text):
val = m.group(1)
if re.fullmatch(r"\d{3}-\d{7}-\d{7}", val):
add("order_amazon", val)
continue
if not re.search(r"\d", val):
continue
if val.lower() in _ORDER_STOP:
continue
# Require at least 5 chars and a digit; reject long alpha+token noise
if len(val) < 5:
continue
if re.search(r"[a-zA-Z]{6,}", val) and not re.search(r"\d{4,}", val):
continue # e.g. MGdHZfSeDzi_BhM6 without enough digits
conf = "high" if re.fullmatch(r"\d{5,12}", val) else "medium"
add("order_id", val, confidence=conf)
for m in _ITEM_LABEL.finditer(text):
val = m.group(1)
if re.fullmatch(r"2\d{8}", val):
add("item_no", val, confidence="high")
elif re.search(r"\d", val) or re.search(r"[A-Z]{2,}", val, re.I):
add("item_or_sku", val, confidence="medium")
for m in _ITEM_NO_BARE.finditer(text):
add("item_no", m.group(1), confidence="medium")
for m in _ACCOUNT.finditer(text):
val = m.group(1)
if val.lower() in {"within", "number", "setup", "your", "the"}:
continue
if not re.search(r"\d", val) and len(val) < 6:
continue
add("account_ref", val, confidence="medium")
if re.search(r"\b(fedex|tracking|shipment|shipped|carrier)\b", text, re.I):
for m in _FEDEX.finditer(text):
# skip USPS/item_no already captured lengths that are tracking
val = m.group(1)
if val.startswith("9") and len(val) >= 20:
continue
if re.fullmatch(r"2\d{8}", val):
continue
add("tracking_fedex_candidate", val, confidence="low")
return hits
def extract_entities_from_issue(issue: dict[str, Any]) -> list[dict[str, str]]:
return extract_entities(_text_from_issue(issue))
def format_entities_line(entities: Iterable[dict[str, str]]) -> str:
parts = []
for e in entities:
conf = e.get("confidence") or "high"
if conf == "high":
parts.append(f"{e['type']}:{e['value']}")
else:
parts.append(f"{e['type']}:{e['value']}({conf})")
return "; ".join(parts)
redmine_ticket_filters.py
+199
View File
@@ -0,0 +1,199 @@
#!/usr/bin/env python3
"""Saved property filters and filesystem queue materialization.
Saved filters live as text files under a directory (default: ./saved-filters).
Each file contains one property expression (same language as search --where).
Comments (# ...) and blank lines are ignored; remaining lines are joined with AND
unless the file is a single multi-line expression with parentheses.
Load by:
- basename without extension: helpdeskEmailMatchCompany
- alias.helpdeskEmailMatchCompany
- @helpdeskEmailMatchCompany
- path/to/file.expr
"""
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Any, Iterable
from redmine_ticket_query import QueryError, filter_by_query
DEFAULT_SAVED_FILTERS_DIR = Path("saved-filters")
DEFAULT_QUEUES_DIR_NAME = "queues"
def default_saved_filters_dir(repo_root: Path | None = None) -> Path:
root = repo_root or Path.cwd()
return root / "saved-filters"
def strip_filter_source(text: str) -> str:
lines: list[str] = []
for raw in text.splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
lines.append(line)
return " ".join(lines).strip()
def resolve_filter_name(raw: str) -> str:
name = raw.strip()
if name.startswith("alias."):
name = name[len("alias.") :]
if name.startswith("@"):
name = name[1:]
return name.strip()
def load_saved_filter_expression(name_or_path: str, search_dirs: Iterable[Path]) -> str:
"""Load expression from a saved filter name or filesystem path."""
raw = name_or_path.strip()
path = Path(raw)
candidates: list[Path] = []
if path.is_file():
candidates.append(path)
else:
name = resolve_filter_name(raw)
for directory in search_dirs:
for ext in (".expr", ".filter", ".wtf", ".txt", ""):
candidate = directory / f"{name}{ext}"
if candidate.is_file():
candidates.append(candidate)
break
if candidates:
break
if not candidates:
searched = ", ".join(str(d) for d in search_dirs)
raise QueryError(f"Saved filter not found: {name_or_path!r} (looked in {searched})")
expression = strip_filter_source(candidates[0].read_text(encoding="utf-8"))
if not expression:
raise QueryError(f"Saved filter file is empty: {candidates[0]}")
return expression
def expand_where_list(
where_items: list[str] | None,
saved_dirs: Iterable[Path],
) -> list[str]:
"""Expand @alias / alias.name / bare saved names; leave plain expressions as-is.
Heuristic: if the string starts with @ or alias. or is a single identifier
matching a saved file, load it; otherwise treat as inline expression.
"""
out: list[str] = []
for item in where_items or []:
text = item.strip()
if not text:
continue
if text.startswith("@") or text.startswith("alias."):
out.append(load_saved_filter_expression(text, saved_dirs))
continue
# bare identifier → try saved filter first
if re_fullmatch_identifier(text):
try:
out.append(load_saved_filter_expression(text, saved_dirs))
continue
except QueryError:
pass
out.append(text)
return out
def re_fullmatch_identifier(text: str) -> bool:
import re
return bool(re.fullmatch(r"[A-Za-z_][A-Za-z0-9_-]*", text))
def load_queue_definitions(path: Path) -> list[dict[str, Any]]:
"""Load queue definitions from JSON.
Format:
{
"queues": [
{
"name": "customer-service-triage-needed",
"where": ["assigned_to IS NULL", "category.name IS NULL"],
"max_tickets": 200,
"order": "newest-updated-first"
}
]
}
"""
data = json.loads(path.read_text(encoding="utf-8"))
if isinstance(data, list):
queues = data
elif isinstance(data, dict):
queues = data.get("queues") or data.get("filters") or []
else:
raise ValueError(f"Invalid queue config: {path}")
if not isinstance(queues, list):
raise ValueError(f"queues must be a list in {path}")
return [q for q in queues if isinstance(q, dict) and q.get("name")]
def materialize_queue(
*,
name: str,
issues: list[dict[str, Any]],
cache_dir: Path,
project: str,
queues_root: Path,
) -> dict[str, Any]:
"""Create/refresh a directory of symlinks to matching ticket JSON files.
Layout:
<queues_root>/<name>/
12345.json -> <cache>/<project>/12345.json
...
.queue-manifest.json
"""
queue_dir = queues_root / name
queue_dir.mkdir(parents=True, exist_ok=True)
# Remove old ticket symlinks (keep non-matching cleaned)
for existing in queue_dir.glob("*.json"):
if existing.name.startswith("."):
continue
if existing.is_symlink() or existing.is_file():
existing.unlink()
project_dir = Path(cache_dir) / project
linked = 0
missing = 0
for issue in issues:
issue_id = issue.get("id")
if issue_id is None:
continue
target = project_dir / f"{issue_id}.json"
link_path = queue_dir / f"{issue_id}.json"
if not target.is_file():
missing += 1
continue
# relative symlink when possible
try:
rel = os.path.relpath(target, start=queue_dir)
link_path.symlink_to(rel)
except OSError:
link_path.symlink_to(target)
linked += 1
manifest = {
"name": name,
"project": project,
"cache_dir": str(cache_dir),
"linked": linked,
"missing_source": missing,
"issue_ids": [int(i["id"]) for i in issues if i.get("id") is not None],
}
(queue_dir / ".queue-manifest.json").write_text(
json.dumps(manifest, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
return manifest
redmine_ticket_meta.py
+264
View File
@@ -0,0 +1,264 @@
#!/usr/bin/env python3
"""Resolve human-friendly Redmine names to IDs for set-fields / mutations.
Caches project metadata under <cache-dir>/meta/<project>/ so agents can use
--status Closed instead of --status-id 5 without hammering the API every run.
"""
from __future__ import annotations
import json
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable
def now_iso() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
def meta_dir(cache_dir: Path, project: str) -> Path:
path = Path(cache_dir) / "meta" / project
path.mkdir(parents=True, exist_ok=True)
return path
def _load_json(path: Path) -> dict[str, Any] | None:
if not path.is_file():
return None
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
return data if isinstance(data, dict) else None
def _save_json(path: Path, data: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
tmp.replace(path)
def _fresh(path: Path, max_age_seconds: int) -> bool:
if not path.is_file():
return False
age = time.time() - path.stat().st_mtime
return age <= max_age_seconds
def normalize_name(value: str) -> str:
return " ".join(str(value).strip().lower().split())
def fetch_issue_statuses(client: Any) -> list[dict[str, Any]]:
data = client.request("GET", "/issue_statuses.json")
statuses = data.get("issue_statuses") if isinstance(data, dict) else None
return list(statuses) if isinstance(statuses, list) else []
def fetch_issue_priorities(client: Any) -> list[dict[str, Any]]:
data = client.request("GET", "/enumerations/issue_priorities.json")
priorities = data.get("issue_priorities") if isinstance(data, dict) else None
return list(priorities) if isinstance(priorities, list) else []
def fetch_project_categories(client: Any, project: str) -> list[dict[str, Any]]:
data = client.request("GET", f"/projects/{project}/issue_categories.json")
cats = data.get("issue_categories") if isinstance(data, dict) else None
return list(cats) if isinstance(cats, list) else []
def fetch_project_memberships(client: Any, project: str) -> list[dict[str, Any]]:
"""Memberships include nested user objects for assignee resolution."""
members: list[dict[str, Any]] = []
offset = 0
total = None
while total is None or offset < total:
data = client.request(
"GET",
f"/projects/{project}/memberships.json",
{"limit": 100, "offset": offset},
)
page = data.get("memberships") if isinstance(data, dict) else []
if not isinstance(page, list):
page = []
members.extend(page)
total = int(data.get("total_count", len(members))) if isinstance(data, dict) else len(members)
if not page:
break
offset += len(page)
return members
def users_from_memberships(memberships: list[dict[str, Any]]) -> list[dict[str, Any]]:
users: list[dict[str, Any]] = []
seen: set[int] = set()
for m in memberships:
user = m.get("user") if isinstance(m, dict) else None
if not isinstance(user, dict) or user.get("id") is None:
continue
uid = int(user["id"])
if uid in seen:
continue
seen.add(uid)
users.append(
{
"id": uid,
"name": user.get("name") or f"{user.get('firstname', '')} {user.get('lastname', '')}".strip(),
"login": user.get("login"),
}
)
return users
def refresh_meta_cache(
client: Any,
cache_dir: Path,
project: str,
*,
force: bool = False,
max_age_seconds: int = 6 * 3600,
) -> dict[str, Path]:
root = meta_dir(cache_dir, project)
paths = {
"statuses": root / "issue_statuses.json",
"priorities": root / "issue_priorities.json",
"categories": root / "issue_categories.json",
"users": root / "project_users.json",
}
if not force and all(_fresh(p, max_age_seconds) for p in paths.values()):
return paths
statuses = fetch_issue_statuses(client)
priorities = fetch_issue_priorities(client)
categories = fetch_project_categories(client, project)
memberships = fetch_project_memberships(client, project)
users = users_from_memberships(memberships)
_save_json(paths["statuses"], {"fetched_at": now_iso(), "issue_statuses": statuses})
_save_json(paths["priorities"], {"fetched_at": now_iso(), "issue_priorities": priorities})
_save_json(paths["categories"], {"fetched_at": now_iso(), "issue_categories": categories})
_save_json(paths["users"], {"fetched_at": now_iso(), "users": users})
return paths
def _items_from_file(path: Path, key: str) -> list[dict[str, Any]]:
data = _load_json(path) or {}
items = data.get(key)
return list(items) if isinstance(items, list) else []
def resolve_id_by_name(
items: list[dict[str, Any]],
name: str,
*,
id_key: str = "id",
name_keys: tuple[str, ...] = ("name",),
) -> int:
target = normalize_name(name)
if target.isdigit():
return int(target)
matches: list[dict[str, Any]] = []
for item in items:
if not isinstance(item, dict):
continue
candidates = []
for nk in name_keys:
val = item.get(nk)
if val is not None:
candidates.append(normalize_name(str(val)))
# also full first+last style
if item.get("firstname") or item.get("lastname"):
candidates.append(
normalize_name(f"{item.get('firstname', '')} {item.get('lastname', '')}")
)
if any(c == target for c in candidates if c):
matches.append(item)
continue
# substring match as fallback
if any(target in c for c in candidates if c):
matches.append(item)
if not matches:
available = []
for item in items[:30]:
if isinstance(item, dict):
available.append(str(item.get("name") or item.get("login") or item.get(id_key)))
raise LookupError(
f"No match for {name!r}. Examples: {', '.join(available) or '(empty meta cache)'}"
)
if len(matches) > 1:
# prefer exact name over substring
exact = [
m
for m in matches
if any(normalize_name(str(m.get(k) or "")) == target for k in name_keys)
or normalize_name(f"{m.get('firstname', '')} {m.get('lastname', '')}") == target
]
if len(exact) == 1:
return int(exact[0][id_key])
labels = [str(m.get("name") or m.get("login") or m.get(id_key)) for m in matches[:8]]
raise LookupError(f"Ambiguous match for {name!r}: {', '.join(labels)}")
return int(matches[0][id_key])
def load_meta_lists(cache_dir: Path, project: str) -> dict[str, list[dict[str, Any]]]:
root = meta_dir(cache_dir, project)
return {
"statuses": _items_from_file(root / "issue_statuses.json", "issue_statuses"),
"priorities": _items_from_file(root / "issue_priorities.json", "issue_priorities"),
"categories": _items_from_file(root / "issue_categories.json", "issue_categories"),
"users": _items_from_file(root / "project_users.json", "users"),
}
def resolve_field_names(
cache_dir: Path,
project: str,
*,
status_name: str | None = None,
assignee_name: str | None = None,
priority_name: str | None = None,
category_name: str | None = None,
client: Any | None = None,
refresh: bool = False,
) -> dict[str, int]:
"""Return numeric id fields from optional name strings.
If meta cache is empty and client is provided, refresh automatically.
"""
lists = load_meta_lists(cache_dir, project)
need_refresh = refresh or (
client is not None
and (
(status_name and not lists["statuses"])
or (assignee_name and not lists["users"])
or (priority_name and not lists["priorities"])
or (category_name and not lists["categories"])
)
)
if need_refresh and client is not None:
refresh_meta_cache(client, cache_dir, project, force=True)
lists = load_meta_lists(cache_dir, project)
out: dict[str, int] = {}
if status_name:
out["status_id"] = resolve_id_by_name(lists["statuses"], status_name)
if priority_name:
out["priority_id"] = resolve_id_by_name(lists["priorities"], priority_name)
if category_name:
if normalize_name(category_name) in {"", "none", "clear", "null"}:
out["category_id"] = 0
else:
out["category_id"] = resolve_id_by_name(lists["categories"], category_name)
if assignee_name:
if normalize_name(assignee_name) in {"", "none", "unassigned", "nobody", "null", "0"}:
out["assigned_to_id"] = 0
else:
out["assigned_to_id"] = resolve_id_by_name(
lists["users"],
assignee_name,
name_keys=("name", "login"),
)
return out
redmine_ticket_nmi.py
+628
View File
@@ -0,0 +1,628 @@
#!/usr/bin/env python3
"""Bridge Redmine ticket entities → NMI agent CLI (mrscClient-php bin/nmi).
Pattern extract stays in redmine_ticket_entities.py (no NMI I/O).
This module shells out to the NMI CLI for resolve / cs-brief using those hints.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Iterable, Optional
# Prefer env overrides for agents/profiles
DEFAULT_NMI_BIN = os.environ.get(
"NMI_CLI",
"/home/jason/projects/mrscClient-php/bin/nmi",
)
DEFAULT_ENV_FILE = os.environ.get(
"NMI_ENV_FILE",
"/home/jason/projects/mrscClient-php/examples/modern/.env",
)
DEFAULT_MODE = os.environ.get("NMI_MODE", "dev")
# Entity types that map cleanly into NMI resolve ids
_TRACKING_TYPES = {
"tracking_ups",
"tracking_usps",
"tracking_fedex_candidate",
}
_ORDER_TYPES = {"order_amazon", "order_id"}
_ITEM_TYPES = {"item_no", "item_or_sku"}
_ACCOUNT_TYPES = {"account_ref"}
# NMI deep link templates (from obsidian-ldr/INBOX/WIP/Linking to Entities Within NMI.md)
NMI_BASE = "https://app.ldrprep.com"
NMI_LINKS = {
"item_no": {
"manager": lambda v: f"{NMI_BASE}/?page=itemManager&itemNo={v}",
"statement": lambda v: f"{NMI_BASE}/?page=itemStatement&itemNo={v}",
},
"item_or_sku": {
"manager": lambda v: f"{NMI_BASE}/?page=itemManager&itemNo={v}",
},
"tracking_ups": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"},
"tracking_usps": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"},
"tracking_fedex_candidate": {"box_history": lambda v: f"{NMI_BASE}/?page=boxFinder&tracking_number={v}&action=search"},
"order_id": {"order_summary": lambda v: f"{NMI_BASE}/?page=requests&filter=view&id={v}"},
"request_id": {"package_history": lambda v: f"{NMI_BASE}/?page=boxFinder&request_id={v}"},
"account_ref": {"user_info": lambda v: f"{NMI_BASE}/?page=userInfo&accountNo={v}"},
}
# LDR unit item numbers commonly 9 digits starting with 2
_ITEM_NO_BARE = re.compile(r"\b(2\d{8})\b")
@dataclass(frozen=True)
class NmiRuntimeProfile:
"""Resolved NMI execution settings plus a preflight result."""
mode: str
env_file: Path
nmi_bin: Path
endpoint: Optional[str]
ready: bool
problem: Optional[str] = None
def resolve_runtime_profile(
*,
mode: Optional[str] = None,
env_file: Optional[str] = None,
nmi_bin: Optional[str] = None,
endpoint: Optional[str] = None,
environ: Optional[dict[str, str]] = None,
) -> NmiRuntimeProfile:
"""Resolve the exact profile passed to PHP before an NMI subprocess starts.
An env file that is empty or missing is always rejected. It is too easy for
the checked-in example `.env` to otherwise select a surprising dev/default
path while making the TUI look successful.
"""
env = os.environ if environ is None else environ
requested_mode = (mode or env.get("NMI_MODE") or DEFAULT_MODE).strip().lower()
normalized_mode = "production" if requested_mode in {"prod", "production"} else requested_mode
selected_env_file = Path(env_file or env.get("NMI_ENV_FILE") or DEFAULT_ENV_FILE).expanduser()
selected_nmi_bin = Path(nmi_bin or env.get("NMI_CLI") or DEFAULT_NMI_BIN).expanduser()
mode_upper = normalized_mode.upper()
selected_endpoint = (
endpoint
or env.get(f"NMI_ENDPOINT_{mode_upper}")
or env.get("NMI_ENDPOINT")
or None
)
if not selected_env_file.is_file():
return NmiRuntimeProfile(
mode=normalized_mode,
env_file=selected_env_file,
nmi_bin=selected_nmi_bin,
endpoint=selected_endpoint,
ready=False,
problem=f"NMI env file does not exist: {selected_env_file}",
)
if selected_env_file.stat().st_size == 0:
return NmiRuntimeProfile(
mode=normalized_mode,
env_file=selected_env_file,
nmi_bin=selected_nmi_bin,
endpoint=selected_endpoint,
ready=False,
problem=f"NMI env file is empty: {selected_env_file}",
)
if not selected_nmi_bin.is_file():
return NmiRuntimeProfile(
mode=normalized_mode,
env_file=selected_env_file,
nmi_bin=selected_nmi_bin,
endpoint=selected_endpoint,
ready=False,
problem=f"NMI CLI does not exist: {selected_nmi_bin}",
)
return NmiRuntimeProfile(
mode=normalized_mode,
env_file=selected_env_file,
nmi_bin=selected_nmi_bin,
endpoint=selected_endpoint,
ready=True,
)
@dataclass
class NmiLookupPlan:
"""Deduped identifiers to try against NMI, highest confidence first."""
tracking: list[str] = field(default_factory=list)
order_id: list[str] = field(default_factory=list)
item_no: list[str] = field(default_factory=list)
account: list[str] = field(default_factory=list)
skipped: list[dict[str, str]] = field(default_factory=list)
def is_empty(self) -> bool:
return not (self.tracking or self.order_id or self.item_no or self.account)
def enrich_entities_with_item_nos(
entities: list[dict[str, str]], text: str
) -> list[dict[str, str]]:
"""Add bare LDR-style item numbers when not already present."""
seen = {(e.get("type"), (e.get("value") or "").upper()) for e in entities}
out = list(entities)
for m in _ITEM_NO_BARE.finditer(text or ""):
val = m.group(1)
key = ("item_no", val)
if key in seen:
continue
# Avoid treating pure tracking-ish long numbers: item is exactly 9 digits
out.append({"type": "item_no", "value": val, "confidence": "medium"})
seen.add(key)
return out
def plan_from_entities(entities: Iterable[dict[str, str]]) -> NmiLookupPlan:
plan = NmiLookupPlan()
seen_t: set[str] = set()
seen_o: set[str] = set()
seen_i: set[str] = set()
seen_a: set[str] = set()
# Sort high confidence first
ordered = sorted(
entities,
key=lambda e: {"high": 0, "medium": 1, "low": 2}.get(e.get("confidence") or "high", 3),
)
for e in ordered:
kind = e.get("type") or ""
val = (e.get("value") or "").strip()
conf = e.get("confidence") or "high"
if not val:
continue
if kind in _TRACKING_TYPES:
if conf == "low" and kind == "tracking_fedex_candidate":
plan.skipped.append(e)
continue
u = val.upper()
if u not in seen_t:
seen_t.add(u)
plan.tracking.append(val)
elif kind in _ORDER_TYPES:
if val not in seen_o:
seen_o.add(val)
plan.order_id.append(val)
elif kind in _ITEM_TYPES or kind == "item_no":
# Prefer pure numeric item_no; SKU-like may still help as order/item search later
if re.fullmatch(r"\d{6,12}", val):
if val not in seen_i:
seen_i.add(val)
plan.item_no.append(val)
else:
plan.skipped.append({**e, "reason": "sku_not_numeric_item"})
elif kind in _ACCOUNT_TYPES:
if val not in seen_a:
seen_a.add(val)
plan.account.append(val)
else:
plan.skipped.append(e)
return plan
def _php_bin() -> str:
return shutil.which("php") or "php"
def run_nmi(
args: list[str],
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
timeout: int = 60,
) -> subprocess.CompletedProcess[str]:
cmd = [_php_bin(), nmi_bin, *args, f"--mode={mode}", f"--env-file={env_file}"]
if endpoint:
cmd.append(f"--endpoint={endpoint}")
return subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
check=False,
)
def try_resolve_ids(
plan: NmiLookupPlan,
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
include_pictures: bool = False,
max_attempts: int = 6,
) -> dict[str, Any]:
"""Try NMI resolve with identifiers from the plan until one succeeds."""
attempts: list[dict[str, Any]] = []
candidates: list[list[str]] = []
for item in plan.item_no:
candidates.append(["resolve", f"--item-no={item}"])
for tn in plan.tracking:
candidates.append(["resolve", f"--tracking={tn}"])
for oid in plan.order_id:
candidates.append(["resolve", f"--order-id={oid}"])
for acct in plan.account:
candidates.append(["resolve", f"--account={acct}"])
if include_pictures:
for c in candidates:
c.append("--pictures")
for args in candidates[:max_attempts]:
proc = run_nmi(args, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint)
attempt = {
"args": args,
"returncode": proc.returncode,
"stderr": (proc.stderr or "").strip()[:500],
}
if proc.returncode == 0 and (proc.stdout or "").strip():
try:
data = json.loads(proc.stdout)
except json.JSONDecodeError as exc:
attempt["error"] = f"json_decode: {exc}"
attempts.append(attempt)
continue
attempt["ok"] = True
attempts.append(attempt)
return {"ok": True, "result": data, "attempts": attempts, "used_args": args}
attempt["ok"] = False
attempts.append(attempt)
return {"ok": False, "result": None, "attempts": attempts, "used_args": None}
def try_cs_brief(
plan: NmiLookupPlan,
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
max_attempts: int = 6,
) -> dict[str, Any]:
"""Try cs-brief with plan identifiers until one returns text."""
attempts: list[dict[str, Any]] = []
candidates: list[list[str]] = []
for item in plan.item_no:
candidates.append(["cs-brief", f"--item-no={item}"])
for tn in plan.tracking:
candidates.append(["cs-brief", f"--tracking={tn}"])
for oid in plan.order_id:
candidates.append(["cs-brief", f"--order-id={oid}"])
for acct in plan.account:
candidates.append(["cs-brief", f"--account={acct}"])
for args in candidates[:max_attempts]:
proc = run_nmi(args, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint)
attempt = {
"args": args,
"returncode": proc.returncode,
"stderr": (proc.stderr or "").strip()[:500],
}
text = (proc.stdout or "").strip()
if proc.returncode == 0 and text and "=== NMI CS BRIEF ===" in text:
attempt["ok"] = True
attempts.append(attempt)
return {"ok": True, "text": text + ("\n" if not text.endswith("\n") else ""), "attempts": attempts, "used_args": args}
attempt["ok"] = False
if text:
attempt["stdout_preview"] = text[:200]
attempts.append(attempt)
return {"ok": False, "text": None, "attempts": attempts, "used_args": None}
def brief_for_issue(
issue: dict[str, Any],
entities: list[dict[str, str]],
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
with_history: bool = True,
) -> dict[str, Any]:
profile = resolve_runtime_profile(
mode=mode,
env_file=env_file,
nmi_bin=nmi_bin,
endpoint=endpoint,
)
runtime = {
"mode": profile.mode,
"env_file": str(profile.env_file),
"nmi_bin": str(profile.nmi_bin),
"endpoint": profile.endpoint,
"ready": profile.ready,
"problem": profile.problem,
}
# Prefer public extract path text rebuild without private imports
try:
from redmine_ticket_entities import extract_entities_from_issue as _ex
# rebuild text via extract side-channel: re-run extract on issue already given
# Use journals/subject from issue for bare item_no enrichment only
chunks = []
for key in ("subject", "description"):
if issue.get(key):
chunks.append(str(issue[key]))
journals = issue.get("journals")
if isinstance(journals, list):
for j in journals[-15:]:
if isinstance(j, dict) and j.get("notes"):
chunks.append(str(j["notes"]))
text = "\n".join(chunks)
_ = _ex # silence unused if tree-shaken
except Exception:
text = str(issue.get("subject") or "") + "\n" + str(issue.get("description") or "")
ents = enrich_entities_with_item_nos(entities, text)
plan = plan_from_entities(ents)
out: dict[str, Any] = {
"issue_id": issue.get("id"),
"runtime": runtime,
"plan": {
"tracking": plan.tracking,
"order_id": plan.order_id,
"item_no": plan.item_no,
"account": plan.account,
"skipped": plan.skipped,
},
"entities": ents,
}
if not profile.ready:
out["ok"] = False
out["error"] = "nmi_runtime_not_ready"
return out
if plan.is_empty():
out["ok"] = False
out["error"] = "no_nmi_candidates"
return out
brief = try_cs_brief(plan, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint)
out["brief"] = brief
out["ok"] = bool(brief.get("ok"))
if not out["ok"]:
# still try resolve for structured debug
out["resolve"] = try_resolve_ids(
plan, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint
)
out["error"] = "nmi_lookup_failed"
# Pull item history (new getItemHistory) when requested and item_no present
if with_history:
out = enrich_with_history(out, mode=mode, env_file=env_file, nmi_bin=nmi_bin, endpoint=endpoint)
return out
def format_private_note(payload: dict[str, Any]) -> str:
"""Build a Redmine private note body from nmi brief result.
Enhanced with:
- Deep NMI links (Item Manager, Box History, Request pages etc.) per the WIP note
- Item history from the new getItemHistory endpoint (billings, who moved, condition changes)
"""
lines = [
"=== Auto NMI pre-fetch (agent) ===",
f"issue_id: {payload.get('issue_id')}",
]
plan = payload.get("plan") or {}
lines.append(
"candidates: "
+ json.dumps(
{
"tracking": plan.get("tracking"),
"order_id": plan.get("order_id"),
"item_no": plan.get("item_no"),
"account": plan.get("account"),
},
sort_keys=True,
)
)
# Deep links from the WIP note
ents = payload.get("entities") or []
links = generate_nmi_links(ents)
if links:
lines.append("")
lines.append("=== NMI Deep Links (Item Manager / Box History / Request pages / userInfo) ===")
lines.extend(links[:12])
brief = payload.get("brief") or {}
if brief.get("ok") and brief.get("text"):
lines.append("")
lines.append(brief["text"].rstrip())
else:
lines.append("")
lines.append("NMI lookup did not resolve cleanly.")
if payload.get("error"):
lines.append(f"error: {payload['error']}")
attempts = brief.get("attempts") or (payload.get("resolve") or {}).get("attempts") or []
for a in attempts[:5]:
lines.append(f"- attempt {a.get('args')}: rc={a.get('returncode')} err={a.get('stderr') or a.get('error')}")
# New history
hist = payload.get("item_history") or {}
if hist.get("ok"):
lines.append("")
lines.append("=== Item History (getItemHistory) ===")
h = hist.get("history") or {}
if isinstance(h, dict) and h:
lines.append(json.dumps(h, indent=2, default=str)[:1600])
elif hist.get("text"):
lines.append(hist["text"][:1400])
else:
lines.append("(see raw for full history)")
elif "item_history" in payload:
lines.append("")
lines.append("Item history note: " + str((payload.get("item_history") or {}).get("error", "no data")))
lines.append("")
lines.append("_Generated by redmine_ticket_nmi; verify before customer-facing use._")
return "\n".join(lines) + "\n"
def generate_nmi_links(entities: list[dict[str, str]]) -> list[str]:
"""Produce a list of useful deep links based on resolved entities."""
links = []
seen = set()
for e in entities:
kind = e.get("type")
val = e.get("value")
if not kind or not val:
continue
templates = NMI_LINKS.get(kind, {})
for label, tmpl in templates.items():
url = tmpl(val)
key = (kind, val, label)
if key in seen:
continue
seen.add(key)
links.append(f"- {label} for {kind} {val}: {url}")
return links
def try_item_history(
item_no: str,
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
limit: int = 50,
snapshot: bool = True,
) -> dict[str, Any]:
"""Call nmi item history for a single item_no. Returns structured result."""
cmd = [
"php",
nmi_bin,
"item",
"history",
f"--item-no={item_no}",
f"--limit={limit}",
]
if snapshot:
cmd.append("--snapshot")
if mode:
cmd.extend(["--mode", mode])
if env_file:
cmd.extend(["--env-file", env_file])
if endpoint:
cmd.extend(["--endpoint", endpoint])
try:
res = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if res.returncode != 0:
return {"ok": False, "error": res.stderr.strip() or res.stdout.strip(), "cmd": cmd}
# Try JSON first
try:
data = json.loads(res.stdout)
return {"ok": True, "history": data, "raw": res.stdout}
except Exception:
return {"ok": True, "text": res.stdout.strip(), "raw": res.stdout}
except Exception as exc:
return {"ok": False, "error": str(exc)}
def enrich_with_history(
payload: dict[str, Any],
*,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
endpoint: Optional[str] = None,
) -> dict[str, Any]:
"""If an item_no is present in the plan, fetch its history and attach."""
plan = payload.get("plan") or {}
item_no = plan.get("item_no")
if not item_no:
return payload
hist = try_item_history(
str(item_no),
mode=mode,
env_file=env_file,
nmi_bin=nmi_bin,
endpoint=endpoint,
limit=60,
snapshot=True,
)
payload["item_history"] = hist
return payload
def try_auto_link_redmine_contact(
*,
contact_id: int,
tracking: list[str] | None = None,
order_id: list[str] | None = None,
item_no: list[str] | None = None,
mode: str = DEFAULT_MODE,
env_file: str = DEFAULT_ENV_FILE,
nmi_bin: str = DEFAULT_NMI_BIN,
) -> dict[str, Any]:
"""If we have a Redmine contact_id and identifiers from a ticket, try to resolve in NMI
and automatically create the redmine_contact_links entry.
This implements the pattern: ticket mentions tracking/order/item -> successful
cs-info/resolve -> the returned customer owns the shipment -> link the Redmine
contact to that NMI user_account using the new junction table.
"""
if not contact_id:
return {"ok": False, "reason": "no_contact_id"}
ids: list[str] = []
if tracking:
for t in tracking:
ids.extend(["--tracking", str(t)])
if order_id:
for o in order_id:
ids.extend(["--order-id", str(o)])
if item_no:
for i in item_no:
ids.extend(["--item-no", str(i)])
if not ids:
return {"ok": False, "reason": "no_identifiers"}
cmd = [nmi_bin, "link-contact", "--redmine-contact-id", str(contact_id), "--mode", mode]
if env_file and env_file != DEFAULT_ENV_FILE:
cmd.extend(["--env-file", env_file])
cmd.extend(ids)
try:
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=90)
ok = proc.returncode == 0
result = {
"ok": ok,
"contact_id": contact_id,
"cmd": " ".join(cmd),
"stdout": (proc.stdout or "")[:2500],
"stderr": (proc.stderr or "")[:800],
}
if ok and proc.stdout:
try:
if proc.stdout.strip().startswith("{"):
result["parsed"] = json.loads(proc.stdout)
except Exception:
pass
return result
except Exception as e:
return {"ok": False, "contact_id": contact_id, "error": str(e)}
redmine_ticket_ocr.py
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""OCR / text extract helpers for scanned ticket attachments.
Primary path: pdftotext (fast, works on digital PDFs).
Fallback: pdftoppm + tesseract for image-heavy scans.
Also handles common image attachments via tesseract.
Does not write to Redmine — callers queue subject/notes drafts.
"""
from __future__ import annotations
import re
import shutil
import subprocess
import tempfile
from pathlib import Path
from typing import Any
def _run(cmd: list[str], *, timeout: int = 120) -> subprocess.CompletedProcess[str]:
return subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
check=False,
)
def extract_text_from_pdf(path: Path, *, max_chars: int = 12000) -> tuple[str, str]:
"""Return (text, method). method is pdftotext|tesseract|empty."""
path = Path(path)
text = ""
pdftotext = shutil.which("pdftotext")
if pdftotext:
proc = _run([pdftotext, "-layout", "-nopgbrk", str(path), "-"], timeout=60)
text = (proc.stdout or "").strip()
if len(re.sub(r"\s+", "", text)) >= 40:
return text[:max_chars], "pdftotext"
# OCR fallback: first 2 pages as PNG
pdftoppm = shutil.which("pdftoppm")
tesseract = shutil.which("tesseract")
if not pdftoppm or not tesseract:
return text[:max_chars] if text else "", "empty" if not text else "pdftotext-thin"
with tempfile.TemporaryDirectory(prefix="rtl-ocr-") as td:
out_prefix = Path(td) / "page"
_run([pdftoppm, "-png", "-f", "1", "-l", "2", "-r", "200", str(path), str(out_prefix)], timeout=90)
chunks: list[str] = []
for png in sorted(Path(td).glob("page*.png")):
proc = _run([tesseract, str(png), "stdout", "-l", "eng"], timeout=90)
if proc.stdout:
chunks.append(proc.stdout.strip())
ocr_text = "\n\n".join(chunks).strip()
if ocr_text:
return ocr_text[:max_chars], "tesseract"
return text[:max_chars] if text else "", "empty"
def extract_text_from_image(path: Path, *, max_chars: int = 12000) -> tuple[str, str]:
tesseract = shutil.which("tesseract")
if not tesseract:
return "", "empty"
proc = _run([tesseract, str(path), "stdout", "-l", "eng"], timeout=90)
text = (proc.stdout or "").strip()
return (text[:max_chars], "tesseract") if text else ("", "empty")
def extract_text_from_file(path: Path, *, max_chars: int = 12000) -> tuple[str, str]:
path = Path(path)
suffix = path.suffix.lower()
if suffix == ".pdf":
return extract_text_from_pdf(path, max_chars=max_chars)
if suffix in {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".webp", ".gif"}:
return extract_text_from_image(path, max_chars=max_chars)
if suffix in {".txt", ".md", ".csv"}:
try:
return path.read_text(encoding="utf-8", errors="replace")[:max_chars], "read"
except OSError:
return "", "empty"
return "", "empty"
_PERSON_LINE = re.compile(
r"^(?:name|employee|applicant|patient|to|for)\s*[:\-]\s*(.+)$",
re.I | re.M,
)
_COMPANY_LINE = re.compile(
r"^(?:company|organization|employer|vendor|client)\s*[:\-]\s*(.+)$",
re.I | re.M,
)
_DATE_LINE = re.compile(
r"\b((?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+\d{1,2},?\s+\d{4}|\d{1,2}[/-]\d{1,2}[/-]\d{2,4})\b",
re.I,
)
_DOC_HINTS = (
(r"\bw-?2\b", "W-2"),
(r"\bw-?4\b", "W-4"),
(r"\bi-?9\b", "I-9"),
(r"\bdriver.?s?\s+license\b|\bdl\b", "Driver License"),
(r"\bpassport\b", "Passport"),
(r"\binvoice\b", "Invoice"),
(r"\bpacking\s+slip\b", "Packing Slip"),
(r"\bbill\s+of\s+lading\b|\bbol\b", "Bill of Lading"),
(r"\bresume\b|\bcv\b", "Resume"),
(r"\boffer\s+letter\b", "Offer Letter"),
(r"\bnda\b|non-?disclosure", "NDA"),
(r"\bhandbook\b", "Handbook"),
(r"\bdirect\s+deposit\b", "Direct Deposit Form"),
)
def suggest_subject_from_text(
text: str,
*,
original_subject: str = "",
max_len: int = 120,
) -> str:
"""Heuristic subject for scanned-doc tickets."""
clean = re.sub(r"[ \t]+", " ", text or "")
lines = [ln.strip() for ln in clean.splitlines() if ln.strip()]
# Drop very short / noise lines
content_lines = [ln for ln in lines if len(ln) >= 4 and not re.fullmatch(r"[\d\W]+", ln)][:40]
doc_type = "Scanned document"
joined = "\n".join(content_lines[:30])
for pat, label in _DOC_HINTS:
if re.search(pat, joined, re.I):
doc_type = label
break
person = None
m = _PERSON_LINE.search(joined)
if m:
person = re.sub(r"\s+", " ", m.group(1)).strip(" .,-")[:60]
company = None
m = _COMPANY_LINE.search(joined)
if m:
company = re.sub(r"\s+", " ", m.group(1)).strip(" .,-")[:60]
if not company:
for ln in content_lines[:25]:
if re.search(r"\b(disposal|inc\.?|llc|ltd|corp\.?|co\.)\b", ln, re.I) and 8 <= len(ln) <= 60:
if re.search(r"please|payable|www\.|http|account number|invoice number", ln, re.I):
continue
company = re.sub(r"\s+", " ", ln).strip(" .,-")[:60]
break
inv_m = re.search(r"\binvoice\s*(?:number|#)?\s*[:.]?\s*(\d{5,})", joined, re.I)
invoice_no = inv_m.group(1) if inv_m else None
if doc_type == "Invoice" and invoice_no:
doc_type = f"Invoice {invoice_no}"
# Fallback: first substantial title-ish line (skip bullets / fee-table noise)
title_line = None
for ln in content_lines[:8]:
if len(ln) < 8 or len(ln) > 80:
continue
if ln.lstrip().startswith(("-", "•", "*")):
continue
if re.search(r"https?://|@|page\s+\d|return check fee|service charge", ln, re.I):
continue
title_line = ln
break
date = None
dm = _DATE_LINE.search(joined)
if dm:
date = dm.group(1)
parts: list[str] = [doc_type]
if person:
parts.append(f"— {person}")
elif company:
parts.append(f"— {company}")
elif title_line and title_line.lower() not in doc_type.lower():
parts.append(f"— {title_line[:50]}")
if date and date not in " ".join(parts):
parts.append(f"({date})")
subject = " ".join(parts)
subject = re.sub(r"\s+", " ", subject).strip()
if len(subject) > max_len:
subject = subject[: max_len - 1].rstrip() + "…"
if not subject or subject == "Scanned document":
# last resort: original if not generic
orig = (original_subject or "").strip()
if orig and not re.search(r"scanned\s+document", orig, re.I):
return orig[:max_len]
if title_line:
return f"Scanned document — {title_line[:80]}"[:max_len]
return subject
def attachment_is_ocr_candidate(att: dict[str, Any]) -> bool:
name = str(att.get("filename") or att.get("name") or "").lower()
ctype = str(att.get("content_type") or "").lower()
if name.endswith(".pdf") or "pdf" in ctype:
return True
if any(name.endswith(ext) for ext in (".png", ".jpg", ".jpeg", ".tif", ".tiff", ".webp")):
return True
if ctype.startswith("image/"):
return True
return False
def looks_like_generic_scan_subject(subject: str) -> bool:
s = (subject or "").strip().lower()
if not s:
return True
return bool(
re.search(
r"scanned\s+document|scan\s+from|hp\s+scan|scanner|untitled|document\s+from\s+scan",
s,
)
)
def build_ocr_note(
*,
issue_id: int,
files: list[dict[str, str]],
proposed_subject: str,
) -> str:
lines = [
"=== OCR / attachment review (agent) ===",
f"issue_id: {issue_id}",
f"proposed_subject: {proposed_subject}",
"",
]
for f in files:
lines.append(f"## {f.get('filename')} ({f.get('method')})")
lines.append("")
snippet = (f.get("text") or "")[:2500].strip()
lines.append(snippet or "_(no extractable text)_")
lines.append("")
lines.append("_Generated by redmine_ticket_ocr; verify before customer-facing use._")
return "\n".join(lines) + "\n"
redmine_ticket_query.py
+365
View File
@@ -0,0 +1,365 @@
#!/usr/bin/env python3
"""Local property-based query helpers for cached Redmine ticket JSON.
Small expression language (intentionally not a full DSL):
path == value
path != value
path IS NULL / path IS NOT NULL
path.includes("substring")
path.since("20 minutes") # relative window for date-ish fields
path == other.path
expr AND expr
expr OR expr
( expr )
Paths use dot notation into issue JSON, e.g.:
subject
status.name
helpdesk_ticket.contact.email
assigned_to.name
When a dict is compared to a string, its ``name`` field is used if present.
"""
from __future__ import annotations
import re
from datetime import datetime, timedelta, timezone
from typing import Any, Callable
class QueryError(ValueError):
"""Invalid property query expression."""
_TOKEN_RE = re.compile(
r"""
\s*
(
(?:AND|OR|IS|NOT|NULL)\b
| &&
| \|\|
| \.
| \(
| \)
| ==
| !=
| "[^"\\]*(?:\\.[^"\\]*)*"
| '[^'\\]*(?:\\.[^'\\]*)*'
| [A-Za-z_][A-Za-z0-9_]*
| \d+(?:\.\d+)?
)
""",
re.VERBOSE | re.IGNORECASE,
)
_RELATIVE_RE = re.compile(
r"^(?P<n>\d+)\s*(?P<unit>d|day|days|h|hour|hours|m|min|mins|minute|minutes|w|week|weeks)\s*(?:ago)?$",
re.IGNORECASE,
)
def tokenize(expression: str) -> list[str]:
text = expression.strip()
if not text:
raise QueryError("Empty query expression.")
tokens: list[str] = []
pos = 0
while pos < len(text):
match = _TOKEN_RE.match(text, pos)
if not match:
raise QueryError(f"Unexpected input near: {text[pos:pos + 40]!r}")
token = match.group(1)
tokens.append(token)
pos = match.end()
if pos != len(text):
raise QueryError(f"Unparsed trailing input: {text[pos:]!r}")
return tokens
def _unquote(token: str) -> str:
if len(token) >= 2 and token[0] == token[-1] and token[0] in "\"'":
body = token[1:-1]
return bytes(body, "utf-8").decode("unicode_escape") if "\\" in body else body
return token
def parse_relative_delta(value: str) -> timedelta:
match = _RELATIVE_RE.fullmatch(value.strip())
if not match:
raise QueryError(
f"Invalid relative time {value!r}; use forms like '20 minutes', '2h', '5 days'."
)
amount = int(match.group("n"))
unit = match.group("unit").lower()
if unit.startswith("w"):
return timedelta(weeks=amount)
if unit.startswith("d"):
return timedelta(days=amount)
if unit.startswith("h"):
return timedelta(hours=amount)
return timedelta(minutes=amount)
def parse_dt_loose(value: Any) -> datetime | None:
if value is None:
return None
if isinstance(value, datetime):
dt = value
else:
text = str(value).strip()
if not text:
return None
if text.endswith("Z"):
text = text[:-1] + "+00:00"
if "T" not in text and len(text) == 10:
text = text + "T00:00:00+00:00"
try:
dt = datetime.fromisoformat(text)
except ValueError:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
def get_path(issue: dict[str, Any], path: str) -> Any:
current: Any = issue
for part in path.split("."):
if current is None:
return None
if isinstance(current, dict):
current = current.get(part)
continue
return None
return current
def display_value(value: Any) -> Any:
"""Coerce nested Redmine objects to a comparable scalar when useful."""
if isinstance(value, dict):
if value.get("name") is not None:
return value.get("name")
if value.get("email") is not None:
return value.get("email")
if value.get("id") is not None:
return value.get("id")
return value
def values_equal(left: Any, right: Any) -> bool:
left_d = display_value(left)
right_d = display_value(right)
if left_d is None and right_d is None:
return True
if left_d is None or right_d is None:
return False
if isinstance(left_d, (int, float)) and isinstance(right_d, (int, float)):
return left_d == right_d
return str(left_d).lower() == str(right_d).lower()
def value_includes(haystack: Any, needle: str) -> bool:
if haystack is None:
return False
text = str(display_value(haystack))
return needle.lower() in text.lower()
class _Parser:
def __init__(self, tokens: list[str]):
self.tokens = tokens
self.i = 0
def _peek(self) -> str | None:
if self.i >= len(self.tokens):
return None
return self.tokens[self.i]
def _advance(self) -> str:
token = self._peek()
if token is None:
raise QueryError("Unexpected end of expression.")
self.i += 1
return token
def _accept_keyword(self, *keywords: str) -> str | None:
token = self._peek()
if token is None:
return None
upper = token.upper()
for keyword in keywords:
if upper == keyword.upper():
self.i += 1
return token
return None
def parse(self) -> Callable[[dict[str, Any]], bool]:
node = self._parse_or()
if self._peek() is not None:
raise QueryError(f"Unexpected token {self._peek()!r}")
return node
def _parse_or(self) -> Callable[[dict[str, Any]], bool]:
left = self._parse_and()
while True:
if self._accept_keyword("OR") or self._peek() == "||":
if self._peek() == "||":
self._advance()
right = self._parse_and()
prev_left, prev_right = left, right
left = lambda issue, a=prev_left, b=prev_right: a(issue) or b(issue)
continue
break
return left
def _parse_and(self) -> Callable[[dict[str, Any]], bool]:
left = self._parse_primary()
while True:
if self._accept_keyword("AND") or self._peek() == "&&":
if self._peek() == "&&":
self._advance()
right = self._parse_primary()
prev_left, prev_right = left, right
left = lambda issue, a=prev_left, b=prev_right: a(issue) and b(issue)
continue
break
return left
def _parse_primary(self) -> Callable[[dict[str, Any]], bool]:
if self._peek() == "(":
self._advance()
node = self._parse_or()
if self._peek() != ")":
raise QueryError("Expected ')'")
self._advance()
return node
return self._parse_comparison()
def _parse_path(self) -> str:
parts: list[str] = []
token = self._advance()
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", token):
raise QueryError(f"Expected path starting identifier, got {token!r}")
parts.append(token)
while self._peek() == ".":
# Lookahead: path.includes( / path.since( → leave method name for comparison parser
# tokens: ['.', 'includes', '(']
if self.i + 2 < len(self.tokens):
method_name = self.tokens[self.i + 1]
after = self.tokens[self.i + 2]
if method_name in {"includes", "since"} and after == "(":
break
self._advance() # consume '.'
nxt = self._peek()
if nxt is None:
raise QueryError("Dangling '.' in path")
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", nxt):
raise QueryError(f"Invalid path segment {nxt!r}")
parts.append(self._advance())
return ".".join(parts)
def _parse_value_or_path(self) -> tuple[str, Any]:
"""Return ('path', path_str) or ('lit', literal)."""
token = self._peek()
if token is None:
raise QueryError("Expected value or path")
if token[0] in "\"'" or re.fullmatch(r"\d+(?:\.\d+)?", token):
self._advance()
if token[0] in "\"'":
return "lit", _unquote(token)
if "." in token:
return "lit", float(token)
return "lit", int(token)
path = self._parse_path()
return "path", path
def _parse_comparison(self) -> Callable[[dict[str, Any]], bool]:
left_path = self._parse_path()
# method form: path.includes("x") / path.since("20 minutes")
method = None
base_path = left_path
if self._peek() == ".":
# path was stopped before method name
self._advance()
method = self._advance()
if method not in {"includes", "since"}:
raise QueryError(f"Unknown method .{method}; supported: includes, since")
base_path = left_path
if method is not None:
if self._advance() != "(":
raise QueryError(f"Expected '(' after .{method}")
arg_tok = self._advance()
if not arg_tok or arg_tok[0] not in "\"'":
raise QueryError(f".{method}() argument must be a string literal")
arg = _unquote(arg_tok)
if self._advance() != ")":
raise QueryError(f"Expected ')' after .{method} argument")
if method == "includes":
return lambda issue, p=base_path, a=arg: value_includes(get_path(issue, p), a)
delta = parse_relative_delta(arg)
def since_pred(issue: dict[str, Any], p=base_path, d=delta) -> bool:
raw = get_path(issue, p)
dt = parse_dt_loose(raw)
if dt is None:
return False
now = datetime.now(timezone.utc)
return dt >= (now - d)
return since_pred
# IS NULL / IS NOT NULL
if self._accept_keyword("IS"):
not_null = bool(self._accept_keyword("NOT"))
if not self._accept_keyword("NULL"):
raise QueryError("Expected NULL after IS [NOT]")
if not_null:
return lambda issue, p=left_path: get_path(issue, p) is not None and display_value(get_path(issue, p)) not in (None, "")
return lambda issue, p=left_path: get_path(issue, p) is None or display_value(get_path(issue, p)) in (None, "")
op = self._peek()
if op not in {"==", "!="}:
raise QueryError(f"Expected comparison operator after path {left_path!r}, got {op!r}")
self._advance()
kind, right = self._parse_value_or_path()
if kind == "lit":
if op == "==":
return lambda issue, p=left_path, r=right: values_equal(get_path(issue, p), r)
return lambda issue, p=left_path, r=right: not values_equal(get_path(issue, p), r)
# path == path
if op == "==":
return lambda issue, a=left_path, b=right: values_equal(get_path(issue, a), get_path(issue, b))
return lambda issue, a=left_path, b=right: not values_equal(get_path(issue, a), get_path(issue, b))
def compile_query(expression: str) -> Callable[[dict[str, Any]], bool]:
tokens = tokenize(expression)
# Normalize parentheses tokens already fine
return _Parser(tokens).parse()
def issue_matches(issue: dict[str, Any], expression: str) -> bool:
return compile_query(expression)(issue)
def filter_by_query(
issues: list[dict[str, Any]],
expression: str | None = None,
expressions: list[str] | None = None,
) -> list[dict[str, Any]]:
preds: list[Callable[[dict[str, Any]], bool]] = []
if expression:
preds.append(compile_query(expression))
for expr in expressions or []:
preds.append(compile_query(expr))
if not preds:
return list(issues)
out: list[dict[str, Any]] = []
for issue in issues:
if all(pred(issue) for pred in preds):
out.append(issue)
return out
redmine_ticket_textual_tui.py
+1190
View File
File diff suppressed because it is too large Load Diff
redmine_ticket_tui.py
+1141
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More