Skip to content

Latest commit

 

History

History
119 lines (92 loc) · 18.3 KB

AGENTS.md

File metadata and controls

119 lines (92 loc) · 18.3 KB

last30days Skill

Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, Grok (xAI), and 50+ other Agent Skills hosts. Python scripts with multi-source search aggregation.

Structure

  • skills/last30days/SKILL.md — canonical skill definition / runtime spec the model reads when the slash command fires; its reference-routing table gates the stage and mode instructions in skills/last30days/references/
  • skills/last30days/scripts/last30days.py — main research engine
  • skills/last30days/scripts/lib/ — search, enrichment, rendering modules
  • skills/last30days/scripts/lib/vendor/bird-search/ — vendored X search client
  • docs/solutions/ — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (module, tags, problem_type)
  • CONCEPTS.md — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
  • CONFIGURATION.md — user-facing knobs (env vars, flags, per-host install patterns); keep in sync per the rules below
  • CHANGELOG.md — structured release history built by towncrier at release time (launch copy lives in GitHub Releases)
  • changelog.d/ — per-PR news fragments; feature PRs write here, never edit CHANGELOG.md directly
  • CONTRIBUTING.md — setup, fragments, and release notes for humans and agents (towncrier is release-only)
  • .github/scripts/prepare_release.py — lockstep version bump + towncrier build (release PRs only)
  • HERMES_SETUP.md — install instructions for the Hermes harness specifically
  • mcp/ — Go MCP server (.mcpb bundle for Claude Desktop) exposing research and preflight (what a research run would read, write, and contact) tools. It embeds a copy of skills/last30days/scripts/ (gitignored mcp/internal/engine/vendored/, regenerated by mcp/scripts/sync-engine.sh) and shells out to python3. See mcp/README.md.

Orientation

  • This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (/last30days <topic> in most harnesses); scripts/last30days.py is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
  • Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration (the root or a routed reference) is incomplete — the model invoking the skill won't know the flag exists.
  • README and PR examples show /last30days <topic> first. Direct CLI invocation (python3 scripts/last30days.py ...) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
  • Slash commands don't pass shell mechanics through. /last30days OpenClaw --emit=html | pbcopy is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full python3 ... with explicit flags and a real shell).

Commands

# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only).
# Saves to $LAST30DAYS_MEMORY_DIR when set in shell or ~/.config/last30days/.env;
# add --save-dir <path> for a one-off override. Mirrors LAST30DAYS_STORE convention.
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y   # copies skill into ~/.agents/skills/<name>/ (frozen at install time); re-run to sync working-tree edits — see Rules below

# Tests (pytest; tests/ discovery is configured in pyproject.toml)
uv run pytest                              # full suite
uv run pytest tests/test_dedupe_v3.py      # single file
uv run pytest tests/test_dedupe_v3.py -k some_case   # single case
uv run pytest --cov                        # with coverage (skips lib/vendor/)
PYTHONPATH=skills/last30days/scripts uv run python -m unittest tests.test_subproc -v   # subprocess tests (Windows CI job; unittest skips conftest.py, so set the path)
uv run pytest tests/eval -x -s             # research-quality eval (separate CI job)

# MCP server (Go, under mcp/)
bash mcp/scripts/sync-engine.sh            # re-mirror the Python engine into the Go embed dir
(cd mcp && go test -race ./...)

# Release prep (maintainers / release automation — not feature PRs):
# Prefer GitHub Actions → "Prepare release". Local equivalent:
uv run python .github/scripts/prepare_release.py --bump patch   # or --version X.Y.Z

Python 3.12+ required. Use uv for the env; the venv lives at .venv/. CI (.github/workflows/validate.yml) runs the full suite in parallel with coverage (uv run pytest -n auto --dist worksteal --cov, failing below the fail_under floor), plus the subprocess tests, the eval suite, and the Go MCP tests after sync-engine.sh.

Changelog and releases (agents)

Agents open most PRs. Follow this so CHANGELOG.md stops conflicting and versions stay lockstep:

  1. Feature/fix PRs: add changelog.d/<pr-or-issue>.<type>.md (added / changed / fixed / removed / deprecated / security) when the change belongs in the next release notes. See changelog.d/README.md and CONTRIBUTING.md. Fill the PR template’s Summary, Agent disclosure, and Relationship sections.
  2. Never edit CHANGELOG.md in a feature PR. Never bump version strings in pyproject.toml, SKILL.md, plugin/marketplace JSON, or uv.lock outside a release PR. CI (changelog-guard.yml) enforces this.
  3. Nothing for release notes: omit the fragment, check Skip changelog in the template, and add the skip-changelog label. Dependabot PRs (author dependabot[bot]) skip that requirement automatically.
  4. Cutting a release: run Actions → Prepare release (patch/minor/major). That opens a chore(release): bump version to X.Y.Z PR which runs towncrier and bumps every lockstep surface. Merging to main triggers Tag release, which pushes vX.Y.Z and existing release.yml publishes .skill / .mcpb artifacts. Do not hand-edit ten version files. Contributors do not need a global towncrier install — uv sync --group dev (or the Action) provides it for release prep only.
  5. Lockstep gate remains tests/test_plugin_contract.py::test_versions_match_across_manifests. Workflow contract: tests/test_changelog_workflow.py.

Rules

  • lib/__init__.py must be bare package marker (comment only, NO eager imports)
  • One-time setup: npx skills add . -g -y copies the skill into ~/.agents/skills/<name>/ (real directory) and, for harnesses that support symlinked skill dirs, drops a per-host symlink pointing at that copy. Working-tree edits do NOT propagate automatically — the ~/.agents/skills/<name>/ copy is frozen at install time. To sync after edits, re-run npx skills add . -g -y. For live-edit on a dev machine, move that real directory to a backup outside ~/.agents/skills/ first, then create the working-tree symlink: ln -sfn "$PWD/skills/last30days" ~/.agents/skills/last30days (run from the repo root). ln -sfn replaces an existing symlink but does not replace a real directory; with the install copy still present, it creates a link inside that directory and leaves the stale skill active.
  • Git remote: origin = public (mvanhorn/last30days-skill)
  • Do not reduce fail_under in pyproject.toml ([tool.coverage.report]) without documenting why in the PR. The coverage gate is a floor meant to rise over time, not to be relaxed when new code is under-tested.
  • Every lib/*.py call to log.source_log(...) must pass tty_only=False. The default is True, which silently drops every line when stderr isn't a TTY (Claude Code, Codex, CI, captured output) — turning source observability into invisible failure. Enforced by tests/test_source_log_visibility.py.
  • CLI-gated optional sources (Digg via digg-pp-cli, YouTube via yt-dlp) activate only when shutil.which resolves the binary on the agent subprocess PATH — not merely when the file exists on disk. First-run setup installs Digg through @mvanhorn/printing-press-library (default $HOME/.local/bin); Hermes/OpenClaw gateways often need that directory on PATH. Setup must distinguish PATH-visible installs from off-PATH binaries and must not claim "now active" unless the engine gate would pass. See docs/solutions/integration-issues/digg-cli-agent-path-setup-wizard.md.
  • First-run onboarding is consent-driven, model-led, and host-split. Spec lives in the root SKILL.md gate and Step 0 of references/setup-wizard.md; contract locked by tests/test_onboarding_contract.py (and tests/test_grok_bot_host_contract.py).
    • The setup subprocess does only mechanical work (cookie reads, tool installs, GitHub device-auth, emitting the engine-owned welcome via --welcome). It cannot prompt, so consent lives in the skill instructions.
    • Step 0 has THREE branches:
      • Claude Code Modal Flow for hosts with modals — the restored v3.0.0 AskUserQuestion-driven NUX (welcome, Auto/Manual/Skip, cookie consent, ScrapeCreators offer, INCLUDE_SOURCES opt-in, first-topic picker). The welcome pitch is embedded in the modal's question, because the modal is the only always-fully-visible surface — a separate welcome message or --welcome Bash run gets buried behind "ctrl+o to expand".
      • Non-Modal Prose Flow for hosts without modals (OpenClaw, Codex, Cursor, Gemini CLI, Grok CLI) — relays last30days.py --welcome verbatim.
      • Grok Bot Prose Flow for LAST30DAYS_HOST=grok-bot — leads with the bot's own X lanes (its built-in X tools first, then the X for Grok Bot connector; LAST30DAYS_X_HOST_LANE=1 plus a --x-posts envelope the model fetches itself, exported only when the fetch returned posts), persists any backup key (X_BEARER_TOKEN / XAI_API_KEY) only through the engine's setup --store-key path, and never reads a browser session.
    • The logged-in Reddit session has its own consent question (references/reddit-session.md, routed from the root): asked after the X cookie answer (yes or skip) in the Modal and Non-Modal flows (a Reddit yes is its own consent and outranks a saved BROWSER_CONSENT=false; only FROM_BROWSER=off blocks it), once for returning users through the root gate's REDDIT_OFFER token (REDDIT_SESSION_OFFERED records it), and never on a Grok Bot host. X consent never implies it.
    • Do NOT collapse the modal flow back into a bare silent setup call or flatten it to prose-only — the guided modals are the feature (they eroded once and were restored).
    • The first two flows ask before reading cookies and surface the macOS Full Disk Access fix on permission-denied; all three offer the ScrapeCreators GitHub signup (10,000 free calls) on every first run.
    • GitHub device auth is a two-command split: setup --github-start returns the code fast (foreground, copies to clipboard), setup --github-poll waits for authorization (setup --github chains both for back-compat). Success persists SCRAPECREATORS_API_KEY via setup_wizard.write_api_key (0o600) and masks the key in stdout.
    • Step 5 source opt-in has two tiers, both comment-enabled: Recommended (TikTok + Instagram posts and top comments, plus YouTube comments — INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments) and Everything (also Threads + Pinterest). Comments are on by default (posts on → comments on for all three platforms). Threads and Pinterest are the only opt-in extras — they appear only in the Step 5 Everything option, never in the welcome or the Step 4 offer.
  • Comment ranking. Instagram comments come from ScrapeCreators (/v2/instagram/post/comments, ranked by comment_like_count) with vote-weighting parity to YouTube/TikTok (a dedicated _instagram_engagement carve-out plus _VOTE_LOG_REFERENCE/label/threshold entries). The "Top Community Comments" list (render._render_top_comments) is the report's quote pool, chosen and ordered by position, never by like count across items: each item (thread, video, reel, story) gives its top comment and its #2 by its own stored order (metadata["top_comments"]: score for Reddit, like count for TikTok and Instagram, YouTube's "top comments" order, Hacker News thread order), nothing below. All top comments come first, in the order of the stories (shown clusters as numbered, then other kept items as ranked), each story's standalone X/Threads/Bluesky/LinkedIn posts (10+ likes, labelled post, a LinkedIn post only with a real author) and what its YouTube, TikTok or Instagram video said (one line per video with 10+ likes, labelled said: YouTube's transcript_highlights, or the best highlight youtube_yt.extract_transcript_highlights draws from a reel's or TikTok's spoken caption) after its top comments, then all #2 comments in the same order; within a story and position, entries on a candidate that cleared _BEST_TAKE_FUNNY_FLOOR lead and end funny (fun_score belongs to the candidate). The per-platform display floors (YouTube 50, TikTok 500...) do not apply; on a platform that counts likes, upvotes or reactions (Instagram, TikTok, YouTube, X, Reddit, Threads, Bluesky, LinkedIn, GitHub) an item joins only when its top community comment has at least 5 (a Reddit thread's floor scales with its own score, min(5, max(2, score // 20)), so a small on-topic thread is not shut out), and its #2 needs at least 1, while Hacker News comments are exempt. A bot's comment ([bot] and bot-named authors, "automated review" text, HTML-comment markup), a comment by the item's own author (channel, creator, OP, submitter; read from the stored comment's is_uploader flag where the source provides one, names compared only when it is absent) and one with nothing quotable left are never community comments: the next comment takes the position. Quote text also drops lines quoting the parent (>), and markdown emphasis markers. Every eligibility test runs before duplicates are removed. Up to 40 one-line entries, shown clusters covered first and taking turns for slots (one top comment each, then round by round, so a busy cluster cannot starve a later one), each in the quote line's own shape (dot, author, on r/sub or platform, · count), then metadata: URL, cluster N or re: "title", and top comment / #2 comment / post. Quote text is one line with raw URLs dropped, inner double quotes made single, a sentence-end cut when over 240 characters, and leading reply handles stripped from X posts. Compact output also prints ## Posts by @handle and ## Polymarket Markets (each on-topic market as a ready-to-copy odds line, 📊 Polymarket · {subject}: {outcome} {price} · ..., at most three outcomes, movement in points after the outcome it belongs to, built from the odds summary polymarket._odds_summary stores on the item; then URL and metadata), and the command line (not the renderer) appends the ## Before you write the report checklist to --emit=compact stdout only (lib/report_checklist.py).

Security hygiene

  • Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or .env contents.
  • Use the env-based auth patterns in skills/last30days/scripts/lib/env.py; tests and fixtures must use obvious dummy values only.
  • Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
  • Do not weaken or disable the advisory security workflow (.github/workflows/security.yml) without explaining why in the PR description or review thread.

Maintaining README translations

README.md is the canonical English README. When changing it, reflect the same substantive updates in README.fr.md, README.de.md, README.es.md, README.pt-BR.md, README.ja.md, README.zh-CN.md, README.ar.md, README.fa.md, and README.tr.md, preserving commands, links, tables, and reciprocal language navigation.

Maintaining CONFIGURATION.md

CONFIGURATION.md is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from SKILL.md (the canonical runtime spec).

Update CONFIGURATION.md when:

  • adding a new env var (e.g. LAST30DAYS_*, BSKY_*, *_API_KEY)
  • adding a new CLI flag that affects configuration (e.g. --store, --web-backend)
  • adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Grok, Hermes…)
  • adding a new optional source that requires its own credential
  • changing the priority order of config layers (per-run flag > env > .env file > defaults)

Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / .env → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.

When a new config concept lands in SKILL.md or AGENTS.md, mirror the user-facing knob in CONFIGURATION.md so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.

Plugin manifests (Grok)

The repo doubles as a native Grok Build plugin via .grok-plugin/plugin.json + .grok-plugin/marketplace.json. Grok also reads .claude-plugin/* for compatibility; the native pair is the first-class lane and what an official xAI marketplace listing points at. The self-hosted catalog uses a bare Git URL source ({"source":"url","url":"https://github.com/mvanhorn/last30days-skill.git"}) so grok plugin marketplace add mvanhorn/last30days-skill tracks HEAD — not a self-referential local path: "." (Grok does not enumerate those). Version lockstep with Claude/Codex/Gemini manifests is enforced by tests/test_plugin_contract.py. Validate with grok plugin validate ..

Submitting to the xAI plugin marketplace

Listing in xAI's official catalog (xai-org/plugin-marketplace) is a post-merge outbound PR to their repo that pins a full 40-char commit SHA — unlike our own .grok-plugin/marketplace.json, which uses a bare URL source and tracks HEAD. Step-by-step procedure: docs/reference/xai-marketplace-submission.md.

Beta channel

Experimental changes get tested on mvanhorn/last30days-skill-private, which installs as a parallel /last30days-beta slash command. Beta-only changes never ship to public without a review PR here. Workflow guide (and the plan that established this setup) lives at BETA.md in the private repo.