Orchestra-Research/AI-Research-SKILLs instruction manual

1. Purpose, scope, and Claude Code classification

AI Research Skills Library manual

This public MIT-licensed repository is a collection of research playbooks rather than a model, training framework, or hosted service. Its files teach compatible coding agents how to handle research tasks spanning idea formation, experimentation, engineering, evaluation, and writing. A typical skill contains a `SKILL.md` entry point, deeper `references/`, and optional `scripts/`, templates, examples, or assets.

**Claude Code classification:** this is primarily a **native skill/context-injection library with orchestration**, distributed both through a Claude Code marketplace and a cross-agent installer. Installed Markdown skills are placed where Claude Code can discover them; the `autoresearch` skill coordinates other skills through a two-loop research workflow. The repository also documents a reusable skill-file structure, so it has a secondary **skill-authoring** relationship. The supplied evidence does not establish lifecycle interception or a general MCP/tool server.

The README repeatedly states **98 skills in 23 categories**, but its detailed category table and named inventory enumerate only **91**. The package README also contains stale category counts, while the changelog says an inventory drift check was added. This manual therefore explains every skill actually named in the supplied files and does not invent the seven identities missing from that inventory. Repository dates and 2026 release claims are reported as first-party claims, not independently verified facts.

2. Installation, selection, updating, and removal

The recommended installer auto-detects supported agents, lets you choose all skills, a quick-start bundle, categories, or individual skills, and downloads the selection. Global installation stores one canonical copy under `~/.orchestra/skills/`, then creates agent-specific symlinks; the README says Windows falls back to copying. Run the documented interactive entry point:

npx @orchestra-research/ai-research-skills

Non-interactive and maintenance operations documented by the package are:

npx @orchestra-research/ai-research-skills install --all
npx @orchestra-research/ai-research-skills install post-training
npx @orchestra-research/ai-research-skills list
npx @orchestra-research/ai-research-skills update

For a project-specific, version-controlled copy, use `--local`. Local mode copies files into directories such as `.claude/skills/` and records the selection in `.orchestra-skills.json`:

npx @orchestra-research/ai-research-skills install --all --local
npx @orchestra-research/ai-research-skills install --category post-training --local
npx @orchestra-research/ai-research-skills list --local
npx @orchestra-research/ai-research-skills update --local
npx @orchestra-research/ai-research-skills uninstall --local

Claude Code also has a documented marketplace path:

/plugin marketplace add orchestra-research/AI-research-SKILLs
/plugin install fine-tuning@ai-research-skills
/plugin install post-training@ai-research-skills
/plugin install inference-serving@ai-research-skills
/plugin install distributed-training@ai-research-skills
/plugin install optimization@ai-research-skills

The installer also targets Hermes Agent, OpenCode, OpenClaw, Cursor, Codex, Gemini CLI, Qwen Code, and shared `.agents/` locations. Auto-activation means a compatible agent selects a skill when the conversation matches its topic; the supplied files do not define a guaranteed routing algorithm or minimum Claude Code version.

3. Operating model: skill files, autoresearch, and research artifacts

Start a task in ordinary language and name the relevant domain when useful. A skill’s `SKILL.md` supplies triggers, quick patterns, examples, and links; reference files supply APIs, tutorials, issue solutions, release history, and codebase navigation. Optional scripts and assets are part of individual skills, not a universal executable interface. Inspect project-specific dependencies, credentials, and licenses before using them.

**Autoresearch** is the central coordinator. It uses an inner optimization loop for constrained experiments and an outer synthesis loop for literature review, interpretation, pivots, and paper writing. The README says it can route to domain skills, preserve state in `research-state.yaml`, `findings.md`, and `research-log.md`, and organize `literature/`, `experiments/`, `src/`, `data/`, and `to_human/`. Claude Code `/loop` and OpenClaw heartbeat support are claimed for continuous operation, but no exact `/loop` command is supplied here, so none is invented.

The three **Agent-Native Research Artifact (ARA)** skills preserve and critique research. **ARA Compiler** converts papers, repositories, logs, or notes into claims, concepts, evidence, an exploration graph, configurations, and code stubs. **ARA Research Manager** extracts decisions, experiments, dead ends, pivots, and provenance-tagged records into `ara/` after a task. **ARA Rigor Reviewer** scores evidence relevance, falsifiability, scope calibration, coherence, exploration integrity, and methodology, then emits severity-ranked findings and a recommendation.

Treat agent output as assistance, not validation. Claims, citations, experiment controls, security implications, generated code, and venue rules still require human review.

4. Catalog I: foundations, training, safety, and infrastructure

The following are the plain-language functions of every named skill in the first half of the inventory.

• **Ideation:** Research Brainstorming applies ten lenses to identify promising directions; Creative Thinking uses bisociation, structure mapping, and constraint manipulation to seek novelty.
• **ML Paper Writing:** ML Paper Writing structures conference papers, provides venue templates, and emphasizes citation verification; Academic Plotting creates architecture diagrams with Gemini or publication charts with matplotlib/seaborn.
• **Model architecture:** LitGPT offers clean LLM implementations and recipes; Mamba covers linear-time state-space models; NanoGPT teaches a compact GPT; RWKV explains an RNN/Transformer hybrid; TorchTitan covers PyTorch-native distributed Llama training.
• **Tokenization:** HuggingFace Tokenizers covers fast BPE, WordPiece, and Unigram pipelines; SentencePiece covers language-independent subword training.
• **Fine-tuning:** Axolotl provides YAML-driven model tuning; LLaMA-Factory provides WebUI-based tuning; PEFT covers LoRA, QLoRA, DoRA, and related parameter-efficient methods; Unsloth targets faster QLoRA.
• **Mechanistic interpretability:** TransformerLens exposes hooks and cached activations; SAELens trains and analyzes sparse autoencoders; pyvene performs declarative causal interventions; nnsight traces or modifies model internals locally or through NDIF.
• **Data processing:** NeMo Curator handles GPU-accelerated curation and deduplication; Ray Data handles distributed streaming data pipelines.
• **Post-training:** TRL Fine-Tuning covers transformer reinforcement learning; GRPO RL Training covers group-relative policy optimization; OpenRLHF covers full Ray/vLLM RLHF; SimPO performs preference optimization without a reference model; verl combines HybridFlow with training and serving backends; slime combines Megatron and SGLang; miles adds enterprise MoE, FP8, INT4, and speculative-RL paths; torchforge covers PyTorch-native RL infrastructure.
• **Safety:** Constitutional AI uses written principles for self-improvement; LlamaGuard classifies unsafe inputs/outputs; NeMo Guardrails implements programmable Colang controls; Prompt Guard detects prompt injection and jailbreaks.
• **Distributed training:** DeepSpeed covers ZeRO and parallelism; FSDP2 covers `fully_shard` and DTensor; Accelerate simplifies distributed setup; Megatron-Core handles large-model parallel training; Lightning supplies a high-level Trainer; Ray Train coordinates multi-node training and tuning.
• **Infrastructure:** Modal provides Python-defined serverless GPUs; Lambda Labs covers reserved/on-demand GPU resources; SkyPilot orchestrates jobs across clouds with spot recovery.

5. Catalog II: optimization, evaluation, applications, and emerging methods

The remaining named skills provide these functions:

• **Optimization:** Flash Attention reduces attention time and memory; bitsandbytes supplies 8/4-bit quantization; GPTQ, AWQ, and HQQ provide distinct low-bit weight-quantization methods; GGUF covers llama.cpp’s portable quantized format and K-quants.
• **Evaluation:** lm-evaluation-harness benchmarks language models across many tasks; BigCode Evaluation Harness measures code models with HumanEval, MBPP, MultiPL-E, and pass@k; NeMo Evaluator runs container-oriented benchmarks across local Docker, Slurm, or cloud backends.
• **Inference:** vLLM serves models with PagedAttention and batching; TensorRT-LLM optimizes NVIDIA inference; llama.cpp supports CPU and Apple Silicon execution with GGUF; SGLang accelerates structured generation and agent workloads.
• **MLOps:** Weights & Biases tracks runs, sweeps, artifacts, and models; MLflow tracks, registers, and deploys models; TensorBoard visualizes metrics, profiles, embeddings, images, and scalars.
• **Agents:** LangChain provides agent patterns and integrations; LlamaIndex builds data-connected and RAG applications; CrewAI coordinates role-based agent teams; AutoGPT provides autonomous workflows and a visual builder. A supplied reference also documents **A-Evolve**, which iteratively improves file-based prompts, skills, memory, and optionally tools using benchmark feedback and Git snapshots.
• **RAG:** Chroma is a local/cloud embedding database; FAISS performs large-scale similarity search; Pinecone is managed vector search; Qdrant supplies filtered and hybrid vector search; Sentence Transformers creates multilingual semantic embeddings.
• **Prompt engineering:** DSPy optimizes declarative prompt programs; Instructor validates structured output with Pydantic; Guidance constrains generation with regexes or grammars; Outlines uses finite-state constraints for structured text.
• **Observability:** LangSmith traces and evaluates LLM applications; Phoenix provides OpenTelemetry-based tracing and evaluation.
• **Multimodal:** CLIP aligns images and text; Whisper transcribes speech; LLaVA supports image dialogue; BLIP-2 handles captioning and visual QA; Segment Anything performs prompted segmentation; Stable Diffusion generates or controls images; AudioCraft generates music and sound.
• **Emerging techniques:** MoE Training activates sparse experts; Model Merging combines model weights; Long Context extends context with RoPE, YaRN, or ALiBi; Speculative Decoding accelerates generation; Knowledge Distillation transfers behavior to smaller models; Model Pruning creates sparse networks.

6. Selected documented workflows and copy-paste examples

The GRPO skill’s workflow is dataset → rewards → training → deployment. Its supplied template uses Qwen 2.5 1.5B Instruct, LoRA, and three format/correctness rewards; its library contains correctness, formatting, length, style, and multi-objective rewards. The guidance says to test rewards separately, add them incrementally, monitor reward metrics and `reward_std`, and begin with 4–8 generations. These are repository recommendations, not guarantees.

For sparse-autoencoder analysis, the documented installation is:

pip install sae-lens

SAELens loads a TransformerLens model and pretrained SAE, caches a layer activation, then calls `sae.encode(activations)` for sparse features and `sae.decode(features)` for reconstruction. For causal intervention work:

pip install pyvene

pyvene’s `IntervenableConfig` selects a layer, component, and intervention type; `IntervenableModel` then swaps, adds, zeros, collects, or learns rotated-space interventions. For tracing:

pip install nnsight

Within `model.trace(...)`, nnsight returns proxy objects; `.save()` makes selected activations or logits available after execution. Remote NDIF tracing is documented with `remote=True` and an `NDIF_API_KEY` environment variable.

Paper templates cover ICML, ICLR, NeurIPS, ACL-family venues, AAAI, and COLM. The documented bibliography build sequence is:

pdflatex main.tex
bibtex main
pdflatex main.tex
pdflatex main.tex

Alternatively, `latexmk -pdf main.tex` handles dependencies. Verify the current venue’s official template, anonymization rules, and page limits before submission; the repository explicitly notes that templates change annually.

7. A-Evolve API documented inside the library

The included A-Evolve reference describes a separate Python package for benchmark-driven agent evolution. It requires Python 3.11+, Git, and provider credentials. Its minimal API constructs `ae.Evolver(agent=..., benchmark=...)`; `run(cycles=10)` performs solve, evaluate, snapshot, evolve, reload, and convergence checks, returning an `EvolutionResult` with final score, completed cycles, convergence state, and score history. `EvolveConfig` controls batch size, cycles, holdout ratio, mutable layers, model, token budget, and plateau detection; `from_yaml()` loads those fields.

Three extension protocols are documented. A `BaseAgent` must implement `solve(task) -> Trajectory`; it can `remember(...)`, `export_to_fs()`, and `reload_from_fs()`. A `BenchmarkAdapter` implements `get_tasks(split, limit)` and `evaluate(task, trajectory) -> Feedback`. An `EvolutionEngine` implements `step(workspace, observations, history, trial) -> StepResult` and may implement `on_cycle_end(accepted, score)`.

`AgentWorkspace` exposes prompt functions (`read_prompt`, `write_prompt`, fragment read/write/list), skill functions (`list_skills`, `read_skill`, `write_skill`, `delete_skill`), draft functions (`list_drafts`, `write_draft`, `clear_drafts`), memory functions (`add_memory`, `read_memories`, `read_all_memories`), tool functions (`read_tool_registry`, `write_tool_registry`, `read_tool`, `write_tool`), and history/metrics readers. `EvolutionHistory` can return recent or failed observations and a score curve.

Built-in engines are AEvolveEngine for broad LLM-driven mutation, GuidedSynthesisEngine for memory-first skill curation, AdaptiveEvolutionEngine for score-responsive changes, and AdaptiveSkillEngine for skill-focused discovery. Documented benchmarks are SWE-bench Verified, MCP-Atlas, Terminal-Bench 2.0, SkillsBench, and ARC-AGI-3. Workspaces must not be mutated concurrently.

8. Demos, verification, limitations, licensing, and citation

The demo gallery illustrates intended use rather than guaranteed reproducibility. NeMo Evaluator compares Llama model sizes on GPQA; a LoRA reproduction combines TRL and GRPO; llama.cpp plus GGUF explores layer-wise quantization; FAISS measures multilingual embedding alignment. Two autoresearch demos show autonomous hypothesis testing and synthesis: one rejects an ETF-overlap hypothesis and pivots to embedding norm heterogeneity, while the other compares DPO, RLOO, and GRPO using SVD and sparse-autoencoder analyses. Treat their numerical findings as repository-reported results.

The academic-plotting demo uses a six-part Gemini prompt—framing, style, colors, layout, connections, constraints—and best-of-three selection for an architecture diagram. Its data-chart workflow uses serif defaults, a colorblind-safe palette, 300 DPI, and PDF/PNG export. The README states that chart data are synthetic distributions matching paper-reported results, an important qualification.

Operational checks should match the task: inspect generated code, run appropriate project tests, verify citations against primary sources, review plots against raw data, and check model/library versions. Some workflows require GPUs, cloud services, API keys, local runtimes, or third-party packages. External services may receive submitted prompts or data under their own policies. Individual referenced libraries can have licenses different from this repository’s MIT license.

The repository provides no single command that validates every skill, no compatibility matrix, and no independent audit of claimed speed, accuracy, cost, or research quality. It also contains count drift across supplied READMEs. Cite the project, if desired, as Orchestra Research’s *AI Research Skills Library* using the BibTeX, APA, Chicago, or IEEE forms in the root README, and check the repository’s `LICENSE` plus each dependency’s license before redistribution.