Per-language evaluation harness

Measures MASSIVE intent accuracy, macro-F1, ECE, confidence, and per-case output for selected languages.

Parent repository GitHub

Type: research command
Repository: NandhaKishorM/laya
Readiness: Usable for evaluation and carefully validated deployments, but shipped calibration, multilingual task quality, option-order sensitivity, and workload-specific accuracy require local testing before automated actions.
Keywords: 6

Location

Repository path

research/eval/laya_eval.py

Invocation

python research/eval/laya_eval.py --model MODEL --langs LANGS [--out FILE]

Setup

Installation / activation

Direct Python script execution

Keywords

evaluationMASSIVElanguageECEJSONcheckpoint

Repository context

NandhaKishorM/laya

Apache-2.0 multilingual decision-model toolkit that provides Python and TypeScript SDKs, local CLI and HTTP serving, ONNX runtimes, framework integrations, evaluation harnesses, and an optional MCP server for fast typed choice, ordinal-score, and yes/no decisions; no native Claude Code plugin, hook, skill, or Claude-specific installation path is evidenced.

machine learningnatural language processingclassificationdecision routingmultilingual inferenceguardrailscontent moderationsupport automationmodel routingevaluation

Open full repository research