LLM Systems in Production: research, evaluation, and reliability
AbstractApplied LLM engineer building production LLM and agentic systems. My work centers on autonomous AI agent pipelines, RAG retrieval, and agent memory across proprietary and open-source models — with a focus on reliability, cost, and latency in production. In production this powers AuditAgent, an AI security auditor trusted by teams like Uniswap, Lido, and UBS; in research it produced a first-author paper in IEEE Network Magazine on fine-tuning LLMs for network analysis.
At a glance
Selected outcomes
Measured on shipped systems — not benchmarks run in isolation.
| 40% | recall of professional audit findings | AuditAgent |
| 67% | score on the EVMBench benchmark | AuditAgent |
| −27% | prompt size via agent memory + unified RAG | AuditAgent |
| −96% | manual network-analysis time | BGP-LLaMA |
Where I've worked
AI Engineer — AuditAgent
Built an autonomous, multi-stage LLM agent pipeline for smart-contract security: detection, validation, and persistent agent memory over a unified RAG index. Also built the in-house multi-provider LLM router (OpenAI · Anthropic · Google) with streaming, retries, and cost tracking.
AI Researcher — LLMs for Network Analysis
Built BGP-LLaMA, cutting manual network-analysis time by 96% with a fine-tuned LLaMA and few-shot prompted GPT. First-author publication in IEEE Network Magazine (Mobile-LLaMA), including a purpose-built instruction-tuning dataset.
Selected work
AuditAgent ↗︎
AI security auditor for smart contracts. A multi-agent pipeline reaching 40% recall of professional findings and 67% on EVMBench, scanning up to 12k LoC per run with resumable checkpoints.
Opencode Agent Pod ↗︎
Turns opencode — a CLI coding agent — into a programmable, fully autonomous agent behind an HTTP/SSE API; each request runs a complete tool loop in an isolated sandbox. General-purpose agent capability any project can drop in, well beyond coding.
AI Customs ↗︎
Self-hosted LLM that flags discrepancies in customs declarations — Gemma-3-27B on local GPUs (vLLM/TGI) with Docling + OCR parsing and JSON-schema-constrained output, so sensitive trade data never leaves the operator's environment.
BGP-LLaMA ↗︎
Natural-language BGP anomaly detection: a fine-tuned LLaMA (served with vLLM) and GPT behind a React/FastAPI app that cut manual network-analysis time by 96%.
Mobile-LLaMA ↗︎
Instruction fine-tuning of open-source LLMs for 5G network analysis. First-author paper in IEEE Network Magazine, with a purpose-built training dataset (+90 pts accuracy).
Contact me
Always happy to compare notes on agents, retrieval, and evaluation.