Agent context infrastructureOpen source · active
Repo2GPT
An MCP, API, and CLI system that lets coding agents inspect a repository, budget context, and recover from oversized or noisy snapshots.
- 3
- MCP tools
- 500 KB
- default file guard
- 10
- GitHub stars
Alexei Korol · AI/ML engineer
I build and evaluate LLM-powered systems: agents, RAG, and evals.
Production-minded AI work with bounded context, observable behavior, explicit failure paths, and measurements that survive the repository click.
- 83RAG eval questions
- 0.952document R@10
- 3agent MCP tools
90-second review path
- Ask the systemSee citations and degraded behavior.
- Inspect the architecturePrompt contract, cost math, eval table.
- Open working evidenceCode, tests, retrieval results, tradeoffs.
Flagship · cited portfolio RAG
Ask my portfolio
Answers are constrained to projects, engineering notes, and the resume. Every supported claim links to its source.
Projects as evidence
Three systems. Deep enough to audit.
Evaluated retrieval systemOpen source · evaluated
SongCraft RAG
Hybrid retrieval over a 45-document songwriting corpus, with local embeddings, cross-encoder reranking, cited answers, and a committed 83-question golden set.
- 0.952
- document recall@10
- 0.711
- exact chunk recall@5
- 7,370
- unique chunks
LLM evaluation and guardrailsOpen source · calibration stage
Prosody Judge
An eight-rubric LLM-as-a-judge pipeline with async batching, multi-run self-consistency, structured outputs, uncertainty flags, checkpoints, and spend limits.
- $0.067
- four-item calibration
- 24
- judge calls per full item
- 2.0 → 8.7
- authenticity separation
Engineering notes
I built it. Here is what broke and what moved.
Build-time Markdown, route-level metadata, RSS, per-post social cards, code, numbers, and the wrong turn. No generic AI explainers.
Read lab notesHybrid retrieval made my top result worse
I added BM25 to a dense RAG retriever, watched recall@1 regress, and used a cross-encoder to turn a wider candidate pool into a measurable win.
I stopped treating one LLM score as a measurement
A single judge call hid variance. Three runs per rubric cost more, but exposed disagreement and created a concrete human-review gate.
I built this portfolio RAG to fail usefully
The model call is optional; retrieval, citations, refusal checks, and a useful degraded answer continue when quota or configuration fails.