AI Lab
Research notes and experiment results from the AVELIN team. Each publication is a page on this site with the full paper attached as a PDF.
Publications
- PDFWhere the Time of Long-Context Requests Goes on One Production Server: 62% Generating, 16% Reading the Prompt, 17% WaitingWhere the time of 18,364 requests goes on one production inference server with a context window of a million tokens that serves agentic sessions, from the server's own counters over 31.5 hours. 62% of the summed request time is generating, 16% reading the prompt and 17% waiting in the queue. The first token comes within a second for 71% of requests; the 8% that take more than ten seconds hold at least 61% of all time to the first token.
- PDFLong Prompts of One Operator's Coding Agents: 19% of Requests Carry 200K Tokens or More, Two or More Such Requests Overlap 9% of the TimeA study of 104,088 requests made by one operator's coding agents through one gateway, from token counts alone. The median prompt is 74K tokens; 19.3% of requests carry 200K or more and hold 60% of all prompt tokens. Where cache counters exist (70% of requests), 94.1% of prompt tokens are cache reads. Two or more requests of 200K or more are in flight 8.9% of the time, with a peak of 11.
- PDFA Host-RAM Tier for the KV Cache: Up to 33× Sooner After Eviction, 16% of Cache Misses Served in ProductionAn evicted conversation normally has to be recomputed. With each KV-cache block also kept in host RAM, a conversation of about 200K tokens returned in 1.37 and 0.78 seconds instead of 44.62 and 12.37. Over 26 hours of production on one H200 server, the RAM pool supplied 41 million tokens in 517 restores, 16% of what the GPU cache had missed, for 24 seconds of transfer.
Questions about the AI Lab
What is published in the AI Lab?
Research notes and results of experiments run by the AVELIN team.
Is there a PDF for each publication?
Yes. Each publication has the full paper attached as a PDF, with a download link on the page.
Can I cite or share a publication?
Yes. Link to the page or to the PDF. Each page lists its authors and publication date.
How is the AI Lab different from News?
News covers company and product announcements. The AI Lab holds research write-ups and experiment results.