Circular Buffer Performance Trick

Update 12/04/2024: Read at the end of the post for more info.

I have been hacking on AI agents recently for both fun and profit as part of the work I’m doing for one of my clients.

They’re mostly text-to-speech (TTS) agents leveraging LLMs for generating text which is then turned into voice by a trained TTS model.

As you [probably] know, maintaining conversation with LLMs over a longer period of time requires maintaining the conversational context and sending it back to the LLM along with your follow-up prompts to prevent the LLMs from “hallucinating” from the get-go.

[Read More]