Chose local LLM over cloud API for my side project, rough start though

I picked a local Llama 3 model over a cloud API for my email summarizer project because I wanted privacy and zero ongoing costs. Took me about 6 hours to get it running on my RTX 3060, and the first output was garbage, like it mixed up two clients and added a random recipe line. Lowered the context to 2048 tokens and switched to a 4-bit quant, now it works but still takes 40 seconds per email. Anyone else tried this route and found a better middle ground, like a hybrid setup?

1 comments

Log in to join the discussion

Log In

1 Comment

blair_davis10d ago

My llama once summarized a grocery list as a haiku, so honestly that recipe win sounds about right.

7