ResearchPractical AIProjectsOpen SourceAbout
Research/Running AI on Your Own Computer: What Makes It Faster, and What Quietly Changes the Answers
Local AICost AnalysisSpeed vs AccuracyBenchmarkingOriginal Research

Running AI on Your Own Computer: What Makes It Faster, and What Quietly Changes the Answers

I tested what happens when a small AI model sorts 500 customer-support tickets on one ordinary graphics card, changing one setup choice at a time. The best setup was 2.2x faster than the default and cost a fraction of a cent per thousand tickets, against about $22 per thousand through a paid AI service. Overall accuracy barely moved. But individual answers did: the faster, more compressed setups gave a different answer on about 1 ticket in 12.

Bart Porcincula · Published · Updated

Finding

On one consumer graphics card (16 GB), sorting 500 support tickets with the open Qwen3.5-9B model, the best setup (llama-server, 4-bit compression, 4 tickets processed at once) ran 2.2x faster than the Ollama default (178 vs 80 tickets per minute) and cost $0.0032 per 1,000 tickets in electricity, against $22.43 per 1,000 for Claude on the same task. Accuracy did not measurably change, but the 4-bit model gave a different queue answer than the higher-quality 8-bit one on 8% of tickets.

Local AI setup study on a single 16 GB graphics card (Oct 2026)

26 setup variations, each run 3 times on 500 support tickets (3 questions per ticket), using the Qwen3.5-9B model with fixed settings and one fixed set of instructions. Measured speed, the graphics card's power draw, how many answers changed against a higher-quality reference setup, and cost per 1,000 tickets. Scored by a fixed script against a protocol written before the tests were run.

MeasurementValueNote
Out-of-the-box setup (Ollama)80.3 tickets/minthe starting point everything else is compared to
Best setup found (llama-server, 4-bit, 4 tickets at once)178.4 tickets/min2.22x the out-of-the-box speed
Same model file, two different programs: llama-server vs Ollama142.1 vs 92.1 tickets/min1.54x faster, with nothing else changed
Tickets whose answer changed, 4-bit vs 8-bit model (category / urgency)8.0% / 5.2%overall accuracy differed by under 1.4 points, which is within the margin of error
One request asking all 3 questions vs 3 separate requests99.1 vs 142.1 tickets/min30% slower, and changed 16-19% of category and urgency answers
Electricity cost per 1,000 tickets, best setup$0.0032at PHP 15/kWh; $0.005-0.009 including the card's price spread over an assumed 4 years
Claude on the same task, 1 request per ticket$22.43 per 1,000 ticketsthe pay-per-use price of a run that was actually covered by a subscription plan

The short version

Some businesses run an AI model on their own hardware instead of paying a provider for every request. The way that model is set up matters a lot, but not in the way most people assume. I tested it on one ordinary graphics card with a realistic job: sorting 500 customer-support tickets.

  • Speed: the best setup was 2.2x faster than the default, with no new hardware. The fix was swapping one free program for another.
  • Cost: about $0.003 of electricity per 1,000 tickets. The same job through a paid AI service (Claude) comes to about $22 per 1,000, thousands of times more.
  • Accuracy: it did not measurably change between setups.
  • The catch: individual answers did. A smaller, more compressed version of the model gave a different answer on about 1 ticket in 12. The overall score was the same, but the tickets that went wrong were different ones.

Takeaway: for steady, repetitive work like sorting, tagging or triaging, running a small model locally can be dramatically cheaper. But the savings come from testing the setup, not trusting the defaults, and "slightly different answers" has to be an acceptable trade for the task.

What I tested

Imagine a support team receiving 500 tickets. For each one, the AI answers three questions: which of 10 teams should handle it (the queue), how urgent it is (low, medium or high), and whether it is a service incident.

I ran this on a single RTX 5060 Ti graphics card with 16 GB of memory, using an open model called Qwen3.5-9B. The model, the instructions, and the 500 tickets stayed identical every time. Only the setup around the model changed. I wrote the test plan down before running anything, so I could not adjust it after seeing results.

This is not a contest of which AI is smartest. Every setup, Claude included, scored below a plain keyword-matching program on the 10-way queue question (0.29-0.34 against 0.418). That tells me the example labels are noisy. Read the accuracy numbers as "did the setup change anything?", not "how good is this AI?".

A quick key to the terms

  • Tickets per minute: how many tickets get sorted each minute. Higher is better.
  • 4-bit, 8-bit, BF16: how compressed the model is. Think of image quality: 4-bit is a smaller, faster file with some detail lost, 8-bit is in the middle, and BF16 is the uncompressed original.
  • Ollama and llama-server: two free programs that run an AI model on your own machine. Ollama is the popular, easy one. llama-server is the more hands-on one.
  • Tickets at once: how many tickets the card works on in parallel, like a kitchen cooking four orders together instead of one at a time.
  • Changed answers: how often a setup gave a different answer than the 8-bit reference setup. It does not mean wrong, only different.

What moved, and what did not

Across 26 setup variations, each run three times, accuracy stayed in a narrow band: queue 0.29-0.34, urgency 0.40-0.41, incident 0.75-0.76. Apart from a few reference runs on a different program (vLLM), no setup was reliably better or worse than the 8-bit reference; the differences were inside the margin of error. Speed, on the other hand, varied by more than ten times.

What did change was individual answers. Going from 8-bit to 4-bit changed 8.0% of queue answers and 5.2% of urgency answers, with no measurable change in accuracy. Two setups can score the same and still disagree on many tickets. That matters if anyone downstream audits or compares results.

SetupTickets per minuteAnswers changed vs 8-bit (category / urgency)
Ollama out of the box (4-bit, one ticket at a time)80.312.4% / 6.6%
llama-server, 8-bit (the reference)128.6-
llama-server, 4-bit142.18.0% / 5.2%
llama-server, 4-bit, 4 tickets at once (best)178.47.8% / 5.4%
Ollama, same model file as the 4-bit row92.19.4% / 5.8%
llama-server, uncompressed BF16 (too big for the card)48.60.8% / 1.8%
llama-server 4-bit, one request asking all 3 questions99.119.2% / 16.4%

What each choice did, one at a time

  • Compression. Each step toward less compression costs 4-6% in speed but gives more consistent answers: the 5-bit version changed 3.4% / 3.8% of answers, the uncompressed version under 2%. The catch is that the uncompressed model (a 17.9 GB file) does not fit in 16 GB, so part of it has to run on the computer's main processor, and it ran 2.6x slower than 8-bit. The spill is inferred from the file size against the card's memory; the server log did not record it directly. A model that does not fit on your card is the most expensive mistake I saw. In an earlier experiment the same thing made a run 3.6x slower.
  • Which program runs the model. With the identical model file and one ticket at a time, llama-server was 1.54x faster than Ollama.
  • Handling several tickets at once. llama-server gained 20% at two tickets at once and 5% more at four, then slowed down. Ollama stayed flat from two upward. Its log explains why: for this type of model it reports that parallel requests are not supported and runs a single lane. That is a property of one Ollama version and one model type, not of Ollama in general.
  • Asking all three questions in one request. It sounds more efficient, but it was 30% slower per ticket and changed 16-19% of answers. The model writes words faster in the combined version, but it has to write the whole reply before the ticket is done.
  • Forcing a strict answer format, and reusing instructions between requests. A format constraint was free and produced identical output to the unconstrained run, because the model was already giving valid answers. Reusing the instructions (called prompt caching) made no difference, since nothing could be reused across the three questions on this model.
  • Compressing the model's short-term memory (the "KV cache"). It changed neither speed nor answers beyond normal noise.
  • A reference point. A more demanding program (vLLM, running on Linux inside Windows) with a different compression format reached 522 tickets per minute, almost 3x the best setup here. It also lowered queue accuracy by 1.8 points (range -3.2 to -0.4). Treat it as a marker of what is possible, not a fair comparison.

What it costs

Electricity for the best setup is $0.0032 per 1,000 tickets (at PHP 15 per kWh). Including the graphics card's price spread over an assumed four years, it is $0.005-0.009. Claude on the same task, one request per ticket, is $22.43 per 1,000. That is the pay-per-use price of a run that was actually covered by a subscription plan.

Buying the card for this job pays back at about 430 tickets a month. Electricity is negligible, so that figure comes from comparing the card's price against Claude's per-ticket cost. It is cost per decision, not per correct decision: both land below the simple keyword baseline on this task.

A caveat about my own pass/fail rule

Before testing, I set a rule for when a setup counts as "answers did not change": its answers must differ from the reference by no more than twice its own run-to-run randomness (and at least 2%), and its accuracy must not drop by more than 3 points. The best setup passed. But an identical earlier run of the same settings failed.

The reason is that the allowance grows with measured randomness, and handling several tickets at once adds randomness (2-5% from run to run, against exactly zero when one ticket is processed at a time). So the rule rewards noisy setups. I report it as written. My own reading is that 4-bit compression, not parallel processing, causes most of the drift.

Limits

  • One graphics card, one model family, one task, one sample of 500 tickets, one random seed.
  • The answers are only a few words long, so this job is dominated by reading the ticket and per-request overhead. Jobs that write long answers would rank the setups differently.
  • Steady load only. I did not measure how it copes with sudden bursts of tickets.
  • Energy is the graphics card's power draw only. The extra power used when the uncompressed model spills onto the main processor is real but unmeasured.
  • The exchange rate, four-year life and usage assumptions in the cost section are mine.

What I would take from it

  1. Test before accepting defaults. Apart from the all-in-one-request variants, the default was the slowest setup that fit on the card, and the fix was swapping software, not buying hardware.
  2. Do not trust accuracy alone to say a change is safe. Same score, 8% different answers.
  3. Check what a feature really does on your model. Caching, parallel processing and combined requests all behaved differently from the folklore on this model.
  4. Write the pass/fail rule first, then read where it is odd. My rule was gamed by noise, and writing that down is more useful than hiding it.

The protocol, raw per-run files, scoring script and charts are in the project repository.

Is local AI cheaper for sorting customer enquiries?

The graphics-card electricity cost was very small on this ticket-sorting task. It was not a full operating-cost comparison: hardware, the rest of the computer's power, setup, maintenance, and human review need their own budget. The Claude figure is the historical API-equivalent value of a subscription-covered run, not an invoice paid or a current price quote. Test your own categories and include correction time.

Cite this research

Bart Porcincula. "Running AI on Your Own Computer: What Makes It Faster, and What Quietly Changes the Answers." Published 2026-10-06; updated 2026-11-08. https://porsync.com/research/serving-choices-speed-not-accuracy-local-llm

Permanent article URL