ResearchPractical AIProjectsOpen SourceAbout
Research/Before You Build: A Way to Test Whether a Small AI Can Do the Job on Your Device
Edge AISmall Language ModelsQuantisationFeasibilityAI StrategyMethodology

Before You Build: A Way to Test Whether a Small AI Can Do the Job on Your Device

Most projects that put AI on a phone, kiosk or other small device start by training a model and end by discovering it never fit the device. This is the reverse order: a measurement-first method that answers "can a small AI model do this job on this hardware at all?" before any training, using a simple pass/fail gate based on how much quality survives compression and whether it fits the device.

Bart Porcincula · Published · Updated

Finding

Feasibility for running a small language model on an edge device can be decided without training anything: benchmark candidate models at several compression levels (quantisation) and treat retention (score at that compression ÷ score at 8-bit) as the gate. Below 0.85 is degradation, below 0.70 is collapse. If no candidate clears the gate inside the device's memory and speed limits, fine-tuning cannot save the project and should not be started.

Edge small-model feasibility framework: decision thresholds

The fixed measures and go/no-go thresholds of the framework. These are the numbers a feasibility verdict is read against. They are set before any model is measured so the verdict cannot be negotiated after the fact.

MeasurementValueNote
Quality kept after compression: acceptable≥ 0.85score after compression ÷ score at 8-bit; keeps at least 85%
Quality kept after compression: degraded0.70 – 0.85usable only if the task tolerates errors
Quality kept after compression: collapsed< 0.70hard fail for that model at that compression
Behavioural adherence% of prompts where the model follows its assigned rolee.g. giving a hint without giving away the answer, or returning correctly formatted data
Hardware envelopespeed · time to first word · peak memorymeasured for each model size and compression level, then matched to a class of device
Training runs required0the entire verdict is produced with public models and prompts only

The short version

The usual plan for putting AI on a small device is: collect data, train a model, then squeeze it onto the hardware. It fails in a specific and expensive way. Months into training, the team discovers that at the compression the device actually requires, the model stops working properly. The training was real. The feasibility was assumed.

The fix is to reverse the order and test the physics before building the product. Take existing public models, compress them the way the device demands, and see whether any of them can do the job. If none can, you have your answer for the cost of a few evenings on a desktop, not a training budget.

Takeaway: before committing to an on-device AI project, run a feasibility test with a pass/fail line drawn in advance. It costs a small fraction of the project and shows whether the rest is worth spending.

This piece describes a method, not a result. The thresholds below are design choices fixed in advance, not measured findings, and the full framework has not yet been run end to end on a project. One part of it has: in the distillation study, compressing the served model from 16-bit to 4-bit kept 97% of its accuracy, comfortably above the 0.85 gate.

Why this order matters

Fine-tuning improves a model that basically works. It does not rescue one that has collapsed under compression. So the first question is not "how do we train it?" but "does any candidate model, at the compression this hardware demands, do the job at all, using nothing but instructions?"

The four parts of the framework

Four fixed components, defined before anything is measured:

1. A fixed test set, not a training set. A few hundred items with known correct answers, covering the real task, including its hardest part (for a multilingual task, the language with the least available data; for a reasoning task, the multi-step questions). It stays small on purpose, because it is a ruler. The moment it grows into training data, the study has quietly become the product.

2. Measures fixed in advance. Accuracy, plus the behaviours that decide real usability. Does the model stay in its role (a tutor that hints without revealing answers, a data extractor that returns correctly formatted output)? Report the hardest slice separately and never average it away. A score that blends an easy majority with a failing minority is how infeasible projects get approved.

3. Quality kept after compression as the headline number. Compression makes a model smaller and faster, like lowering the quality of a photo, but it can cost the model abilities. For each model, take the 8-bit score as the baseline, then re-run the identical test at each lower compression level. The retention ratio is the score after compression divided by the 8-bit score. Above 0.85 the compression is roughly free. Between 0.70 and 0.85 the task decides whether the loss is tolerable. Below 0.70 that model at that compression is out, however good it looked uncompressed. Abilities with the least data behind them, and multi-step reasoning, usually collapse first, which is exactly why the hardest slice is reported separately.

4. A hardware envelope that is measured, not estimated. Speed, time to the first word, and peak memory for each model size and compression level, matched against what the target device actually has. A model that passes the quality gate but needs 6 GB of memory on a 3 GB device fails feasibility just as hard as one that collapsed.

Reading the verdict

The go/no-go is mechanical by design. At least one model at one compression level must clear the retention gate on every measure, including the hardest slice, and fit inside the hardware limits. If one does, the project is feasible, and the same test becomes the safety net for every later round of fine-tuning. If none does, the honest conclusions are "wait for better small models" or "change the hardware target". The framework has delivered that answer without a training budget.

The discipline that makes this work is separating measurement from judgement. The tests produce numbers, and a human reads them against thresholds that were locked before the first run. Feasibility studies fail as instruments when the thresholds move after the results arrive.

Why this is a strategy tool, not just a test

"Should we build this?" is the most expensive question in applied AI, and most organisations answer it by piloting the product. This framework answers it by piloting the physics: what the model can do under the real deployment constraints, at a small fraction of the cost. The same pattern applies to every adoption question worth asking: define the pass/fail line before you fall in love with the build.

Cite this research

Bart Porcincula. "Before You Build: A Way to Test Whether a Small AI Can Do the Job on Your Device." Published 2026-07-05; updated 2026-10-07. https://porsync.com/research/edge-slm-feasibility-framework

Permanent article URL