LoRA: when it helps, when a good prompt beats a bad adapter

Training a full model is out of reach for most people. A low-rank adapter is a much smaller file that adjusts an existing model's behaviour towards whatever you trained it on — a person, a product, a visual style.

It is quick to train, small to store, and can be applied at variable strength or combined with others. That accessibility is why it is everywhere.

It is also frequently the wrong tool, reached for when a well-written prompt or a reference image would have done the job in a fraction of the time.

Specific — a particular face, a particular product, a particular visual treatment. Not a general category, which the base model already handles.

Repeatedly — across many images, over weeks. Training has a fixed cost that a handful of images will not repay.

Not describable — if a paragraph and a reference image get you there, you do not need training. Identity genuinely resists description; most styles do not.

Fail any one and the honest answer is a better prompt, a reference image, or accepting the base model's version.

The instinct is to supply your best photographs. It is wrong, and it produces an adapter that reproduces those photographs rather than the subject.

What you want is the subject held constant and everything else varied — angle, distance, lighting, background, expression. Twenty varied images consistently beat sixty similar ones.

Include some unflattering and awkward frames. A dataset of only ideal images teaches the model that the subject only exists under ideal conditions, and it will refuse to place them anywhere else.

Caption the background, the lighting, the angle, the clothing — the things that change between images. Do not describe the subject's permanent features in detail.

The logic is that anything you describe becomes something the model associates with that description rather than with the subject itself. Describe the face carefully and the face becomes conditional on those words; leave it undescribed and it becomes what the adapter is for.

Use a consistent trigger word that does not collide with ordinary language. Something distinctive, used in every caption, so you can invoke the adapter deliberately rather than having it colour everything.

At maximum strength an adapter dominates the base model, and you lose the general capability you were adding to. Poses become limited, backgrounds drift towards the training set, and the image becomes a variation on the dataset.

Start well below full and raise it until the subject is recognisable, then stop. That threshold is usually lower than expected.

Where you combine several adapters, reduce each further. Two at high strength compete and the result resembles neither.

Generate the subject in a setting, pose and light that appear nowhere in the training data. If it holds, the adapter learned the subject. If it drags a training background along or refuses the pose, it memorised.

The fix for memorisation is always the dataset — more variety, fewer near-duplicates — rather than more training. Additional steps on a narrow dataset make the problem worse.

Keep the dataset alongside the adapter file. Retraining with two more angles is straightforward; reconstructing which images you used a year later is not.