Why AI colorizers turn old photos sepia, and how we fixed it

October 2026 · 6 min read

If you've used AI colorization, you've probably seen it: you upload a crisp black-and-white photo and get back something brown, as if it had been dipped in tea. It isn't your photo. It's a habit the model picked up during training.

Measuring the problem

Most colorization models are tested on modern color photos that have been turned gray, because then the "right answer" is known. That hides the problem. Real old photos look different: faded blacks, film grain, soft focus, dust, and tones that old film recorded differently from a modern camera.

So we built a second test: 300 genuine black-and-white photos, kept out of training, and counted how often each model's result came out as a single warm, brownish tint instead of real colors. Midway through training, our own model did this to 24% of them. That's one family photo in four.

Where the brown comes from

We found three causes, and they add up.

1. The training photos taught it. Our model learns color from about 1.4 million openly licensed photographs. A few percent of "color" photos are actually old prints that were scanned in color, or modern photos with a vintage filter. To the model, they say: when a photo looks old, the answer is brown.
2. When in doubt, models average. Part of training rewards being close on average. When a model isn't sure whether a coat was blue, green or red, the safest average is a muddy brown. Old photos are exactly where models are least sure.
3. Old film saw color differently. Early "orthochromatic" film was almost blind to red, so red lips and red cloth print nearly black, and blue skies print almost white. A model trained only on modern grayscale conversions has never seen that, and gets confused.

We also noticed something we didn't expect: fading matters more than blur. Photos whose blacks had faded to gray came out sepia or gray three times as often as crisp ones, while blurry photos did no worse.

What we changed

• Cleaned the training data. We removed about 64,000 warm-tinted photos and every photo from accounts that mainly post scanned black-and-white archives.
• Taught it what old photos look like. More than half of the training photos now get simulated aging: fading, grain, blur, dust, scratches, JPEG damage and low resolution. Many are also converted to gray the way old film would have recorded them, not the way a modern camera would.
• Practiced on real old photos. During training, the model also colorizes genuine black-and-white photos (never the 300 we test on). A second network, which learns to spot fake-looking results, pushes it away from brown and gray answers.
• Rewarded honest color. The color reward now only kicks in when a result is duller than the real photo, so the model isn't pushed to oversaturate either.
• Checked the whites. The brightest parts of a photo (snow, white shirts, an overcast sky) are almost never yellow in reality. A final white-balance step removes any warm cast those areas picked up, fading out toward the shadows. It took yellow-tinted highlights from 15% of old photos to almost none, and slightly improved accuracy on photos where we know the real colors.

The result

Note the middle column: a model can avoid brown simply by leaving photos gray. We wanted both numbers down.

It isn't perfect. A few scenes still lean golden, and colors are always an estimate. That's why Ungray has Recolor: if you know grandma's coat was blue, tap it and make it blue. Only the color changes; every detail of the photo stays exactly as it was.