Information-Driven Design of Imaging Systems

There's a quiet assumption baked into how we judge cameras, microscopes, and telescopes: if the image looks good, the system is good. A team behind a new NeurIPS 2025 paper is pushing back on that, and their argument is pretty convincing once you sit with it.

Their point is that plenty of imaging systems never produce a human-readable picture at all. Your phone's raw sensor data gets processed before you see a photo. MRI machines record frequency-space data that means nothing until it's reconstructed. Self-driving cars feed raw camera and LiDAR streams straight into neural nets, no visual step involved. In all these cases, what actually matters is how much information about the object survived the noise, not whether a human would call the measurement pretty.

So the researchers built an estimator for mutual information, the classic Shannon-style measure of how much a measurement reduces your uncertainty about what produced it, using only noisy measurements plus a known noise model. The clever part is splitting the problem: since photon shot noise is Poisson and readout noise is Gaussian, the noise-only entropy term is calculable directly, so the hard part is learning the total variation in measurements with something like a PixelCNN, a full Gaussian, or a stationary Gaussian process, depending on how much accuracy versus speed you need. Earlier information-theoretic approaches either treated cameras as idealized communication channels, ignoring real lens and sensor limits, or required an explicit model of the objects being photographed. This one sidesteps both traps by working straight from the data.

They tested the idea across four very different domains: Bayer-pattern color filters in ordinary cameras, telescope-array placement for radio astronomy, mask-based lensless cameras whose raw output looks like static, and LED-array microscopes predicting protein expression. In every case, higher estimated information tracked with better downstream results, matching what expensive neural-network reconstruction or classification pipelines would have told you anyway, but without needing to build or train those pipelines first.

Then they went a step further with something called IDEAL, which uses gradient ascent directly on the information score to tune imaging hardware parameters, skipping the decoder network entirely. Starting from a randomly arranged color filter array, IDEAL converged on a design matching what full end-to-end training (hardware plus decoder, trained jointly) produces, but with far less memory and compute, since there's no decoder to backpropagate through. The team thinks the same trick could generalize to other sensing setups, chemical or biological sensors included, anywhere you've got a known encoder and a known noise model.