Trusting Trust Again

How much can you trust code produced by LLMs? How much can you trust any code at all? Ken Thompson wrote “Reflections on Trusting Trust” in 1984. Do you think it’s a solved problem by now?

Thompson used a compiler as an example. But of course it’s much more than this. Basically, any tool used in software development is a potential source of problems. We have trust issues with software packages (NPM is a good example, since it’s popular but others aren’t better). We even have trust issues with our CPUs.

Even before LLMs, it was already a huge problem. We’ve been trusting strangers whose code or tools we use. The problem with an LLM is that it’s not even a person - it’s an “alien mind”. Nobody, not even its creators, truly knows what’s inside. No matter how human-like its language is, no matter how capable the model is, there’s always a possibility that it might do something you didn’t want. There might be a “devil” inside of it and nobody can prove that it’s not there.

So do you expect me to talk about potential solutions now? Well, if a Turing Award winner didn’t give them to us then I have a much lower chance of doing so. But we can learn at least one thing:

You can’t trust code that you did not totally create yourself.

No amount of verification or type safety can fix this. Especially if you let the LLM write the specs and proofs too. What should we do about this? Should we not use LLMs to generate code? I don’t think so. If that was the point, we should have stopped using compilers a long time ago.

At least write your proofs yourself.