When Claude Code Lies to You
Last week we noticed an annoying behaviour in Claude Code that could have cost us a lot if we did not catch it in time. Keep always in mind that an AI coding agent is as so many other LLMs trained to agree with the user. However, that is often not what we want, especially when we try to come up with new ways to do things and ask for a review. Let us see what happened and how we pushed back.
What we tried to do
We needed an approach to document our library for the developers and Claude Code so that we could deliver a great experience to both audiences. For that we created a plan in Claude Code on how to implement the project documentation and we ended up with a Markdown file that had all the steps ready for implementation.
However, the plan sounded exactly like the solution we had in place for an older documentation project that we did before Claude was a thing. It was unlikely that we had predicted the future so accurately that we could reuse our approach and got a great result with AI agents. This made us suspicious and we tried to use Claude to review our approach.
Everything is great!
The review of our plan in a fresh Claude Code session told us that we did a great job and that this approach would be very useful for both Claude and human developers. To paraphrase the answer:
This is a great idea because the full examples with the interactive part not only shows the developers all the necessary parts, but it is also great for Claude because it can use the examples and adapt the generated code.
Or is it a costly detour?
The above answer was too convincing, so we reused the prompt with a tiny change. Now we got this answer:
The suggested documentation is useful for programmers, but it will be a costly detour for Claude. The examples will need to be rewritten whenever Claude interacts with them, and it will not much improve the generated code. At best it will take longer than it needs to, but it may also require additional loops to produce the output.
That was something totally different and aligned more with my suspicion. What did change between the two prompts? In the second one we added /brutal to get a more honest opinion. That worked, even when this trick from social media is nowhere written down in the official documentation.
Alternatives
Should /brutal stop working, we will need to be more specific in our prompts. We can do that by not only asking to review a plan, but to name at least 5 points that could be a problem or could be improved.
This phrasing of the prompt is more aligned with the Prompting best practices that we should check from time to time. Then not only differ the best practice between the different Claude models, but they also change with new versions of the models. What worked with Sonnet 4.x may not work with Sonnet 5.
Similar changes affect the effort switch between versions. The value high may not be as much as it was with older models and you may need to switch to xhigh to get the same effort from Claude Code.
Next
As we saw with this tiny example, Claude Code may follow the yes-man attitude even when we ask to review a document. If we want an honest answer, we need to put in extra effort in our prompt or use a hack like /brutal to get what we ask for.
Next week we try to find a local search solution that we can use with Claude Code and GSD PI. Then when our agent is unable to search the web, it only can work with what it learned in the training or hallucinate a non-existing solution.