It's a bit more principled; in international agreements you have two or more independent and sovereign entities hammering out the rules by which they commit themselves to act. Since there is no World Prosecutor making certain that all parties stick to what they have agreed you need safeguards and conflict resolution built in in the deal. Trust only comes in after that (and I agree Trust in the UK is a scarce good these days).
Brits have the misconception that their doctrine of Parliamentarian sovereignty can play a relevant role in the system of safeguards and conflict resolution. More than that they think it can play this role unilaterally. Now we could point to article 27 of the Vienna Convention, but Brexiteers in general don't understand legal niceties very well. Before you know it they are talking of Brexiteering out of the UN to be truly sovereign. But, as the US ratio for not ratifying the Vienna Convention, but still applying its principles, shows us there is something deeper than a written rule for nations to allow infringements on their day to day execution of sovereignty. That ratio is that what the VC enshrines is longstanding practice already anyway.
There is a reason for this longstanding practice; if we would not live by rules as in the VC and every party would have the right to unilaterally change the rules, then you would have ask yourself the question why you bother with striking deals at all. If one side can retro-actively change the mandate for an international agreement you will have to assume as opposite party that whoever is sent doesn't have an actual mandate to negotiate at all. With so much uncertainty built in into the system the system no longer makes any sense. And once sides realize the system no longer makes sense, they revert to the default of international relations; the right of the strongest.
Brexiteers think that they stand a chance against the EU without the protections of a legal framework. That is another grave misconception. A clean Brexit makes them more subservient to the EU than any deal they could strike.