I also have experienced this and I decided to do some research. I took the four v-wheels off my Z plate, took out the bearings and washers and measured some things.
According to this drawing, the ridge inside the v-wheel should be 1mm +0.00/-0.08. Since I'm old-school I use inches, so that means it should measure between 0.039" - 0.036".
The ridges in my wheels measured 0.031", 0.032", 0.033" and 0.036". Which means they are out of tolerance and too small.
The washer that is sold by Inventables and Openbuilds is supposed to be 1mm thick. The drawing on Openbuilds shows the thickness as 1mm +0.076/-0.00. Which means it should measure between .039" - .042". My washers all measured .043" which means they are barely out of tolerance on the thick side.
All of this taken together means that there is 0.007" - 0.012" of play in the wheels. Which doesn't sound like a lot until you think about the upper wheels being pushed one way and the lower wheels being pushed the other way, which equals 0.024" and over a distance of about 6" gives us about 0.3 degrees off.
So the wheel ridges are too small and the washers are too thick. This is the worst-case tolerance condition. Even if the wheel ridges were small but still in tolerance (0.036") and the washers were too thick but still in tolerance (0.042") we would still have slop of 0.006". Which is not great.
But as it stands we are double that.
I'm not sure what the correct fix is. It would be nice if the wheels were in tolerance at least. Washers that actually measured 1mm would help.
McMaster-Carr sells precision shims that are 6mm id x 9.78mm od x .031" thick (yes, they are listed using both metric and imperial units) which would give a little pre-load on some of the bearings. They are $10.54 for 10, and of course I need 12.

I'm going to poke around some more and see if I can't find other shims. I'm also sending this info to Carbide support.