Mina handles so many skarvar that I can't count them.
If I sand too early and it is not completely dry, the edge can "give way", but as long as the entire sanding block is against the surface and it is dry, I find that it lasts much longer than regular sandpaper.
I guess it's the technique and that you aren't keeping the sanding surface flat against the substrate.