Play in the worm-side of the clip
The goal here is to keep the clip perpendicular to the worm and router body at all times. The easiest way to do so is to make the hole longer and have it fit the worm as closely as possible so the elongated hole can’t tilt on the worm shaft.
The threads on the worm have a 1/2-inch diameter, and the clip has a 1/2-inch hole for it. The clip’s hole has 1/8-inch of clearance all the way around its perimeter, making for a 3/4-inch area to work with.
A Hillman bronze machine bushing from Lowes is 1/2-inch long, has a 1/2-inch hole through the center, and has a 3/4-inch diameter flange on one side They come in packs of one for $3.91, and it has a Lowes item number of 1772.
Attaching this bushing to the side of the clip will extend the length of the hole from 5/8-inches to 1 1/8 inches, where the latter 1/2-inch of the hole doesn’t have any other mechanisms to present any additional slop, like the clip does.