I finally got the drifting problems sorted out! I added a lot of data logging to the code and spotted the switch bouncing when I ran it drawing circles over and over.

[(98)=100][(82)=84][(101)=204][(116)=118]
[(103)<104>=105]
[(0)=183]
[(40)=42][(133)=214][(87)=89][(103)=150]

Its not terribly easy to read as I needed the data compact so I could scan through large amounts of it. What is happening though is that each step is taking anywhere from 40-130ms to complete, but then there is a point where the switch fires at 103ms 104ms 105ms after receiving an instruction to move. This means the switch is closing, but then opening and closing again within 2ms.

I added some simple code to reject any switch changes that happen too quickly to the Arduino Sketch:

if(millis()-lastclick_millis[axis]<20){
    // De bouncing
}
lastclick_millis[axis] = millis();

Since the normal switch changes were happening at around 40ms and up, and the bounces were all 2ms and less I figured rejecting anything below 20ms was a fairly safe boundary.

Then I set my H shape test cutting, and it worked perfectly! Here it is after a quick light sand and a coat of mineral oil, with a 9v battery for scale:

Now on to getting the auto start/stop working and cutting more things!