Saturday, March 30, 2013

Allen Key Entity Extraction I: Mazak the Misaligned

(Disclaimer: this gets heavy on technical stuff. If you have zero interest in machines, you should probably stop reading now. Part 2 and/or 3 will have more general entertainment. If you like machines, keep reading. You might learn something.)

     It's been a while since I've posted anything. That's largely because grad school life doesn't lend itself to many interesting stories (at least not my grad school life). I spend about half my time sleeping, commuting, eating, and generally taking care of myself. The other half of my time is split between class, research, errands, dicking around, and a little side project I'm working on for fun with these guys: Basically, the most exciting thing that happens to me on a regular basis is recognizing total strangers on the T during my morning commute.

     Incidentally, and not totally unrelated to the bore-factor of my life, research can move really slowly sometimes. For the last three months in particular, not much has been accomplished in lab on my part. There's a few reasons for this:
  1. IAP. Why would I work that hard when I'm used to taking all of January (for the last four years) to do absolutely nothing?
  2. Up until recently, I didn't have a well-defined project. It's hard to move forward on a project when you don't... really... have... a project?
  3. The lathe I need to make some critical components for my test fixture has been pretty massively misaligned for ages
     As it turns out, I am the responsible lab member for maintenance of the lathe. I was given this job after I voluntarily started performing repairs on the machine last semester. I had worked in this lab my junior year, and had spent a lot of time with the machine already. So much so that people would joke I was having an affair with "Billy Mazak" (Mazak being the brand of lathe). Anyway, my penchant for machining and restoration of old broken equipment (see: my cars) has led me to be quite involved with the Mazak these last few months. It's a very capable machine: the Mazak Super Quick Turn-15MS, or Billy for short, is a production style machine, similar to this one: The major difference is that our Mazak is from 1993, so it's not quite that nice. But it does have a 15 kW primary spindle, and the milling option and secondary spindle (critically, as you'll come to learn, no controllable y-axis).

     Since the Mazak is from 1993, and is in a research lab staffed primarily by graduate students, it's seen a fair number of collisions and misuses in the last 20 years. So it's not miracle that a few things are misaligned and out of calibration. I started off with the simple things in December. The tool eye (a super precise touch sensor, basically), used for touching off tools so the machine knows their position relative to the parts, was misaligned. The tool eye swings out on a little motorized arm like the one shown below.

     Once the arm is out, the operator touches the tool to two of the four faces of the tool eye (one X face, one Z face. The +Z face is being used below). This lets the machine know where things are. It's similar to when you turn a part on a manual lathe, and you measure the diameter of a test cut to calibrate your tools. Except this is more precise (0.001 mm precise), repeatable, and automated.

     The tool eye needs to be aligned and calibrated. If it is slightly rotated and the sensor faces aren't normal to the axes they're measuring in, things get inconsistent. Imagine your +Z face is slightly off-normal. If you move your tool in X, it will move further or closer to your +Z face, and your measurements will vary. The tool eye also needs to be clean. The little holes at the base are actually air nozzles that keeps dirt and grime from landing on the ground surfaces of the eye. After those two criteria are met, some test cuts are made, parts are measured, and then the tool eye position has to be derived and set as a software parameter. Seemed like a good place to start.

     I grabbed some manuals and read up on the procedure. Basically, a calibration block, consisting of two ground flat faces about an inch apart, is bolted to the +Z side of the tool eye, and a dial indicator (this one in particular) is mounted to the turret (the part with all the tools that rotates into place), The turret is moved in X, and the difference in Z between the two edges of the calibration block is measured. The four machine screws holding the tool eye down are loosened, and then snugged up. Then, using a hammer and punch on the back side of the calibration block, small impacts are used to rotate the tool eye into place. Yes. This is the official, published procedure. If you are unaware, precision machine alignment is often a matter of hammering something that's really hard to move into place, one micron at a time. The trick is having precise measurement equipment.

     So I did this alignment. As far as the indicator read, the two edges of the block were in plane. The indicator is guaranteed 3 micron accuracy and has a resolution of 1 micron. The process really wasn't that bad (not nearly as bad as aligning the head on a Bridgeport, at least). So I started getting confident. When I got back from winter break in January, I decided it was time to align the main spindle. This is where things started getting frustrating.

     The main spindle's axis of rotation on a lathe has to be parallel to the Z axis of the carriage/turret. If this isn't the case, an undesired taper will be introduced to your parts (e.g. if you turn a bar of constant diameter, one end of it will be larger than the other). So I attempted to measure this runout. My first attempt involved putting a large drill rod in the spindle and moving a dial indicator axially along its surface. Seeing that the runout was large, I pulled off the spindle covers. This turned into a bit of production.

     The Mazak is an enclosed CNC. It is designed to run unattended on a factory floor, and at fairly high speeds. The machinery is enclosed by thick steel panels designed to keep operators safe from runaway bricks of steel, moving parts, flood coolant, and high voltage. However, to get at the headstock for alignment, these panels need to be removed. Our machine is buried in the back corner of our shop, on the first floor of a 103 year old academic building. We have way too much equipment, and it gets a bit cramped back there. Seeing as nobody had needed to access this part of the machine in recent years, a fair amount of stuff had accumulated: disassembled lab benches, broken down shelving units, rubber mats, and probably some deceased wildlife (I suspect I blocked that part out from memory). I began clearing things out. After lifting some two inch thick wooden table tops out from behind the machine, I finally reached the bottom of the pile. All that remained were a few rubber mats. I began tossing those aside. At the very bottom was a roll of rubber. I went to toss it aside, and almost threw out my back trying to pull it off the floor. At this point, it occurred to me that this was actually a four foot long, six inch diameter roll of lead sheet, used for shielding our home-made x-ray cabinet. You can see the end of the roll against the wall in the image below. After struggling for a while, I eventually gave up. I don't know how that roll got there, but I suspect it's going to stay there for many years to come. 

     After freeing up some space to work, I had to remove four large steel panels. Two of them involved some precarious use of the engine hoist in a cramped corner to pick them up and move them. Thankfully, no photographic evidence exists. The end result was an exposed spindle and associate high voltage (the machine is fed with 480 V. Turns out 15 kW is a lot. An average house draws about 2 kW). Fortunately, between the Electric Vehicle Team and working for Tesla's Battery Safety and Technology Lab, this wasn't my first time working around high voltage. 

     After all this work, I realized by that a drill rod long enough to give reasonable resolution (I was measuring microns over ~400 mm, in the end) will have a large enough variation in straightness to invalidate the entire measurement. Realizing this, I decided to use the correct technique.

     The published method for checking spindle alignment is to take a long, large diameter piece of material, and turn the outside diameter. After this, the diameter at the two ends of the turned surface should be measured, and if there is a diametrical variation, the spindle and Z axes are not aligned. So I did just this. I had specifically bought a 25-50 mm and a 50-75 mm micrometer for this purpose a few months earlier. The misalignment was massive! It was 84 microns over 300 mm. Or so it seemed. It turns out that when you have a 40 mm aluminum rod unsupported for 300 mm, it's really hard to take a cut on the end of it without deflecting it slightly. This kind of invalidates the entire taper measurement. So I had to step it up to a larger diameter piece of stock. Unfortunately, the largest collet we have is 40 mm, so I had to swap the collet head out for a three jaw hydraulic chuck. 

     Turns out leaning into a machine, holding a 49 lb chuck with one hand, and using the other to screw a large nut inside of the chuck onto the end of a hydraulic cylinder is difficult to manage. So rather than asking a labmate for help, I opted to make a rig. I grabbed a large piece of round aluminum and placed it over the open door, with one end on either side of the opening. Using a hoisting strap and some zipties, I made a sling and mounted that on the rod. using some aluminum sheet and some aluminum blocks, I made a pair of wedges to keep the bar from rolling around. I then placed the chuck in the improvised sling and let it bear the weight, freeing up a hand for carefully maneuvering the chuck into place. Once the chuck was mounted, I obtained some larger diameter stock and threw it in the machine for another go at machining the OD.

     Initial results were promising. The taper was reduced, and there was no audible chatter while machining the end. At this point, I assumed the spindle was, in fact, going to need alignment. So I went ahead and started looking for a wrench to loosen the six large hold down bolts on the headstock. The first problem: whatever is used to loosen the bolts will be used to tighten them. Tightening them requires a torque wrench, which means they need to be square drive. Each bolt has a 14 mm hex key socket. The largest square drive hex bit we have in lab is 8 mm. After spending a few days scouring various shops on campus, my housemate's automotive tools, and local hardware stores, I gave up on finding a 14 mm socket drive hex bit. Buying one online will cost a lot of money and more time, so I gave up and decide to make my own. I grabbed a piece of oversized hexagonal steel, and using my lab's wire EDM, I made it into the appropriately sized Allen wrench. I then used a regular socket wrench to drive it.

     The second problem: each of these hold down bolts is spec'd at 250 ft-lbs of torque. For perspective, my car's engine puts out about 100 ft-lbs of torque. We build medical devices. We don't have torque wrenches large enough for this job. So I spend another few days trying to find a torque wrench large enough. After exhausting all but one resource, I went to a shop on campus  (which will remain unnamed) and was kindly lent one under the table.

     And so, I went to align the spindle. Fortunately, I accidentally stepped on the pedal which activates the hydraulic chuck. This meant the part unfixtured, and the previous OD turn was no longer a valid reference surface for alignment. I say "fortunately" because had this not happened, I would have misaligned the spindle. After unfixturing by mistake, I performed a new OD turn and found that the taper was reduced. I had removed less material on this pass than before, which seemed to indicate that the bar was still deflecting during the operation. As such, I carefully tweaked my cutting parameters, and eventually the taper disappeared (well, 2 um over 300 mm. 1/150000 is basically zero), and the results were repeatable. And such, after 2.5 weeks of "spindle alignment", it turned out that the spindle had never been misaligned to begin with. I was a little annoyed. So I tried to make the most of the situation.

     The covers I removed had exposed the mechanism for the parts catcher, which had stopped working many years ago. Basically, the parts catcher is a box on a rotating shaft that is actuated by a little hydraulic motor, and it flips up during a parting (cut-off) operation to catch the finished part and deposit it in a box, rather than at the bottom of the coolant pan. It had mysteriously stopped working long before the current batch of grad students started. I began debugging the hydraulics system. Eventually, I found the appropriate solenoid. I triggered the mechanical override on the valve and the parts catcher flipped up, very slowly. I adjusted the flow control valves until it moved at a reasonable rate.

     Clearly, the problem was not of a hydraulic nature. So I began diagnosing the electrical system. I pulled the connectors on the solenoids and got a multimeter. The solenoids nominally get about 99 VAC. I tried issuing the parts catcher command, and observed no change. So I pulled off some of the other solenoid controls for working components (like the caliper for the spindle's disc brake), and tested them. It seems that the difference between on and off is about 1 VAC. Strange. Well, it was clear that the solenoid was not getting the appropriate signal. Just to be thorough, I swapped the control wires from the disc brake to the parts catcher, and tested to ensure that the solenoid worked when it received the proper signal. It did. Unfortunately, this meant that the problem was either a controller problem, or a software problem. Since I have little faith in software, and because it's a little easier to get at than the CNC controller, I decided to pursue a the software route.

     Turns out that 1993 was not a good year for CNC controller software. Nor were any of the adjacent years, for that matter. All of the parameters for machine settings are hidden in these parameter menus, where every parameter is given a number (such as B49 or Q152). To understand what that parameter controls, you take the number, and you look it up in the manual, where an explanation is given. For some parameters, such as the ones that control the tool eye offsets, you put in some number that has physical meaning. For others, you put in some number derived from a formula. Some parameters are simply eight bits, and you input a string of ones and zeros to turn certain features on and off. Seeing as the machine does not have non-volatile memory, and it is 20 years old, when it loses power for extended periods of time, the back-up battery gets depleted and all parameters are reset. I know this has happened a few times. So my best guess at this point was that one of the parameters was not set correctly. The obvious solution would be to look through the parameters manual and find the one that corresponds to the parts catcher. Unfortunately, nothing is ever really easy with this machine. Mazak doesn't publish about 70% of the parameters, because they don't want users changing them. As such, I wound up looking online, and there aren't very many resources available. Eventually, I started digging through ancient documentation our lab keeps in the annex. I found a copy of the parameters list, with many years of edits scribbled onto it. Unfortunately, of the hundreds of eight digit parameters, only a few are labeled, and none of the labels or comments mention the parts catcher. So, in a brute-force approach, I wound up going through the entire list, checking each parameter one by one, until I found that a pair of bits were swapped. Sure enough, swapping the ones and zeros resulted in the parts catcher suddenly working again.

     After making a note on the paper parameters list about the parts catcher, I decided to pursue fixing another long-running problem on the Mazak: the inability to single-point right hand threads. When single-pointing threads, a triangular cutting tool (i.e. a tool with the profile of a single thread) is moved across the surface of  the part, and its motion is synchronized to rotation of the spindle, so that multiple passes can be made along the same exact trajectory, eventually leading to a nice spiral thread, as seen here (incidentally, we just this exact lathe in lab. It's a little benchtop unit. I was responsible for coordinating the purchase and getting it set up, which was a whole other set of headaches. The threading only happens in the second half of the video). Single-pointing is great. It allows you to quickly and easily cut custom threads that are nice and concentric, with way less hassle than using a tap or die. For some reason though, the Mazak doesn't like it. It'll approach the part, and then just stop and not move. No errors will be thrown. However, the operation works fine if the spindle is spinning in reverse. This basically means we can only cut left-hand threads. After many hours of searching through the parameters list, nothing came to fruition. I went to the internet, and after determining that our machine is the only one with this problem, I asked a machinist forum for help. Using some procedures they passed along, I recalibrated the spindle encoder. This involved sticking some o-scope probes inside of a box with some high voltage in it and carefully adjusting tiny potentiometers with a screwdriver while the motor was spinning until the waveform was correct. That was unpleasant, and frankly, unhelpful. The spindle speed tracking improved, but the threading problem was not resolved. I eventually gave up and put the covers back on the machine. Maybe it's time to call Mazak and see what they say. I doubt that'll be terribly helpful though.

     With everything back together, I realized it was time to pursue the major alignment project: the turret. The turret has been notoriously misaligned for years, meaning that all the tooling was off by about 150 um radially, in the Y direction. Since the machine has no control over this axis, there's no calibration or software solution. While the vertical error is acceptable for simple OD and ID turns, it was proving to be problematic when using live tooling, as any holes drilled in the face of the part would be in the incorrect location, which often resulted in non-functional parts. Unfortunately, I did not realize how long the turret alignment would take, nor how many resources. Look for the next two parts to learn how fire, diamonds, cavemen, and precision machinery all come together into one miserable experience.


  1. It might be true to some that talks about machines and tools could be boring or downright alien. But the mechanisms in each machine are unique and equally interesting. And to be able to perform repairs, maintenance checks, and even calibrations, the right tools are as important as your passion and knowledge. Good going. I hope to see the final product of this project. Good luck!

    Jeanette West @ Zo-Air Company Inc.

  2. You babble on too much. Stick to the point. We don't need to know (what you had for lunch...)