One of the most hated days in the Twitter data center was rack push days. All the Site Operations Technicians would come in extra early to move the new racks off of a truck. Our goal was to move them into the specific room. Coming in early wasn’t a problem, but the amount of manual labor required to move these heavy racks were. Let me explain.
Why Rack Push Days Sucked: Getting The Racks Off The Truck
The first reason why rack push days sucked because getting the racks off the truck was a hurry up and wait. Our warehouse crew had to use forklifts to move the wooden boxes containing the racks off of the truck and place them onto the dock. Since we only had two forklifts that could take some time. Plus, the warehouse crew couldn’t move too fast with the forklifts due to the size of the dock. Plus, moving too fast can be dangerous.
Once the boxes were on the dock us techs worked to unbox them. That meant removing all the plastic and packing materials and moving a heavy ramp from box to box to slide the rack down.
Now the procedure I laid out is when the truck was on time. Unfortunately, we had to wait for the truck to arrive many times. Traffic in Atlanta is especially bad in the morning and sometimes the truck would get stuck in it. Or the truck got held up due to bad weather and we learned too late it wouldn’t arrive until the afternoon.
Why Rack Push Days Sucked: The Racks Were Heavy
Oh, did I forget to mention these racks weren’t empty? Yes, I did. That’s right: Twitter had their vendors build out each rack for them (which isn’t uncommon), which meant installing all the servers, networking equipment, cabling, etc. And that’s the reason why these days sucked. The racks were heavy and hard to push. A “lightweight” rack could weigh as little as 1800 pounds give or take. However, our heaviest racks were 2700 pounds if I remember correctly.
So to move these it took a pair of techs to each rack. And sometimes it took three techs because some of the techs on the team weren’t that strong. Now, I must say that moving racks like this isn’t common for some companies. I know that Google has a separate team (probably a contracting company) to move their equipment. And their techs don’t do this type of tasks as we had to do. So why didn’t Twitter hire a moving team to help us? Money. We were already there in the data center getting our salary (which was hourly by the way) so it was more economical for the company to get us to move the equipment.
Time To Push!
Once we had all the racks out of the boxes it was time to push! The team split into two: One pushed the racks onto the elevator to move them upstairs and the second team pushed them down the hallway into the specific room.
Oh I wish I had a picture of the hallway to show you but I don’t. Because I couldn’t take any pictures of the data center which is regular security practice. Don’t fret, however! I’m good with descriptions. So picture this: A long hallway spanning nearly a quarter of a mile or over 400 meters. Yes, I’m serious. The hallway was that long. And Twitter had rooms in different places all throughout that hallway.
It was a great day when the racks went into the room closest to the elevator. Hence, we didn’t have to expend that much labor. Oh, but we had to push those racks to the other end of the hallway we complained! Yet, we got smart. We pushed the racks to a halfway point and then another team took over and pushed the rack the rest of the way.
Finally, Fit The Racks In Place
And the final reason rack push sucked because fitting the racks in their place in the room took precision. You see, there were already racks in place in the room working properly. So we had to be careful not to bump into those racks and disrupt them. Also, we didn’t want to damage those racks or the equipment we pushed.
Finally, fitting the racks took more precision because in many of the rooms used elevated flooring. Thus, all the power connections was underneath. And that meant each tile had a hole in the middle for the cabling to fit through. You see where I’m going, right? We had to be extremely careful not to get a wheel underneath the rack stuck in the hole or it could tip over.
The Aftermath
The aftermath of rack push was a bunch of tired and hungry and thirsty Site Operations Technicians. Some ate the breakfast biscuits our manager brought in. Others, like myself, drank a cold beverage and relaxed for several minutes before starting our regular tasks of diagnosing and fixing defective servers.