UKG outage remediation teams praised for response

If your instincts tell you to play it safe during difficult times, there’s plenty of evidence to suggest that extreme adversity breeds innovation. The Space Race was a moonshot for scientific progress. Microsoft was founded during an economic downturn. More than 800 craft distillers across the U.S. made hand sanitizer during the first wave of the pandemic.

Rather than hunker down when times get tough, consider heeding Machiavelli’s advice: “Never waste the opportunities offered by a good crisis.”

The University of Utah and University of Utah Health arguably faced “a good crisis” on December 12, 2021, when the Ultimate Kronos Group (UKG, formerly known as Kronos) announced it had discovered a security breach. A ransomware attack had impacted approximately 2,000 UKG customers that use the company’s Kronos Private Cloud product, taking payroll systems around the world offline.

“This unprecedented situation required considerable time, energy, and resources, but we responded with tremendous resiliency,” Chief Information Officer Steve Hess, Ph.D., said. “Though we constantly strive to minimize vulnerabilities in our systems, including when relying on external vendors for critical services, managing risk is part of keeping an entity the size of the University of Utah in operation. I’m incredibly proud of how our HR, Finance, and IT teams responded so that U employees could continue to receive their paychecks, especially during the holidays.”

Soon after news of the cyberattack on UKG broke, a university-wide task force assessed potential impacts on U employees and institutional operations. A collaborative team of software developers set out to create two interim web applications to track hours and pay codes — “UTime” — that campus employees could access in Campus Information Services (CIS) and an employee performance management (EPM) timekeeping application for U of U Health staff.

“The teams that came together shared a common purpose — taking care of our colleagues so that they could take care of patients, students, and our community,” said Sarah Sherer, U of U Health chief human resources officer. “There were no egos or focus on titles as the teams came to brainstorm solutions … their innovation allowed us to problem solve in an arena we had not experienced before, and for that, we are all extremely grateful.”

Jeff Herring, U of U chief human resources officer,  added, “As challenging as the UKG outage was, it was great to see the collaboration across campus.  I am proud of all of our university teams that came together to find creative solutions to a very difficult situation.”

Considering the timeframe, Matt Edgren, senior web software engineer for the HR/Auxiliary Engineering Team in UIT University Support Services, said he “was extremely pleased with how quickly we were able to move when we needed to.”

“That was awesome,” Edgren said. “I was also happy to be part of something that really helped people in a crisis.”

The payroll process at the U is part of the larger PeopleSoft environment comprising many systems and interdependent applications and services. Developing UTime happened in a parallel fashion — with Edgren leading a team charged with creating the front-end web application and Mitch West, senior software design engineer, working on building the backend system in PeopleSoft.

“Both applications are compatible with each other,” West said, “so I would know that the web team was successful if an operation still worked in my application and vice-versa. The final version is not going to be perfect and you’re not going to use everything you designed, but it’s kind of crazy how robust UTime ultimately was when you consider the compressed timeline.”

UTime, which featured an administrative tool tacked on to PeopleSoft that allowed payroll staff to adjust data on their end, was positively received by most managers and employees, according to Doug Kenner, associate director for the USS HR/Auxiliary Team.

“I still get comments about how the UTime app, from a usability standpoint, was better than UKG is currently,” Kenner said.

Though UTime was built on a PeopleSoft framework, there wasn’t a template, or playbook, to create a tool with such far-reaching impact in mere weeks.

“What we created in the end did not look like the original prototype, so it was basically built from scratch,” West said.

Kenner praised the dedication of all teams involved in the project, “especially going into it knowing this could potentially suck them into working weekends and holidays.”

“It’s a lot better when you’re working with nice people,” West said. “All things considered, we were all in fairly good spirits despite the long hours.”

UTime and U of U Health’s interim timekeeping tool went live in the PeopleSoft production environment on December 20, only eight days after the crisis began. UKG restored access to the core functionality of Kronos Private Cloud more than a month later, on January 22, 2022.

The process of reconciling overpayments and underpayments at the university continued through the end of March 2022, West said.

“Reconciliation of payments was probably as big of an effort or even bigger than the outage, but at that point, it wasn’t the same kind of crisis,” he said.

As for the UTime application, West said it “could be up capturing data in a matter of minutes” if needed. It’s deployed, he said, but not accessible in the production environment.

“I suppose it’s comforting to know it’s still there, should we need it,” he said.

Hess again expressed his “deep appreciation to everyone.”

“I have so much gratitude for the unified response and tireless efforts that went into keeping our timekeeping and payroll systems going, including all the reconciliation work long after the outage itself was over. I wish I could personally thank each and every person who made this happen,” he said.

Here are some of the teams involved in the UKG outage remediation effort:

University of Utah

• University Human Resource Management, specifically the Payroll and HR Information Systems (HRIS) Teams
• USS HR/Auxiliary, Quality Assurance, and Content Management & Usability Teams
• UIT Project Management Office
• UIT Software Platform Services (Chief Technology Officer organization)
• UIT Strategic Communication
• University Marketing & Communications

---

U of U Health

• UUHC Human Resources Teams
• Human Resources Information Technology (HRIT) Teams
• Finance Team
• Information Technology Services (ITS)
• Kronos Team
• Centralized Nursing Payroll
• Project Management Teams

 

Node 4 Search