Emails: Amid public celebration, data glitches were piling up during DPU billing system transition
On the morning of May 26, Richmond’s Department of Public Utilities announced it had successfully launched two new systems meant to track customer accounts and improve service in the field.
An “abundance of testing, training, dress rehearsals and validations” had laid the groundwork for a successful rollout, DPU said, but officials would continue to watch for issues that might impact customers.
Around the same time that news release went out at 10 a.m., DPU Senior Deputy Director Jamie Porter noticed some problems.
“Team,” Porter wrote in a 9:53 a.m. email. “I am getting error messages from Kubra.”
A data connection between the KUBRA payment platform and DPU’s new Oracle Customer Cloud Service, which held customer account info, seemed off. A data integration process called “CCSKubra_GetBill” was repeatedly throwing up error notifications.
“I’ve received about 80 of these emails,” Porter wrote seven minutes later, just as DPU’s celebratory news release was going out.
The data issues were causing several problems, according to internal emails obtained through a Freedom of Information Act request. The FOIA request sought records between May 26, the day DPU announced it had successfully launched the new system, and June 1, the first time DPU sent a news release acknowledging there’d been a “system glitch.”
Customer service agents were unable to pull up bill information in the new system, and couldn’t schedule service appointments. Some service orders weren’t going through at all.
“Until these issues are resolved; to have documentation of the issue and the customer's needs, we are submitting tickets,” wrote contractor Tiffany Blackaby.
There also seemed to be issues with meter read data coming in from the field.
“I imported the file. The main route is only showing 588 meters. Last month there were 8515 meters,” DPU business systems specialist Sherri Porter wrote on May 26.
The problems seemed particularly serious for data involving PromisePay plans, which allow customers to pay off past due balances by splitting them up into monthly installments.
According to the emails, some PromisePay accounts were showing no balances, no service address or no due dates. In data terms, those fields were “null.”
“I wanted to check in again on the null due dates: Instead of leaving them as null, can we please set the due date to the next date a bill will be issued?” wrote a contractor for the PromisePay vendor.
At one point, more than 1,000 accounts were showing up with a null service address.
At least one account was showing late charges, which isn’t supposed to happen with PromisePay customers who are trying to get caught up. That forced the programming team to try to find a fix in the Oracle system to detach late charges from PromisePay accounts.
On June 1, there seemed to be confusion about which aspects of the system were going live when.
A DPU employee informed the Kubra team the city wasn’t receiving some payments through its online billing platforms because the systems weren’t connecting right.
“We were receiving transactions from your test system to our production prior to go live,” wrote DPU’s Max Pop. “Would you kindly have a look and see if you are attempting to send us transactions.”
Faisal Alvi, a senior project manager at KUBRA, indicated the connection at issue wasn’t up and running yet.
“It is part of the MyHQ/EZPAY changes that will Go live at a later date,” Alvi wrote.
Some staffers seemed to have trouble using the live customer system instead of the test version. One email sought help for someone whose system kept going back to the testing status.
“Can you assist or advise what she needs to do to get in the production screen,” a DPU customer care specialist wrote. “She has reset her password, log out the system and logged back in. I[t] goes back to test.”
The emails offer a narrow window into some of the issues, not a complete picture. The Richmonder attempted to file a broader FOIA request to capture emails across a wider time period. The city said the broader requests would return thousands of emails and cost thousands of dollars.
Graham Moomaw
At a news conference this month, Mayor Danny Avula and other officials acknowledged the city hadn’t done the greatest job of communicating with the public about the DPU billing problems.
“This is one of many technology deployments that we've done over the past year and a half or so,” said Chief Administrative Officer Odie Donald II. “But we've had some challenges.”
Despite the issues, DPU Director Scott Morris said the overall data integrity of customer accounts “has remained intact the entirety of the transition.”
“So if a customer does have concern with any of the accuracy of their account, we can happily give them an audit of their account and show that there is an accurate accounting of any type of metering or consumption on their bill,” Morris said.
City officials said this month that the billing issues seem to be stabilizing, a trend reflected in declining call volumes to DPU’s customer service center.
Once the transition issues have been worked, DPU has said, the new systems will help city residents by making bills clearer and improving the efficiency of DPU work being done in the field.
System and coding hiccups are fairly common in large-scale software projects. Given the extent of the problems, the mayor said this month the city will own it “when we miss the mark on something.”
Officials have acknowledged there were premature declarations of the project’s success, some of which also shows up in the internal emails.
On May 28, a few days before DPU acknowledged the glitch an email went out with the heading “Extraordinary thank you to an extraordinary Team!”
Porter, the DPU deputy director, thanked the Oracle project team for "successfully implementing our new cloud-based Customer Information System in just 22 months.”
“Congratulations to everyone involved in the significant accomplishment,” Porter wrote. “Project Synergy!”
Contact Reporter Graham Moomaw at gmoomaw@richmonder.org
