Neudesic + DTE Energy
Deploying a Next Generation Data Platform Provides Big Gains in Energy Theft Reduction, Customer Satisfaction, and Operational Growth
DTE Energy chose Neudesic to accelerate their move to Microsoft Azure and Azure Data Services when its existing data platform failed to meet expectations for actionable insights. Now it’s making big gains in energy security, customer satisfaction and operational growth.
When DTE Energy began its enterprise data analytics program in mid-2018, the plan was to create a central, high-level repository of up-to-the-minute information that would fuel operational strategy and decision-making across the business. The Detroit-based energy company, serving 2.2 million Michigan residents with reliable electricity and 1.3 million more with natural gas, rolled out its Advanced Meter Infrastructure (AMI) smart-meter systems over several years and by the time the process was almost complete in 2018, had begun receiving colossal amounts of data from customers.
It was an incredible opportunity to equip field workers with new insights, improve customer satisfaction, promote operational safety, drive down instances of fraud and, in general, fuel business growth. But DTE quickly discovered that its initial pick for the new system, a Cloudera- and Hadoop-based platform, did not provide the functionality, scalability or support it required.
“We had a fragmented architecture with lots of data silos,” says Uma Poduturu, IT Manager at DTE. “This led to 50 percent longer lead times before data was available to business partners and internal developers, leading to complex pipelines that limited data visibility. It was hard for developers and business stakeholders to get insights reliably.”
But the unreliable data pipeline was only one problem. The systems were also plagued by inflexibility, and the costs were prohibitive. Not only did the legacy environment not support data science or analytics, but it was also large and overly complex, meaning DTE had to retain a high-level skillset just to keep everything working.
“Our data platform went into production early in 2019,” explains Joyce LePage, IT Manager, Enterprise Data and Analytics at DTE. “But it never really met our needs. It did help us build the central data lake and get started, and we were able to get a fair amount of real time AMI data. But, as we were growing, it became apparent that it couldn’t be a long-term solution.”
With terabytes of data flooding in, the platform quickly neared capacity and DTE was forced to decide whether to upgrade the entire stack, with a range of new licenses and a lot more cash investment or seek out an alternative that better suited its needs. It was a frustrating situation for DTE’s various business units, which were flushed with information but starved of insight, so the decision was taken to begin again from scratch, building a new environment to address the various issues holding the business back.
“We had nothing in place, so we brought in the expertise we needed and started to plan how we would build a solution for a data lake on the cloud that fitted our strategic direction,” adds LePage.
To assess its existing environment and come up with a new roadmap, DTE selected Microsoft partner Neudesic for a six-week consulting engagement. Neudesic determined that DTE’s legacy infrastructure couldn’t handle DTE’s data science and scalability requirements and, in the near term, would limit its ability to use data effectively for these purposes.
“They couldn’t pull apart computation and storage, they couldn’t scale, and their previous cloud solution really lacked data science capabilities,” says David Bess, Director of Solutions, Energy, and Utilities at Neudesic.
DTE’s internal IT experts agreed with this assessment and, based on further consultation, DTE opted to extend Neudesic’s contract to help it migrate its existing environment to Microsoft Azure.
With terabytes of data flooding in, the platform quickly neared capacity and DTE was forced to decide whether to upgrade the entire stack, with a range of new licenses and a lot more cash investment or seek out an alternative that better suited its needs. It was a frustrating situation for DTE’s various business units, which were flushed with information but starved of insight, so the decision was taken to begin again from scratch, building a new environment to address the various issues holding the business back.
“We had nothing in place, so we brought in the expertise we needed and started to plan how we would build a solution for a data lake on the cloud that fitted our strategic direction,” adds LePage.
To assess its existing environment and come up with a new roadmap, DTE selected Microsoft partner Neudesic for a six-week consulting engagement. Neudesic determined that DTE’s legacy infrastructure couldn’t handle DTE’s data science and scalability requirements and, in the near term, would limit its ability to use data effectively for these purposes.
“They couldn’t pull apart computation and storage, they couldn’t scale, and their previous cloud solution really lacked data science capabilities,” says David Bess, Director of Solutions, Energy and Utilities at Neudesic.
DTE’s internal IT experts agreed with this assessment and, based on further consultation, DTE opted to extend Neudesic’s contract to help it migrate its existing environment to Microsoft Azure.
“We are now more proactive and aware of the issues in our distribution systems. We gather this data and proactively send it to our engineers for them to look at and give them more information on customers to help them understand what’s going on,”
Furthermore, although it is the early days of migration, the business has already experienced real-world benefits as a result of the upgrade. Neudesic helped DTE move more than 30TB of data, 125 workloads and 2,500 users from its legacy system to Microsoft Azure in a two-phased approach that has reduced DTE’s infrastructure and maintenance costs.
This has enabled the business to adapt quickly and meet its obligations under the Green Button Initiative, a government program to bring increased clarity to consumers as to their energy usage with easy access to personal data via a website. DTE customers can now log on and view their energy use from the previous year, potentially encouraging them into behavior changes that will cut waste and reduce bills.
Not only is the new platform empowering DTE stakeholders and customers, but it’s also helping to fight crime by detecting when a system may have been tampered with, voltage on a disconnected meter, or consumption at a vacant house.
With the machine-learning analytics and IoT capabilities of Azure and Azure Databricks, DTE can analyze data at terabyte scale to detect anomalies in energy usage and begin enforcement action. According to Neudesic modelling, faster insights could save DTE considerable money each year in fraud-discovery alone.
“Energy theft is a thing that people don’t think a lot about in the United States,” says Bess.“It happens pretty regularly in the electric utilities business. Internal projections showed that the build-out of its energy-theft analytics solution leveraging Azure’s cloud computing and analytics services could save DTE roughly $2.5 million per year in revenue protection by identifying energy theft quickly and accurately.”
If reducing power theft, cutting waste, saving money, enhancing strategy and delivering happy customers wasn’t enough, DTE field employees are now also armed with data to keep them safer and more informed every working day, according to Bess.
“Our enhanced data access now gives DTE decision-level support and situational awareness at all levels of the business, whether it’s to the field workers or someone working in the back office,” he says. “This ensures DTE provides the broadest access to the most critical data that drives reliability and safety across its grid operations.”
This has enabled the business to adapt quickly and meet its obligations under the Green Button Initiative, a government program to bring increased clarity to consumers as to their energy usage with easy access to personal data via a website. DTE customers can now log on and view their energy use from the previous year, potentially encouraging them into behavior changes that will cut waste and reduce bills.
Not only is the new platform empowering DTE stakeholders and customers, but it’s also helping to fight crime by detecting when a system may have been tampered with, voltage on a disconnected meter, or consumption at a vacant house.
With the machine-learning analytics and IoT capabilities of Azure and Azure Databricks, DTE can analyze data at terabyte scale to detect anomalies in energy usage and begin enforcement action. According to Neudesic modelling, faster insights could save DTE considerable money each year in fraud-discovery alone.
“Energy theft is a thing that people don’t think a lot about in the United States,” says Bess.“It happens pretty regularly in the electric utilities business. Internal projections showed that the build-out of its energy-theft analytics solution leveraging Azure’s cloud computing and analytics services could save DTE roughly $2.5 million per year in revenue protection by identifying energy theft quickly and accurately.”
If reducing power theft, cutting waste, saving money, enhancing strategy and delivering happy customers wasn’t enough, DTE field employees are now also armed with data to keep them safer and more informed every working day, according to Bess.
“Our enhanced data access now gives DTE decision-level support and situational awareness at all levels of the business, whether it’s to the field workers or someone working in the back office,” he says. “This ensures DTE provides the broadest access to the most critical data that drives reliability and safety across its grid operations.”
Serving customers, meeting regulations, and fueling growth
“With Microsoft’s Azure Data Services, our enhanced data access now gives DTE decision-level support and situational awareness at all levels of the business, whether it’s to the field workers or someone working in the back office. This ensures DTE provides the broadest access to the most critical data that drives reliability and safety across its grid operations.” – David Bess, Director of Solutions, Energy and Utilities, Neudesic
The future of DTE’s platform is developing and operationalizing data and analytic capabilities to fuel growth. LePage and Li argue DTE is now able to further develop unstructured information sources, use the data capabilities of the tool to better forecast energy as resource, and expose the utility to critical business applications – to maintain more efficient operations that pass savings on to users, all while empowering them to take control.
“We are a regulated business. We are going to use this platform to analyze the load and the generation sources so we can keep the electric grid safe and reliable. This is a requirement from energy regulators,” says Li.
“Our goal is to fully enable our business partners to be the data scientists,” adds LePage. “We want to give them the tools and data they need so we can collaborate with them to move their capabilities higher up the maturity scale.”
“We are a regulated business. We are going to use this platform to analyze the load and the generation sources so we can keep the electric grid safe and reliable. This is a requirement from energy regulators,” says Li.
“Our goal is to fully enable our business partners to be the data scientists,” adds LePage. “We want to give them the tools and data they need so we can collaborate with them to move their capabilities higher up the maturity scale.”