How Normal Behavior Modeling Turns Data Into Insights for Industrials


Clive Humby’s famous statement “data is the new oil” was a bold stance when first published in 2006, but it’s become a widely accepted truth today because, like oil, data enables a huge swath of our economy and those that control it become the power brokers. And also, like oil, which in its crude state is considerably less functional than the refined products we make with it, raw data is a gooey mess until we transform it into valuable insights.

In a recent SparkCognition webinar entitled AI Demystified: Transforming Industrial Operations With Deep Learning and Autoencoders, we examined the power of neural networks in industrial operations to allow organizations to cross the chasm between raw data and actionable insights. This webinar offers an in-depth explainer of how SparkCognition uses normal behavior modeling to detect known and unknown issues on otherwise highly reliable assets. 

Andrea Schmidt and Sreenivasa Gorti discuss normal behavior modeling during a SparkCognition webinar

Too much data, not enough insights?

We typically think of having lots of sensor data as a good thing. But in this webinar, Sreenivasa Gorti, VP of Product Management at SparkCognition, described how the increasing sensorization of industrial assets has (in many cases) led to a spiraling problem of too much data. After all, each sensor can generate over 30M data points each year. He explained: “So when you start thinking about how traditional physics-based models and statistical approaches handled this much data, they’re quickly hitting their limit in terms of producing actionable insights from this data.” 

He also noted how hard it can be to anticipate an asset failure in industrial equipment, which is generally very reliable—until it’s not. “There is this idea that industrial equipment or a specific piece of critical equipment may have 99% reliability in isolation,” he said. “[But] when you combine a large number of these into an overall system view, you’ll end up with system reliability at something like 80, 85%”. Gorti laid out two paths an organization might take to overcome the kind of reliability issues that are challenging to predict but extremely important to avoid. 

Path #1 is the traditional physics-based analytics approach. Although state-of-the-art for many years, this path is tricky to operationalize, sustain, and scale well. One of the biggest issues with this approach is that it is resource-intensive for subject matter experts (SMEs). He explained: “Subject matter expertise is really an important part of how these approaches are developed and maintained over time. And as the amount of data becomes more and more, our subject matter experts are getting overloaded with data processing tasks and just maintaining these models as assets age. The models have to be maintained because they’re no longer as accurate as they used to be.” 

Another challenge in industrial operations is what Gorti calls the ‘human database’ problem: “What we have are industrial experts who have maintained a knowledge base that they’ve grown over a long period of time and have essentially become the repository for best practices.” Of course, this approach fails to scale well across an enterprise. 

Pivoting to Path #2, Gorti talked about organizations recognizing the value of an AI approach. But when they start with their own internal data science teams, it can be problematic: 

“It’s one thing to develop a model in a lab environment. It’s a completely different level of effort to operationalize these with uninterrupted flows of data from large number of sensors—putting in all the software infrastructure and the plumbing required to operationalize solutions. 

[Another] complexity is that industrial systems are (by and large) designed to be reliable. There’s a lot of engineering effort that goes into them. So failure data is not necessarily available in an operational sense. And so when you develop these models, you have to really start looking at unsupervised approaches. 

And then, of course, the broader characteristics of an industrial environment [means] there are a lot of operational interventions. On an ongoing basis, you have to account for how the models include these operational interventions as part of their learning history. You have to account for the fact that these models need to be updated and trained periodically to account for aging assets and processes. 

We’ve already talked about the shifting workforce as a big concern area. How do you take these sorts of concerns and develop AI-based approaches that you can start to implement at scale in organizations? 

So some of the key requirements, I would say, are we have to look at existing workflows. That involves subject matter experts and operators. And how do you incorporate AI? Not as a replacement but as an assistant that can take away a lot of the grunt work that’s typically associated with operations—and have the AI system do that piece of the work.” 

Normal behavior modeling: Overcoming the challenge of structured vs unstructured data to create a more reliable risk score

With that, Gorti handed off to Andrea Schmidt, Director of Solution Architecture at SparkCognition, for a deep dive into normal behavior modeling. Anyone wanting a better understanding of just how SparkCognition uses neural networks to detect anomalies in operating equipment will appreciate her detailed explainer in the full webinar

After a brief treatment of predictive analytics in the pre-industrial and industrial eras, Schmidt talked about how the computer era launched the industrial space into automatic data collection, opening up a world of statistical process control and condition-based monitoring techniques. As artificial intelligence R&D advanced, the familiar question of whether and how to apply supervised and unsupervised machine learning methods appeared, with the ensuing pros and cons of both approaches. Schmidt explained the vexing problem with supervised learning in industrials is the lack of multi-class examples, i.e., there’s not enough failure data to train a model that’s not biased. With unsupervised learning approaches, we face a different challenge: not having enough examples to describe the entire space.

As she put it: “What we really need is something in between a supervised approach and an unsupervised approach that allows us to model and alert on what we do know about the data but also on what we don’t know about the data.”

She continues, explaining how and why we leverage normal behavior modeling for our customers: 

“The approach that we use is called normal behavior modeling, which in our experience, has proven to be effective in predicting failures with extending lead times. And we’re able to do so far beyond what current SCADA systems are able to do today. And with longer lead times, our customers can be more effective at diagnosing, planning, and maintenance activities. 

What we choose to do is this semi-supervised approach, which is really a blend of the two approaches of unsupervised and supervised. We use this approach to leverage what we already know about the dataset—because we do know some things about our predictive maintenance date. We know when it’s running well. We can have somebody go out on the floor and say, everything is running perfectly, looking at the data center. So we do know that. But we might not know all of the different failure modes that occur. 

What we do is selectively choose our trading data set to only represent that normal operating condition. And by training on good data only, we can get the accuracy in the alarming that we need in order to detect both the known issues and those unknown issues. And due to the high reliability of equipment, we should have plenty of that data to choose from. 

We typically ask our customers to provide a year or more worth of data. And then what we do is we filter that data out. We selectively choose when the equipment is down, or some maintenance was performed. We ignore those times and we really just focus on what is normal. We do that by looking at maintenance logs and some of the historical data along with the sensor data. By doing so, what we’re able to do is train a model to be able to look at normal. 

We’re taking a ton of sensors, and we’re representing that in a single output of a risk score. And that risk score is going to be (we expect) low on the risk level. And then as we approach a downtime, as we approach some issue occurring within the operations, we expect to see that risk score rise, and that’s how we measure ourselves.”

Schmidt and Gorti offer up a wealth of information, answer some frequently asked questions, and detail real-world case studies from SparkCognition deployments for major industrial companies in this fast-moving on-demand webinar. You’ll learn how SparkCognition’s proven AI capabilities enable a sustainable competitive advantage for today’s leading organizations, including:

  • How neural networks allow operators to understand the dynamics of aging assets reducing failure by up to 20%.
  • How Normal Behavior Modeling is significantly reducing alert fatigue and boosting worker productivity.
  • How machine learning can augment human knowledge and retain critical asset history increasing uptime by 5%.


As always, find all of our upcoming and on-demand webinars here.

Latest blogs

Abstract depiction of our Generative AI Platform created by DALL-E
Campbell LeFlore

What’s Inside our Generative AI Platform?

Perfected over ten years of real-world engagements serving many of the world’s largest brands in energy, manufacturing, transportation, utilities, financial services, and other industries, SparkCognition’s

Read More
Campbell LeFlore

The Top Challenges HSE Managers Face Today

HSE managers are the behind-the-scenes heroes enabling today’s high-throughput industrial environments to operate at peak performance. Their determined efforts ensure the well-being and protection of

Read More
SparkCognition is committed to compliance with applicable privacy laws, including GDPR, and we provide related assurances in our contractual commitments. Click here to review our Cookie & Privacy Policy.