We're excited to announce a new feature that addresses a significant need in time series analysis: stateful functions. These functions enable more advanced and nuanced data analysis, a crucial tool for users who want to extract deeper insights from their time series data.
One of the most powerful aspects of this feature is the integration of stateful functions into Sift's rules engine. This allows users to write more advanced and powerful logic that we automatically evaluate in real-time, enabling proactive monitoring to catch anomalies and ensure optimal system performance.
Understanding Stateful Functions
Sift has always supported stateless functions—such as sin, string length, or list contains—that operate independently on each row of data without retaining any memory of previous rows. While useful, these functions can fall short when it comes to more complex analyses where the context of multiple data points is essential.
Stateful functions, on the other hand, track data across multiple rows. This allows for more sophisticated operations like:
- Rolling Average (Moving Average): Smooth out noisy data to reveal underlying trends. This is essential for time series analysis where raw sensor data can be erratic.
- Take a one minute rolling average
avg($1, rolling("1m"))
- Take a one minute rolling average
- Persistence (Pending Period): Define rules that only trigger alarms when a condition has been met for a minimum amount of time, reducing noise from brief, insignificant fluctuations.
- Alert when the vehicle state is in "RESTARTING" for more than 30 seconds
persistence($1 == "RESTARTING", 30)
- Alert when the vehicle state is in "RESTARTING" for more than 30 seconds
- Rate of Change (Derivative): Calculate how quickly values are changing over time.
- Determine the change in force per millisecond by taking the derivative of mass times acceleration
deriv($1 * $2)
- Determine the change in force per millisecond by taking the derivative of mass times acceleration
- Periodic Count: Determine the number of data points within specific intervals, such as points per second.
- Measure the data rate by counting the number of points per second
count($1, rolling(1))
- Measure the data rate by counting the number of points per second
What makes stateful functions even more accessible is their familiar syntax. Users with experience in Excel will find writing expressions familiar and can quickly perform powerful transformations with simple commands. These new capabilities are fundamental for anyone looking to do more than just surface-level analysis of their sensor data.
Why Stateful Functions Matter
Stateful functions are akin to aggregates in databases—they enable the grouping of data and computation of a result for each group. A common example is a rolling average, which in SQL can be computed using a windowed aggregate. This means that for each data point, Sift can compute a rolling window, calculate the average, and output a smoother series of data points. Windowed aggregates are versatile and can be used for various other operations like min/max, sum, or median, making stateful functions even more powerful and flexible for different analytical needs.
With stateful functions, users can configure the data tracked by a duration in time, providing flexibility for various analytical needs.
Practical Applications
In practice, stateful functions open up new possibilities for monitoring and analyzing system performance. For instance, by applying a rolling average to your sensor data, you can filter out the noise and focus on meaningful trends. Similarly, calculating the rate of change can help you identify sudden spikes or drops, indicating potential issues that need attention.
Conclusion & Next Steps
Stateful functions are a game-changer for time series analysis in Sift. They provide the tools needed to move beyond basic data examination, offering deeper insights and more precise monitoring capabilities. We're excited to see how our users leverage this new feature to enhance their data analysis and system performance.
Customers with accounts can login and read more details here:
