Heuristic 4.1: Experimental Evaluation

Heuristic 4.1: The importance of session/persistent-state modelling raises with lower knowledge of the corresponding user-given parameter values.

Summary of findings: Similarly to Heuristic 3.1, whenever there is any uncertainty about the value of the user-given parameter, which hence needs to be in the stateless model modelled with probabilities, the model may become very imprecise. The reason for this fact is that while in the stateful model, the parameter (session/persistent) value for the whole system execution often remains unchanged or is changed very rarely (the uncertainty about the state value is included on only one place, in the usage profile), the stateless model includes also the behaviours reflecting the unrealistic cases of state changes during system execution (similarly to the phenomenon observed in Heuristic 3.1). Interestingly, the deviation of the stateless model from the stateful results tends to exhibit a common phenomenon regarding the probability distribution of the reported values. In particular, while the mean and the median of the results use to be the same (or very similar), the variance of the stateless results tends to be significantly higher, with much smaller best value (fastest response) and much higher worst value (slowest response) compared to the accurate results of the stateful model.

Go back to the Stateful Software Performance Engineering website.

Example 1

Consider a simple model of a sale service performed in a supermarket system. Before starting the sale, the cashier may scan a member card, which then customizes system processing of the sale. In our simple example, the sale service scans the products (in a loop) and then processes the payment. After each scan of a product, the system may recompute the price (which may be time consuming) if the customer is a member (has the member card).

Figure 1: Model of the sale() Service

Figure 2: Usage Model

Figure 3: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 4: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 5: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 2

Figure 5: Usage Model

This example follows the model of Example 1 with the only difference that in Example 2, we have the certainty that the customer is a member.

Figure 6: Stateful (BLUE) vs. Probabilistic (RED) Model - Probability Distribution

Figure 7: Stateful (BLUE) vs. Probabilistic (RED) Model - Cumulative Probability Function

Figure 8: Stateful (BLUE) vs. Probabilistic (RED)) Model - Time Series

Example 3

Figure 8: Usage Model

This example follows the model of Example 1 with the only difference that in Example 3, we have the certainty that the customer is not a member.

Figure 9: Stateful (BLUE) vs. Probabilistic (RED) Model - Probability Distribution

Figure 10: Stateful (BLUE) vs. Probabilistic (RED) Model - Cumulative Probability Function

Figure 11: Stateful (BLUE) vs. Probabilistic (RED)) Model - Time Series

Discussion

The three examples demonstrate that the deviation of the probabilistic results from the stateful results can be observed only when there is an uncertainty about the state value in the session, which has been demonstrated already in Heuristic 3.1.

In these examples, the uncertain information is the membership of a customer entering the supermarket, which is at that moment unknown and is very likely to be different in each session of system usage by the cashier. A naive solution of model customization for each possible session/persistent state value would be to have a separate model copy for each value. However, there may be thousands of such state values (simply consider that the values are not only member and non-member, but a more fine-grained information for each member about the amount of money they have spent in the supermarket in the past). In such a case, the maintainability of the model would become a critical issue.

Model	MEAN	MEDIAN	VARIANCE
Stateful Example 1:	1,80	4,46	3,34
Probabilistic Example 1:	1,58	1,49	1,99
Stateful Example 2:	4,5	4,5	0,0
Probabilistic Example 2:	1,84	4,48	3,10
Stateful Example 3:	0,4	0,4	0,0
Probabilistic Example 3:	1,35	4,95	2,58