Heuristic 4.2: Experimental Evaluation

Heuristic 4.2: The importance of session/persistent-state modelling raises with higher correlation of subsequent state-driven decisions -- which is typically very high.

Summary of findings: The evaluation is built on a set of examples analogical to the set employed in the evaluation of Heuristic 2.3. Moreover, it discusses that the correlation can be very high, since the state value (for both the session and persistent state) is highly stable along system execution (i.e. also between the state-dependent decisions).

Go back to the Stateful Software Performance Engineering website.

Example 1

Consider a simple model of a sale service performed in a supermarket system (used already in Heuristic 4.1). Before starting the sale, the cashier may scan a member card, which then customizes system processing of the sale. In our simple example, the sale service scans the products (in a loop) and then processes the payment. After each scan of a product, the system may recompute the price (which may be time consuming) if the customer is a member (has the member card).

Figure 1: Model of the sale() Service

Figure 2: Usage Model

Figure 3: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 4: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 5: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 2

Figure 5: Model of the sale() Service

This example follows the model of Example 1 with the only difference that the expected number of products scanned during the sale increases from 3 to 6, which in the end increases the number of subsequent state-dependent branches also from 3 to 6.

Figure 6: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 7: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 8: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 3

Figure 8: Model of the sale() Service

This example follows the model of Example 1 with the only difference that the expected number of products scanned during the sale increases from 3 to 9, which again increases the number of state-dependent branches.

Figure 9: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 10: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 11: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 4

Figure 11: Model of the sale() Service

This example follows the model of Example 1 with the only difference that the expected number of products scanned during the sale is specified with a random variable saying that in 30% of cases loop count = 3, in 50% of cases loop count = 6, and in 20% of cases loop count = 9.

Figure 12: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 13: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 14: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Discussion

The set of examples above demonstrates that the deviation of the probabilistic results from the stateful results increases with the higher number of state-related control-flow decisions (branches in this case). At the same time, notice that the nature of the session/persistent state implies that the state value is highly stable along system execution, since it is likely to persist along the execution of a single session or even o sequence of sessions. Thus, all the control-flow decisions dependent on the state value will in the given session (resp. set of sessions) be fully positively correlated.

Model	MEAN	MEDIAN	VARIANCE
Stateful Example 1:	1,80	4,46	3,34
Probabilistic Example 1:	1,58	1,49	1,99
Stateful Example 2:	6,17	6,97	5,98
Probabilistic Example 2:	1,49	7,97	1,88
Stateful Example 3:	8,28	9,97	1,49
Probabilistic Example 3:	1,722	6,08	2,944
Stateful Example 4:	1,15	6,97	1,52
Probabilistic Example 4:	1,6	1,49	2,14