Heuristic 2.3: Experimental Evaluation

Heuristic 2.3: The importance of internal/global-state modelling raises with higher correlation of subsequent state-driven decisions.

Summary of findings: In the experimental evaluation, we have used a number of models analogous to the example outlined in [1], with an internal action in between of the branches. The experiments below show that even if the probabilities of the branches accurately reflect the usage profile, the results computed from the stateless model can be very imprecise. Notice that already in a very simple model (one service with two or three branches), the probability distribution (mainly the variance and best/worst case) of the stateless-model results deviates significantly from the stateful-model distribution. The mean and median values tend to be quite stable for simple examples, and start to deviate when more complexity is introduced into the models.

Go back to the Stateful Software Performance Engineering website.

Example 1

This heuristic can already be validated on a very simple model depicted in Figure 1, with two branches and one internal action in between. The first branch action (BranchAB) defines two alternative executions, one starting with an empty behaviour (let us call it A) and one with an internal action intB (let us call it B). The second branch action (BranchCD) again splits the execution into two alternatives, one with an internal action intC (let us call it C) and one with an empty behaviour (let us call it D).

Figure 1: Stateful Model for dependent branches

Figure 2: State Holder for dependent branches

The experiment configuration was defined by the usage profile with a closed workload of a population size equal to 10. The simulation results (using PCMSimuBench) have shown that 6 out of 10 requests were directed in the branches A followed with C, and four requests to B followed with D. In comparison to this model, in the probabilistic abstraction model (where P(A)=P(C)=0.6 and P(B)=P(D)=0.4), the combination of A followed with D and B followed with C are also possible, which significantly disturbs the probability distribution of the results. The following simulation results illustrate the system response time implied by the discussed usage profile.

The deviation of the stateless-model results from the stateful-model results is clearly visible on the histograms and distribution functions below. Mainly notice much higher variability of the stateless (probabilistic) model, visible mainly on the probability distribution of the results. Additionally, the difference in the results is nicely visible on the time series, where the response times per request are illustrated.

Figure 3: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 4: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 5: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 2

This example extends the Example 1 by adding additional dependent BranchEF into the sequence of branches. The difference in the results grows with the complexity and is nicely visible on histograms and time series, where the response times per request are illustrated.

Figure 3: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 4: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 5: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Model	MEAN	MEDIAN	VARIANCE
Stateful Example 1:	112	100	1,057
Probabilistic Example 1:	237	199	1,199
Stateful Example 2:	136	100	9,59
Probabilistic Example 2:	316	299	2,07