Heuristic 3.1: Experimental Evaluation

Heuristic 3.1: The importance of allocation/configuration-state modelling raises with lower knowledge of deployment/configuration parameters.

Summary of findings: The experimental evaluation reveals that whenever there is any uncertainty about the value of the parameters, which hence needs to be in the stateless model modelled with probabilities, the model may become very imprecise. The reason for this fact is that while in the stateful model, the parameter value for the whole system execution remains the same (the uncertainty about the parameter value is included on only one place), the stateless model includes also the behaviours reflecting the unrealistic cases of parameter changes during system execution. Interestingly, the deviation of the stateless model from the stateful results tends to exhibit a common phenomenon regarding the probability distribution of the reported values. In particular, while the mean and the median of the results use to be the same (or very similar), the variance of the stateless results tends to be significantly higher, with much smaller best value (fastest response) and much higher worst value (slowest response) compared to the accurate results of the stateful model.

Go back to the Stateful Software Performance Engineering website.

Example 1

First imagine a very simple reference example of a system with one service (inside one component) executing a certain functionality in a sequence of loops with the loop count dependent on an allocation state defining the maximal queue length of the component. In this example, we have a full certainty about the value of the queue length, which is q=5. The usage model of this example is very simple, including only a single call to the discussed service.

Figure 1: Model of the Analysed Service

Figure 2: Usage Model

The results below show that in such a case, the probabilities in the model can simply be designed in a way that accurately reflect the reality. Stateful modelling hence brings some benefit only if the model is about to be reused, or the allocation state value may change in the future.

Figure 3: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 4: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure 5: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Example 2

This example is much more interesting than Example 1 because it shows that the results of the stateful and probabilistic model start to deviate significantly once there is some uncertainty about the allocation/configuration state value. Assume such an uncertainty is expressed probabilistically, e.g. with the probability 30% q=1, and q=10 otherwise (with the probability 70%).

In the stateful model, the allocation state value is encoded into a component parameter (we are talking about the component-specific allocation state), which is configured before the execution is triggered - either to q=1 or q=10. Hence, during the whole execution, the queue length is either considered to be 1 or 10. No combinations of these two values are possible, opposite to the probabilistic model, where the uncertainty needs to be encoded into the control flow to every place where the current value of the state is queried.

Figure 6: Model of the Analysed Service

Figure 7: Usage Model

Figure 8: Stateful (RED) vs. Probabilistic (BLUE) Model - Probability Distribution

Figure 9: Stateful (RED) vs. Probabilistic (BLUE) Model - Cumulative Probability Function

Figure10: Stateful (RED) vs. Probabilistic (BLUE) Model - Time Series

Model	MEAN	MEDIAN	VARIANCE
Stateful Example 1:	10	10	0
Probabilistic Example 1:	14,986	4,954	2,91
Stateful Example 2:	1,1	0	0
Probabilistic Example 2:	1,5	4,9	3,08