Heuristic 1.2: Experimental Evaluation
Heuristic 1.2: The importance of protocol-state modelling raises with higher complexity of component's environment.
Summary of findings: In some situations common in complex systems, it may be very hard or even impossible to estimate probabilities for the stateless model precisely. A simple exemplary model illustrating this phenomenon can be build on the fact that the probabilistic abstraction can hardly be foreseen in the models where the same service is called twice and each time behaves differently based on the actual protocol-state value that may change in the meantime. In such a case, two models of the same service would need to be present in the stateless system model to make it accurate. Otherwise, all the response-time characteristics of the probabilistic model (even the mean value, which uses to be very stable) may significantly deviate from the values of the more-precise stateful model.
Go back to the Stateful Software Performance Engineering website.
Example 1
Consider a component managing a file, which can be opened, active for modification and closed. The component is initially in the state when it accepts only the request for opening the file (see Figure 1). After that, it moves to the state, where the file can be either closed or activated for modification. Closing takes the component again to the initial state (from both the opened and active state).
The file component implements four services: openFile(), activate(), modifyFile() and closeFile(). While openFile(), activate() and closeFile() only execute a simple internal action with a very low resource demand and then change the state, the modifyFile() service executes a complex internal action with a high resource demand. After defining a CPU rate (one CPU cycle in one time unit, -ms- in our case), the simple internal actions last 1 ms, while the complex action lasts 10 ms.
The usage model (see Figure 6) considers a simple usage profile with a single scenario (for each of 10 users in the system) calling a sequence of openFile(), modifyFile(), activate(), modifyFile(), and closeFile(). While in the stateful model, the executability of each service is given by the actual value of the protocol state, for the probabilistic model, we need to estimate the transition probabilities for all the branches in the SEFFs (the probabilities that the simple internal actions open, activate, close and complex internal action do are executed). While one can see that open, activate and close are executed whenever the corresponding services are called from the usage profile, the situation is not that simple with the complex do action inside modifyFile(). The modifyFile() service is once called in the opened state and once in the active state, so the likelihood of executing do is 50%.
The figures below summarize the difference in the predicted system response time in the stateful and probabilistic version of the model. Although the mean values are quite similar (165 ms in the stateful model, vs. 149 ms in the probabilistic model), the median already differs fairly (227 ms in the stateful vs. 129 ms in the probabilistic model), and the variance differs significantly (8.77 ms in the stateful vs. 1.06 ms in the probabilistic model).
To observe the deviation of the results in a more realistic setting, we slightly modified the resource demands in the example to reflect expected variability of the internal-action executions. We added dependency of the execution time of modelled actions on the size of files. The assigned values of data size distribution have been determined according to the Spec JMS Benchmark (data size distribution = DoublePDF[ (1.0; 0.00) (2.0; 0.01) (3.0; 0.03) (4.0; 0.08) (5.0; 0.13) (6.0; 0.58) (7.0; 0.11) (8.0; 0.06) ]). The deviation of the probabilistic and stateful model is in this setting even more visible. The results for the overal responce time of the scenario are summarized below. One can again see that both the median and variance of the probability distributions differ significantly. The mean values are quite similar (9,260 s in the stateful model, vs. 9,377 s in the probabilistic model), the median already differs fairly (11,096 s in the stateful vs. 7,658 s in the probabilistic model), and the variance differs significantly (1,989 s in the stateful vs. 5.308 s in the probabilistic model).
Example 2
This example extends Example 1 with a new execute() service called inside modifyFile() instead of the internal do() action. The usage profile is unchanged. In this example, do() is executed inside execute(), which is again state dependent (do() is executed only if the component is in the open state). The initial issue in this example is the estimation of the probabilities guarding the execution of the do() action. There are two intuitive strategies of the probability estimation. The first one only evaluates how often the component is in the given state (thus "activated" AND "opened", which is 50% in our case), the second examines all execution combinations and the likelihood of their passing through the do() action (which could result in the estimation of 0% or 25% dependent of the experience of the engineer).
The results of both the cases are depicted in the figures below (stateful model vs. probabilistic model estimation of 50% on the left, stateful model vs. probabilistic model estimation of 0% on the right). Again, one can see a significant deviation of the variance of the histogram of stateful vs. stateless results (7,43 ms in the stateful vs. 1.06 ms in the probabilistic model), and considerable deviation of the median (207 ms in the stateful vs. 129 ms in the probabilistic model) and the mean (158 ms in the stateful vs. 149 ms in the probabilistic model).
Again, we have also examined the modification of the resource demands to a more realistic case (similarly to Example 1). However, the graphs still report analogous observations. The mean values are quite similar (9,3 s in the stateful model, vs. 9,01 s in the probabilistic model), the median already differs fairly (9,75 s in the stateful vs. 7,475 s in the probabilistic model), and the variance differs significantly (6,23 s in the stateful vs. 3,832 s in the probabilistic model).
