Comparison of Evolutionary Results with Random Search

We conducted a run of PerOpteryx in which candidates are randomly created within the defined degrees of freedom. With the problem formulation as presented in the latest work, we can easily randomly create candidates by choosing a value for each design option from its range randomly.

The used case study is a slightly modified version to the one in the paper and evaluated with the latest LQNS version 4.3.

Random Search Results

For the random search, we created 1199 random candidates. Together with the predefined initial candidate, we evaluated the resulting 1200 candidates for response time, cost and probability of failure on demand as described in this work. 24 Pareto-optimal candidates were found.

Below you find the results in two diagrams.

Response Time vs. Cost
Probability of Failure on Demand vs. Cost

Evolutionary Results

For an easier comparison, we also present the results of the evolutionary algorithm here. We used a slightly different model here, so the results are different than already shown in this work. 70 Pareto-optimal candidates were found in 1206 evaluated candidates.

Response Time vs. Cost
Probability of Failure on Demand vs. Cost

Comparison

We compare the found optimal candidates of both runs in the following.

Response Time vs. Cost (all results of both runs)
Probability of Failure on Demand vs. Cost (all results of both runs)
Response Time vs. Cost (overall optimal candidates only)
Probability of Failure on Demand vs. Cost (overall optimal candidates only)

We can see that the evolutionary approach is mostly superior to the random search. It finds more Pareto-optimal solutions (70 candidates) than the random search (24 candidates), and thus explores the Pareto-front better. When comparing the two Pareto-optimal result sets, we see that most optimal candidates of the evolutionary approach are superior to optimal candidates of the random search. We have analysed the union of both result sets for Pareto-optimality by removing dominated solutions from that union. The results are shown above marked with "(overall optimal candidates only)". The overall set contains 70 candidates: 64 come from the evolutionary search and 6 from the random search. That means that only 6 of the 70 Pareto-optimal solutions of the evolutionary search are dominated by solutions from the random search, whereas 18 of the 24 Pareto-optimal solutions of the random search are dominated by solutions from the evolutionary search.

Looking at the performance cost tradeoff, the evolutionary search explored candidates with low cost and good response time much better. However, the random search found some candidates with very low response times (down to 0.34 with cost 148.5) that were not found by the evolutionary search (the best found response time here is 0.45 with cost 95). There seems to be potential to improve the evolutionary algorithm in this area. For the reliability cost tradeoff, the evolutionary search found a superior Pareto-front.

Additional requirements limiting the search to candidates with certain quality attributes (e.g. response time lower than 5) could make the search focus on more interesting regions.

With either search strategy, we can automatically find candidates that are superior to the initial candidate defined by the software architect without manual effort. The Pareto-optimal results can be used for trade-off analysis: The software architects can choose a Pareto-optimal candidate with quality properties fitting to the systems requirements. For example, software architects can choose the least expensive candidates with acceptable performance and reliability.