modopt.benchmarking

`modopt.benchmarking.plot_performance_profiles`(data)	Plot the performance profiles for the given data.
`modopt.benchmarking.generate_performance_profiles`(data)	Compute performance profiles and return them along with their corresponding performance ratio (Tau) values.
`modopt.benchmarking.filter_cutest_problems`([...])	Filter CUTEst problems based on the number of variables, number of constraints, and problem tags.

modopt.benchmarking.plot_performance_profiles(data, save_figname='performance.pdf')[source]

Plot the performance profiles for the given data.

Parameters

datadict: Dictionary containing the performance data for each solver. The keys are the (problem_name: str, solver_name: str) and the values are dictionaries containing ‘time’ and ‘success’ as keys with corresponding values denoting the time (float) taken for solver_name to solve problem_name and the success (bool) of the solver. Additionally, if the number of evaluations is available, the dictionary can also contain ‘nev’ as a key with the corresponding int value denoting the number of evaluations.
save_fignamestr, default=’performance.pdf’: Path to save the plot with the performance profiles. If the number of evaluations is available in data, the performance profiles for the number of evaluations is also plotted and saved to the path with save_figname appended with _nev before the extension.

Examples

>>> from modopt.benchmarking import plot_performance_profiles
>>> data = {('problem1', 'solver1'): {'time': 0.1, 'success': True},
...         ('problem1', 'solver2'): {'time': 0.2, 'success': True},
...         ('problem2', 'solver1'): {'time': 0.3, 'success': True},
...         ('problem2', 'solver2'): {'time': 0.4, 'success': False}}
>>> plot_performance_profiles(data, save_figname='performance.pdf')
Total number of problems: 2 

Solver: solver1
--------------------------------------------------
Number of problems solved: 2
Percentage of problems solved: 100.0
-------------------------------------------------- 

Solver: solver2
--------------------------------------------------
Number of problems solved: 1
Percentage of problems solved: 50.0
-------------------------------------------------- 

modopt.benchmarking.generate_performance_profiles(data)[source]

Compute performance profiles and return them along with their corresponding performance ratio (Tau) values.

Depending on the input data, the function returns either two or four outputs:

If ‘nev’ exists in list(data.values())[0]:
- Tau (np.ndarray): Array of Tau values for the primary (time) performance profiles.
- performance_profiles (np.ndarray): Performance profiles corresponding to Tau.
- Tau_n (np.ndarray): Array of Tau values for secondary (nev) performance profiles.
- performance_profiles_n (np.ndarray): Performance profiles corresponding to Tau_n.
Otherwise:
- Tau (np.ndarray): Array of Tau values for the primary (time) performance profiles.
- performance_profiles (np.ndarray): Performance profiles corresponding to Tau.

Parameters

datadict: Dictionary containing the performance data for each solver. The keys are the (problem_name: str, solver_name: str) and the values are dictionaries containing ‘time’ and ‘success’ as keys with corresponding values denoting the time (float) taken for solver_name to solve problem_name and the success (bool) of the solver. Additionally, if the number of evaluations is available, the dictionary can also contain ‘nev’ as a key with the corresponding int value denoting the number of evaluations.

Returns

Taunumpy.ndarray: Array of log-scaled performance ratios.
performance_profilesdict: Dictionary containing the performance profiles for each solver. The keys are the solver names and the values are the proportion of problems solved under the performance ratio corresponding to entries in Tau.
Tau_nnumpy.ndarray: Array of log-scaled performance ratios for the number of evaluations. Only returned if the number of evaluations ‘nev’ is available in the data.
performance_profiles_ndict: Dictionary containing the performance profiles for the number of evaluations. The keys are the solver names and the values are the proportion of problems solved under the performance ratio corresponding to entries in Tau_n. Only returned if the number of evaluations ‘nev’ is available in the data.

Examples

>>> from modopt.benchmarking import generate_performance_profiles
>>> data = {('problem1', 'solver1'): {'time': 0.1, 'success': True},
...         ('problem1', 'solver2'): {'time': 0.2, 'success': True},
...         ('problem2', 'solver1'): {'time': 0.3, 'success': True},
...         ('problem2', 'solver2'): {'time': 0.4, 'success': False}}
>>> Tau, performance_profiles = generate_performance_profiles(data)
Total number of problems: 2 

Solver: solver1
--------------------------------------------------
Number of problems solved: 2
Percentage of problems solved: 100.0
-------------------------------------------------- 

Solver: solver2
--------------------------------------------------
Number of problems solved: 1
Percentage of problems solved: 50.0
-------------------------------------------------- 

>>> print(Tau) 

modopt.benchmarking.filter_cutest_problems(num_vars=[0, 1], num_cons=[0, 0], tags=[], return_metadata=False)[source]

Filter CUTEst problems based on the number of variables, number of constraints, and problem tags.

Parameters

num_varslist, default=[0,1]: List of two integers denoting the minimum and maximum number of variables.
num_conslist, default=[0,0]: List of two integers denoting the minimum and maximum number of constraints.
tagslist, default=[]: List of tags indicating the desired optimization problem types. For example, [‘UC’] for unconstrained problems, [‘BC’] for bound-constrained problems, [‘QP’] for quadratic programming problems, [‘LP’] for linear programming problems, [‘FP’] for feasible point problems, and any valid combination of these tags for problems that satisfy multiple criteria. A problem is selected if it contains all the specified tags.
return_metadatabool, default=False: If True, also return a dictionary containing the metadata of the filtered problems.

Returns

filtered_problemslist: List of CUTEst problems that have the specified tags and the number of variables and constraints within the specified range.
metadatadict, optional: Dictionary containing the metadata of the filtered problems. The keys are the filtered problem names and the values are dictionaries containing the number of variables, number of constraints, tags, and the cutest classification string for the corresponding filtered problem. Only returned if return_metadata is True.

Examples

>>> from modopt.benchmarking import filter_cutest_problems
>>> problem_names = filter_cutest_problems(num_vars=[1,5], num_cons=[1,1])
>>> len(problem_names)
45
>>> pnames, metadata = filter_cutest_problems(num_vars=[1,2], tags=['BC'], return_metadata=True)
>>> len(pnames)
24
>>> pnames[0]
'BQP1VAR'
>>> metadata[pnames[0]]
{'nx': 1, 'nc': 0, 'tags': ['BC', 'QP'], 'cutest_cls': 'QBR2-AN-1-0'}