- Using the Prolog IDE
With the Prolog Benchmark Tool (PBT) you can:
Before running the PBT you have to update two property files and adjust them to your system:
In this file you have to define some directories and files:
In this file you have to add all Prolog versions you want to run benchmarks on. It's always the same pattern:
"Name"|"Path to executable"
Example: To compare the run-time of your queries on SWI-Prolog 7.1.13, 6.2.0 and 6.0.2 specify
SWI 7.1.13|C:\prolog\7.1.13\bin\swipl-win.exe SWI 6.2.0|C:\prolog\6.2.0\bin\swipl-win.exe SWI 6.0.2|C:\prolog\6.0.2\bin\swipl-win.exe
To create a configuration, you just have to start the Program (main class: org.cs3.plbenchmarks.RunPBT) without any additional parameters. Depending on your property files, you will see something similar to this:
In the “Configuration” section, you can:
In the “Benchmark code” section, you can write the goals which you want to check. Every line is a unique goal 1). For every goal the benchmark tool will count the number of results and measure the time it took to get all these results.
There are additional flags you can add to the goals.
Sometimes your result set contains duplicates. If you want to filter these duplicates and only count the unique results, you can use the following flag:
Actually this means, that the goal will be executed twice, so you should just use it if the predicate is cheap or if you really need it.
In many cases the first execution of a predicate might be slower, because Prolog is building up index structures on demand. Any subsequent executions can make use of these index structures and are therefore faster. If you want to measure the influence of the index creation you can use the following flag:
This means, that the goal is executed five times. In the results you can distinguish between the first call and any subsequent call, to see how the runtime changed.
After you are done with your configuration, you can click the button “Add to batch” and your configuration will appear in the list on the right. You can add as many different configurations as you like. If you are changing something, you always have to click “Update current” to apply the changes, otherwise the changes will be ignored.
In the small toolbar on the bottom of the list you can:
For running the benchmarks, you don't need to use the GUI. Just use the GUI to create the configurations and save them to a file. Now you can start the program by adding an additional parameter (the absolute path the your configuration file) and it will start directly, without any GUI.
Make sure, that your factbase folder only contains the pl files and not the qlf files. For the qlf files you need an additional subfolder (in the factbase folder) called “qlf”. If it is not there, create it manually, and the PBT will do the rest.
This means, that there is a bug in your code (e.g. some predicate could not be found). Do you need to check the “JT” box in the PBT (because some of your predicates uses JT-specific predicates)? Did you try to run the benchmark code manually in a Prolog process? Any exception in the code will lead to this message. Unfortunatelly there is currently no precise handling of exceptions, so you just have to check for bugs manually.
Before starting the Prolog process, the PBT will create a lockfile and wait for the Prolog process to delete it. If Prolog crashes the file will not be deleted and the PBT thinks, that Prolog is still busy. Delete the lockfile manually (refresh the project in eclipse and you should see it) and the PBT will continue.
If you change the pl file, you also have to make sure, that you delete all qlf files (in the qlf subfolder) which belonged to this pl file. The PBT doesn't check if a qlf file and a pl file are fitting (and it will just load the old qlf file, while ignoring the new pl file).
This might be a case, where your benchmark code uses derived facts (e.g. ast_node_type_value). These facts are not created automatically by the PBT (because it takes time and they are not always used). If you need these facts, just add “create_derived_facts” as first goal in your benchmark code.