Apache Benchmark (ab.exe) on steroids
Superbenchmarker is a command-line load generator tool for performance testing HTTP APIs and websites. Inspired by Apache Benchmark, it is meant to become Apache Benchmark (ab.exe) on steriods. It displays final results at the end of the test in the terminal window but it also constantly reports back in a web interface.
Superbenchmarker (sb or sbcore) runs on Windows or Mac (not tested yet on Linux) and requires .NET 4.52+ or .NET Core 2.1+ installed on the box.
Easiest way to install sb is to use chocolatey.
> cinst SuperBenchmarkerand to update your version of sb:
> cup SuperBenchmarkerIn this release, use sb to run SuperBenchmaker:
> sb> cinst SuperBenchmarkerCoreand to update your version of sb:
> cup SuperBenchmarkerCoreIn this release, use sbcore to run SuperBenchmaker:
> sbcoreYou can also download the lastest version from the Download folder of this github repository. This is a single exe with all dependencies IL-merged.
Currently, until brew is sorted out, you need to build from the source:
git clone https://github.com/aliostad/SuperBenchmarker
cd SuperBenchmarker
./build.shAnd then you can run using dotnet command:
dotnet ./src/SuperBenchmarker/bin/Debug/netcoreapp2.0/SuperBenchmarker.dll -u https://google.com -N 10To run it, just point it to a website or API using -u (--url) parameter (NOTE: In all these examples, if you have installed SuperBenchmarkerCore, replace sb with sbcore):
sb -u "http://example.com"This command fires 100 GET requests using a single thread, which is equivalent to running command below:
sb -u "http://example.com" -c 1 -n 100So you can use -c and -n to change concurrency or total number of requests. Be careful not to use a very high -c on a single box since it will affect the result of the test. Watchout for CPU consumption on the box and make sure it is =. These parameters include jsonCount or regex extraction (see below). By default, value of the parameter is truncated and capped to 50 characters. To prevent that, use flag -C (--dontcap)`
Sometimes you might be interested to save the whole response body. In this case, use flag -z to store responses in "responses" subfolder. If you would like to store in another folder, use -w:
sb -u "http://example.com/api/cars" -z -w "anotherFolder"Occasionaly, all you need is a small piece of information out of the response body. For example, you would like to see how many records returned in the JSON response. You can use option -j (--jsonCount) with the tree location of element containing array. For example, in this JSON response:
{
"foo": {
"bar": [
{...},
{...},
...
]
}
}the array is located at path "foo/bar" and you would use this command to capture the count in the run.log:
sb -u "http://example.com/api/cars" -j "foo/bar"if the result from the server itself is an array at the root, use empty string as the path:
sb -u "http://example.com/api/cars" -j ""Sometimes you would like to capture part of the response body. You can use option -R to provide a regex to capture the substring of interest. You would use a regex with one group, denoting the value to capture. For example, let's imagine result is an HTML:
Your ID is: 12313554
you would use the commadn below to capture the ID in the log file:
sb -u "http://example.com/api/yourid" -R "Your ID is: (\d+)"Use -x if you need the client to use default proxy. Using option -g you can set the TLS version. For example, to use TLS 1.2:
sb -u "https://example.com/api/things" -g 2You can use -W option to provide number of seconds for warmup where the results are not included in the test.
Not all features can be done via command line parameters and sometimes more flexibility is needed that must be done via a plugin. SuperBenchmarker provides plugin interfaces for these two scenarios:
In order to build a plugin, just add a reference to sb.exe in chocolatey lib folder (if you have installed .NET Core package, add a reference to superbenchmarker.dll) and build your DLL.
You need to include in your DLL a public class implementing IValueProvider. SuperBenchmarker search through public types in the DLL and if it finds an implementation of IValueProvider, it uses it.
public class MyPlugin: IValueProvider
{
private IList _ids; // e.g. populated from a database
public IDictionary GetValues(int index)
{
// return a dictionary with name of the parameters and their corresponding values.
// index is the 0-based count of the requests sent. For example, if index is 9, it is the 10th request
...
return new Dictionary { {"ID", _ids(index)} };
}
}You need to include in your DLL a public class implementing IResponseStatusOverride. SuperBenchmarker search through public types in the DLL and if it finds an implementation of IResponseStatusOverride, it uses it.
public class TeaPot : IResponseStatusOverride
{
public HttpStatusCode OverrideStatus(HttpStatusCode statusReceived,
byte[] responseBuffer,
HttpResponseHeaders responseHeaders,
HttpRequestMessage request)
{
// look into response buffer (byte array) and then return a status
return (HttpStatusCode) 418;
}
}Then run this commmand (NOTE name of the parameter returned in the dictionary is the same as the one defined below):
sb -u "https://example.com/api/car/{{{ID}}}" -p c:/path/to/my/plugin/myplugin.dllBear in mind, in most cases you need to pass the full path to the DLL.
…No open issues yet, or sync has not completed.