Tutorial: Getting started with Microsoft RESTler

In this tutorial, we will cover the steps to start using the Microsoft RESTler, an open-source stateful REST API fuzzing tool.

Created at Microsoft Research, RESTler is described by the collaborators as

The first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services. For a given cloud service with an OpenAPI/Swagger specification, RESTler analyzes its entire specification, and then generates and executes tests that exercise the service through its REST API.

Pre-requisites

Install Python 3.8.2 and .NET core SDK 3.1, according to your appropriate OS.

Set up RESTler

Create a directory to place the RESTler binaries:

Clone the RESTler repository, switch to its root directory and then run the following Python script:

Once it is over, four folders will have been created (compiler, engine, restler, and resultsAnalyzer).

Time to play

📑 Disclaimer: For this tutorial, we are going to use the PokeAPI as our example REST API . As it is a real API and not a fake or demo one, it servers over 60,000,000 API calls each month, the execution of the last mode, Fuzz, will be skipped because it is the most aggressive one and may create outages in the service under test.

RESTler runs in 4 main ordered modes and we will get through all of them now.

In this first step, the tool analyzes the Swagger/OpenAPI spec and generates a fuzzing grammar, which contains information about parameters and responses for individual requests and the dependencies between requests.

To begin let’s create a new directory where will run RESTler on our example.

Then download the pokeapi.yaml and save it toC:\restler-test so we can compile it with RESTler.

The compile command creates a new subdirectory of the same name (Compile) where the results of the compilation are stored in several files, such as grammar.py, grammar.json, dict.json , engine_settings.json , and config.json.

Now that we have the compiled grammar, the tool will attempt to execute successfully all request types of the Swagger specification at least once in order to identify request types that are unreachable/exercisable with the current test setup.

This step may take a while but once it is done the results are stored in a new subdirectory of the same name (Test). Looking at the subdirectory C:\restler-test\Test\RestlerResults\experiment<...>\logs, there will be a file named main.txt which lists one-by-one all the API requests in the Swagger spec and whether the request has been successfully executed (VALID*) or not (INVALID), and also reveals the total spec coverage at the bottom of that file.

*In this context, a valid request means RESTler executed the request and also get a 20x HTTP status code as a response.

Here is the main.txt output I got for the PokeAPI grammar:

In Fuzz-lean mode, RESTler executes once every <endpoint, method> in the compiled RESTler PokeAPI grammar with a default set of checkers looking for bugs that can be found quickly.

This step may also take time to finish. Likewise the past modes, the results are stored in a new subdirectory of the same name (FuzzLean) and at the C:\restler-test\FuzzLean\RestlerResults\experiment<...>\logs\ subdirectory you will find the main.txt file with the same coverage results as the previous Test run. However, in the Fuzz-lean mode, a new bug_buckets directory is created containing four files but for now you can pay attention to bug_buckets.txt which includes a list of each unique sequence that found a bug*.

*There are a few types of bug that RESTler is able to detect, see more details on that matter here.

Here is the .txt output I got for the PokeAPI grammar:

In the last step, RESTler will fuzz the service under test for a longer period of time with the purpose of catching even more bugs and issues (resource leaks, perf degradation, backend corruptions, etc.).

⚠️The Fuzz mode is the more aggressive and may create outages in the service under test if the service is poorly implemented.

📑 Disclaimer: As I had mentioned before, the Fuzz mode was not executed with the PokeAPI because it is an authentic service and our aim to use it was to show how the tool handles a real API and not to provoke outages to external applications.

In order to not encourage you to run this mode with a service that you do not have permission to do so, we will not write the command here but its syntax is available in the RESTler docs.

Overview

  • It is a brand new tool so naturally there are initial bugs to solve but the collaborators seem to be actively working on those.
  • Its documentation is very informative, instructive, and well written.
  • It provides a self-contained demo service so the user can easily play around with all the available modes before applying it to the target API.
  • It provides informative execution logs.
  • The syntax of its commands is intuitive.
  • Its output patterns (directories structure, files naming, files content) make sense and are also self-explanatory enough.

That said, Microsoft RESTler seems a really promising open-source tool for REST API fuzzing and I am excited to read more reviews about how it handles huge professional APIs.

If you try it out, feel free to share your thoughts and experiences in the comments below.