Tutorial: Getting started with Microsoft RESTler

In this tutorial, we will cover the steps to start using the Microsoft RESTler, an open-source stateful REST API fuzzing tool.

Photo by Christina @ wocintechchat.com on Unsplash

Created at Microsoft Research, RESTler is described by the collaborators as

The first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services. For a given cloud service with an OpenAPI/Swagger specification, RESTler analyzes its entire specification, and then generates and executes tests that exercise the service through its REST API.

Pre-requisites

Install Python 3.8.2 and .NET core SDK 3.1, according to your appropriate OS.

Set up RESTler

Create a directory to place the RESTler binaries:

mkdir C:\restler_bin

Clone the RESTler repository, switch to its root directory and then run the following Python script:

python ./build-restler.py --dest_dir C:\restler_bin

Once it is over, four folders will have been created (compiler, engine, restler, and resultsAnalyzer).

Time to play

📑 Disclaimer: For this tutorial, we are going to use the PokeAPI as our example REST API . As it is a real API and not a fake or demo one, it servers over 60,000,000 API calls each month, the execution of the last mode, Fuzz, will be skipped because it is the most aggressive one and may create outages in the service under test.

RESTler runs in 4 main ordered modes and we will get through all of them now.

1. Compile

In this first step, the tool analyzes the Swagger/OpenAPI spec and generates a fuzzing grammar, which contains information about parameters and responses for individual requests and the dependencies between requests.

To begin let’s create a new directory where will run RESTler on our example.

mkdir C:\restler-test
cd C:\restler-test

Then download the pokeapi.yaml and save it toC:\restler-test so we can compile it with RESTler.

C:\restler_bin\restler\Restler.exe compile --api_spec C:\restler-test\pokeapi.yaml

The compile command creates a new subdirectory of the same name (Compile) where the results of the compilation are stored in several files, such as grammar.py, grammar.json, dict.json , engine_settings.json , and config.json.

2. Test

Now that we have the compiled grammar, the tool will attempt to execute successfully all request types of the Swagger specification at least once in order to identify request types that are unreachable/exercisable with the current test setup.

C:\restler_bin\restler\Restler.exe test --gr
ammar_file  C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

This step may take a while but once it is done the results are stored in a new subdirectory of the same name (Test). Looking at the subdirectory C:\restler-test\Test\RestlerResults\experiment<...>\logs, there will be a file named main.txt which lists one-by-one all the API requests in the Swagger spec and whether the request has been successfully executed (VALID*) or not (INVALID), and also reveals the total spec coverage at the bottom of that file.

*In this context, a valid request means RESTler executed the request and also get a 20x HTTP status code as a response.

Here is the main.txt output I got for the PokeAPI grammar:

2020-11-21 22:25:50.269: Going to fuzz a set with 2 requests
2020-11-21 22:25:50.269: Request-0: Value Combinations: 1
2020-11-21 22:25:50.269: Request-1: Value Combinations: 2
2020-11-21 22:25:50.269: Avg. Value Combinations per Request: 1.5
2020-11-21 22:25:50.269: Median Value Combinations per Request: 1.5
2020-11-21 22:25:50.269: Min Value Combinations per Request: 1
2020-11-21 22:25:50.269: Max Value Combinations per Request: 2
2020-11-21 22:25:50.269: Total dependencies: 0
2020-11-21 22:25:50.271: Starting directed-smoke-test
2020-11-21 22:25:50.288: Will attempt to render 2 requests found
Rendering request 0 from scratch
2020-11-21 22:31:51.911: Request 0
2020-11-21 22:31:51.912: Endpoint - /pokemon/{idOrName}
2020-11-21 22:31:51.912: Hex Def - 68da971679008d9273ce4bbcb33f7faa1522c900
2020-11-21 22:31:51.912: Sequence length that satisfies dependencies: 1
2020-11-21 22:31:51.932: Rendering INVALID
2020-11-21 22:31:51.951: A bug code was received after rendering this request.
- restler_static_string: 'GET '
- restler_static_string: '/'
- restler_static_string: 'api'
- restler_static_string: '/'
- restler_static_string: 'v2'
- restler_static_string: '/'
- restler_static_string: 'pokemon'
- restler_static_string: '/'
+ restler_fuzzable_int: ['0', '1']
- restler_static_string: ' HTTP/1.1\r\n'
- restler_static_string: 'Accept: application/json\r\n'
- restler_static_string: 'Host: pokeapi.co\r\n'
- restler_static_string: '\r\n'
Rendering request 1 from scratch
2020-11-21 22:35:53.090: Request 1
2020-11-21 22:35:53.090: Endpoint - /pokemon
2020-11-21 22:35:53.090: Hex Def - b093ad80f75a0b14181970c3262f13d41f570641
2020-11-21 22:35:53.090: Sequence length that satisfies dependencies: 1
2020-11-21 22:35:53.113: Rendering INVALID
2020-11-21 22:35:53.135: A bug code was received after rendering this request.
- restler_static_string: 'GET '
- restler_static_string: '/'
- restler_static_string: 'api'
- restler_static_string: '/'
- restler_static_string: 'v2'
- restler_static_string: '/'
- restler_static_string: 'pokemon'
- restler_static_string: ' HTTP/1.1\r\n'
- restler_static_string: 'Accept: application/json\r\n'
- restler_static_string: 'Host: pokeapi.co\r\n'
- restler_static_string: '\r\n'
2020-11-21 22:35:53.481: Final Swagger spec coverage: 0 / 2
2020-11-21 22:35:53.481: Rendered requests: 2 / 2
2020-11-21 22:35:53.481: Rendered requests with "valid" status codes: 0 / 2
2020-11-21 22:35:53.481: Num fully valid requests (no resource creation failures): 0
2020-11-21 22:35:53.481: Num requests not rendered due to invalid sequence re-renders: 0
2020-11-21 22:35:53.481: Num invalid requests caused by failed resource creations: 0
2020-11-21 22:35:53.481: Total Creations of Dyn Objects: 0
2020-11-21 22:35:53.481: Total Requests Sent: {'gc': 0, 'main_driver': 3}
2020-11-21 22:35:53.481: Bug Buckets: {'main_driver_timeout': 2}

3. Fuzz-lean

In Fuzz-lean mode, RESTler executes once every <endpoint, method> in the compiled RESTler PokeAPI grammar with a default set of checkers looking for bugs that can be found quickly.

C:\restler_bin\restler\Restler.exe fuzz-lean --gr
ammar_file  C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

This step may also take time to finish. Likewise the past modes, the results are stored in a new subdirectory of the same name (FuzzLean) and at the C:\restler-test\FuzzLean\RestlerResults\experiment<...>\logs\ subdirectory you will find the main.txt file with the same coverage results as the previous Test run. However, in the Fuzz-lean mode, a new bug_buckets directory is created containing four files but for now you can pay attention to bug_buckets.txt which includes a list of each unique sequence that found a bug*.

*There are a few types of bug that RESTler is able to detect, see more details on that matter here.

Here is the .txt output I got for the PokeAPI grammar:

main_driver_timeout: 2
Total Buckets: 2
-------------
main_driver_timeout - Bug was reproduced - main_driver_timeout_1.txt
Hash: main_driver_timeout_466d56fb305ae37cfc6d92596aeb99ca258d6d54
GET /api/v2/pokemon/1 HTTP/1.1\r\nAccept: application/json\r\nHost: pokeapi.co\r\n\r\n
--------------------------------------------------------------------------------
main_driver_timeout - Bug was reproduced - main_driver_timeout_2.txt
Hash: main_driver_timeout_2b385ce87900b8609cb646d75fd7cd963584586a
GET /api/v2/pokemon HTTP/1.1\r\nAccept: application/json\r\nHost: pokeapi.co\r\n\r\n
--------------------------------------------------------------------------------

4. Fuzz

In the last step, RESTler will fuzz the service under test for a longer period of time with the purpose of catching even more bugs and issues (resource leaks, perf degradation, backend corruptions, etc.).

⚠️The Fuzz mode is the more aggressive and may create outages in the service under test if the service is poorly implemented.

📑 Disclaimer: As I had mentioned before, the Fuzz mode was not executed with the PokeAPI because it is an authentic service and our aim to use it was to show how the tool handles a real API and not to provoke outages to external applications.

In order to not encourage you to run this mode with a service that you do not have permission to do so, we will not write the command here but its syntax is available in the RESTler docs.

Overview

Cons

• It is a brand new tool so naturally there are initial bugs to solve but the collaborators seem to be actively working on those.

Pros

• Its documentation is very informative, instructive, and well written.
• It provides a self-contained demo service so the user can easily play around with all the available modes before applying it to the target API.
• It provides informative execution logs.
• The syntax of its commands is intuitive.
• Its output patterns (directories structure, files naming, files content) make sense and are also self-explanatory enough.

That said, Microsoft RESTler seems a really promising open-source tool for REST API fuzzing and I am excited to read more reviews about how it handles huge professional APIs.

If you try it out, feel free to share your thoughts and experiences in the comments below.