# Cucumber

This documentation describes the technical aspects of our cucumber test suite.

- See also: [on how to write Cucumber tests](testing.md).
- See also: [about Cucumber in the OSRM wiki](https://github.com/Project-OSRM/osrm-backend/wiki/Cucumber-Test-Suite).
- See also: [the Cucumber docs](https://github.com/cucumber/cucumber-js/tree/main/docs).

## tl;dr

Run the Cucumber tests with:

``` bash
$ npm test
```

## Single OSRM Configuration

An OSRM configuration consists of a *routing algorithm* and a *data load method*. OSRM
currently supports the routing algorithms:
- `ch` (Contraction Hierarchy), and
- `mld` (Multi-Level-Dijkstra)

and the data load methods:

- `directly` (load the files into memory),
- `mmap` (use memory mapped files), and
- `datastore` (use shared memory).

To test all scenarios with a single OSRM configuration, say:

``` bash
$ npx cucumber-js -p home -p mld -p mmap --parallel 8 --fail-fast
```
Explanations follow:

### Profiles

Profiles are chosen with the `-p` commandline argument.  Cucumber profiles allow you to
change multiple configuration items with just one commandline argument.  If you set
more than one profile they are all merged into one configuration.

Note: Cucumber profiles should not be confused with OSRM profiles.  Cucumber profiles
are defined in `cucumber.mjs`.  OSRM profiles reside in the `profiles/*.lua` files.

Our implementation offers following stock profiles. You should always use one base
profile followed by zero or more additional profiles.

| Name      |                                                                |
| --------- | -------------------------------------------------------------- |
| home      | Base profile to use on a developer machine                     |
| github    | Base profile to use on the github CI server                    |
| ch        | Additional profile that selects the CH algorithm               |
| mld       | Additional profile that selects the MLD algorithm              |
| mmap      | Additional profile that selects the mmap data load method      |
| directly  | Additional profile that selects the directly data load method  |
| datastore | Additional profile that selects the datastore data load method |
| stress    | Additional profile that selects only @stress tests             |
| todo      | Additional profile that selects only @todo tests               |
| all       | Additional profile that selects all tests                      |

### Arguments

Here is a  [description of all
arguments](https://github.com/cucumber/cucumber-js/blob/main/docs/configuration.md#options)
you can pass to Cucumber. The interesting ones probably are: `--fail-fast`, `--format`,
`--parallel`, and `--tags`.

Note: when using `--parallel N` make sure there are `N` contiguous free ports at the
configured port number (eg. at ports 5000--5000+N).

## All OSRM Configurations

We provide a shortcut to run all 6 configurations:

``` bash
$ npm test
```
This is how the tests are run on the CI server. You can pass the same arguments as
mentioned above.

## Cache

To speed up subsequent runs with the same parameters, the files generated by Cucumber
and the by the OSRM extraction chain are held in a cache directory. This cache is
located by default in `test/cache` and should be cleaned periodically:

``` bash
$ rm -rf test/cache
```

## Configuration

The whole configuration is done in `cucumber.mjs`. You can either edit `worldParameters`
in `cucumber.mjs` or use environment variables to override single defaults.

| worldParameters | Environment Variable     | Defaults to  |                         |
| --------------- | ------------------------ | ------------ | ----------------------- |
|                 | `CUCUMBER_TIMEOUT`       | 5000         | Scenario timeout in ms. |
| `httpTimeout`   | `CUCUMBER_HTTP_TIMEOUT`  | 2000         | HTTP timeout in ms.     |
| `testPath`      | `CUCUMBER_TEST_PATH`     | `test`       | The test directory      |
| `profilesPath`  | `CUCUMBER_PROFILES_PATH` | `profiles`   | The profiles directory  |
| `logsPath`      | `CUCUMBER_LOGS_PATH`     | `test/logs`  | The logs directory      |
| `cachePath`     | `CUCUMBER_CACHE_PATH`    | `test/cache` | The cache directory     |
| `buildPath`     | `OSRM_BUILD_DIR`         | `build`      | Path to the binaries    |
| `loadMethod`    | `OSRM_LOAD_METHOD`       | datastore    | Data load method        |
| `algorithm`     | `OSRM_ALGORITHM`         | ch           | Routing algorithm       |
| `ip`            | `OSRM_IP`                | 127.0.0.1    | IP Address              |
| `port`          | `OSRM_PORT`              | 5000         | IP Port                 |

The default Cucumber timeout can be changed by setting the environment variable
`CUCUMBER_TIMEOUT`. This is discouraged, because the default timeout of 5 seconds is
plenty for the problem sizes we are dealing with. The probable reasons for a test timing
out are that `osrm-routed` died or that sync between `osrm-datastore` and `osrm-routed`
was lost.

### Other environment variables

`OSRM_RASTER_SOURCE` is set by 'Given the raster source' and is supposed to be read back
in your `profiles/*.lua` profile by `os.getenv('OSRM_RASTER_SOURCE')`.

`OSRM_PROFILE` See: [Pull Request #4516](https://github.com/Project-OSRM/osrm-backend/pull/4516)


## Tags

Single scenarios or whole feature files can be tagged. Tag names can be selected
arbitrarily although it is best to conform to the tags already used. Eg. the tag
`@guidance` can be used to run only those tests related to the guidance feature:

``` bash
$ npm test -- --tags @guidance
```

We also support following special tags:

| Tag                                 | A scenario thus tagged ...                                   |
| ----------------------------------- | ------------------------------------------------------------ |
| `@isolated`                         | will not run while any other scenario is running in parallel |
| `@with_(datastore\|directly\|mmap)` | will be executed iff the load method matches                 |
| `@no_(datastore\|directly\|mmap)`   | will be executed unless the load method matches              |
| `@with_(ch\|mld)`                   | will be executed iff the algorithm matches                   |
| `@no_(ch\|mld)`                     | will be executed unless the algorithm matches                |
| `@with_(linux\|darwin\|win32)`      | will be executed iff the OS matches                          |
| `@no_(linux\|darwin\|win32)`        | will be executed unless the OS matches                       |

A test that calls `osrm-datastore --spring-clean` should not run concurrently with any
other test, thus the tag `@isolated` should be applied. A test that runs or kills
`osrm-routed` should not run while testing the datastore load method, and thus should be
labeled with the tag `@no_datastore`.
