Adding a Test Case for Nebula Graph

min.wu
2021-07-28

Adding a Test Case for Nebula Graph

Before We Start

Nebula Graph is a product under continuous development, so the format of the returned result may be updated as required, which may cause parse errors to your applications.

For example, after the release of Nebula Graph v2.0.0, the updated SHOW JOBS syntax caused crash of Nebula Graph Studio because it cannot parse the statement correctly.

To avoid such errors, the Nebula Graph team have developed a new testing framework to help its users add new test cases without too much effort.

Preparing Environment

According to Install Nebula Graph by compiling the source code, prepare the environment as follows:

a. Get a development machine ready. Here is the minimum requirement:

  • Python 3.7 or later.
python3 --version

b. Follow the steps described in the documentation. Create a branch from the master branch.

git checkout -b  dev tck

For further details, refer to the steps of How to Contribute in Nebula Graph Database Manual.

c. Install the environment for executing the test cases.

$ cd nebula-graph/tests/
$ make init-all

If an error occurs informing that the module version is too old, you should upgrade it according to the prompt. For example, run the command pip3 install --upgrade keyrings.alt.

d. Execute the existing cases.

make test

Adding New Test Cases

Learn from Existing Cases

There have been a lot of test cases in the tests/tck/features/ directory of the nebula-graph repository on GitHub. Each feature has its own subdirectory.

Take MATCH as an example. Look at the match/MatchById.feature file.

Feature: Match By Id

  Background:
    Given a graph with space named "basketballplayer"     # Choose a ready graph space named basketballplayer for unit testing.

  Scenario: single node                                   # Give the new scenario (test case) a name.
    When executing query:                              # The line surrounded by three double quotes is the statement sent to Nebula Graph.
      """                                                            # This statement is to obtain a vertex with the "James Harden" ID.
      MATCH (n) WHERE id(n) == 'James Harden' RETURN n
      """                                                            # The "Then" line gives the expected result.                
    Then the result should be, in any order, with relax comparison:
      | n                |                                          # The expected result should be composed of one column named "n".
      | ("James Harden") |                                # A record with "James Harden" surrounded by parentheses, representing a vertex
    When executing query:                            # The next statement. 
      """
      MATCH (n) WHERE id(n) == 'not_exist_vertex' RETURN n
      """
    Then the result should be, in any order, with relax comparison:
      | n |

nebula-test is a BDD-based testing framework. All these feature files are written in Gherkin. For more information about BDD, refer to BDD-Based Integration Testing Framework for Nebula Graph: Part I.

Here is an introduction to the sections of a feature file:

  • Feature: Gives the subject of the file.
  • Background: Gives the steps that are common to all the scenarios in the same feature file. In this example, a ready graph space named “basketballplayer”, with schema, indexes, and data, is selected.
  • Scenario: Composed of a pair of When and Then steps. A scenario is the smallest unit of concurrent execution.
  • When executing query: Sends a series of commands to Nebula Graph.
  • Then the result should be: Obtains the result from Nebula Graph and compares the result with the expected result.

The returned result can be compared with the expected result with one of these methods:

  • the result should be: The results must be exactly the same, which means the content of the rows, columns, and records are exactly the same.
  • the result should contain: If the returned result contain the mentioned content in the expected result, it is considered valid.
  • the result should be, in any order: The order of rows in the returned result is not used to determine the legality of the result.
  • the result should be, in any order, with relax comparison: The returned result should comply with the given format but the order of the rows does not matter. Besides, the result must contain the specified content. For example, each record must has one or more “Tiago Splitter” strings.
  • the execution should be successful: The execution is successful.
  • an ExecutionError should be raised at runtime: not existed! The execution failed. The ExecutionError exception is thrown, and the error code is not existed!.

Writing a Test Case

If you want to do a check of the format of the returned result of SHOW JOBS, you can add a feature file named JobCommands.feature under the directory: nebula-graph/tests/tck/features/job/.

@mintest                                                         # A tag. This is for debugging.

Feature: Job compact, flush, rebuild_index

  Background:                                     # Choose a graph space with a dataset.
    Given a graph with space named "basketballplayer"

  Scenario: submit job
    When executing query:
      """
      SUBMIT JOB COMPACT;
      """
    Then the result should contain:
      | New Job Id |
      | /\d+/          |              # Write a regular expression for positive integers
    When executing query:
      """
      SUBMIT JOB FLUSH;
      """
     When executing query:
      """
      SUBMIT JOB STATS;

      SHOW JOBS;
      """
    Then the result should contain:   # Do a check of the result format
      | Job Id | Command | Status  | Start Time                               | Stop Time                 |
      | /\d+/  | "STATS"      | /\w+/  | /\d+-\d+-\d+T\d+:\d+:\d+/ | /\d+-\d+-\d+T\d+:\d+:\d+/ |

Debugging a Test Case

nebula-graph/tests> make fmt

Do a check of the format and get the file format valid.

nebula-graph/tests>  pytest -m "mintest"

Run the test case. In general, the case may need some changes. You can use # to comment out a line or use @skip to skip a scenario.

nebula-graph/tests>  pytest -n4

If the memory capacity is not big enough, use -n4 to control the concurrency.

For more information, refer to the test manual.

Submitting Test Case via Git

If the test case passes, remove the @mintest tag and submit a pull request to the Nebula Graph repository on GitHub. For more information, refer to How to Contribute.

If you are interested in learning more about Nebula Graph, welcome to join the Nebula Slack channel!