Baike.dev
Log in
Back to tool/Back to issues
#1937·reviewdog

`-fail-level` not respecting configured runner level (in e.g.: `reviewdog.yml`) (not sure if a bug or not)

Author: DrowzeCreated Sep 20, 2024Updated Aug 6, 2026

Hello

Just ran on this today - it seems that the new -fail-level argument is not respecting the runner level that was configured in .reviewdog.yml config file. I'm not sure if this is expected or not tbh, but seems like a changed behaviour from the last version we were using (version v0.18.1 with -reporter=github-pr-check -fail-on-error=true)

Reproducing it:

Minimal repro:

yaml
#.reviewdog.yml
runner:
  error_level_runner:
    cmd: 'true'
    errorformat:
      - '%f:%l: %m'
    level: error

  info_level_runner:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42: example info"'
    errorformat:
      - '%f:%l: %m'
    level: info
$ reviewdog -fail-level=error
time=2024-09-20T14:16:29.796-03:00 level=INFO msg="reviewdog: [start] runner=error_level_runner"
time=2024-09-20T14:16:29.797-03:00 level=INFO msg="reviewdog: [start] runner=info_level_runner"
time=2024-09-20T14:16:29.837-03:00 level=INFO msg="reviewdog: [finish] runner=error_level_runner"
time=2024-09-20T14:16:29.960-03:00 level=INFO msg="reviewdog: [finish] runner=info_level_runner"
foo.rb:42: [info_level_runner] example info
reviewdog: found at least one issue with severity greater than or equal to the given level: error
Complete repro (all -fail-level=+"runner level" combinations) (click to expand)
yaml
# .reviewdog.yml
runner:
  i_runner:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42: example"'
    errorformat: [ '%f:%l: %m' ]
    level: info

  i_runner_i:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:I: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: info

  i_runner_w:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:W: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: info

  i_runner_e:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:E: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: info

  w_runner:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42: example"'
    errorformat: [ '%f:%l: %m' ]
    level: warning

  w_runner_i:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:I: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: warning

  w_runner_w:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:W: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: warning

  w_runner_e:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:E: example"'
    errorformat: [ '%f:%l:%t: %m' ]
    level: warning

  e_runner:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42: example"'
    errorformat: [ '%f:%l: %m' ]
    level: error

  e_runner_i:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:I: example"'
    errorformat: [ '%f:%l %t: %m' ]
    level: error

  e_runner_w:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:W: example"'
    errorformat: [ '%f:%l %t: %m' ]
    level: error

  e_runner_e:
    cmd: |
      ruby -e '$stdout.puts "foo.rb:42:E: example"'
    errorformat: [ '%f:%l %t: %m' ]
    level: error

Testing all runner levels/fail-level combinations:

bash
runners=(i_runner i_runner_i i_runner_w i_runner_e w_runner w_runner_i w_runner_w w_runner_e e_runner e_runner_i e_runner_w e_runner_e)

for runner in "${runners[@]}"; do
  runner_level="$(yq ".runner.$runner.level" .reviewdog.yml)"
  reviewdog -runners="$runner" -fail-level=info &>/dev/null
  exit_code=$?
  printf "%18s:%20s%25s%15s\n" "fail-level=info" "runner=$runner" "runner_level=$runner_level" "exit_code=$exit_code"
done

for runner in "${runners[@]}"; do
  runner_level="$(yq ".runner.$runner.level" .reviewdog.yml)"
  reviewdog -runners="$runner" -fail-level=warning &>/dev/null
  exit_code=$?
  printf "%18s:%20s%25s%15s\n" "fail-level=warning" "runner=$runner" "runner_level=$runner_level" "exit_code=$exit_code"
done

for runner in "${runners[@]}"; do
  runner_level="$(yq ".runner.$runner.level" .reviewdog.yml)"
  reviewdog -runners="$runner" -fail-level=error &>/dev/null
  exit_code=$?
  printf "%18s:%20s%25s%15s\n" "fail-level=error" "runner=$runner" "runner_level=$runner_level" "exit_code=$exit_code"
done

Output:

   fail-level=info:     runner=i_runner        runner_level=info    exit_code=1
   fail-level=info:   runner=i_runner_i        runner_level=info    exit_code=1
   fail-level=info:   runner=i_runner_w        runner_level=info    exit_code=1
   fail-level=info:   runner=i_runner_e        runner_level=info    exit_code=1
   fail-level=info:     runner=w_runner     runner_level=warning    exit_code=1
   fail-level=info:   runner=w_runner_i     runner_level=warning    exit_code=1
   fail-level=info:   runner=w_runner_w     runner_level=warning    exit_code=1
   fail-level=info:   runner=w_runner_e     runner_level=warning    exit_code=1
   fail-level=info:     runner=e_runner       runner_level=error    exit_code=1
   fail-level=info:   runner=e_runner_i       runner_level=error    exit_code=0
   fail-level=info:   runner=e_runner_w       runner_level=error    exit_code=0
   fail-level=info:   runner=e_runner_e       runner_level=error    exit_code=0
fail-level=warning:     runner=i_runner        runner_level=info    exit_code=1
fail-level=warning:   runner=i_runner_i        runner_level=info    exit_code=0
fail-level=warning:   runner=i_runner_w        runner_level=info    exit_code=1
fail-level=warning:   runner=i_runner_e        runner_level=info    exit_code=1
fail-level=warning:     runner=w_runner     runner_level=warning    exit_code=1
fail-level=warning:   runner=w_runner_i     runner_level=warning    exit_code=0
fail-level=warning:   runner=w_runner_w     runner_level=warning    exit_code=1
fail-level=warning:   runner=w_runner_e     runner_level=warning    exit_code=1
fail-level=warning:     runner=e_runner       runner_level=error    exit_code=1
fail-level=warning:   runner=e_runner_i       runner_level=error    exit_code=0
fail-level=warning:   runner=e_runner_w       runner_level=error    exit_code=0
fail-level=warning:   runner=e_runner_e       runner_level=error    exit_code=0
  fail-level=error:     runner=i_runner        runner_level=info    exit_code=1
  fail-level=error:   runner=i_runner_i        runner_level=info    exit_code=0
  fail-level=error:   runner=i_runner_w        runner_level=info    exit_code=0
  fail-level=error:   runner=i_runner_e        runner_level=info    exit_code=1
  fail-level=error:     runner=w_runner     runner_level=warning    exit_code=1
  fail-level=error:   runner=w_runner_i     runner_level=warning    exit_code=0
  fail-level=error:   runner=w_runner_w     runner_level=warning    exit_code=0
  fail-level=error:   runner=w_runner_e     runner_level=warning    exit_code=1
  fail-level=error:     runner=e_runner       runner_level=error    exit_code=1
  fail-level=error:   runner=e_runner_i       runner_level=error    exit_code=0
  fail-level=error:   runner=e_runner_w       runner_level=error    exit_code=0
  fail-level=error:   runner=e_runner_e       runner_level=error    exit_code=0

From the repros above, I believe:

  • when the runner message has a level on it, reviewdog always use it to calculate the runner result level.
  • when the runner message does not have a level on it, reviewdog always calculate the runner result as level error, regardless of the configured runner level.

So essentially it seems that the runner level (as configured in .reviewdog.yml) is not being taken into consideration for calculating the exit code.

Source: reviewdog/reviewdog

View original on GitHubView discussion on GitHub