How to Require Minimal Code Coverage for Github Pull-Requests with Coveralls

2017-06-12

Why Code Coverage Shouldn't be Ignored

Marco Pivetta wrote a post about automated code coverage check with PHPUnit few years ago. In the post he explains how important is code coverage for code quality of open-source projects:

"[Code Coverage] is not an universal metric to define if our code works, and there is tons of examples of how your code can still be buggy even if every line of code was executed at least once. It should anyway not be ignored, and it is up to us to decide how hard we want to test each part of our application."

He shows setup for Travis CI, that will

• generate coverage.xml with phpunit
• computed total coverage in % with PHP script
• check it against your minimal requirement
• fails CI if it is lower than that

Simple setup to let computer handle "please, add tests" response for you.

"Setup and forget"

• You move responsibility to a tool, so you don't have to check code coverage for every project of yours manually. CI looks after it for you.
• You don't have to explain at PRs, that "tests are missing". In my experience, less people read CONTRIBUTING.md than red Travis CI error note.
• Let's be honest - do you check your coding standards manually? If so, start with this.

3 Steps to Automate Minimal Coverage Check With Coveralls

Nowadays, Coveralls CI can handle this process completely for you.

Here is how:

1. Create .coveralls.yml

service_name: travis-ci
coverage_clover: coverage.xml # file generated by phpunit
json_path: coverage.json # file generated by php-coveralls

2. Send coverage.xml to Coveralls in .travis.yml

If you already generate coverage with phpunit skip to after_script section.

language: php

php:
  - 7.1

install:
  - composer install

script:
  - vendor/bin/phpunit --coverage-clover coverage.xml

after_script:
  # upload coverage.xml file to Coveralls to analyze it
  # minimal required coverage is set to 80+ %
  - wget https://github.com/satooshi/php-coveralls/releases/download/v1.0.1/coveralls.phar
  - php coveralls.phar --verbose

Performance note: This is the simplest setup with single build. If you run more builds with multiple PHP versions or ENV values, you should run coverage only once, because code coverage uses xdebug extension and thus is very slow.

3. Setup Minimal Require Code Coverage in Coveralls.io

Add your repository on Coveralls.io, go to "Settings" and find "Coverage Threshold fo Failure".

There you can put a value you're ok with. No need to push yourself too hard. I personally start with coverage bit below current level.

That's all setup you need. Just push your pull-request and see the result.

WTFs You Might Meet

1. "I cannot see Coveralls check under PR"

Travis or Scrutinizer appear there right after you create a PR, because they're set up as services (Settings → Integrations & Services).

This is not the case for Coverrals.

Don't worry, it's there. It will appear when Travis will upload coverage.xml to it:

after_script:
  # ...
  - php coveralls.phar --verbose

2. "There is no Report about Required Limit under PR"

As you can see above if build fails due to low coverage, Coveralls actually doesn't show the number for required minimal code coverage.

This is the reason I've put a note to .travis.yml:

after_script:
  # upload coverage.xml file to Coveralls to analyze it
  # minimal required coverage is set to 80+ %

And yes, there is issue for that. Feel free to upvote it or suggest a better solution.

These 2 were surprises for me and took me a while to figure out. Let me know in comments below if you came across any other.

To Sum Up

• Coveralls can help you not only with coverage, but also with its minimal limit.
• Use CI in cases where you can save your attention leaks. You'll have more time for your friends, family and other joys of life - like coding.
• Setup your repository and forget this post.

Enjoy your day!