How to Verify a PHP Upgrade with PHPUnit

A migration is not finished when the scanner is clean. It is finished when the test suite is green on the new runtime. This is how to run that suite so the result actually means something, how to read the output, and how to turn the failures into the next batch of work.

The short answer

Run PHPUnit on exactly the PHP version and the extensions you are going to deploy, with the deprecations and the warnings visible, and with the deprecations failing the run while you are still migrating. Read the result as two numbers: the pass rate and the number of failures. Compare the pass rate with the one you recorded before the upgrade, because a stable rate with a different set of failures is progress and a lower rate is a regression.

That is the whole method. The details below are about the ways it goes wrong.

Run the suite on the new runtime

The number one source of a false green is running the tests somewhere other than where the code will run. Three things have to match:

• The PHP version. Not "PHP 8", but the exact version: php -v in the CI container and on the production host have to agree. A bug that only exists in 8.1.12 will not show up on 8.3.
• The extensions. php -m on both sides. A missing intl or bcmath turns a passing test into a fatal error, and the stack trace points at your code rather than at the missing extension.
• The configuration. php.ini values such as memory_limit, max_execution_time and error_reporting behave differently in a container and on a host.

The most reliable setup is one container image for the tests and the same image for the deployment:

docker run --rm -v "$PWD":/app -w /app php:8.3-cli \
  composer install --no-interaction --prefer-dist
docker run --rm -v "$PWD":/app -w /app php:8.3-cli \
  vendor/bin/phpunit --display-deprecations --display-warnings

Make the deprecations visible

PHP 8.1 to 8.3 are deprecation releases. The notices they raise are the PHP 9 work list, so they have to be visible while you are in the code:

vendor/bin/phpunit --display-deprecations --display-warnings --fail-on-deprecation

Use --fail-on-deprecation during the migration and remove it once the suite is clean, otherwise it blocks every pull request for a reason that is not a regression.

Note that --display-deprecations was added in PHPUnit 10. On PHPUnit 9 the equivalent is to make sure your phpunit.xml does not silence the deprecations:

<phpunit>
    <php>
        <ini name="error_reporting" value="E_ALL"/>
        <ini name="display_errors" value="true"/>
    </php>
</phpunit>

Read the output

The end of the PHPUnit output is the part that matters:

PHPUnit 10.5.0 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.6
Configuration: /app/phpunit.xml

..........F....F.................................................  58 / 58 (100%)

Time: 00:04.213, Memory: 38.00 MB

There were 2 failures:

1) Tests\Order\OrderProcessorTest::testAppliesDiscount
Failed asserting that 90.0 matches expected 100.0.

/app/tests/Order/OrderProcessorTest.php:42

2) Tests\Order\OrderProcessorTest::testRoundsTotal
Failed asserting that 10.5 matches expected 10.0.

/app/tests/Order/OrderProcessorTest.php:57

ERRORS!
Tests: 58, Assertions: 214, Failures: 2.

Four numbers carry the information:

• The pass rate. 58 tests, 2 failures means 56 of 58 passed. This is the number you compare before and after each batch.
• Failures versus errors. A failure means the assertion did not hold. An error means the test could not run, because of an exception, a fatal error or a missing fixture. Errors are the more urgent of the two, because they usually mean the code path is broken rather than the expectation being wrong.
• The file and the line of every failure. That is the work list.
• The deprecation count, when you enabled the display flags. Any number above zero is a PHP 9 item.

php2shell parses this output and turns it into exactly that list: the pass rate, the number of failures and errors, and the failing test cases with their locations. The parsing happens on the output text, never on your source code.

Turn failures into a work list

A list of forty failing tests is not a plan. Convert it into one:

1. Group by cause, not by test. Forty failures caused by one changed rounding rule is one task. Forty failures caused by forty different issues is forty tasks, and you should look for the shared cause first, because PHP 8 changes tend to be systemic.
2. Separate the errors from the failures. Errors first: they usually mean the code does not run at all on the new version.
3. Decide the direction per group. For each group, decide whether the test is wrong or the code is wrong. PHP 8 changed some behaviours on purpose, so a failing test can be a test that encodes the old behaviour. Updating the expectation is correct in that case, and it has to be a deliberate decision, not a reflex.
4. Work in groups of related fixes and re-run the suite after each one. The pass rate after every run is your progress metric.

The order that works best: the removed functions first, because they are fatal; then the type strictness changes, because they change behaviour; then everything else.

When the suite was already failing

Plenty of projects start a migration with a red suite. That is fine, as long as you measure it before you change anything:

# before the upgrade
vendor/bin/phpunit --testdox | tail -5
# Tests: 58, Assertions: 214, Failures: 6, Errors: 1.

Now you have a baseline: 51 passing, 7 not. After each batch of fixes, compare:

• Same pass rate, different failures: progress. The rate held and the failing set changed, so the fixes went somewhere.
• Lower pass rate: a regression. Find the change that caused it before you do anything else.
• Higher pass rate: expected, but check that the new passes are the ones you fixed and not a test that now stops asserting anything.

Write the baseline down. A number you remember is a number you will doubt after the third batch.

Cover the changed paths

Coverage percentage is a poor migration metric. What matters is whether the code paths that the upgrade touched are exercised:

• For every class the scanner flagged, there should be at least one test that reaches it. A flagged class with no test is an unknown.
• For every rule you fixed automatically, the test that covered it should still pass. The automatic fixer changes the shape of the code, not the behaviour, so a failure means the transformation was wrong.
• The error paths matter more than the happy path. The null handling change only shows up when the value is null, and that is exactly the branch the happy path test never reaches.

A useful technique during a migration is a temporary test that asserts the new behaviour explicitly, for example that a deprecated function is no longer called. It fails on the old code, passes on the new one, and it documents the change.

Run it in CI

The suite belongs in the pipeline from the first day of the migration, not at the end. A minimal job:

name: php-upgrade-verification
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    container: php:8.3-cli
    steps:
      - uses: actions/checkout@v4
      - run: composer install --no-interaction --prefer-dist
      - run: vendor/bin/phpunit --display-deprecations --fail-on-deprecation --coverage-text

Two things to add as the migration progresses:

• A matrix over the target versions. Run the suite on 8.1, 8.2 and 8.3, not only on the newest one. It is the cheapest way to find a version specific behaviour.
• The compatibility scan as a separate job. The scan says what is incompatible, the suite says what still works. Both belong in the pipeline, and they answer different questions.

PHPUnit 9 to 10 to 11

A PHP upgrade usually drags a PHPUnit upgrade along with it, and the configuration format changes between the major versions. These are the changes that break a CI pipeline:

PHPUnit 10

• The XML configuration is validated against a new schema. Unknown attributes are a hard error, not a warning. In particular the attributes moved from the root element to the <coverage> and <source> elements.
• The <coverage><include> element replaced the old <filter> element.
• --coverage-text now needs a coverage driver: Xdebug or PCOV. Without one, the job fails instead of silently reporting no coverage.
• The convertDeprecationsToExceptions attribute was removed, because deprecations are handled by --fail-on-deprecation instead.
• The --whitelist option was replaced by --coverage-filter.

PHPUnit 11

• PHP 8.2 is the minimum supported version.
• The <source> element and the include and exclude attributes replaced the older coverage configuration entirely.
• Metadata in a docblock is validated strictly. An attribute that does not exist is a fatal error in the test, not a warning.

The practical advice: change the PHPUnit version and the PHP version in one step, and expect to spend an hour on the configuration file rather than on the code. It is a good week to keep the application code changes separate, so you can tell a configuration problem from a compatibility problem.

Frequently asked questions

Should the test suite pass before I start the upgrade?

Yes, ideally. A green suite before the upgrade is the only reference you have: after the upgrade, a failure tells you that the upgrade broke something, not that the test was already broken.

What if the test suite is already failing?

Record the exact pass rate before you change anything and compare it with the rate after each batch of fixes. A stable rate with a different set of failures is progress; a lower rate is a regression.

How many tests do I need?

Enough to cover the code paths the upgrade touches. If a class is flagged by the scanner, there should be at least one test that exercises it. Coverage of the changed paths matters more than the overall percentage.

Where to go next

• How to upgrade PHP 7.4 to 8.3 — the full step-by-step migration plan.
• 12 PHP 8 error messages — what the failures look like and how to fix them.
• PHPCompatibility vs Rector — which tool covers which half of the migration.
• The FAQ — plans, credits, licences and how your code is handled.