How to make a Dynamic Matrix in GitHub Actions

Do you want split monorepo for each package? Instead of 20 workflows with copy-paste steps, you can define just one with a static matrix for packages.

Yet, nothing in real life is static but rather dynamic. A new package can be added, old can be removed. How could we automate this even more with a dynamic matrix?

In the last post we talked about monorepo split with GitHub Actions.

Today we'll look on a rather general idea for any GitHub Action - dynamic matrix.

Static Matrix

We've already talked about the use case for the split of many packages into many repositories. Instead of repeating each workflow with a different package, we can use a static matrix.

A typical static matrix looks like this:

        runs-on: ubuntu-latest

                    # list your packages here
                    - coding-standard
                    - phpstan-rules

After you define the strategy, add steps that use ${{ matrix.package }}.

        # ...
            -   uses: actions/checkout@v2

                uses: symplify/github-action-monorepo-split@master
                    GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN }}
                    package-directory: 'packages/${{ matrix.package }}'
                    split-repository-organization: 'symplify'
                    split-repository-name: '${{ matrix.package }}'

For each item in package: a new workflow run will be triggered. So, in this case - matrix triggers 3 parallel runs.

Matrix is an Array

The above could be written in PHP like:

$matrix = [
    'packages' => [

foreach ($matrix['packages'] as $package) {
    $packageDirectory = 'packages/' . $package;
    $splitRepositoryName = $package;
    // ... do the magic

Is there new package? Add it to the matrix:

                    # list your packages here
                    - coding-standard
                    - phpstan-rules
+                   - easy-coding-standard

That's it!

Any static opens up the door to troubles... How easy do you think is to forget to extend the workflow after adding a new package?

From Static Array to JSON

GitHub Actions are ready to make our life easier. Since April 2020, there is a fromJson() function to help us. What does it do?

It converts a json to an array, like this:

-                 package:
-                   - coding-standard
-                   - phpstan-rules
+                 package: ${{ fromJson(["coding-standard", "phpstan-rules"]) }}

You are probably thinking, "Well, that's just terrible, Tomas". Thank you, and you're right.

  • But where is JSON, there is... an endpoint.
  • And where is an endpoint, there is a space for dynamic output.

The Symplify\MonorepoBuilder is using this trick to get all the packages from /packages directory in handy json format:

vendor/bin/monorepo-builder packages-json


This version is not final, but very roughly the command above would be written like this:

-           matrix: ${{ fromJson(["coding-standard", "phpstan-rules"]) }}
+           matrix: ${{ fromJson(vendor/bin/monorepo-builder packages-json) }}

From Json to Fully Dynamic Matrix

If we run the command above as it is, it would fail. Setting up a matrix is like the setUp() method in a PHPUnit test case - there is zero code executed before it.

  • That's why we use setUp() method to actually set what we need first.
  • Then run test() method.

We have to do the same here:

  • In the 1st step, we create the JSON with all packages
  • In the 2nd step, we use this JSON as input for the matrix, that will create a standalone run for each package

How does the Workflow look Like?

    # phase 1
        runs-on: ubuntu-latest

            # git clone + use PHP + composer install
            -   uses: actions/checkout@v2
            -   uses: shivammathur/setup-php@v2
                    php-version: 7.4
            -   run: composer install --no-progress --ansi

            # here we create the json, we need the "id:" so we can use it in "outputs" bellow
                id: set-matrix
                run: echo "::set-output name=matrix::$(vendor/bin/monorepo-builder packages-json --names)"

        # here, we save the result of this 1st phase to the "outputs"
            matrix: ${{ steps.set-matrix.outputs.matrix }}

    # phase 2
        # this means, wait for the "provide_packages_json" phase 1 to finish
        needs: provide_packages_json

        runs-on: ubuntu-latest
            # ↓ the real magic happens here - create dynamic matrix from the json
                package: ${{ fromJson(needs.provide_packages_json.outputs.matrix) }}

            # ...

That's it!

This way, we'll never forget to split nor test a new package. Never.

Check fully functional .github/workflows/split_monorepo.yaml in Symplify monorepo for more details.

Is this something niche? Not really. This week, Kamil has added this approach to Sylius too.

Use for ~~Monorepo Split~~ Anything Dynamic!

This fromJson() trick is not something exclusive to monorepo package splitting. It's just one of the possible use cases.

The primary use case is already in your mind.

  • What is repeated static in your CI?
  • What lists can be delegated to simple command that already knows the data?
  • How could you use the dynamic matrix in your workflows today?

Happy coding!