How to make a Dynamic Matrix in GitHub Actions

2020-11-16

In the last post we talked about monorepo split with GitHub Actions.

Today we'll look on a rather general idea for any GitHub Action - dynamic matrix.

Static Matrix

We've already talked about the use case for the split of many packages into many repositories. Instead of repeating each workflow with a different package, we can use a static matrix.

A typical static matrix looks like this:

jobs:
    monorepo_split:
        runs-on: ubuntu-latest

        strategy:
            matrix:
                package:
                    # list your packages here
                    - coding-standard
                    - phpstan-rules

After you define the strategy, add steps that use ${{ matrix.package }}.

        # ...
        steps:
            -   uses: actions/checkout@v2

            -
                uses: symplify/github-action-monorepo-split@master
                env:
                    GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN }}
                with:
                    package-directory: 'packages/${{ matrix.package }}'
                    split-repository-organization: 'symplify'
                    split-repository-name: '${{ matrix.package }}'

For each item in package: a new workflow run will be triggered. So, in this case - matrix triggers 3 parallel runs.

Matrix is an Array

The above could be written in PHP like:

$matrix = [
    'packages' => [
        'coding-standard',
        'phpstan-rules',
    ]
];

foreach ($matrix['packages'] as $package) {
    $packageDirectory = 'packages/' . $package;
    $splitRepositoryName = $package;
    // ... do the magic
}

Is there new package? Add it to the matrix:

            matrix:
                package:
                    # list your packages here
                    - coding-standard
                    - phpstan-rules
+                   - easy-coding-standard

That's it!

Any static opens up the door to troubles... How easy do you think is to forget to extend the workflow after adding a new package?

From Static Array to JSON

GitHub Actions are ready to make our life easier. Since April 2020, there is a fromJson() function to help us. What does it do?

It converts a json to an array, like this:

            matrix:
-                 package:
-                   - coding-standard
-                   - phpstan-rules
+                 package: ${{ fromJson(["coding-standard", "phpstan-rules"]) }}

You are probably thinking, "Well, that's just terrible, Tomas". Thank you, and you're right.

• But where is JSON, there is... an endpoint.
• And where is an endpoint, there is a space for dynamic output.

The Symplify\MonorepoBuilder is using this trick to get all the packages from /packages directory in handy json format:

vendor/bin/monorepo-builder packages-json

↓

[
    "coding-standard",
    "phpstan-rules"
]

This version is not final, but very roughly the command above would be written like this:

-           matrix: ${{ fromJson(["coding-standard", "phpstan-rules"]) }}
+           matrix: ${{ fromJson(vendor/bin/monorepo-builder packages-json) }}

From Json to Fully Dynamic Matrix

If we run the command above as it is, it would fail. Setting up a matrix is like the setUp() method in a PHPUnit test case - there is zero code executed before it.

• That's why we use setUp() method to actually set what we need first.
• Then run test() method.

We have to do the same here:

• In the 1st step, we create the JSON with all packages
• In the 2nd step, we use this JSON as input for the matrix, that will create a standalone run for each package

How does the Workflow look Like?

jobs:
    # phase 1
    provide_packages_json:
        runs-on: ubuntu-latest

        steps:
            # git clone + use PHP + composer install
            -   uses: actions/checkout@v2
            -   uses: shivammathur/setup-php@v2
                with:
                    php-version: 7.4
            -   run: composer install --no-progress --ansi

            # here we create the json, we need the "id:" so we can use it in "outputs" bellow
            -
                id: set-matrix
                run: echo "::set-output name=matrix::$(vendor/bin/monorepo-builder packages-json --names)"

        # here, we save the result of this 1st phase to the "outputs"
        outputs:
            matrix: ${{ steps.set-matrix.outputs.matrix }}

    # phase 2
    split_monorepo:
        # this means, wait for the "provide_packages_json" phase 1 to finish
        needs: provide_packages_json

        runs-on: ubuntu-latest
        strategy:
            # ↓ the real magic happens here - create dynamic matrix from the json
            matrix:
                package: ${{ fromJson(needs.provide_packages_json.outputs.matrix) }}

        steps:
            # ...

That's it!

This way, we'll never forget to split nor test a new package. Never.

Check fully functional .github/workflows/split_monorepo.yaml in Symplify monorepo for more details.

Is this something niche? Not really. This week, Kamil has added this approach to Sylius too.

Use for Monorepo Split Anything Dynamic!

This fromJson() trick is not something exclusive to monorepo package splitting. It's just one of the possible use cases.

The primary use case is already in your mind.

• What is repeated static in your CI?
• What lists can be delegated to simple command that already knows the data?
• How could you use the dynamic matrix in your workflows today?

Happy coding!