How I use Agents to Extract Hidden Feedback from Github and Improved Rector

I was randomly drifting through Github trending PHP repositories, and checking how popular projects use rector.php. A lot of them use it extensively, with prepared sets and even levels.

What I didn't like to see is the skip section. It's like a baseline in PHPStan, but I take it really seriously. In 90 % of cases, it means something is broken in Rector and the user had to skip it to avoid unwanted changes over and over.

Feeling the frustration, I thought: Is there something we can do about it?

"Skip" over Bug Report

So how did it start? 3 weeks ago, I opened the rector.php of the mautic/mautic project (versioned here) and quickly scanned the skip section. I noticed the rules are applied = wanted, but skipped for a file here and there.

I asked Honza, Mautic project merge lead and a friend from Prague PHP meetups I organized in pre-covid times, if he would be interested in a couple of tiny fixes here and there. He said "yes", so I gave it a go.

Before making any changes in the Mautic project itself, I thought, let's figure out if AI agents can figure out why the files are skipped. If it looks like a bug, propose a PR, make it pass and merge it.


My first prompt sounded something like this:

Hey, can you check this https://github.com/mautic/mautic/blob/135aabc643f0ac65e7fa105ea70665c063608cf4/rector.php 
ignore section of Rector, and check particular rule + files combinations?
Figure out, if that's a bug or valid ignore, and if a bug, propose a PR in 
a particular repository and try to fix it.

Once done, open it in my Chrome browser.

Agent work has started...

What is amazing about open-source is that every file is available to anyone. I don't have to copy file contents and write them into the terminal (using Claude Code currently) so the agent has the very important context. I can just tell him (yes, agent Smith) to fix the file and thanks to the FQN path, he can access it. E.g. this part from withSkip().

// handle later, case by case as lot of chnaged code
RemoveAlwaysTrueIfConditionRector::class => [
    // watch out on this one - the variables are set magically via $$name
    // @see app/bundles/FormBundle/Form/Type/FieldType.php:99
    __DIR__.'/app/bundles/FormBundle/Form/Type/FieldType.php',
],

There is:

That's all the agent (or me) needs to figure out a proper fix.


To be honest, the first prompt took a while to figure out. The agent would have to learn about Rector, about rector.php config, most likely explore the documentation to figure out what all the available options there are, then figure out that I talk about the ->withSkip() and nothing else.


I was getting impatient, so I wrote a bit more narrow prompt:

The rule RemoveAlwaysTrueIfConditionRector breaks https://github.com/mautic/mautic/blob/7.x/app/bundles/FormBundle/Form/Type/FieldType.php 
file

Figure out why, make test fixture, fix via PR, open new Chrome tab once CI passes

The agent understood this prompt much more. It took ~20 % of the original time to get the success, and I could also just copy-paste a template:

The rule <rule> breaks <file> file
Figure out why, make test fixture, fix via PR, open new Chrome tab once CI passes

Tip: if a too wide prompt takes too long, try to narrow it down to its essence. What, where, how, when. I like to first shot and merge, because it works best for me. Rector's CI setup is very strong, fast and is only getting better. That's why it fits nicely.

Tip 2: I use the caveman extension not only to save tokens (never had a problem), but to save token input to my own eyes.

If you can, use Caveman to extend your attention span by saving tokens throughput to your brain.


Discovering new Feature missing in Rector

It took roughly <2 mins to fix such a bug in Rector, because all the context was there. The longest delay was me copy-pasting the rule and file to the prompt (at least I had some rest :) . Making bugfixes that would help everyone using this rule is already great motivation to fix them as soon as possible. I never thought we could read some open-source repository, find rector.php and extract a good-enough bug report from 2 lines.

Suddenly, what took a human reporter at least 20-30 minutes to figure out (sometimes more) - the rule, the minimal reproducer, and create a Rector demo link where we could actually see the error. Then it took us a couple of days (or weeks) to process, 10-20 minutes to look at and make a fix, took 2 minutes. Full circle.

This opened a door: what else can we extract from this config?

Here is an interesting section:

// Avoiding breaking BC breaks with forced return types in public methods
ReturnTypeFromReturnNewRector::class => [
    __DIR__.'/app/bundles/IntegrationsBundle/Sync/SyncProcess/Direction/Integration/ObjectChangeGenerator.php',
    __DIR__.'/app/bundles/IntegrationsBundle/Sync/SyncProcess/Direction/Internal/ObjectChangeGenerator.php',
],

Nobody wants to add a BC break, but I want people to use Rector as much as possible to automate 100 % of the safe work without much worrying or nitpicking work. Run a new rule, ignore on this file and 3 more. Run another rule, ignore on these 7 rules... blah.

Safety and fun coding beats stress and anxiety coding.


So how is a type BC break born when it comes to open-source? Rector would be able to figure out a strict type return in here:

 abstract class AbstractModel
 {
-    /**
-     * @return string
-     */
-    public function getName()
+    public function getName(): string
     {
          return 'common';
     }
 }

But what if this class is meant to be extended by a 3rd party? We have to skip it.


Moreover, Rector can figure out a param type declaration based on a private method with an existing type declaration:

 class SomeModel abstract extneds AbstractModel
 {
-    /**
-     * @param int $amount
-     */
-    public function resolve($amount)
+    public function resolve(int $amount)
     {
         return $this->adjust($amount);
     }
     
     // strict types here
     private function adjust(int $amount): int
     {
          return $amount * 100;
     }
 }

But again, this would require updating AbstractModel, which we can't afford. If we do, we'll create a BC break for everyone who overrides the adjust() method (that is meant to override).

So what now? We can create a huge skip in rector.php, that includes all the parent and children class list:

   ->withSkip([
        SomeTypeRule::class => [
            __DIR__ . '/src/Model/AbstractModel.php',
            __DIR__ . '/src/Model/FirstModel.php',
            __DIR__ . '/src/Model/SecondModel.php',
            __DIR__ . '/src/Model/ThirdModel.php',
        ],
        // and 20 more...
    )

And that's just a single type rule. Rector has over 80 of those. Isn't there something we can do better?

Introducing Type Guarded Classes

I was wondering, what if we could automate this "skip this and that" game. So I asked my agent:

There is a group of rules, that add param type declarations and return type 
declarations. You can find them in <file with all the rules>.

We have a problem that these rules add param + return type declarations to 
classes meant to be extended. This creates BC break as people do not have 
types filled yet.

I want you to add a configuration option to list guard these classes. 
If a type is about to be added, check if this is current or child class of 
any guard class. If so, skip it.

I didn't know if that was clear enough, possible or hypercomplex. But the agent handled it very well and added the feature - you can see it here.


Where before there were 20 random skip lines (with the risk of a new child class being missed), now there are a couple of lines:

    ->withTypeGuardedClasses([
        Mautic\CoreBundle\Controller\AbstractStandardFormController::class,
        Mautic\CoreBundle\Controller\CommonController::class,
        Mautic\CoreBundle\Controller\AbstractFormController::class,
    ])

Wow, I was amazed. I know about so much frustration in open-source projects (from Symfony, through Doctrine, Laravel, Mautic...) where people ran Rector, trusted it, only to find a bug after a commit. Then never used the type declarations again. Even though there are 80 % of safe rules that can be applied anywhere.

This solves it all. If a class is in this list, it's safe from all BC breaking type changes.


If it breaks something, we've missed a rule to be protected. Let us know


Great! One repository and so much hidden feedback we can extract from it and make the tool more useful and safer to use for everyone.

More Repositories

What else can we do? If there is one project with rector.php on Github, there must be another one, or 10... or 100.

So I asked my agent:

Give me top 100 repositories from Github, that use Rector and have `rector.php` 
config in the repository. Use Github search.

From these repositories, extract all the skips: full rule, rule + path. 
Skips sole paths for now.

Then show me a list of 20 the most skipped rules + paths.
Include also link to these files, so I can open it right away in my browser.

A little bit of crunching and then... he spat out 30 more cases I could explore. Then I re-used the previous prompt with rule + path and the agent handled those in a couple of sessions.

More Sources

Not all repositories that use Rector are popular on Github, but maybe somewhere else. I want to explore more sources, like Packagist. There, every package has a "dependents" link. When you click on it, we see packages that require the package.


So I told my agent:

add resources from https://packagist.org/packages/rector/rector/dependents?order_by=downloads 
as well, find `rector.php` for top 300 projects and do the same data extaction 
as I mentioned above

A couple of mins later the agent gave more data from a new source.

Visual Presentation

The data were present in the CLI, but it's quite difficult to read through a couple of pages in the terminal, mixed with agent output and my instructions. Let's visualize it:

take all these data and create a simple HTML table that will show:
* most ignored rules by class (no file)
* most ignored rules + file combination - again, include link to the file

...and voilà!

Now I can quickly scan through the HTML file and open interesting cases directly in the browser. The links are clickable and lead to Github, where I can explore more. Also, I can close my agent session and still have results.

But, we're devs, right? We can do one more step to get the most value out of this.

Automate Sessions You'll get Back to Later

After all this work, I want to encode the knowledge in a script. Basically the same thing we do when we help projects to automate code-reviews and avoid stuck PR pipelines.


3 things:

init git repo
put all this knowhow into couple PHP scripts - one for Github, one for Packagist
include HTML page generation in the end of run 

Thank you, great job!

Next time I want to fix a couple of Rector bugs, I can just run the script from this repo (without an agent) and instantly see a beautiful HTML page I can scan through.

I just wanted to share my experience with you, because I was surprised where a single rector.php file can lead me. I had no idea it would turn into an automated tool for open-source bug reporting extraction in one day. What a time to be an open source developer!

(Written by hands and sweat, with my brain GPT.)


Happy coding!