[paulhorowitz]

Description of the proposed changes

Shopify CLI 5.0.0 (released 27 Oct 2026) changed the JSON output of the theme
commands. The workflow installs @shopify/cli@latest and its jq still expected
the 4.x format, so Theme Check has failed on every theme PR since then with
Cannot index boolean with string "path". The preview deploy depends on that
job, so no preview themes have been created either.

As discussed, this reads the 5.x format only and keeps @latest.

• theme check: annotations come from the flat issues array. Positions are
  now 1-indexed, so the + 1 conversion is removed. Counts come from the
  top-level errorCount / warningCount
• theme list: themes are now under .themes. Without this the preview job
  would fail the same way once Theme Check passed
• theme push: the preview URL is now .theme.previewUrl. Without this the PR
  comment would be posted with an empty link
• fail the preview job with an error if push returns no preview URL. Push can
  exit 0 without one (status cancelled, or a null previewUrl)

shopify-theme-preview-cleanup.yml and shopify-deploy.yml don't need
changes for this. Cleanup only runs theme delete and doesn't parse its
output. Deploy uses the CLI version in each app's yarn.lock, not @latest.

Testing

• Ran the Theme Check step as written against real CLI 5.0.0 on a test theme
  with 2 errors and 1 warning: fails, with 3 annotations at the correct
  file/line/column, all matching the problem matcher pattern
• Same step on a clean theme: passes with 0 errors / 0 warnings
• Confirmed the old jq reproduces the CI error on real 5.0.0 output
• Not yet tested: theme list / theme push in the preview job. These need a
  store and token, so they'll be verified by pointing
  aligent/fmc-shopify-theme#37 at this branch before merging

Useful Docs
https://shopify.dev/docs/api/shopify-cli/migrate/5-0
https://github.com/Shopify/cli/releases/tag/5.0.0

Time Tracking

FSM-131: Shopify theme PR check fails on every PR since Shopify CLI 5.0.0 | Code Review

🤖 Generated with Claude Code

[tmthrgd-aligent]

@paulhorowitz Just wondering re:

Deploy uses the CLI version in each app's yarn.lock, not @latest.
is there any risk of future failures once app's update their yarn.lock to v5? From my look at the deploy workflow, I don't think there's any issue, just wanted to confirm you're thoughts.

[paulhorowitz]

@tmthrgd-aligent I ran this past Claude since it's outside my area. Its take: you're right that the JSON change doesn't affect deploy, because deploy doesn't read the CLI's output. But it flagged one possible problem.

The deploy step always sets both SHOPIFY_APP_AUTOMATION_TOKEN and SHOPIFY_CLI_PARTNERS_TOKEN, so whichever secret the app didn't pass ends up as an empty string. According to Claude, CLI 5.0.0 picks the token variable by whether it exists, checking service account → app automation → partners in that order. If the one it picks is empty, it's a hard error ("is set but empty"). It pointed to packages/cli-kit/src/private/node/session/automation-token.ts at tag 5.0.0 and the changelog entry for #8731.

If that's right, any app that passes only shopify_cli_token would fail to deploy once its lockfile moves to v5, while apps passing shopify_app_automation_token would be fine. Its suggested fix was to only export the token variable that actually has a value. One thing it couldn't check: whether GitHub Actions really sets a missing secret as an empty variable rather than leaving it out. That's the part the whole theory depends on.

It also mentioned that both variables are deprecated in v5 in favour of SHOPIFY_SERVICE_ACCOUNT_TOKEN, and that variable errors if either of the others is set alongside it, even empty.

I can't vouch for any of this myself, so feel free to sanity-check it!

[tmthrgd-aligent]

@paulhorowitz Interesting. That's it's own issue I suppose so I'm happy with this workflow, might just ask for a second review and then we can merge this. If the deploy is broken by that variable, we can update it separately.