Fixing Lighthouse CI Score Fluctuations In GitHub Actions

Ensuring consistent performance and accessibility scores is crucial for maintaining a high-quality web application. Intermittent failures in Continuous Integration (CI) can undermine developer trust and delay releases. This article addresses how to stabilize Lighthouse CI performance and accessibility scores, aiming for a minimum of 0.85 for performance and 0.9 for accessibility in GitHub Actions.

Background: The Intermittent Failures

Recently, our GitHub Actions workflow for Frontend Quality Checks, specifically the Lighthouse CI, began failing due to performance scores dropping below the required threshold. We encountered an error message indicating that the performance score was 0.65, which is less than the expected minimum score of 0.85. This issue blocked the CI process, despite local runs consistently passing with high scores. Local runs using pnpm lhci against http://localhost:3000/ showed:

• Performance scores consistently above 0.9.
• Total network payload of approximately 95 KiB across 6 requests.
• No third-party resources and minimal blocking work.

Additionally, diagnostic audits revealed internal Lighthouse errors related to frame_sequence, which, while concerning, didn't directly impact the category scores. Given our project plan to maintain a Lighthouse performance score of ≥ 85 on desktop, this discrepancy between local and CI environments was a significant concern.

The inconsistency suggested potential environment or configuration drift, or simply flakiness within the GitHub CI environment, rather than a fundamental performance issue with the frontend. This inconsistency is the core problem we aim to address.

The Problem: Discrepancies Between Local and CI

Issue Identified

In our CI environment, Lighthouse CI occasionally reports performance scores below 0.85, sometimes as low as 0.65, leading to job failures. However, when running the same pnpm lhci command locally, using identical configurations and URLs, the performance scores consistently exceed the threshold. This inconsistency raises concerns about the reliability of our CI pipeline.

Impact of the Discrepancy

This issue undermines the core acceptance criterion of maintaining a Lighthouse performance score of ≥ 85 on desktop. More critically, it erodes developer trust in the CI system. False negatives and flaky gatekeeping can lead to developers questioning the validity of the CI results, potentially causing them to bypass or ignore critical feedback. Trust in the CI pipeline is crucial for maintaining code quality and ensuring timely releases.

The Goal: Consistent and Reliable Scoring

Our primary goal is to ensure that Lighthouse CI consistently reports the following scores in GitHub Actions for both the develop branch and pull requests:

• categories.performance >= 0.85
• categories.accessibility >= 0.9

Achieving these scores in the CI environment should align with the results obtained locally, providing a reliable and consistent measure of our application's performance and accessibility. The accessibility target of 0.9 is a stretch goal to ensure our UI remains robust and accessible as it evolves.

Scope: Stabilizing Lighthouse and Minor Enhancements

To achieve our goal, we have defined the following scope:

1. Stabilize Lighthouse in CI

This involves:

• Aligning Lighthouse configuration and runtime between local development environments and the GitHub Actions environment. This includes ensuring that both environments use the same settings, versions, and dependencies.
• Reducing flakiness by adjusting the number of runs and throttling settings. Flakiness can occur due to variations in the execution environment, such as CPU load or network conditions.
• Ensuring that we are testing the built Next.js app under realistic desktop conditions. This means running Lighthouse against the production build rather than a development server.

2. Minor Performance & Accessibility Polish

In addition to stabilizing Lighthouse, we will also address any remaining low-hanging accessibility or performance issues that could potentially drag the score below our target threshold as the dashboard gets more complex. This proactive approach will help us maintain a high level of quality as we continue to develop the application.

Current Observations: Environment-Specific Issues

From the latest Lighthouse HTML reports generated for http://localhost:3000/, we have made several key observations:

Payload Analysis

The total transfer size is approximately 95 KiB, which includes 6 requests (4 scripts, 1 stylesheet, and 1 document). This small payload size indicates that the application should perform well under normal conditions.

Configuration Review

The configuration is set to formFactor: "desktop" with throttling and screen emulation handled by Lighthouse. This ensures that we are measuring performance and accessibility under desktop conditions, as required by our project plan.

Diagnostic Insights

We have identified internal errors related to largest-contentful-paint-element and render-blocking-resources in Lighthouse 12.1.0. These errors, which are likely due to known upstream issues, do not appear to be affecting the category scores directly. However, they warrant monitoring to ensure they do not become a blocker in the future.

These observations suggest that the issues we are encountering in CI are more likely related to environment differences (such as CPU contention, variations in Node or Chrome versions, or the build mode) rather than inherent performance problems within the application. The 0.65 score observed in CI strongly indicates that these environmental factors are at play.

Acceptance Criteria: Defining Success

To ensure we meet our goals, we have established the following acceptance criteria:

• [ ] The GitHub Actions workflow running pnpm lhci passes consistently on the develop branch and for all pull requests. This is the most critical criterion, as it ensures that our CI pipeline is reliable.
• [ ] In the CI environment, the Lighthouse report for http://localhost:3000/ shows categories.performance >= 0.85 and categories.accessibility >= 0.90. This confirms that we are meeting our target performance and accessibility scores in the CI environment.
• [ ] The threshold is enforced via lighthouse.config.cjs assertions, rather than relying solely on visual inspection of the HTML report. This ensures that the scores are programmatically validated.
• [ ] CI and local runs use the same Lighthouse configuration, including the same @lhci/cli version, formFactor, and key settings. Consistency in configuration is crucial for ensuring that the results are comparable.
• [ ] A short note is added to the documentation (README, ProjectOverview, or ProjectPlan) explaining how to run Lighthouse locally and what thresholds we enforce. This helps ensure that developers are aware of the performance and accessibility standards and how to test against them.

Suggested Implementation: A Step-by-Step Approach

To achieve our goals, we propose the following implementation steps:

1. Align Lighthouse Configuration Between Local and CI

• Verify lighthouse.config.cjs:

  • Ensure it uses formFactor: "desktop" and screenEmulation.disabled: true (or equivalent) to measure desktop performance as required.

  • Set collect.numberOfRuns to at least 3 for the main URL and rely on median scores to reduce flakiness.

  • Include explicit assertions for performance and accessibility:

    // Example
    assertions: {
      'categories:performance': ['error', { minScore: 0.85 }],
      'categories:accessibility': ['warn', { minScore: 0.9 }],
    }
    

• In the GitHub Action:

  • Ensure the same @lhci/cli version as local (0.14.0) and Node version are being used. Consistent versions across environments help prevent unexpected behavior.
  • Verify the job runs after a production build of the web app (e.g., pnpm build + pnpm start) rather than a development server. Production builds often have optimizations that can significantly impact performance scores.

2. Reduce CI Flakiness

• Dedicated Job: Run Lighthouse in a dedicated job with minimal concurrent workload. This avoids running heavy tests simultaneously on the same runner, which can lead to resource contention and inconsistent results.

• LHCI Configuration:

  • Use simulated throttling (throttlingMethod: 'simulate') consistently between local and CI. Simulated throttling provides a more consistent environment for testing.
  • Optionally, loosen timeouts (maxWaitForFcp, maxWaitForLoad) slightly to account for slower GitHub runners. This can help prevent false failures due to timeouts.

• Retry Mechanism: If we continue to see sporadic drops, allow one retry for the LHCI step and fail only if both attempts are below the threshold. This adds a layer of robustness to our CI pipeline.

3. Small Performance & A11y Hardening

Even though the current page is lightweight, we should proactively address potential issues before the dashboard becomes more complex:

• Performance:

  • Keep bundles small by ensuring any new charts or libraries (e.g., Plotly/Recharts) are lazily loaded on routes that need them. Lazy loading improves initial load times.
  • Use Next.js dynamic imports with ssr: false for heavy client-only components (e.g., complex charts). This defers the loading of these components until they are needed.
  • Ensure images (once added) use next/image with appropriate sizes and lazy loading. next/image provides built-in optimizations for image delivery.

• Accessibility:

  • Ensure all interactive components from shadcn/ui have:

    • Accessible labels (aria-label, aria-labelledby).
    • Proper focus styles and keyboard navigation.

  • Add alt text to all img elements (or rely on next/image with proper alt props). alt text is crucial for screen readers.

  • Maintain semantic structure using elements like <main>, <header>, <nav>, and <footer>, and ensure correct heading hierarchy. Semantic HTML improves accessibility and SEO.

4. Documentation & DX

• Frontend Dev Docs: Add a short “Lighthouse checks” section to the frontend dev docs (ProjectOverview/README) covering:

  • How to run Lighthouse locally: cd web && pnpm lhci.
  • What thresholds we enforce (performance ≥0.85, accessibility ≥0.90, desktop).
  • Where to find CI artifacts (.lighthouseci reports). Clear documentation ensures that developers understand the process and expectations.

Notes / Gotchas: Important Considerations

• Lighthouse Errors: The internal Lighthouse errors in largest-contentful-paint-element and render-blocking-resources are likely upstream issues in Lighthouse 12.1.0. These appear in local reports as well and should be monitored but not treated as a blocker unless they begin affecting category scores. Staying informed about upstream issues helps us avoid chasing false positives.
• Revisit Thresholds: Once the dashboard, Strategy Studio, and RAG views are integrated and the application becomes more complex, we should revisit this gate to confirm that performance remains comfortably above the threshold. Regular reviews ensure that our thresholds remain relevant as the application evolves.

Conclusion

Stabilizing Lighthouse CI performance and accessibility scores in GitHub Actions is essential for maintaining a high-quality web application. By aligning configurations, reducing flakiness, and proactively addressing potential issues, we can ensure a reliable CI pipeline that fosters developer trust and helps us deliver a consistently performant and accessible user experience.

For more information on Lighthouse and web performance optimization, visit Google's Web.dev. This resource provides in-depth guides and best practices for improving web performance and accessibility.