First, a little bit of background

Everything we do at Mozilla revolves around Bugzilla

First, a little bit of background

Talk Outline

  1. Fixing bugs

    {automated tests, code reviews, super reviews, backouts}

  2. Verifying fixes

    {qawanted, dogfooding, built-in profiler, filing bugs}

  3. Regression finding

    {mozregression, regression, regressionwindow-needed}

  4. Monitoring stability

    {release manager, trickle releases, chemspills, crash reporting}

Fixing bugs: Automated Tests

Code changes should be paired with unit tests when possible

Fixing bugs: Automated Tests (cont'd)

  1. compiled-code (B)
  2. xpcshell (X)
  3. JS shell tests (J)
  4. crashtest (C)
  5. reftest (R)
  6. Mochitest (M), Mochitest-other (Moth)
  7. Talos (T)

Fixing bugs: Code Reviews

Fixing bugs: Code Reviews (cont'd)

Fixing bugs: Backouts

Fixing bugs: Backouts

Continuous integration (build with each push, and run all the tests)

Fixing bugs: Test failures

Talk Outline

  1. Fixing bugs

    {automated tests, code reviews, super reviews, backouts}

  2. Verifying fixes

    {qawanted, dogfooding, built-in profiler, filing bugs}

  3. Regression finding

    {mozregression, regression, regressionwindow-needed}

  4. Monitoring stability

    {release manager, trickle releases, chemspills, crash reporting}

Verifying bugs: qawanted

Verifying bugs: People using early test builds

Verifying bugs: Telemetry

Opt-in service that monitors usage statistics and reports back to Mozilla

Keeps track of

Verifying bugs: Telemetry (first-paint evolution)

Verifying bugs: Telemetry (tab anim close histogram)

Verifying bugs: Gecko Profiler

Built-in profiler that is compiled in Nightly and Aurora builds

Can be used to find issues of jank while using Firefox in daily practice or in stress testing

Verifying bugs: Dogfooding and filing bugs

Verifying bugs: Dogfooding and filing bugs

Bugzilla Helper form

Needs to be super-easy to file bugs against products

Better to file a dupe than to not file

Talk Outline

  1. Fixing bugs

    {automated tests, code reviews, super reviews, backouts}

  2. Verifying fixes

    {qawanted, dogfooding, built-in profiler, filing bugs}

  3. Regression finding

    {mozregression, regression, regressionwindow-needed}

  4. Monitoring stability

    {release manager, trickle releases, chemspills, crash reporting}

Regression finding: regressionwindow-needed

a keyword that people can add to a bug to request that someone narrow down the window of changes that caused the regression

if a narrow window can be determined, fixing the bug can become trivial

Regression finding: ./mozregression

Uses binary search on Nightly builds to narrow down cause/fix of bugs

Can track down the cause of a three year old bug in Firefox with 9-10 Nightly builds

Regression finding: regression

bugs that are regressions get the regression keyword

they are often great candidates for some automated testing to make sure that the particular regression doesn't occur again

Talk Outline

  1. Fixing bugs

    {automated tests, code reviews, super reviews, backouts}

  2. Verifying fixes

    {qawanted, dogfooding, built-in profiler, filing bugs}

  3. Regression finding

    {mozregression, regression, regressionwindow-needed}

  4. Monitoring stability

    {release manager, trickle releases, chemspills, crash reporting}

Monitoring stability: Release Management

release driver is someone responsible for the stability of the product

provide guidance to developers as to which bug fixes are important for a given release and also make a range of tree management decisions

tracked bugs are associated with specific releases and monitored closely to make sure that progress is ongoing and that a resolution won't slip past the release

nominated patches are fixes for bugs that need to get uplifted and reach users quicker (higher risk, need sanity check)

Monitoring stability: Trickle release

Auto-update about 5-10% of users and watch for large spikes in crash reports, stability, and incompatibilities

When release seems strong enough, open the flood gates

In the long run, it would be better to have the Aurora and Beta channels with larger audiences so this is not needed as much

Monitoring stability: Crash reporters

Monitoring stability: Crash reporters

And so the cycle continues…