It has been a while since I last blogged and I had something interesting to share so I finally managed to overcome my laziness. In this post, I would like to talk about some of the projects that I have been involved with in 2015 in the A-Team at Mozilla. Before I talk about the projects, I would like to give a shout out to @jmaher, @armenzg, @kmoir, @dminor, @ahal, @gbrown, @wlach who were always there to answer my questions, help me find a way to solve problems when I was stuck and review my patches. I have worked on some exciting problems and made some great progress on the projects in the past 4 months. They are:
1) SETA – Search for Extraneous Test Automation
We run hundreds of builds and thousands of test jobs each day on our integration branches, that is , mozilla-inbound and fx-team. And as more and more platforms are added every month, the load on test machines is ever increasing. But are so many test jobs for each push required? We run the test jobs to catch failures but majority of time, the test jobs pass and the ones who indeed catch failures often have duplicate ones. SETA tries to tackle this problem by being smart about utilizing machine cycles. In SETA, we find the minimum number of jobs that are needed to find all the failures that have occurred in the last six months on integration machines. From this data, we predict that these jobs will be more likely to catch failures than others and therefore other test jobs are set to run less frequently. It is true that we will be wrong certain number of times and when we miss a failure, the sheriffs would need to backfill some jobs to find the root cause. But most of the time, it will work. Joel has done an excellent blog post giving examples and statistics that has been done in this project. This project has been deployed in Mozilla Releng production systems and we have reduced the number of jobs to roughly 150-190 jobs/push from 350-400 jobs/push per day on desktop (linux, osx, win) platforms, a 50% reduction during high load weekdays. To put this into perspective, the past week we have seen the lowest jobs per push since January 1st on both mozilla-inbound and fx-team. I see this as a huge win as it drastically reduces the load on our machines as well as reduces the time the sheriffs need to star intermittents, increasing productivity for all. And this data is for desktop platforms only, android and other platforms are yet to come, after which we should be seeing more gains.
jobs/push per week since January for mozilla-inbound. Lowest on April 20th – April 26th
2) Mozilla CI Tools
MozCI (Mozilla CI Tools) is a python library which allows one to trigger builds and test jobs on treeherder.mozilla.org via command line. It has various use cases like triggering jobs for bisecting intermittent failures, backfilling missing jobs, and personally I use it for bisecting performance regressions. This tool is also used by sheriffs and is aimed for increasing developer productivity.
Project Repo: https://github.com/armenzg/mozilla_ci_tools
Project Documentation: https://mozilla-ci-tools.readthedocs.org/en/latest/index.html
3) Firefox Performance Sheriffing
In late January 2015, I took up the role of a performance sheriff. In this role, I look at the performance data produced by the test jobs and find regressions, root causes and get bugs on file to track issues and bring it to the attention of patch authors.
Sheriff documentation: https://wiki.mozilla.org/Buildbot/Talos/Sheriffing
I have also contributed patches to some other projects like removing android*.json from mochitests (bug 1083347), A-Team bootcamp and Mozregression. If you are looking to contribute to open source projects, I think this is a great time to start contributing to Automation and Tools team at Mozilla and make a big impact. For me, it has been one of the most productive quarters and I plan to keep contributing further. As some of you may know, this summer I will be joining as the A-Team intern at Mozilla in San Francisco with @chmanchester as my mentor, and I am looking forward to do more exciting work here!