Interview with Kyle Mestery, a Distinguished Engineer at IBM who has been involved with Open vSwitch since about 2012, about the Open vSwitch development process. Our conversation was based on Upstream Open Source Networking Development: The Good, The Bad, and the Ugly, a presentation at ONS 2016 given by Kyle along with Justin Pettit from VMware and Russell Bryant from Red Hat. Kyle also gave a version of the talk with Armando Migliaccio at OpenStack Austin. The latter talk was recorded on video.

The focus of the conversation is to present the Open vSwitch development process by comparing it against the process for OpenStack Neutron and OpenDaylight. All three project names begin with “Open,” but there are significant differences in how they develop code!

How do these projects communicate? All of them have mailing lists, although there are subtle differences in how they use them. Open vSwitch has two main lists, ovs-discuss and ovs-dev. OpenStack, despite being a much bigger project, has only a single development mailing list that it divides up using bracketed “topic tags” supported by the GNU Mailman mailing list manager. OpenDaylight, finally, has many mailing lists per subproject. Kyle explains the advantages and disadvantages of each approach.

All of these projects have IRC channels also. Open vSwitch has a single channel #openvswitch and the other projects have multiple, subproject-specific channels.

OpenDaylight stands out as the only project among the three that relies heavily on conference calls.

Are the projects friendly to newcomers? In general, Kyle thinks so. As with any project, regardless of open or closed source, there will be some existing developers who are super-helpful and others who are overworked or overstressed and less helpful initially. In the end, how you cycle through leader and contributors in a project is how the project grows.

The projects handle bugs differently as well. Open vSwitch primarily handles bugs on the mailing list. OpenStack files bugs in Launchpad using a carefully designed template. OpenDaylight has a Bugzilla instance and a wiki with instructions and advice. Kyle thinks that Open vSwitch may need to make heavier use of a bug tracker sometime in the future.

The projects have different approaches to code review. OpenDaylight and OpenStack use Gerrit, a web-based code review system, although many developers do not like and avoid the Gerrit web interface, instead using a command-line tool called Gertty. Open vSwitch primarily uses patches emailed to the ovs-dev mailing list, similar to the Linux kernel patch workflow. In-flight patches can be monitored via Patchwork, although this is only a tracking system and has no direct control over the Open vSwitch repository. Open vSwitch also accepts pull requests via Github.

Kyle mentions some ways that the Open vSwitch development process might benefit from approaches used in other projects, such as by assigning areas to particular reviewers and dividing the project into multiple, finer-grained repositories. OVN, for example, might be appropriate as a separate project in the future.

Kyle's advice: plan ahead, research the projects, give your developers time to become comfortable with the projects, treat everyone with respect, treat everyone equally, and give back to the core of the project. Keep in mind that little maintenance patch are as important as huge new features. Finally, trust your developers: you hired good people, so trust their ability to work upstream.

The interview also touches on:

• How Kyle became involved with Open vSwitch when he was a software engineer at Cisco and how his role at IBM has shifted a little more toward management.
• The “Wild West” explosion of open source software in networking in the last few years.
• The four stages of how companies get involved in open source networking projects: excitement, panic, enlightenment, success. The key to enlightenment, Kyle says, is that you get out what you put in, which includes a “karma cycle” of helping to get other developer's code in, e.g. through code review.
• The importance of giving developers credit for putting time into reviewing and testing code, and how different projects do it, plus the pitfalls in using karma-like systems that can be gamed to the point of becoming “poisonous.”
• “Onboarding” in projects and how a developer becomes a “core team” member or “committer.”

You can reach Kyle as @mestery on Twitter and follow his blog at siliconloons.com.

OVS Orbit is produced by Ben Pfaff. The intro music in this episode is Drive, featuring cdk and DarrylJ, copyright 2013 by Alex. The bumper music is Yeah Ant featuring Wired Ant and Javolenus, copyright 2013 by Speck. The outro music is Space Bazooka featuring Doxen Zsigmond, copyright 2013 by Kirkoid. All content is licensed under a Creative Commons Attribution 3.0 Unported (CC BY 3.0) license.