View Details

DUM-E (“dummy”) and U (“you”) are the names of the robot arms in the Iron Man movies. After watching this movie for the n-teenth time, I have a strong urge to also have robotic arms in a workshop like Tony Stark. You can see the value of the robots clearly throughout the movie. The robots allow Tony to produce suits more quickly, help test the suits, and provide periodic comedic relief. At one point, DUM-E even saves Tony’s life. As a bit of a thought experiment, I considered what it would take to get the same functionality in reality. What this ends up leading to is a configuration management system for manufacturing, much like a build system. This post is going to outline that a bit!

The most popular open-source framework for building robots is ROS (Robotics Operating System). You can add different components like cameras or sensors and program all the functionality you need for your specific use case. The underlying infrastructure works by passing messages, through a pub/subsystem. Elementary Robotics created their own OS called atom. It’s pretty cool, it uses Redis for the messaging layer and docker for packaging and defining the individual components. Need a camera on your robot? Include the camera container in your atom OS config file. You can then pipe the messages from the camera into machine learning in another container. It’s important to know the basics of these frameworks to continue into how we would build DUM-E and U.

Let’s dive in. The end goal here is to be as productive as Tony Stark at building things.

Fire extinguisher robot One of my favorite scenes with DUM-E is when Tony is testing the suits and it’s DUM-E’s job to blast him with a fire extinguisher when he is on fire. For comic relief in the movie, DUM-E messes this up a bunch and blasts Tony when he’s not on fire.

Let’s break this down, starting with a robot that will shoot a fire extinguisher on any fire. First, what you would need is the robotic arm base, maybe you build your own, maybe it’s ABB, Kuka, FANUC, or any other robot arm maker. Let’s assume you have some sort of robotic arm with an SDK/API you can program. You also need a fire extinguisher. Since we are hackers we will just duct tape this to the robot arm and have a trigger on the switch to fire it programmatically. Next, we need a camera. Let’s also duct tape this and all the wires to the robot. We need to know if something in our proximity is on fire and where it is. We will need some code to determine if something is on fire. You could likely train a machine learning model to do this. So when the ML model identifies something as on fire, we need to calculate where it is in relation to the distance from the camera identifying it to the fire extinguisher we duct-taped to the robot. This is all doable and pretty much depends on how well we trained our model.

In the movie, DUM-E is quite bad at identifying fire. It is just a movie but we should consider it might be hard for the model to differentiate fire from the color of the suit when it’s not on fire. If you recall, Iron Man’s suit is crimson and gold which could be misidentified as fire if it’s moving in the same pattern a fire might move. Tony does fly and move around at very fast speeds. This really comes down to how well Tony trains the model. As long as DUM-E continually learns, which he should, by the time Iron Man has been blasted by mistake a few times, the model should know the difference between the two (on fire and suit that looks like fire moving in a weird way). We also get to witness this learning in the movie.

Lifesaver DUM-E, despite his namesake, is very intelligent. A major scene in the movie is when he saves Tony’s life by passing him the reactor to power the magnet in his chest. The reactor is just out of Tony’s reach as he is dying and DUM-E realizes this and passes it to him. This could be programmed in a few different ways.

One way would be the equivalent of hard coding this behavior. Maybe Tony trained DUM-E to pass him the reactor. That’s a bit lame and wouldn’t be very useful outside this context. Let’s assume DUM-E was programmed a different way.

What would be more useful overall is if DUM-E had some programming that when Tony is reaching for an object just outside his reach, DUM-E should know to pass it to him. Again this relies on a camera and a very precise machine learning model. Instead of the fire extinguisher though, we would need a claw to pick up the object and pass it. The machine learning model for this behavior would have to:

• Identify a human
• Identify the difference between a human in a resting state and a human reaching for something, arms extended
• Identify the object the human is reaching for, scan for objects a certain distance from the end of the hand
• We’d also want some code to know when the object is out of reach and the robot should help or if the human is fine on their own, don’t want the robot arm getting in my way when I actually want to grab something. So maybe watch the rate of the reach and calculate if it’s possible for the arm to extend to any objects around where it is headed.

This should all be possible. For bonus points let’s make it even more useful. Tony uses his robots to help him build things in his workshop and at times he asks them to pass him tools. Let’s add a microphone component to the robot and a model to identify when I am asking for an object. Now the robot needs to correctly identify objects based on a name, and let’s hope it parses what I said correctly in the first place. We could also help the robot identify objects, by using the camera to identify if I pointed to a specific object when I asked for it. This would be super helpful and like having another set of arms around.

Assembling the suits Both DUM-E and U help Stark assemble the Iron Man suits. To do this, the robots need to know the final configuration of the suits when put together. They also need to know where on Tony’s body they need to attach the suit. So we need:

• A camera to identify Tony and where to put the parts of the suit, we need to identify the parts of the suit as well
• We need a claw with the tools to do the final welding and putting together of the suit

Hardware mode After building a few suits, Tony’s workshop is viewed in less of an “I am actively building things” configuration. You can see the floor is more clear and there are fewer tools and materials strewn about. It’s basically like someone cleaned up and things have been at rest for a while. When Tony needs to build the reactor to create the element to power the suit, he tells the robots “We are going back into hardware mode.” This had me thinking, wouldn’t it be cool if there were different configurations of factory floor layouts that could be named and switched to on a whim? How would we do this?

Up til now, we’ve programmed all the robots to do what we wanted with code. Assuming we used ROS or Atom, we would have some configuration files and code laying around in a repo somewhere. Let’s assume a repo per robot or a repo per behavior of the robot, either way, we have a single place where code is defined that determines the behavior of the robots. What we need on top of this is a few things:

• A programmatic map of the factory floor with coordinates so we can assign robots to specific coordinates, we could use geocoding or something else for this, imagine a whatever powers your Roomba
• Each robot needs a camera and a tracking mechanism for knowing where to go to get to its defined coordinates for this workflow (if it is stationary), if not the robot might already have a task of moving something from one place to another, more on that later
• A configuration to specify:
  • What robot or machine is being used for this step
  • Where a robot or machine needs to go
  • The code that needs to be loaded into the robot for its tasks now, or if the code is already loaded into the robot, the robot needs to know what code to execute for this configuration
  • Ex. is this code for analyzing an object to make sure it is up to quality? Is this code to pass objects? Should it now go into laser cutting mode?
  • Any artifacts the robots need to work with
  • For example, say in the configuration file our first step was a desktop metal printer printing a certain stl file. Much like a CI pipeline, our artifact would be the finished 3D printed object itself.
  • The next robot in the pipeline might be a robotic arm, assigned to coordinates between the printer and the assembly line. This next configuration in the file would know it needs to take that artifact (since we would define it) and pass it to the assembly line
  • Then the rest of the configuration file could define the assembly line

So now we can have configuration files for several different assembly processes. If we want to start building something different we just load the new file and the robots would update their code. So when Tony says, “we are going back to hardware mode” we can think of this as him telling the system to load the new file.

Sample file

``` name: “hardware-mode” steps: machine: desktop-metal-1 runs: | my-super-cool-stl-file.stl artifact: part-hook machine: dum-e location: near-desktop-metal-1 # or maybe actual code coordinates, would be nice if there were shortcuts that translated to those runs: | part-hook | assembly-line # code the dum-e robot needs to execute, or maybe point to the repo where the code is stored

```

The cool thing about this setup is now our entire factory is configured in code. We can roll back by reverting a commit or we can add more functionality by modifying the file. We also gain tracking the entire history of the factory setup for free. Possibly CAD programs could help generate these files. It is the equivalent of a build pipeline but for manufacturing, I guess it could be considered a physical build pipeline.

Overall, this was a fun thought experiment. I can only hope to get a few robots and try and hack a real pipeline together one day. I do think becoming as productive as Tony Stark could be possible, just need the funds and time to hook it all together. And, of course, something to build!

View Details

Tesla had its first Battery Day on September 22nd, 20201. What a fantastic world we live in that we can witness the first Apple-like keynote for batteries. Batteries are a part of our everyday life; without them, the world would be a much different place. Your cellphone, flashlight, tablet, laptops, drones, cars, and other devices would not be portable and operational without batteries.

At the heart of it, batteries store chemical energy and convert it into electrical energy. The chemical reaction in a battery involves the flow of electrons from one electrode to another. When a battery is discharging, electrons flow from the electrode known as the anode, or negative electrode, to the electrode known as the cathode, or positive electrode. This flow of electrons provides an electric current that can be used to power devices. Electrons have a negative charge; therefore, as the flow of negative electrons moves from one electrode to another, an electrolyte is used to balance the charge by being the route for charge-balancing positive ions to flow.

Let’s break this process down a bit and uncover the chemical reactions at play within batteries. To have an electrical current, we need a flow of electrons. Where do those electrons come from?

Electrons in the anode are produced by a chemical reaction between the anode, or negative electrode, and the electrolyte. Simultaneously, another chemical reaction occurs in the cathode, or positive electrode, enabling it to accept electrons. Through these chemical reactions, a flow of electrons is created, resulting in an electrical current.

A chemical reaction that involves the exchange of electrons is known as a reduction-oxidation reaction, or redox reaction. Reduction refers to a gain of electrons. Thus, half of this reaction, defined as reduction, occurs at the cathode because it gains electrons. Oxidation refers to a loss of electrons. Therefore, half of this reaction, defined as oxidation, occurs at the anode because it loses electrons to the cathode. Each of these reactions, reduction and oxidation, has a particular standard potential. An electrochemical cell can be made up of any two conducting materials that have reactions with different standard potentials since the more robust material, which makes up the cathode, will gain electrons from the weaker material, which makes up the anode.

Batteries can be made up of one or more electrochemical cells, each cell consisting of one anode, one cathode, and an electrolyte, as described above. The electrodes and electrolyte are generally made up of different types of metals or other chemical compounds. Different materials for the electrodes and electrolyte produce different chemical reactions that affect how the battery works, how much energy it can store, and its voltage.

Volts The word “volt” refers to the measure of electric potential. The term came from the Italian scientist Alessandro Volta, who is credited for inventing the first battery. In 1780, Luigi Galvani, another Italian scientist, observed that the legs of frogs hanging on iron or brass hooks would twitch when touched with a probe of some other type of metal. Galvani believed that this was caused by electricity from within the frogs’ tissues. He called it ‘animal electricity.’

Volta believed the electric current came from the two different metal types: the hooks on which the frogs were hanging and the probe’s different metal. He thought the current was merely being transmitted through, not from, the frogs’ tissues. Volta experimented with stacks of silver and zinc layers interspersed with layers of cloth or paper soaked in saltwater and found an electric current flowed through a wire applied to both ends of the pile. Volta also found that the amount of voltage could be increased by using different metals in the pile. Leading to what we know today as the scientific unit of a “volt2.”

There are two ways to increase a battery’s voltage: stack several cells together or increase the cell’s electrochemical potential by choosing different materials.

When cells are combined in a series, it has an additive effect on the battery’s voltage. Essentially, the force at which the electrons move through the battery can be seen as the total force as it moves from the first cell’s anode through the number of cells the battery contains to the last cell’s cathode.

In contrast, when cells are combined in parallel, it increases the battery’s possible current, which is defined as the total number of electrons flowing through the cells, but not its voltage.

Measuring electricity When you buy a light bulb, the box indicates the wattage for the bulb. Watts are a measurement of power. Watts describe the rate of electricity that is being used at a specific moment. Therefore, a 60-watt light bulb uses 60 watts of electricity at any moment while turned on.

Watt-hours (Wh), on the other hand, are a measurement of energy. Watt-hours describe the total amount of electricity used over time. You can derive from the name that watt-hours are a combination of watts, the rate electricity is used, and hours, the length of time used. Going back to our example, a 60-watt light bulb that draws 60 watts of electricity at any moment while turned on uses 60 watt-hours of electricity over one hour.

Watt-hours will only get you so far, however. If you want to measure the electricity used by a large appliance or a household, folks tend to use kilowatt-hours (kWh). A kilowatt is equal to one thousand watts; therefore, one kilowatt-hour is equal to one thousand watt-hours.

If you want to measure the output of a power plant or the amount of electricity used by an entire city, you will use megawatts. A megawatt is one thousand kilowatts or one million watts. Getting even larger, a gigawatt is one thousand megawatts, or one million kilowatts, or one billion watts. Gigawatts is where the namesake for Tesla’s Gigafactories comes from. In 2018, battery production at the Gigafactory in Nevada reached 20 gigawatt-hours (GWh) per year3.

Alkaline batteries Most people are probably familiar with alkaline batteries. These are the batteries that you typically use to power toys, electronics, flashlights, etc. The bulk of alkaline batteries produced are single-use, although there are some rechargeable alkaline batteries in existence. So what makes up an alkaline battery?

Alkaline batteries have zinc as their anode and manganese dioxide (MnO2) as their cathode. Their name, however, comes from the alkaline solution used as the electrolyte. The electrolyte is typically potassium hydroxide (KOH), which can contain a large number of dissolved ions. The more ions the electrolyte solution can absorb, the longer the redox reaction that drives the battery can keep going.

The zinc anode is usually in powdered form. Powder has a greater surface area for a reaction, which means the cell can quickly release its power. The zinc anode gives up its electrons to the manganese dioxide cathode, to which carbon, in the form of graphite, is added to improve its conductivity and help it keep its shape.

Alkaline batteries are popular because they have a low self-discharge rate, giving them a long shelf life, and don’t contain toxic heavy metals like lead or cadmium. They account for the bulk of batteries that are made today, although their place at the top will likely soon be challenged by the lithium-ion batteries in our phones, laptops, cars, and an increasing number of other gadgets.

Lithium-ion batteries Lithium-ion batteries are popular due to their energy density. Because the energy is dense, your phone can last all day and still be the small, portable, handheld device we are all familiar with. As you likely know from the behavior of your phone, lithium-ion batteries are rechargeable. The namesake for the battery comes from the fact that lithium ions (Li+) are involved in the chemical reactions that make up the battery.

In a lithium-ion cell, both electrodes, anode and cathode, are made of materials that can absorb lithium ions. The absorbing action is known as intercalation when charged ions of an element can be stored inside a material without significantly disturbing it. The lithium ions are paired to an electron within the structure of the anode. When the battery discharges, the intercalated lithium ions are released from the anode and travel through the electrolyte solution to be intercalated in the cathode.

A lithium-ion battery starts its life in a state of full discharge: all its lithium ions are intercalated within the cathode, and its chemistry cannot yet produce any electricity. Before the battery can be used, it needs to be charged. As the battery is charged, an oxidation reaction occurs at the cathode, meaning that it loses some negatively charged electrons. An equal number of positively charged intercalated lithium ions are dissolved into the electrolyte solution to maintain the charge balance in the cathode. These travel over to the anode, where they are intercalated, or absorbed, within what is typically graphite. This intercalation reaction also deposits electrons into the graphite anode, to pair with the lithium ion. There are many other types of batteries, but you mostly need to understand lithium-ion batteries as context for this article4.

New technologies Solid-state batteries Counter to the liquid or polymer gel electrolyte found in batteries today, solid-state batteries use a solid electrolyte and solid electrodes. If we recall from earlier, positive ions flow through the electrolyte to balance the electrons’ negative charge. Today, batteries are quite efficient at transferring positive ions since a liquid electrolyte is in contact with the electrodes’ entire surface area. Using a solid makes this a bit harder. Imagine the difference between dipping a chip in soup and dipping it into chopped tomatoes. The chip dipped in the soup will have soup covering more of the chip’s surface area than the chopped tomatoes cover the other chip.

So why even use a solid electrolyte if it is less efficient? Today’s lithium-ion batteries typically rely on flammable liquids as the electrolyte. By using a solid electrolyte, batteries can be less prone to catching fire. Most folks probably remember Samsung’s Galaxy Note 7, which had the unfortunate side effect of catching fire5. Solid electrolytes provide a much safer alternative.

Research and experimentation in solid electrolytes typically tend to be either solid polymers at high temperatures or ceramics at room temperature. The downside of solid polymers at high temperatures is they need to operate at temperatures above 220°F (105°C)6. That is certainly not practical for a handheld device like a phone or tablet, but could be apt for storing energy to power a home.

Quite a few companies are working on using ceramics at room temperature to create a solid-state battery. Toyota has been talking about theirs for years7 and aims to have it completed in 20258. Startups, such as Solid Power and A123 Systems (with the help of Iconic Materials), aim to do the same.

A lot of the novel research being done on solid-state batteries is the work of Jürgen Janek9. Jürgen recently published a benchmark of the performance of all-solid-state lithium batteries10. Another high-profile battery scientist, Gerbrand Ceder, published a paper on interface stability in solid-state batteries11. New and novel research on solid-state batteries is being published quite frequently. While there are many skeptics of solid-state batteries since it has yet to be commercially delivered and scaled, I would not dismiss it entirely from having a seat at the table in the future.

Nuclear batteries Until now, we have only discussed batteries powered by chemical reactions, such as those powering flashlights, phones, and other gadgets. Chemical batteries, also known as galvanic cells, discharge in a given amount of time and either need to be thrown away or recharged. Begging the question: is there a type of battery that could last long term?

Nuclear batteries, also known as atomic batteries, using the energy of beta decay, are being researched to create a battery that lasts longer than those powered by chemical reactions. Batteries powered by beta decay are known as betavoltaics. Radioactive isotopes used in nuclear batteries have half-lives ranging from tens to hundreds of years, so their power output remains nearly constant for a very long time. If nuclear batteries last from tens to hundreds of years, why are we not using them everywhere today? Doesn’t everyone want a phone that could last at least ten years without needing to be charged?

There are a few side effects of nuclear batteries. They cannot be turned off; electrons are continually being produced, even when they are not needed. Research is being done into stimulating beta decay12, which would create more current on-demand, allowing the output to drop to almost nothing when it is turned off. Another downside is the power density of betavoltaic cells is much lower than that of chemical batteries. However, it is interesting to note that betavoltaics were used in the 1970s to power cardiac pacemakers, before being replaced by cheaper lithium-ion batteries, even though lithium-ion batteries have a shorter lifetime.

In 2016, Russian researchers from MISIS presented a prototype betavoltaic battery based on nickel-6313. A downside of using nickel-63 is that it is not readily available, making their research hard to commercialize. CityLabs sells a betavoltaic battery, with a 14.4-year half-life, you can buy today starting at $1,00014, but you would need 1.2 million of these just to have one watt of power. NDB is a startup working on a nano diamond battery that could last for thousands of years15. UPower is another startup working on a megawatt-scale atomic generator.

Silicon anode Today, the material typically used for the anode is graphite because it is economical, reliable, and relatively energy-dense, especially compared to current cathode materials. The limiting factor of lithium-ion batteries is the amount of lithium that can be stored in the electrodes. Using silicon as the material for the anode, rather than graphite, allows around nine times more lithium ions to be held in the anode.

The ability to store more lithium ions using silicon sounds amazing; why isn’t everyone doing this? The problem is a silicon anode swells to 3-4 times its original volume when it absorbs lithium ions. Making the casing bigger doesn’t circumvent the problem because the expansion causes the silicon to fracture, causing the battery to fail. It also gums up with a passivation layer, also known as the solid electrolyte interphase (SEI), formed on electrode surfaces from the decomposition of electrolytes.

“With silicon, the cookie crumbles and gets gooey.” - Elon Musk.

As a solution to this problem, many companies use silicon as a fraction of the anode material. But these materials are expensive and highly engineered. Examples of this include silicon structured in SiO glass ($6.6 per kWh), silicon structured in graphite ($10.2 per kWh), and silicon nanowires (>$100 per kWh)16. Sila Nanotechnologies is using silicon as their anode material17. Amprius claims to use silicon for 100% the anode material with silicon nanowires, a highly engineered, expensive material. Advano, Enevate, and Enovix are startups working on a silicon solution for the anode material.

Tesla’s Battery Day At Tesla’s Battery Day event, they announced many changes to their battery that encompass more than just the materials used. Tesla has on staff one of the most renowned battery scientists, Jeff Dahn. His most recent papers on “A Wide Range of Testing Results on an Excellent Lithium-Ion Cell Chemistry to be used as Benchmarks for New Battery Technologies18” and “Is Cobalt Needed in Ni-rich Positive Electrode Materials for Lithium-Ion Batteries?19” help gives some insight into what Tesla has been working on.

The battery day outcomes increase their vehicles’ range while being more economical; they plan to halve the cost per kilowatt-hour. Most startups20 in this space tend to take a single design decision into account for their products, for example, anode material and focus on that. Tesla, on the other hand, took a very well rounded approach. They took into account not only the materials for the cathode and anode but also the cell design, factory, and integration with the vehicle21.

Source: Tesla’s Battery Day Presentation https://www.youtube.com/watch?v=l6T9xIeZTds

Let’s break down each of these improvements.

Cell design For Tesla’s batteries, while discharging, the positive ions flow over the tabs, while the lithium ions flow from the anode to the cathode, as shown below. The tabs allow the cell’s energy to be transferred to an external source.

Source: Tesla’s Battery Day Presentation https://www.youtube.com/watch?v=l6T9xIeZTds

The Tesla team sought out to increase the cell size to 46 millimeters, which optimizes vehicle range and cost reduction. However, increasing the cells’ size has a negative side effect on supercharging because of thermal issues. To circumvent these issues, the Tesla team removed the tabs, calling their new design tabless.

The tabless design leads to simpler manufacturing, fewer parts, and a five times reduction in the electrical path. Going from 250-millimeter to 50-millimeter electrical path length leads to substantial thermal benefits. The electrical path length is significant because the distance the electron has to travel is much less. Even though the cell is much bigger, the power to weight ratio is better than a smaller cell with tabs.

Source: Tesla’s Battery Day Presentation https://www.youtube.com/watch?v=l6T9xIeZTds

Let’s dive into why this new tabless design matters. Instead of calling it tabless, Tesla could have called it “many tabs” because each of the folded pins is a tab, as shown in the image above. What is the function of a tab?

Growing up, my family would always leave sporting events before they ended to avoid the crowd trying to leave the stadium after the event was over. If we had stayed to the end of the event, it would take more time for us to exit the stadium and be very uncomfortable since everyone would be trying to leave through very few exits at the same time. As people are trying to exit, they get closer and closer to one another, and the environment becomes very hot and rowdy. If we think of people as electrons, a stadium with a single exit is similar to a battery’s behavior with a single tab; electrons are all trying to leave through the single tab and bumping up against one another until they heat up. There are multiple tabs in Tesla’s new design, equivalent to a stadium with lots of exits. Now people, or electrons, can exit quickly while staying cool and calm.

There aren’t many details from the presentation on the new tabless design and its implementation, but it can be attributed to “secret sauce.”

Manufacturing a cell consists of an electrode process where the active materials are coated into films onto foils; the coated foils are then wound in the winding process. The roll is then assembled into the can, sealed, and filled with electrolyte and then sent to Formation where the cell is charged for the first time. If you recall from above, a lithium-ion battery starts its life in a discharged state. For a battery cell with tabs, manufacturing is much more complicated. When the cell with tabs is going through the assembly line, it has to keep stopping where all the tabs are so you can’t do continuous motion production. It is also a lot more error-prone.

“It is really a huge pain in the ass to have tabs from a production standpoint.” - Elon Musk.

The new batteries are 46 millimeters by 80 millimeters, leading to the name 4680. The first two digits refer to the diameter, and the second two digits refer to the length. Previously, an extra zero was added onto the end of the name, but it was removed since it had no purpose.

The 4680 batteries have five times more energy with six times the power and enable a 16% range increase. At the battery pack level, the form factor improvements alone lead to a 14% reduction in cost per kWh.

Cell factory We learned a bit about how removing tabs from the battery cells simplified the manufacturing process above. In an assembly line, you don’t want things to stop and start but continuously move. Any time the process is stopped leads to inefficiency. The Tesla team aims to speed up its process to make one factory have multiple scales of efficiency better than a typical battery factory.

We learned above that the electrode process is where the active materials are coated into films onto foils. The wet process step of the electrode process consists of first: mixing. Mixing occurs when the powders are mixed with either water or a solvent, typically a solvent for the cathode. The mix then goes into a large coat and dry oven, tens of meters long, where the slurry is coated onto the foil and dried. The solvent then has to be recovered. Finally, the coated foil is compressed to the final density. This process is complex and inefficient, especially since humans need to transport the mix from the mixing step to the ovens. It is also inefficient due to the need to put the solvent in and then recover it.

One significant change they are making is skipping the solvent step of the electrode coating’s wet process in favor of a dry process. The dry process transforms the powder directly into film. This technology initially stemmed from Tesla’s acquisition of Maxwell at the beginning of 201922. At battery day, Elon mentioned that since the acquisition, they are now on the 4th revision of the equipment that turns powder into film. Elon noted, “there is still a lot of work to do. There is a clear path to success but a ton of work between here and there.” When this process is scaled up, it results in a ten times reduction in footprint and a ten times reduction in energy, and a massive decrease in CapEx investment.

The manufacturing step known as Formation is where the cell is charged for the first time, and the quality of the cell is verified. Formation is typically 25% of the CapEx investment. The Tesla team improved density and cost-effectiveness by using their knowledge from cars and the powerwall charging and discharging. This led to a 86% reduction in Formation CapEx investment per GWh and a 75% reduction in footprint. For a factory that previously output 150 GWh, this translates to that same factory outputting 1 TWh with the more efficient processes. At the battery pack level, this leads to an 18% reduction in cost per kWh.

Anode material Tesla announced they were moving to silicon as their anode material. Silicon is excellent because it is the most abundant element in the earth’s crust after oxygen. Rather than creating a highly engineered material that would be expensive, Tesla will use the raw silicon found in the earth’s crust and design for it to expand. They will stabilize the silicon’s surface through an elastic, ion-conducting polymer coating and a highly elastic binder and electrolyte.

Tesla’s silicon costs $1.20 per kWh, whereas the solutions we covered earlier cost anywhere from $6 per kWh to upwards of a hundred. Using silicon leads to a 5% reduction in cost per kWh at the battery pack level and a 20% longer range for Tesla vehicles.

Cathode material A helpful analogy for understanding the cathode is to think of the cathode as a bookshelf. In this case, the lithium ions would be books. The most efficient bookshelf holds the most books while still being stable enough to retain its structure as the books get loaned out and returned.

Source: Tesla’s Battery Day Presentation https://www.youtube.com/watch?v=l6T9xIeZTds

The Tesla team aims to increase Nickel in its cathode material since it is the cheapest and has the highest energy density (as shown above). Cobalt is typically used as a cathode material because it is very stable. However, the Tesla team aims to leverage novel coatings and dopants to stabilize Nickel better and remove Cobalt entirely from their materials. Removing Cobalt leads to a 15% reduction in the cathode’s cost per kWh.

The Tesla team made sure to keep in mind the cost of the materials used and the materials’ availability. With silicon for the anode material, availability was not an issue since silicon is readily available. The same goes for lithium, which is also highly accessible. For Nickel, on the other hand, the Tesla team is keeping in mind total Nickel availability by diversifying the amount of Nickel they are using per the type of vehicle.

The team also simplified the cathode manufacturing process by removing all the legacy parts. According to the battery day presentation, the cathode manufacturing process, which is 35% of the cathode cost per kWh, had not had a fresh look in a long time and was wildly inefficient.

“If you take a look at the ‘it’s a small world journey’ of I am a Nickel atom and what happens to me, it’s crazy, you’re going around the world three times, there is a moral equivalent of digging the ditch, filling in the ditch, and digging the ditch again. It’s total madness.” - Elon Musk.

A typical cathode process starts with the metal from the mine being turned into an intermediate material called metal sulfate, which, in turn, is processed again. The Tesla team removed the intermediate step of turning the metal into metal sulfate along with a bunch of other unnecessary steps. They also localized the cathode materials to the US, which decreased the number of miles required for the materials to travel. This leads to a 66% reduction in CapEx investment, a 76% reduction in process cost, and zero wastewater. The cathode material improvements lead to a 12% reduction in cost per kWh at the battery pack level.

Cell vehicle integration In the early days of aircraft, the fuel was carried as cargo. Later, the fuel tanks were made in wing shape. This was a breakthrough because the wings are critical to the airplane’s function but now could be used for another purpose. The fuel tank was no longer cargo but fundamental to the structure of the aircraft. Tesla intends to do the same for cars.

By removing the intermediate structure in the battery pack, they can pack the cells more densely. Instead of having supports and stabilizers in the battery cells, making up the intermediate structural elements, the battery pack itself is structural. Typically, Tesla fills the battery packs with a flame retardant. The new battery packs will be filled with a flame retardant and structural adhesive, giving it stiffness and stability without intermediate structural elements. This makes the structure even stiffer than a regular car.

The cells can now be moved more towards the center of the vehicle because the volumetric efficiency is better, avoiding a side impact potentially contacting the cells. This also allows the car to maneuver better because the polar moment of inertia is improved. Much like an ice skater can turn better with her arms close to her body rather than extended out.

The improvements to the battery pack integration lead to a 10% mass reduction in the car’s body, a 14% range increase, and 370 fewer parts. The smaller, integrated battery and body also help increase the efficiency of manufacturing. This leads to a 55% reduction in CapEx investment and a 35% reduction in floor space. At the battery pack level, the integration improvements lead to a 7% reduction in cost per kWh.

The sum of all these improvements, including cell design, factory, materials, and vehicle integration, achieves the goal to halve the cost per kWh. Cheaper electric vehicles widen Tesla’s market to new buyers reducing the number of gas-powered vehicles on the road.

Summary All in all, it is fantastic to see a technology we all rely on day to day get its time in the spotlight. Although not mentioned at battery day, if Tesla were to achieve 400 watt-hours per kilogram, a zero-emissions jet might just be on the horizon. Now that batteries are vertically integrated into Tesla’s product, you can only imagine that the software will track more data on battery efficiency, leading to more and more improvements in the future.

It is incredible to see Tesla take a fresh look at making the most efficient and cost-effective batteries. The level of thought and detail put into rethinking old processes from first principles to make them more efficient is inspiring. The Tesla team didn’t just look at one angle, but all the angles: cell design, manufacturing, vehicle integration, and materials. There is a clear “why” for every decision made that boils down to economics, not just technical gains. Hopefully, we see another core technology, such as batteries, in the spotlight soon.

1. https://www.tesla.com/2020shareholdermeeting [return]
2. https://www.science.org.au/curious/technology-future/batteries [return]
3. https://www.tesla.com/gigafactory [return]
4. Some people might, however, be interested in a urine-powered battery: https://newatlas.com/urine-battery/42866/ [return]
5. https://time.com/4526350/samsung-galaxy-note-7-recall-problems-overheating-fire/ [return]
6. https://qz.com/1588236/how-we-get-to-the-next-big-battery-breakthrough/ [return]
7. https://cen.acs.org/articles/95/i46/Solid-state-batteries-inch-way.html [return]
8. https://www.caranddriver.com/news/a33435923/toyota-solid-state-battery-2025/ [return]
9. https://iopscience.iop.org/nsearch?terms=J%C3%BCrgen+Janek&nextPage=-1&previousPage=-1¤tPage=1&orderBy=relevance&pageLength=10&searchDatePeriod=anytime&journals=1945-7111&authors=J%C3%BCrgen+Janek [return]
10. https://www.nature.com/articles/s41560-020-0565-1 [return]
11. https://www.nature.com/articles/s41578-019-0157-5 [return]
12. https://www.nist.gov/news-events/news/2016/06/physicists-measured-something-new-radioactive-decay-neutrons [return]
13. https://phys.org/news/2018-06-prototype-nuclear-battery-power.html [return]
14. https://citylabs.net/?option=com_wrapper&view=wrapper&Itemid=20 [return]
15. https://techcrunch.com/2020/08/25/self-charging-thousand-year-battery-startup-ndb-aces-key-tests-and-lands-first-beta-customers/ [return]
16. https://www.youtube.com/watch?v=l6T9xIeZTds&feature=emb_title [return]
17. https://silanano.com/wp-content/uploads/2020/09/The-Future-of-Energy-Storage.pdf [return]
18. https://iopscience.iop.org/article/10.1149/2.0981913jes/meta [return]
19. https://iopscience.iop.org/article/10.1149/2.1381902jes/meta [return]
20. Except for Sila Nanotechnologies, which seems to be most closely aligned with Tesla’s methodology: https://silanano.com/wp-content/uploads/2020/09/The-Future-of-Energy-Storage.pdf [return]
21. Tesla claimed in the presentation there were more aspects they didn’t mention they could improve in the future. [return]
22. https://electrek.co/2019/02/04/tesla-acquires-ultracapacitor-battery-manufacturer/ [return]

View Details

I previously wrote a bit about our internal infrastructure in my post on The Art of Automation. This post is going to go into details about our automated Chief infrastructure Officer (CIO). I joke so much that I automated our CIO that I even named the repo holding the code… cio.

I took the time this weekend to finally clean up some of this code. Previously, our infrastructure was held together with bash, popsicle sticks, glue, and some rust. Now, it is mostly rust and a much more sane architecture to grok. We also get the freedom of caching all our data in a database that we own so we can access it even when services are down. Previously, we called out to each service’s API for every script, bot, or whatever, which can get expensive, slow, and potentially be riddled with rate limits, or worse, downtime.

Let me give you a diagram of what this looks like now:

Sending data to the database At the very bottom of the diagram, you can see where we are using webhooks and cron jobs to pull data out of various services APIs and send it to the database.

Let’s dive into a few of these because it is not as simple as a pipe from an API to a database in most cases.

Applicants Every applicant to Oxide completes our candidate materials. This is a series of questions about things they’ve worked on. Those get submitted with their resume and other details into a Google Form.

A cron job parses the spreadsheet from the Google Form. In doing so, it knows if an application is new and we need to send an email to the applicant that we received it. It will also send an email to the team that we got a new application.

The cron job also parses the materials they submitted and their resume into plain text. Materials can be in the form of HTML, PDF, doc, docx, zip, and even PDF with zip headers ;). The resume and each question in the materials is saved in individual database columns, which makes search and indexing easier when we want to find an application based on something we remember from their materials or resume.

When an applicant gets hired or moved into an interview phase, GitHub issues are opened so we can keep track of their progress through the interview or onboarding.

RFDs We wrote about our RFD process on the Oxide blog in RFD 1 Requests for Discussion.

Each RFD is written in either markdown or asciidoc. We collect the content for each RFD and update it in the database along with its equivalent HTML.

The HTML is used for generating pages in a small website we use for sharing RFDs with folks external to Oxide. These might be friends of Oxide, engineers who we value their expertise and feedback, or potential customers and partners.

By having all the content stored in the database it also makes it easier to search across the content in all the RFDs.

Those are just two examples of APIs we build on top of and enrich as we move data into our database.

GitHub It’s nice to have a cache of certain GitHub API calls for when GitHub is down or we get rate limited. We store a few GitHub endpoints data in our database as well.

Utilizing the data in an easy way Next, we need a way to share all this data with other bots, scripts, fellow colleagues, and apps within the company. This is where the API server comes into play.

The API server acts as the middle-man between the database and any scripts, bots, users, and apps. The API is read-only since we get all the data from external services and APIs.

The API server syncs the database data with Airtable so we can use Airtable as a front-end for viewing all the data in a specific table at once. This turns out to be a great use for Airtable because you can also do joins with other tables in Airtable very easily. It makes for a nice visual experience.

For example, we can relate an RFD from the RFD table to an item in a different table related to the roadmap. As folks push changes to their RFDs, the RFD content will update in Airtable as well.

All in all, this was pretty fun to build, refactor, build, and refactor again. It’s been something I can pick up and work on when I get a free second and easily add functionality to when we want to use our data in a specific way.

For the API server, I got to use our dropshot REST API library for this! Thanks to Dave and Adam for writing that :)

At this point, I can’t imagine working at a company without an internal API for querying everything from Google groups, to applicants, to mailing list subscribers, to RFDs, and more. That’s all for now! I’d love to hear about other ideas you might have for internal infrastructure!

View Details

I have wanted a 3D printer for a very long time. I hope you can tell from my ACM Queue column that I like to do a lot of research and I tend to want the best thing. I had been keeping my eyes on the 3D printer product space for quite some time. This article is going to go over the technical details behind 3D printing as well as my experience with two different products. When I finally decided to buy a 3D printer, I wanted to try ones that used different additive manufacturing processes: material extrusion via fused deposition modeling (FDM) and vat polymerization via stereolithography (SLA)1. While I would have loved to have gotten a printer for each of the seven different additive manufacturing processes, I did not. However, I did dig into the details of all the various additive manufacturing processes and technologies. Let’s dive in!

If you would prefer to skip the research you can jump down to the review.

Additive Manufacturing Processes Popular culture uses the term “3D printing” as a synonym for additive manufacturing processes. In 2010, the American Society for Testing and Materials (ASTM) group “ASTM F42 – Additive Manufacturing”, formulated a set of standards that classify the range of additive manufacturing processes into seven categories2. The processes vary on the material and machine technology used which has effects on the use cases and applications as well as the economics.

Material extrusion Material extrusion defines a process where an object is built by melting and extruding a thermoplastic polymer filament in a predetermined path layer by layer. Imagine if you were building an object and the only material you could use is a tube of toothpaste. You’d slowly build the walls of the object by putting layers of toothpaste on top of each other. Material extrusion is similar.

Material extrusion devices are the most commonly available and the cheapest types of 3D printing technology in the world. It represents the largest installed base of 3D printers globally. The most common applications are electrical housings, form and fit testings, jigs and fixtures, and investment casting patterns. The technology used for the material extrusion process is known as fused deposition modeling or FDM3.

Fused deposition modeling (FDM) FDM, also known as fused filament fabrication (FFF), works with a range of standard thermoplastic filaments, such as acrylonitrile butadiene styrene (ABS), polylactic acid (PLA), polyethylene terephthalate (PET), thermoplastic polyurethane (TPU), nylon, and their various blends.

Let’s break down the FDM process in steps:

• First, a spool of thermoplastic filament is loaded into the printer. Once the nozzle has heated to the correct temperature, the filament is fed to the extrusion head and in the nozzle where it melts.
• Second, the extrusion head is connected to a 3-axis system that allows it to move in the X, Y, and Z dimensions. The melted material is extruded in thin strands and is deposited layer by layer in predetermined locations, where it cools and solidifies. The cooling process can be accelerated by using cooling fans attached to the extrusion head, if the device supports it.
• Third, filling an area requires multiple passes, similar to coloring with a marker or drawing with toothpaste. When a layer is complete, the build platform moves down or the extrusion head moves up, depending on the device, and a new layer is deposited. This process is repeated until the object is complete.

Because of this process, FDM objects tend to have visible layer lines, unless smoothed, and might show inaccuracies around complex features.

Vat photopolymerization Photopolymerization occurs when a photopolymer resin is exposed to the light of a specific wavelength and undergoes a chemical reaction to become solid. This is a common approach additive technologies use to build an object one layer at a time.

Vat polymerization processes are excellent at producing objects with fine details and give a smooth surface finish. This makes them ideal for jewelry, low-run injection molding, dental applications, and medical applications, such as hearing aids. The main limitation of vat polymerization is the brittleness of the produced objects. For this reason it is not suitable for mechanical parts4.

Stereolithography (SLA) Stereolithography was one of the world’s first 3D printing technology, invented by Charles Hull in 19845. SLA resin 3D printers use a laser to cure liquid resin into hardened plastic.

Let’s break down the SLA process in steps:

• First, a liquid photopolymer is filled in a vat or tank.
• Second, a concentrated beam of ultraviolet light or a laser is focused onto the surface of the vat or tank. The beam or laser creates each layer of the desired 3D object using cross-linking or degrading the polymer at a specific location. This step is repeated layer by layer, a 3D physical object is built until completion.

SLA objects have high resolution and accuracy, clear details, and smooth surface finishes. SLA is also quite versatile for many different use cases since photopolymer resin formulations with a wide range of optical, mechanical, and thermal properties to match those of standard, engineering, and industrial thermoplastics have been produced.

Direct light processing (DLP) Direct light processing is near-identical to SLA, except DLP uses a digital light projector screen to flash a single image of each layer all at once. Each layer is composed of square pixels, called voxels, due to the projector being a digital screen. In a way, it is almost like an 8-bit ancestor of SLA in the same way that 8-bit drawings have more defined individual square pixels. Since each layer is exposed all at once, DLP can have faster print times compared to SLA, which solidifies a layer in cross sections.

Continuous direct light processing (CDLP) Continuous direct light processing, also known as continuous liquid interface production (CLIP), produces objects in the same way as DLP. CDLP is called “continuous” since it relies on the continuous motion of the build plate on the Z axis. This results in faster build times because the printer is not required to stop and separate the part from the build plate after each layer is produced.

Powder bed fusion (PBF) Powder bed fusion technologies produce a solid part using a thermal source that induces fusion, sintering or melting, between the particles of a plastic or metal powder one layer at a time. Most PBF technologies have mechanisms for spreading and smoothing thin layers of powder as a part is constructed, resulting in the final component being encapsulated in powder after the build is complete. The most common applications are functional objects, complex ducting (hollow designs), and low run part production.

The main variations in PBF technologies come from different energy sources, such as lasers or electron beams, and the powders used in the process, such as plastics or metals. Polymer-based PBF technologies allow for innovation in that there is no need for support structures. This makes creating objects with complex geometries easier.

Both metal and plastic PBF objects typically are strong and stiff with mechanical properties that are comparable, or sometimes even better, than the bulk material. There is a large range of post-processing methods available which can give objects a very smooth finish. For this reason, PBF is often used to manufacture functional metal parts for applications in the aerospace, automotive, medical, and dental industries.

The limitations of PBF tend to be surface roughness and shrinkage or distortion during processing, as well as the challenges the arise from powder handling and disposal6.

Selective laser sintering (SLS) Selective laser sintering is the most common additive manufacturing technology for industrial applications. The technology originated in the late 1980s at the University of Texas at Austin7. SLS 3D printers use a high-powered CO2 laser to fuse small particles of polymer powder.

Let’s break down the SLS process in steps:

• First, a bed is filled with power.
• Second, lasers, sinter, or coalesce, powdered material create a solid structure. This step is repeated, layer by layer, until the object is complete. Finally, the object, still encased in loose powder, is cleaned with brushes and pressurized air.

Differing from SLA and FDM, the nice thing about SLS is it does not require an object to have support structures. This is due to the unfused powder supporting the part during printing. This makes SLS ideal for objects with complex geometries, including interior features, undercuts, and negative features. Parts produced with SLS printing typically have excellent mechanical characteristics, meaning they are very strong. However, objects with thin walls may not be printed due to the minimum 1mm limitation and thin walls in large models may warp after cooling down.

The most common material for selective laser sintering is polyamide (nylon), a popular engineering thermoplastic with great mechanical properties. Nylon is lightweight, strong, and flexible, as well as stable against impact, chemicals, heat, UV light, water, and dirt. Alumide, a blend of gray aluminum powder and polyamide, and rubber-like materials can also be used.

The combination of low cost per part, high productivity, and established materials make SLS a popular choice among engineers for functional prototyping and a cost-effective alternative to injection molding for limited-run or bridge manufacturing.

Selective laser melting (SLM) and direct metal laser sintering (DMLS) Both selective laser melting and direct metal laser sintering produce objects via a method similar to SLS. Differing from SLS, SLM and DMLS are used in the production of metal parts. SLM fully melts the powder, while DMLS heats the powder to near melting temperatures until it chemically fuses. DMLS only works with alloys while SLM can use single component metals, such as aluminum.

Unlike SLS, SLM and DMLS require support structures to compensate for the high residual stresses generated during the build process. Support structures help to limit the possibility of warping and distortion. DMLS is the most well-established metal additive manufacturing process with the largest installed base.

Electron beam melting (EBM) Electron beam melting uses a high energy beam rather than a laser to induce fusion between the particles of metal powder. A focused electron beam scans across a thin layer of powder which causes localized melting and solidification over a specific cross-sectional area. The nice thing about electron beam systems is that they produce less residual stresses in objects, meaning there is less need for support structures. EBM also uses less energy and can produce layers quicker than SLM and DMLS. However, the minimum feature size, powder particle size, layer thickness, and surface finish are typically lower quality than SLM and DMLS. EBM requires the objects to be produced in a vacuum and the process can only be used with conductive materials8.

Multi jet fusion (MJF) Multi jet fusion is essentially a combination of the SLS and material jetting technologies. A carriage with inkjet nozzles, similar to the nozzles used in inkjet printers, passes over the print area, depositing a fusing agent on a thin layer of plastic powder. Simultaneously, a detailing agent that inhibits sintering is printed near the edge of the part. A high-power infrared radiation (IR) energy source then passes over the build bed and sinters the areas where the fusing agent was dispensed, while leaving the rest of the powder untouched. The process repeats until the object is complete9.

Material jetting Material jetting is most comparable to the inkjet printing process. Like an inkjet printer prints ink layer by layer onto a piece of paper, material jetting deposits material onto the build surface. The layer is then cured or hardened using ultraviolet (UV) light. This is repeated layer by layer until the object is completed. Since the material is deposited in drops, the materials are limited to photopolymers, metals, or wax that cure or harden when exposed to UV light or elevated temperatures.

Material jetting is ideal for realistic prototypes, providing excellent details, high accuracy, and smooth surface finish. Material jetting allows a designer to print in multiple colors and multiple materials in a single print. This makes it great for low run injection molds and medical models. Since material jetting allows multiple materials in a single print, support structures can be printed from a dissolvable material that is easily removed after building. The main drawbacks of material jetting technologies are the high cost and the brittle mechanical properties of the UV activated photopolymers10.

Nanoparticle jetting (NPJ) Nanoparticle jetting is a process by which a liquid, which contains metal nanoparticles or support nanoparticles, is loaded into the printer via a cartridge. The liquid is then jetted, similar to an inkjet printer, onto a build tray through thousands of nozzles in extremely thin layers of droplets. High temperatures inside the building chamber cause the liquid to evaporate leaving behind metal objects11.

Drop-on-demand (DOD) Drop-on-demand material jetting printers have two print jets: one to deposit the build materials, typically a wax-like liquid, and another for a dissolvable support material. Similar to material extrusion, DOD printers follow a predetermined path and deposit material in a pointwise fashion to build layers of an object. These machines also employ a fly-cutter, a single-point cutting tool, that skims the build area after each layer to ensure a perfectly flat surface before printing the next layer. DOD technology is typically used to produce wax-like patterns for lost-wax casting, used to duplicate a metal sculpture that is cast from an original sculpture, and mold making applications12.

Binder jetting A binder jetting process, also referred to as 3DP, uses two materials: a powder and a binder. The binder, which is typically a liquid, acts as the adhesive for the powder. A print head, much like that in an inkjet printer, moves horizontally across the x and y axes to deposit alternating layers of the powder material and the binder. The platform holding the bed of powder, the object is printed on, lowers as each layer is printed. This is repeated until the object is complete. Like SLS, the object does not need support structures since the powder bed acts as support. The powder materials can be either ceramic-based such as glass or gypsum or metal such as stainless steel.

Ceramic-based binder jetting, which uses a ceramic powder as the material, is best for aesthetic applications that need intricate designs such as architectural models, packaging, molds for sand casting, and ergonomic verification. It is not intended for functional prototypes, as the objects created are quite brittle.

Metal binder jetting, which uses a metal powder as the material, is well suited for functional components and more cost-effective than SLM or DMLS metal parts. However, the downside is the metal parts have poorer mechanical properties13.

Direct Energy Deposition (DED) Direct energy deposition creates objects by melting powder material as it is deposited, similar to material extrusion. It is predominantly used with metal powders or wire and is often referred to as metal deposition since it is exclusive to metals. DED relies on dense support structures which are not ideal for creating a part from scratch, which makes it best suited for repairing or adding material to existing objects, such as turbine blades14.

Laser engineered net shape (LENS) Laser engineered net shape utilizes a deposition head which consists of a laser head, powder dispensing nozzles, and inert gas tubing. The deposition head melts the powder as it is ejected from the nozzles to build an object layer by layer. The laser creates a melt pool on the build area and powder is sprayed into the pool, where it is melted and then solidified.

Electron beam additive manufacturing (EBAM) Electron beam additive manufacturing uses an electron beam to create metal objects by welding together metal powder or wire. Differentiating from LENS, which uses a laser, electron beams are more efficient and operate under a vacuum that was originally designed for use in space15.

Sheet lamination Sheet lamination processes include laminated object manufacturing (LOM) and ultrasonic additive manufacturing (UAM)16. You might be familiar with laminators, I had one growing up. To laminate a piece of paper, you would place the paper in what is known as a laminator pouch. The pouch is made up of two types of plastic: polyethylene terephthalate (PET) on the outer layer and ethylene-vinyl acetate (EVA) on the inner layer. A heated roller then adheres the two sides of the pouch together so the paper is fully encased in plastic when it is done.

The ultrasonic additive manufacturing builds metal objects by fusing and stacking metal strips, sheets, or ribbons. The layers are bound together using ultrasonic welding. The process is done on a machine able to computer numerical control (CNC) mill the workpiece as the layers are built. The process requires removal of the unbound metal, often during the welding process. UAM uses metals such as aluminium, copper, stainless steel, and titanium. The process can bond different materials, build at a fast rate, and make large objects practically while requiring relatively little energy since the metal is not melted.

Trying out products Now that we know a bit more about FDM and SLA, I can tell you about my experience with products built using these technologies. As a preface, what I was personally looking for was a product, meaning something easy to set up, easy to use, and including a fully integrated experience between the hardware and the software. I didn’t want something I would have to maintain or debug since I would rather this just work. I can understand how other folks might be in the market for something considering their decision matrix, but this was mine.

For trying out FDM, I decided to get the MakerBot Replicator+. I chose this printer mainly because it is a classic. MakerBot has a great community with Thingiverse, their site for sharing and modifying 3D models. Interestingly, the first Makerbot product was open source and they seem to have snubbed the open source community when they went from an open source model to closed with their later products17.

Makerbot has been around since 2009, I figured through 11 years of experience with 3D printing products they should have, hopefully, nailed it. They also have an iPad app that you can use to print any model from Thingiverse. I use Shapr3D for creating models on my iPad so this seemed super convenient. I could create my model in Shapr3D, upload it to Thingiverse, and print it, all from my iPad. The MakerBot also has a camera so you can watch your 3D print happening from the iPad app.

For SLA, I got the Form Labs Form 3. The software you use for printing your models is called PreForm and works on Mac or Windows. While the Form 3 does not have an iPad app, they do have an online dashboard. You can use this for tracking your print progress. Having an online dashboard is at least moving the right direction towards being able to print from my iPad if they should implement printing from the dashboard in the future. Form Labs, like Makerbot, was part of the Netflix documentary, Print the Legend. The Form 3 is their third revision of their product so I figured all the kinks should be, hopefully, worked out by now.

I am going to first go over the setup process with both machines and then we can compare the quality of the prints and the time each machine took to print the same models.

MakerBot Replicator+

Above is a picture of the printer as I was setting it up. I decided to set up the MakerBot from the iPad app since that would be primarily where I would use it from. If you have ever bought an IoT device you might be familiar with the setup workflow of joining the IoT device’s WiFi network on your mobile device and then configuring the main network to be your WiFi network. This is the same setup process as the MakerBot.

The MakerBot iOS app leaves a bit to be desired. It feels clunky, not snappy, non-native, and slow. Kinda feels like what I would expect an app written by devs with hardware expertise, rather than software expertise, would feel like. Setting up the network failed for me numerous times from my iPad so I decided to try an old Android phone instead. Again, the Android app felt clunky and non-native. It even asked me to go into my Android settings and grant more permissions to the app versus just prompting me for permissions… but finally I got the printer setup through the Android app. Now my printer showed up in my Makerbot account on the Android device and I could get through the setup process.

Being used to the cloud, I expected my printer would just appear on my iPad app since I was logged in to my MakerBot account that I tied the printer to on my Android device. It did not. I had to manually enter the printer’s IP address on my local network to the iOS MakerBot app to add the printer. That seemed like an unnecessary step, my MakerBot account should have stored that information and synced it to my other devices after I completed the setup on my initial device. Or the MakerBot app should be able to scan my local network for printers, but I digress. At least now it was working!

I moved forward with calibrating the device and printing the initial test print. I then continued to print a AAA & AA battery holder, 9V battery holder, and spaceship cookie cutter. I printed these same models on the Form 3 as well, we will go over the comparison later.

Form 3 When the Form 3 arrived, I was thinking “wow this is complex!” The Replicator+ had come in one box while the Form 3 came in 4 separate boxes. I realized after opening this was because I got the printer, the Form Wash, and the Form Cure as well as a few different resins. Below is a picture after I got everything unboxed.

The Form 3 relies on the built-in touch screen for the setup. This was quite nice after the experience with the Replicator+. I very easily got it connected onto my WiFi network and was ready to print. Since the PreForm software requires Windows I had to pull an old Windows desktop out of my pantry for this. The software is easy to use and soon I was printing my first job. The only trouble I got into was the pre-print steps when my first job was uploaded to be printed. The mixer, part of the tank, was getting a bit off track. After searching the forums, I found this is a common issue18 for a first print and after you add some resin in the tank the mixer will perform better. This turned out to be true so it was only a minor glitch!

I then continued to print the AAA & AA battery holder, 9V battery holder, and spaceship cookie cutter just like I had done with the Replicator+.

Result comparison AAA & AA battery holder19 This took 9 hours and 37 minutes on the Replicator+. It took 3 hours and 9 minutes on the Form 3. The model on the left below is from the Replicator+ and the model on the right is from the Form 3. As you can tell the quality from the Form 3 is far smoother. There are fewer build lines, it feels like one continuous piece, and there no strays of filament on the Form 3 model. The only small imperfections in the Form 3 model come from my own work of poorly removing the scaffolding.

9V battery holder20 This took 2 hours and 15 minutes on the Replicator+. It took 1 hour and 47 minutes on the Form 3. The model on the left below is from the Form 3 and the model on the right is from the Replicator+. Again the Form 3 built the smoother model. However, aside from visible lines, the Replicator+ did a fairly good job at this one. The imperfections on the Form 3 model come from the fact I am terrible at removing the scaffoldings.

Spaceship cookie cutter21 This took 1 hour and 48 minutes on the Replicator+. It took 51 minutes on the Form 3. The model on the left below is from the Form 3 and the model on the right is from the Replicator+. While the Replicator+ did a good job on this design, the Form 3 is still smoother quality.

One little detail I really love about the Form 3 is on the base of the prints, that gets removed after printing, is the name of the print, as seen below. I could imagine this coming in handy if you have a bunch of parts being printed that look very similar with small differences.

As shown from the experiments above, the quality and time to build are much better on the Form 3 than the Replicator+. Where the MakerBot wins is in aspects of the user experience. While the iPad app leaves some snappy improvements to be desired, it still exists and works for printing which is on the right track. I also wish the Form 3 had a built-in camera that I could watch as I did with the MakerBot. Since the Form 3 is SLA, I think it would be even more invigorating to watch because I found myself very interested in watching the model rise from the “goo”, aka the resin. Overall, the Form 3 is great and I can only anticipate they continue to improve!

I hope you enjoyed and learned something from this article even if you aren’t in the market for a 3D printer. In the future, I would love if products did automatic support removal because in the pictures above from the Form 3 any imperfections actually came from my removal of the support structures22. I would also love to see some sort of reliable quality monitoring23. While a lot of progress has been made in the 3D printing space, I cannot wait to see what will come in the future. The ability to go from a digital file to a physical object rapidly with many different materials can enable so many folks to create something they could only imagine in their wildest dreams until now.

1. Obviously having two 3D printers is a bit unseemly so I decided after I tried them both I would donate one to a nearby school. [return]
2. https://www.astm.org/Standards/ISOASTM52900.htm [return]
3. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/materialextrusion/ [return]
4. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/vatphotopolymerisation/ [return]
5. http://www.historyofinformation.com/detail.php?id=3864 [return]
6. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/powderbedfusion/ [return]
7. https://www.me.utexas.edu/news/news/selective-laser-sintering-birth-of-an-industry [return]
8. https://www.sciencedirect.com/topics/chemistry/electron-beam-melting [return]
9. https://www.protolabs.com/services/3d-printing/multi-jet-fusion/ [return]
10. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/materialjetting/ [return]
11. https://www.additivemanufacturing.media/blog/post/am-101-nanoparticle-jetting-npj [return]
12. https://www.sciencedirect.com/science/article/abs/pii/S0924424719312701 [return]
13. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/binderjetting/ [return]
14. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/directedenergydeposition/ [return]
15. https://www.researchgate.net/publication/328169730_A_new_3D_printing_method_based_on_non-vacuum_electron_beam_technology [return]
16. https://www.lboro.ac.uk/research/amrg/about/the7categoriesofadditivemanufacturing/sheetlamination/ [return]
17. https://3dprintingindustry.com/news/failure-makerbot-expert-weighs-78926/ [return]
18. https://forum.formlabs.com/t/form-3-mixer-arm-problem/25331 [return]
19. https://www.thingiverse.com/thing:3358129 [return]
20. https://www.thingiverse.com/thing:832281 [return]
21. https://www.thingiverse.com/thing:513900 [return]
22. https://www.arxiv-vanity.com/papers/1904.12117/ [return]
23. https://arxiv.org/pdf/2003.08749.pdf [return]

View Details

My mom has a tendency to buy these really terribly spec’d Windows machines. She’s been doing it for as long as I’ve been alive. I was surprised when on one of our latest Zoom calls she said “You know what, I’m beginning to think that size matters.” I’ve only been telling her this for years! Here’s the problem.

There are a bunch of shitty Windows machines you can buy that cost around $400 dollars and have something like 4GB RAM. For consumers, this is really compelling; the price seems right. The problem is when they start trying to use the machine to do anything, it runs at a snails pace and leaves them with the world’s worst user experience. My mom continually complained about how slow her computer was and I continually said it’s because it’s a shit machine and you have to spend more to actually get good specs.

Apple wouldn’t be caught dead selling a machine with 4GB of RAM. They know better than that and care about the experience the end user has. My sister has been lucky enough to never have to buy a computer since she continually inherits my old ones. After my mom had finished saying that “size matters,” my sister noted that my MacBookPro I gave her in 2012 still runs great and is fast. This is no surprise to me because at the time I bought that computer it was the top of its line and had 16GB of RAM. Today, that model goes up to 64GB of RAM but 16GB is definitely enough for my sister to run a browser and do what she needs for work (although Chrome is really pushing the limits these days).

It infuriates me to no end that consumers have an option of even buying a $400 computer that will give them such a terrible experience. The price is great but the experience is terrible. Even if consumers have a daughter continually telling them that “size matters,” they might still make the very innocent mistake of buying the machine and realizing later that it is a lemon. It is not their fault. Manufacturers of computers should be embarrassed for even selling such a shit machine. I know I would be.

A few articles and papers have surfaced lately on migrating threads and processes to different kernels. One of these is called popcorn1. Another has been dubbed teleforking2. I’m not going to get into the details, but in essence, what people are trying to do is move a process from one computer to another. This is great! This could be a huge problem solver for folks with computers that have terrible specs. It could also mean a lot for the future of consumer computers.

Imagine a computer where if you were running especially hot and your user experience had been compromised… the computer realizes this and forks your process into a remote data center, while maintaining a great user experience locally. It would need to be seamless and invisible to the end user. If the process is a GUI it would need to still have the user interface rendering locally while most of the compute is remote. If the process is a job streaming output into a terminal it is a bit easier. Both should be possible.

Future computers should not have limited computing power, just limited local computing power. This wouldn’t need to just be for your laptops or desktops, your VR headset or gaming console could fork processes to other available computers when they needed more computing power. The remote compute would not always need to be in a data center. An overburdened laptop could fork a process to your gaming console while you were at work and vice-versa while you were playing a game.

Compute should be easily shared and readily available. While consumers should not even have an option of buying a machine with terrible specs that lead to a terrible user experience, the ability to offload processes to another computer would allow them to have a great experience even on a lemon. As I see it, this should be the future of consumer computing. People should be able to create anything they imagine on a computer that gives them unlimited power to do so. To quote one of my favorite lines from Halt and Catch Fire: “Computers aren’t the thing. They’re the thing that gets us to the thing.”

1. https://www.ssrg.ece.vt.edu/theses/MS_Katz.pdf [return]
2. https://thume.ca/2020/04/18/telefork-forking-a-process-onto-a-different-computer/ [return]

View Details

Being cooped up at home got me looking into the new Xbox and PlayStation 5. I was curious about the innovations in the consoles since their successors. Both claim to have ray tracing and support for 8K graphics. This then got me thinking about how prevalent 8K televisions are today. 8K televisions seem to be in the same state as 4K televisions a few years ago. One thing I know through my life is that pixel density will continue to get bigger and bigger. I almost wonder if there is a Moore’s Law equivalent for pixel density… let’s take a look at televisions through the years.

Pixel density through the years Standard definition television The first electronic television was invented in 19271. Cable television systems originated in the United States in the late 1940s and were designed to improve reception of commercial network broadcasts in remote and hilly areas2. We can consider the televisions of this time to be what we know of as “standard definition.” Standard definition television (SDTV) is designed on the assumption that viewers in the typical home setting are located at a distance equal to six or seven times the height of the picture screen — on average some 10 feet away.

High definition television High definition television (HDTV) has its roots in research that was started by Japan’s public broadcaster, NHK, in 19703. For comparison, a 1080i HDTV signal offers about six times the resolution of a conventional 480i SDTV signal. HDTV also features a wider 16:9 aspect ratio format that more closely resembles human peripheral vision than the 4:3 aspect ratio used by conventional TVs in the past. Furthermore, HDTV is based on a system of 3 primary image signal components rather than a single composite signal, thus eliminating the need for signal encoding and decoding processes that can degrade image quality. Perhaps the biggest advantage over the old analog SDTV system is that HDTV is an inherently digital system.

4K resolution In 1984, Hitachi released the CMOS (complementary metal–oxide–semiconductor) graphics processor ARTC HD63484, which was capable of displaying up to 4K resolution when in monochrome mode. The first displays capable of displaying 4K content appeared in 2001, as the IBM T220/T221 LCD monitors4.

8K resolution Just like with HDTV, Japan’s public broadcaster, NHK, was the first to start research and development of 8K resolution in 1995. The format was standardized in October 2007 and the interface was standardized in August 2010 and recommended as the international standard for television in 2012. The world’s first 8K television was unveiled by Sharp at the Consumer Electronics Show (CES) in 20125. Screenings of 2014 Winter Olympics in Sochi and the FIFA World Cup in Brazil in June 2014 were done in 8K6.

While there was a huge gap in time between HD and 4K televisions, HDTV continued to get better and better during those years. It is not like the industry was stagnant, there were more improved and better HDTVs made. It might be fun to take a bet that pixel density will double in size every ten years, but that is just being presumptuous.

What does this mean for bandwidth? While it is fun to stick a finger in the air and try to estimate future pixel density growth, there is another point I want to make. If the next wave of consumer televisions is 8K what does that mean for streaming? Surely, this must have an effect on bandwidth.

For streaming HD, most providers recommend about 18mbps. For streaming 4K, providers recommend 25mbps. For streaming 8K, providers recommend 100mbps. This comes from the fact that 8K televisions have a frame rate of 120fps (frames per second). This is in contrast to 4K televisions that have a frame rate of either 30fps or 60fps. It’s also important to note, this doesn’t take into account that typically multiple devices on a network share the same bandwidth, so if your TV needs 100mbps, it is going to be shared between a computer, iPad, multiple phones, IoT devices, and whatever else is on your network. Typically you would want a multiple of the recommended speed so you can have multiple devices connected at the same time.

Let’s take a look at the average network speed. According to a report7 by speedtest.net in 20188, the average network download speed in the United States was 96.259. In the United Kingdom, the average network download speed was 50.1610. In Spain, the average network download speed was 60.1211. The United States seems to be the highest overall, but is still not cutting it for 8K streaming.

If we know that the pixel density of televisions is only going to increase over time, causing streaming services to need more bandwidth, why are we not seeing a bunch of fiber being laid down or other innovations to get faster internet to the mass market?12

You might remember that Google Fiber was trying to do this exact thing. The problem with the Google Fiber project was what is known as the “last mile” problem. The “last mile” is the last bit of cable to get the connection to your home or business, these are known as drop cables. Most folks have existing cable lines running to their home, which leaves fiber providers deciding between using those existing lines causing a decrease in speed or laying new fiber lines which is very expensive. Google Fiber chose the latter and it paid for that decision.

Fiber is similar to public infrastructure like a freeway, you need to put in the investment upfront but it will pay off dividends over time. Most companies do not get that and want an economic return upfront.

The solution to the “last mile” problem might be wireless, which leads us to the current innovations with satellites.

What about satellites? Startups like Astranis13 claim to be able to provide broadband internet to the masses through satellites. Astranis’ first satellite will offer 7.5 gigabits per second of capacity for Pacific Dataport to use14. Elon’s Starlink15 has the same ambitious mission. Starlink claims they will offer plans to consumers with speeds up to a gigabit per second16.

While most of the world is spending time at home, video chatting and streaming services have become a household essential. As the world continues to move from physical to digital at a rapid pace, we should see high bandwidth internet take a front and center role. As someone who has wanted the dream of fiber, super fast internet for everyone, I can’t wait to see what comes of this. Whether by fiber or satellite, I hope we can reach massive bandwidth speeds across the world.

1. https://bebusinessed.com/history/history-of-the-television/ [return]
2. https://www.britannica.com/technology/standard-definition-television [return]
3. https://ecee.colorado.edu/~ecen4242/marko/TV_History/related%20standards/HDTV_Past.htm [return]
4. https://en.wikipedia.org/wiki/4K_resolution [return]
5. https://www.businesstoday.in/technology/launch/ces-2013-sharp-showcases-worlds-first-8k-tv/story/191438.html [return]
6. https://en.wikipedia.org/wiki/8K_resolution [return]
7. It’s definitely worth understanding why these numbers are so low. It is hard to know from just the data. It might be overall network capacity or possibly the network backbone is at capacity while the “last mile” is not. The former could be solved by more capacity as noted throughout this article, but the latter could be solved by more and better CDNs. Thanks to Alex Rasmussen for pointing this out! [return]
8. I tried to find more recent numbers that were not from a sketchy source and couldn’t, would love if anyone knows of any. [return]
9. https://www.speedtest.net/reports/united-states/2018/#fixed [return]
10. https://www.speedtest.net/reports/united-kingdom/#fixed [return]
11. https://www.speedtest.net/reports/spain/#fixed [return]
12. Another great question is if consumers had 100mbps, would their (typically shitty) wifi setups even let them reap the benefits? Thanks to Scott Andreas for that great question! [return]
13. https://www.astranis.com [return]
14. https://spacenews.com/astranis-will-share-a-falcon-9-for-2020-small-geo-launch/ [return]
15. https://www.space.com/spacex-starlink-satellites.html [return]
16. https://www.fastcompany.com/90458407/spacex-satellite-broadband [return]

View Details

I am unsure if my love of automation comes from a dislike of doing the same thing twice or an overall desire to be more productive and make everything more efficient. Like a lot of programmers, I often ask myself “can this be scripted” when I find myself doing a manual task.

I was inspired recently by reading Wolfram’s writing on his personal infrastructure for productivity1. I, too, have written about my personal infrastructure2, but not at the level of depth or with the same focus on productivity as Wolfram. There is no time like the present to take another swing at it!

Not only do I want to touch on some of the ways I have automated tasks in my life, I also want to spend some time unpacking how common automation patterns are starting to appear in a lot of things people use day to day. Apple’s Shortcuts, home automation, and IFTTT make automation patterns available to the masses in a way that is unprecedented. Before diving into the details, let’s first answer the question of why.

Why automate? Time is one of the most valuable resources in the world. If there was something you could do to free more time for yourself, why wouldn’t you? When I automate myself out of a task I transfer the burden of doing said task to some other script, service, API, or a combination of all of these.

I, personally, feel at my best and most productive when I am building something, solving a problem, or learning something new. In none of those situations does that include “doing something manual that could otherwise be scripted/automated away.” Or when it does… I automate it away. In any circumstance when I find myself in a position to automate something, I will automate it. Especially if I consider the time spent automating the task to be less than any time spent in the future doing the task manually. This is the ultimate pay off: time.

I like to think of automation as the following equation.

$$ \begin{equation} time \ gained = (time \ doing \ task \ manually) - (time \ to \ automate \ task) \end{equation} $$

It’s important to note that in the equation above, the time to automate the task also includes any future bugs you might have to fix in the automation itself.

By automating tasks, I can focus my time on doing the things where I feel at my best and most productive while at the same time being more efficient and getting more done.

Some recent automations For myself, when I automate things, I tend to start by making it work and then making it pretty. In our equation above, the goal is to keep the time automating the task to a minimum. By getting the thing to work first without wasting time on making it pretty, I find I can gain the most time and be most productive. Cleaning up whatever mess I made with scripts or APIs after is a lot easier and faster after you get it to work. You can imagine that most of my automations start out looking like a Rube Goldberg machine. Let’s dive into some of the most recent things I have automated.

On-boarding new hires For our startup, we have been hiring quite a bit quickly. I wanted to make sure our on-boarding process was streamlined and consistent. Adding new folks to GSuite, Zoom, and GitHub teams manually is a huge waste of time and tends to lead to human error. I automated on-boarding new folks into all our tools with a Rust script. This was also a nice excuse for me to tinker with the Rust programming language. I basically automated the role of CIO in Rust.

Now when folks join the company, they get added to a config file which then automatically sets up an email account in GSuite, creates them a Zoom account, adds them to all the right GitHub teams, and then sends an email to them outlining all the tools and their accounts. It’s been improved with every new hire’s feedback as well, which makes it even better.

We open sourced a lot of the libs I used in Rust for doing this at oxidecomputer/cio.

Newsletter RSS feeds Another thing I recently automated was clearing out all the newsletters I get to my email inbox everyday. Most of these are subscriptions to people’s blogs, like The Morning Paper3. Since most of these are actually RSS feeds, I instead now pipe the RSS feed updates to Pocket4. This way everyday I can check Pocket for my list of things to read versus my email inbox being used for that. I found that when these newsletters were going to my inbox, I never actually read them since I tend to use my inbox as a TODO list and I would archive the newsletters right away since they aren’t a priority. Now, I keep my inbox clear of clutter and actually have a place for storing things I want to read later.

Gmail filters Speaking of email inboxes… I am an absolute stickler about Gmail filters. At the time of writing this article, I have 72 different filters. I constantly improve my labeling and automatic archiving of emails through a configuration file. This management system is a little Go tool I made for Gmail filters called gmailfilters5.

For mailing lists, I tend to archive the messages unless they are sent directly to me, whether in cc or to. This keeps my inbox clean, while also making sure each mailing list gets sorted into its own Gmail label so I can easily view all the messages if I need to. By maintaining Gmail filters in a configuration file, versus the user interface, I save a bunch of time trying to find the filter I want to edit, editing it, and saving it. Also, if I make a mistake and want to revert it, I now have a git history of past filters, so this is as simple as git revert.

These are just a few of the things I automate for my day to day life. If you are interested in more of these, please refer to my original posts on my personal infrastructure6.

As developers, automation is not a new concept, we tend to deal with the patterns of automation day to day through continuous integration (CI) and continuous delivery (CD). For the rest of the world, it is interesting to see the patterns of automation are starting to play a role in consumer products.

Automation for the masses Apple Shortcuts Recently, I switched back to an iPhone and got an iPad. I was delighted to play with the new “Shortcuts” feature. A Shortcut allows users to do multiple tasks and streamline them together into one action. For example, you could create a shortcut that on your commute home from the office: gets the latest traffic report, plays your favorite new podcast on the drive home, then turns on your lights when you get home (assuming you have smart lights). You can build anything you like depending on the apps you have installed and your preferences. It’s really quite extensible, while also being approachable by the mass market of iPhone adopters. In the age of COVID-19 and working from home, I’m sure you can think of a different example ;)

Home automation Speaking of lights that can automatically turn on, home automation is another way that wider audiences can create automation patterns for themselves. Between Google Home, Apple’s Homekit, and Amazon Alexa adoption, more and more folks are seeing the power that technology can unleash and time saved by automating everyday tasks. Most of these devices have a concept of creating and using “routines” to chain multiple tasks together.

For example, when I leave the house, turn off all the lights, set the temperature so the AC is no longer running, and turn on the security system. Or, when I say it’s time for bed, turn off all the lights and set the security system to “on and home.” This user experience and ease of use enables consumers to boost their productivity and save time in the same way a developer would through programming and scripting.

There is, of course, a darker side to IoT devices if consumers are uneducated. Whether it’s your lightbulbs, thermostat, home security system, or refrigerator, it is important to research the security of the IoT devices you buy.

IFTTT If-this-then-that (IFTTT) has been around for quite some time, but I wanted to take the time to call it out as an early way that automation was brought to a larger audience without people having to program. IFTTT clones are a dime a dozen now. There is Zapier, Huginn, and automate.io, just to name a few. All these products have one thing in common: promoting personal productivity through combining and chaining various tasks together into a single, automated workflow.

Productivity progress I am glad that the patterns of automation have started to make their way mainstream so that mass market consumers can see the same productivity gains without programming that developers achieve through scripting. The user interface might be different but the goal is the same: saving time and eliminating the need to do manual tasks repeatedly. The feeling developers get after writing a script to make their life easier should not be exclusive. I hope in the future we continue to see easier and more creative ways to automate while granting the same automation superpowers to everyone, not just programmers.

1. https://writings.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/ [return]
2. https://blog.jessfraz.com/post/home-lab-is-the-dopest-lab/ [return]
3. https://blog.acolyer.org/ [return]
4. https://getpocket.com/ [return]
5. https://github.com/jessfraz/gmailfilters [return]
6. https://blog.jessfraz.com/post/personal-infrastructure/ [return]

View Details

A byte of data has been stored in a number of different ways as newer, better, and faster mediums of storage are introduced. A byte is a unit of digital information that most commonly refers to eight bits. A bit is a unit of information that can be expressed as 0 or 1, representing logical state.

In the case of paper cards, a bit was stored as the presence or absence of a hole in the card at a specific place. If we go even further back in time to Babbage’s Analytical Engine, a bit was stored as the position of a mechanical gear or lever. For magnetic storage devices, such as tapes and disks, a bit is represented by the polarity of a certain area of the magnetic film. In modern dynamic random-access memory (DRAM), a bit is often represented as two levels of electrical charge stored in a capacitor, a device that stores electrical energy in an electric field.

In June 1956, Werner Buchholz1 coined the word byte2 to refer to a group of bits used to encode a single character of text3. Let’s go over a bit about character encoding. We will start with American Standard Code for Information Interchange, or ASCII. ASCII was based on the English alphabet, therefore every letter, digit, and symbol (a-z, A-Z, 0–9, +, -, /, “, ! etc) were represented as a 7 bit integer between 32 and 127. This wasn’t very friendly to other languages. In order to support other languages, Unicode extended ASCII. With Unicode, each character is represented as a code-point, or character, for example a lower case j is U+006A, where the U stands for Unicode and after that is a hexadecimal number.

UTF-8 is the standard for representing characters as eight bits, allowing every code-point between 0-127 to be stored in a single byte. If we think back to ASCII this is fine for English characters, but other language’s characters are often expressed as two or more bytes. UTF-16 is the standard for representing characters as 16 bits and UTF-32 is the standard for representing characters as 32 bits. In ASCII every character is a byte, and in Unicode, that’s often not true, a character can be 1, 2, 3, or more bytes. Throughout this article there will be different sized groupings of bits. The number of bits in a byte varies based on the design of the storage medium in the past.

This article is going to travel in time through various mediums of storage as an exercise of diving into how we have stored data through history. By no means will this include every single storage medium ever manufactured, sold, or distributed. This article is meant to be fun and informative while not being encyclopedic. Let’s get started. Let’s assume we have a byte of data to be stored: the letter j, or as an encoded byte 6a or in binary 01001010. As we travel through time, our data byte will come into play in some of the storage technologies we cover. Finally, the article will wrap up with a look at the current and future technologies for storage.

1951

Source for image: http://www.ricomputermuseum.org/Home/interesting_computer_items/univac-magnetic-tape

Our story starts in 1951 with the UNIVAC UNISERVO tape drive for the UNIVAC 1 computer. This was the first tape drive made for a commercial computer. The tape was three pounds of ½ inch wide thin strip of nickel-plated phosphor bronze, called Vicalloy, which was 1,200 feet long. Our data byte could be stored at a rate of 7,200 characters per second4 on tape moving at 100 inches per second. At this point in history, you could measure the speed of a storage algorithm by the distance the tape traveled.

1952

Source for image: https://www.ibm.com/ibm/history/exhibits/storage/storage_PH5-24.html

Let’s fast forward a year to May 21st, 1952 when IBM announced their first magnetic tape unit, the IBM 726. Our data byte could now be moved off UNISERVO metal tape onto IBM’s magnetic tape. This new home would be super cozy for our very small data byte since the tape could store up to 2 million digits. This magnetic 7 track tape moved at 75 inches per second with a transfer rate of 12,500 digits5 or 7,500 characters6 (called copy groups at the time) per second. For reference, this article has 34,128 characters.

7 track tapes had six tracks for data and one to maintain parity by ensuring that the total number of 1-bits in the string was even or odd. Data was recorded at 100 bits per linear inch. This system used a “vacuum channel” method of keeping a loop of tape circulating between two points. This allowed the tape drive to start and stop the tape in a split second. This was done by placing long vacuum columns between the tape reels and the read/write heads to absorb sudden increases in tension in the tape, without which the tape would have typically broken. A removable plastic ring in the back of the tape reel provided write protection. About 1.1 megabytes could be stored on one reel of tape7.

If you think back to VHS tapes, what was required before returning a movie to Blockbuster? Rewinding the tape! The same could be said for tape used for computers. Programs could not hop around a tape, or randomly access data, they had to read and write in sequential order.

1956

Source for image: https://www.computerhistory.org/revolution/memory-storage/8/233

If we move ahead a few years to 1956, the era of magnetic disk storage began with IBM’s completion of a RAMAC 305 computer system to deliver to Zellerbach Paper in San Francisco8. This computer was the first to use a moving-head hard disk drive. The RAMAC disk drive consisted of fifty magnetically coated 24 inch diameter metal platters capable of storing about five million characters of data, 7 bits per character, and spinning at 1,200 revolutions per minute. The storage capacity was about 3.75 megabytes.

RAMAC allowed real-time random access memory to large amounts of data, unlike magnetic tape or punch cards. IBM advertised the RAMAC as being able to store the equivalent of 64,000 punched cards9. Previously to the RAMAC, transactions were held until a group of data was accumulated and batch processed. The RAMRAC introduced the concept of continuously processing transactions as they occurred so data could be retrieved immediately when it was fresh. Our data byte could now be accessed in the RAMAC at 100,000 bits per second10. Prior to this, with tapes, we had to write and read sequential data and could not randomly jump to various parts of the tape. Real-time random access of data was truly revolutionary at this time.

1963

Source for image: https://www.computerhistory.org/timeline/1963/

Let’s fast forward to 1963 when DECtape was introduced. Its namesake stemmed from the Digital Equipment Corporation, known as DEC for short. DECtape was inexpensive and reliable so it was used in many generations of the DEC computers. It was a ¾ inch tape that was laminated and sandwiched between two layers of mylar on a four inch reel.

DECtape could be carried by hand, as opposed to its weighty and large predecessors, making it great for personal computers. In contrast to 7 track tape, DECtape had 6 data tracks, 2 mark tracks, and two clock tracks. Data was recorded at 350 bits per inch. Our data byte, which is 8 bits but could be expanded to 12, could be transferred to DECtape at 8,325 12-bit words per second with a tape speed of 93 +/-12 inches per second11. This is 8% more digits per second than the UNISERVO metal tape in 1952.

1967

Source for image: https://www.computerhistory.org/revolution/memory-storage/8/261/1080

Four years later in 1967, a small team at IBM started working on the IBM floppy disk drive, codenamed Minnow12. At the time, the team was tasked with developing a reliable and inexpensive way to load microcode into the IBM System/370 mainframes13. The project then got reassigned and repurposed to load microcode into the controller for the IBM 3330 Direct Access Storage Facility, codenamed Merlin.

Our data byte could now be stored on read-only 8-inch flexible Mylar disks coated with magnetic material, which are today known as floppy disks. At the time of release, the result of the project was named the IBM 23FD Floppy Disk Drive System. The disks could hold 80 kilobytes of data. Unlike hard drives, a user could easily transfer a floppy in its protective jacket from one drive to another. Later in 1973, IBM released a read/write floppy disk drive, which then became an industry standard14.

1969

Source for image: https://spectrum.ieee.org/tech-history/space-age/software-as-hardware-apollos-rope-memory

In 1969, the Apollo Guidance Computer (AGC) read-only rope memory was launched into space aboard the Apollo 11 mission, which carried American astronauts to the Moon and back. This rope memory was made by hand and could hold 72 kilobytes of data. Manufacturing rope memory was laborious, slow, and required skills analogous to textile work; it could take months to weave a program into the rope memory15. But it was the right tool for the job at the time to resist the harsh rigors of space. When a wire went through one of the circular cores it represented a 1. Wires that went around a core represented a 0. Our data byte would take a human a few minutes (estimated) to weave into the rope.

1977

Source for image: https://en.wikipedia.org/wiki/File:Commodore-Datasette-C2N-Mk2-Front.jpg

Let’s fast forward to 1977 when the Commodore PET, the first (successful) mass-market personal computer, was released. Built-in to the PET was a Commodore 1530 Datasette, meaning data plus cassette. The PET converted data into analog sound signals that were then stored on cassettes16. This made for a cost-effective and reliable storage solution, albeit very slow. Our small databyte could be transferred at a rate of around 60-70 bytes per second17. The cassettes could hold about 100 kilobytes per 30-minute side, with 2 sides per tape. For example, you could fit about 2 of these 55 KB images18 on one side of the cassette. The datasette also appeared in the Commodore VIC-20 and Commodore 64.

1978

Source for image: https://www.youtube.com/watch?v=PRFQm0eUvzs

Let’s jump ahead a year to 1978 when the LaserDisc was introduced as “Discovision” by MCA and Philips. Jaws was the first film sold on a LaserDisc in North America. The audio and video quality on a LaserDisc was far better than the competitors, but too expensive for most consumers. As opposed to the VHS tape which consumers could use to record TV programs, the LaserDisc could not be written to. LaserDiscs used analog video with analog FM stereo sound and pulse-code modulation19, or PCM, digital audio. The disks were 12 inches in diameter and composed of two single sided aluminum disks layered in plastic. The LaserDisc is remembered today as being the foundation CDs and DVDs were built upon.

1979

Source for image: https://www.computerhistory.org/storageengine/seagate-5-25-inch-hdd-becomes-pc-standard/

A year later in 1979, Alan Shugart and Finis Conner founded the company Seagate Technology with the idea of scaling down a hard disk drive to be the same size as a 5 ¼ inch floppy disk, which at the time was the standard. Their first product, in 1980, was the Seagate ST506 hard disk drive, the first hard disk drive for microcomputers. The disk held five megabytes of data, which at the time was five times more than the standard floppy disk. The founders succeeded in their goal of scaling down the drive to the size of a floppy disk drive at 5 ¼ inches. It was a rigid, metallic platter coated on both sides with a thin layer of magnetic material to store data. Our data byte could be transferred at a speed of 625 kilobytes per second20 onto the disk. That’s about a 625KB animated gif21 per second.

1981

Source for image: https://en.wikipedia.org/wiki/History_of_the_floppy_disk#/media/File:Floppy_disk_300_dpi.jpg

Let’s fast forward a couple years to 1981 when Sony introduced the first 3 ½ inch floppy drives. Hewlett-Packard was the first adopter of the technology in 1982 with their HP-150. This put the 3 ½ inch floppy disk on the map and gave it wide distribution in the industry22. The disks were single sided with a formatted capacity of 161.2 kilobytes and an unformatted capacity of 218.8 kilobytes. In 1982, the double sided version was made available and the Microfloppy Industry Committee (MIC), a consortium of 23 media companies, based a spec for a 3 ½ inch floppy on Sony’s original designs cementing the format into history as we know it23. Our data byte could now be stored on the early version of one of the most widely distributed storage mediums: the 3 ½ inch floppy disk. Later a couple of 3 ½ inch floppy disks holding the contents of The Oregon Trail would be paramount to my childhood.

1984

Source for image: https://en.wikipedia.org/wiki/CD-ROM#/media/File:CD-ROM.png

Shortly thereafter in 1984, the compact disk read-only memory (CD-ROM), holding 550 megabytes of pre-recorded data, was announced from Sony and Philips. This format grew out of compact disks digital audio, or CD-DAs, which were used for distributing music. The CD-DA was developed by Sony and Philips in 1982, which has a capacity of 74 minutes. When Sony and Philips were negotiating the standard for a CD-DA, legend has it that one of the four people insisted it be able to hold all of the Ninth Symphony24. The first product released on a CD-ROM was Grolier’s Electronic Encyclopedia, which came out in 1985. The encyclopedia contained nine million words which only took up 12% of the disk space available, which was 553 mebibytes25. We would have more than enough room for the encyclopedia and our data byte. Shortly thereafter in 1985, computer and electronics companies worked together to create a standard for the disks so any computer would be able to access the information.

1984 In 1984, Fujio Masuoka invented a new type of floating-gate memory, called flash memory, that was capable of being erased and reprogrammed multiple times.

Let’s go over a bit about floating-gate memory. Transistors are electrical gates that can be switched on and off individually. Since each transistor can be in two distinct states (on or off), it can store two different numbers: 0 and 1. Floating-gate refers to the second gate added to the middle transistor. This second gate is insulated by a thin oxide layer. These transistors use a small voltage, applied to the gate of the transistor, to denote whether it is on or off, which in turn translates to a 0 or 1.

With a floating gate, when a suitable voltage is applied across the oxide layer, the electrons tunnel through it and get stuck on the floating gate. Therefore even if the power is disconnected, the electrons remain present on the floating gate. When no electrons are on the floating gate it represents a 1, and when electrons are trapped on the floating gate it represents a 0. Reversing this process and applying a suitable voltage across the oxide layer in the opposite direction causes the electrons to tunnel off the floating gate and restore the transistor back to its original state. Therefore, the cells are made programmable and non-volatile26. Our data byte could be programmed into the transistors as 01001010, with electrons trapped in the floating gates to represent the zeros.

Masuoka’s design was a bit more affordable but less flexible than electrically erasable PROM (EEPROM) since it required multiple groups of cells to be erased together, but this also accounted for its speed. At the time, Masuoka was working for Toshiba. He ended up quitting Toshiba shortly after to become a professor at Tohoku University because he was displeased with the company not rewarding him for his work. He sued Toshiba, demanding compensation for his work, which settled in 2006 with a one-time payment of ¥87m, equivalent to $758,000. This still seems light given how impactful flash memory has been on the industry.

While we are on the topic of flash memory, we might as well cover the difference between NOR and NAND flash. We know by now from Masuoka that flash stores information in memory cells made up of floating gate transistors. The names of the technologies are tied directly to the way the memory cells are organized.

In NOR flash, individual memory cells are connected in parallel allowing the random access. This architecture enables the short read times required for the random access of microprocessor instructions. NOR Flash is ideal for lower-density applications that are mostly read only. This is why most CPUs load their firmware, typically, from NOR flash. Masuoka and colleagues presented the invention of NOR flash in 1984 and NAND flash in 198727.

In contrast, NAND Flash designers gave up the ability for random access in a tradeoff to gain a smaller memory cell size. This also has the benefits of a smaller chip size and lower cost-per-bit. NAND flash’s architecture consists of an array of eight memory transistors connected in a series. This leads to high storage density, smaller memory cell size, and faster write and erase since it can program blocks of data at a time. This comes at the cost of having to overwrite data when it is not sequentially written and data already exists in a block28.

1991 Let’s jump ahead to 1991 when a prototype solid state disk (SSD) module was made for evaluation by IBM from SanDisk, at the time known as SunDisk29. This design combined a flash storage array, non-volatile memory chips, with an intelligent controller to automatically detect and correct defective cells. The disk was 20 megabytes in a 2 ½ inch form factor and sold for around $1,00030. This wound up being used by IBM in the ThinkPad pen computer31.

1994

Source for image: https://www.amazon.com/Iomega-100MB-Zip-Plus-Drive/dp/B003UI8POM

One of my personal favorite storage mediums from my childhood was Zip Disks. In 1994, Iomega released the Zip Disk, a 100 megabyte cartridge in a 3 ½ inch form factor, roughly a bit thicker than a standard 3 ½ inch disk. Later versions of the disks could store up to 2 gigabytes. These disks had the convenience of being as small as a floppy disk but with the ability to hold a larger amount of data, which made them compelling. Our data byte could be written onto a Zip disk at 1.4 megabytes per second. At the time, a 1.44 megabyte 3 ½ inch floppy would write at about 16 kilobytes per second. In a Zip drive, heads are non-contact read/write and fly above the surface, which is similar to a hard drive but unlike other floppies. Due to reliability problems and the affordability of CDs, Zip disks eventually became obsolete.

1994

Source for image: https://en.wikipedia.org/wiki/CompactFlash#/media/File:CompactFlash_Memory_Card.svg

Also in 1994, SanDisk introduced CompactFlash, which was widely adopted into consumer devices like digital and video cameras. Like CD-ROMs, CompactFlash speed is based on “x”-ratings, such as 8x, 20x, 133x, etc. The maximum transfer rate is calculated based on the original audio CD transfer rate of 150 kilobytes per second. This winds up looking like R = K ⨉ 150 kB/s, where R is the transfer rate and K is the speed rating. So for 133x CompactFlash, our data byte would be written at 133 ⨉ 150 kB/s or around 19,950 kB/s or 19.95 MB/s. The CompactFlash Association was founded in 1995 to create an industry standard for flash-based memory cards32.

1997 A few years later in 1997, the compact disc rewritable (CD-RW) was introduced. This optical disc was used for data storage, as well as backing up and transferring files to various devices. CD-RWs can only be rewritten about 1,000 times, which, at the time, was not a limiting factor since users rarely overwrote data that often on one disc.

CD-RWs are based on phase change technology. During a phase change of a given medium, certain properties of the medium change. In the case of CD-RWs, phase shifts in a special compound, composed of silver, tellurium, and indium, cause “reflecting lands” and “non-reflecting bumps”, each representing a 0 or 1. When the compound is in a crystalline state, it is translucent, which indicates a 1. When the compound is melted into an amorphous state, it becomes opaque and non-reflective, which indicates a 033. We could write our data byte 01001010 as “non-reflecting bumps” and “reflecting lands” this way.

DVDs eventually overtook much of the market share from CD-RWs.

1999 Let’s fast forward to 1999, when IBM introduced the smallest hard drives in the world at the time: the IBM microdrive in 170 MB and 340 MB capacities. These were small hard disks, 1 inch in size, designed to fit into CompactFlash Type II slots. The intent was to create a device to be used like CompactFlash but with more storage capacity. However, these were soon replaced by USB flash drives, covered next, and larger CompactFlash cards once they became available. Like other hard drives, microdrives were mechanical and contained small, spinning disk platters.

2000 A year later in 2000, USB flash drives were introduced. These drives consisted of flash memory encased in a small form factor with a USB interface. Depending on the version of the USB interface used the speed varies. USB 1.1 is limited to 1.5 megabits per second, whereas USB 2.0 can handle 35 megabits per second, and USB 3.0 can handle 625 megabits per second34. The first USB 3.1 type-C drives were announced in March 2015 and had read/write speeds of 530 megabits per second35. Unlike floppy and optical disks, USB devices are harder to scratch but still deliver the same use cases of data storage and transferring and backing up files. Because of this, drives for floppy and optical disks have since faded out of existence in favor of USB ports.

2005

Source for image: https://en.wikipedia.org/wiki/Hard_disk_drive#/media/File:Laptop-hard-drive-exposed.jpg

In 2005, hard disk drive (HDD) manufacturers started shipping products using perpendicular magnetic recording, or PMR. Quite interestingly, this happened at the same time the iPod Nano announced using flash as opposed to the 1 inch hard drives in the iPod Mini, causing a bit of an industry hoohaw36.

A typical hard drive contains one or more rigid disks coated with a magnetically sensitive film consisting of tiny magnetic grains. Data is recorded when a magnetic write-head flies just above the spinning disk, much like a record player and a record except a record needle is in physical contact with the record. As the platters spin, the air in contact with them creates a slight breeze. Just like air on an airplane wing generates lift, the air generates lift on the head’s airfoil37. The write-head rapidly flips the magnetization of one magnetic region of grains so that its magnetic pole points up or down, to denote a 1 or a 0.

The predecessor to PMR was longitudinal magnetic recording, or LMR. PMR can deliver more than three times the storage density of LMR. The key difference of PMR versus LMR is that the grain structure and the magnetic orientation of the stored data of PMR media is columnar instead of longitudinal. PMR has better thermal stability and improved signal-to-noise ratio (SNR) due to better grain separation and uniformity. It also benefits from better writability due to stronger head fields and better magnetic alignment of the media. Like LMR, PMR’s fundamental limitations are based on the thermal stability of magnetically written bits of data and the need to have sufficient SNR to read back written information.

2007 Let’s jump ahead to 2007, when the first 1 TB hard disk drive from Hitachi Global Storage Technologies was announced. The Hitachi Deskstar 7K1000 used five 3.5 inch 200 gigabytes platters and rotated at 7,200 RPM. This is in stark contrast to the world’s first HDD, the IBM RAMAC 350, which had a storage capacity that was approximately 3.75 megabytes. Oh how far we have come in 51 years! But wait, there’s more.

2009 In 2009, technical work was beginning on non-volatile memory express, or NVMe38. Non-volatile memory (NVM) is a type of memory that has persistence, in contrast to volatile memory which needs constant power to retain data. NVMe filled a need for a scalable host controller interface for peripheral component interconnect express (PCIe) based solid state drives39, hence the name NVMe. Over 90 companies were a part of the working group to develop the design. This was all based on prior work to define the non-volatile memory host controller interface specification (NVMHCIS). Opening up a modern server would likely result in finding some NVMe drives. The best NVMe drives today can do about 3,500 megabytes per second read and 3,300 megabytes per second write40. For the data byte we started with, the character j, that is extremely fast compared to a couple of minutes to hand weave rope memory for the Apollo Guidance Computer.

Today and the future Storage class memory (SCM) Now that we have traveled through time a bit (ha!), let’s take a look at the state of the art for storage class memory (SCM) today. SCM, like NVM, is persistent, but SCM goes further by also providing performance better than or comparable to primary memory as well as byte addressability41. SCM aims to address some of the problems faced by caches today such as the low density of static random access memory (SRAM). With dynamic random access memory (DRAM)42, we can get better density, but this comes at a cost of slower access times. DRAM also suffers from requiring constant power to refresh memory. Let’s break this down a bit. Power is required since the electric charge on the capacitors leaks off little by little, meaning without intervention, the data on the chip would soon be lost. To prevent this leakage, DRAM requires an external memory refresh circuit which periodically rewrites the data in the capacitors, restoring them to their original charge.

To solve the problems with density and power leakage, there are a few SCM technologies developing: phase change memory (PCM), spin-transfer torque random access memory (STT-RAM), and resistive random access memory (ReRAM). One thing that is nice about all these technologies is their ability to function as multi-level cells, or MLCs. This means they can store more than one bit of information, compared to single-level cells (SLCs) which can store only one bit per memory cell, or element. Typically, a memory cell consists of one metal-oxide-semiconductor field-effect transistor (MOSFET). MLCs reduce the number of MOSFETs required to store the same amount of data as SLCs, making them more dense or smaller to deliver the same amount of storage as technologies using SLCs. Let’s go over how each of these SCM technologies work.

Phase change memory (PCM) Earlier we went over how phase change works for CD-RWs. PCM is similar. It’s phase change material is typically Ge-Sb-Te, also known as GST, which can exist in two different states: amorphous and crystalline. The amorphous state has a higher resistance, denoting a 0, than the crystalline state denoting a 1. By assigning data values to intermediate resistances, PCM can be used to store multiple states as a MLC43.

Spin-transfer torque random access memory (STT-RAM) STT-RAM consists of two ferromagnetic, permanent magnetic, layers separated by a dielectric, meaning an insulator that can transmit electric force without conduction. It stores bits of data based on differences in magnetic directions. One magnetic layer, called the reference layer, has a fixed magnetic direction while the other magnetic layer, called the free layer, has a magnetic direction that is controlled by passing current. For a 1, the magnetization direction of the two layers are aligned. For a 0, the two layers have opposing magnetic directions.

Resistive random access memory (ReRAM) A ReRAM cell consists of two metal electrodes separated by a metal oxide layer. We can think of this as slightly similar to Masuoka’s original flash memory design, where electrons would tunnel through the oxide layer and get stuck in the floating gate or vice-versa. However, with ReRAM, the state of the cell is determined based on the concentration of oxygen vacancy in the metal oxide layer.

While these technologies are promising, they still have downsides. PCM and STT-RAM have high write latencies. PCMs latencies are ten times that of DRAM, while STT-RAM has ten times the latencies of SRAM. PCM and ReRAM have a limit on write endurance before a hard error occurs, meaning a memory element gets stuck at a particular value44.

In August 2015, Intel announced Optane, their product build on 3DXPoint, pronounced 3D cross-point45. Optane claims performance 1,000 faster than NAND SSDs with 1,000 times the performance, while being four to five times the price of flash memory. Optane is proof that storage class memory is not just experimental. It will be interesting to watch how these technologies evolve.

Hard disk drives (HDDs) Helium hard disk drive (HHDD) A helium drive is a high capacity hard disk drive (HDD) that is helium-filled and hermetically sealed during manufacturing. Like other hard disks, as we covered earlier, it looks much like a record player with a magnetic-coated platter rotating. Typical hard disk drives would just have air inside the cavity, however that air is causing an amount of drag on the spin of the platters.

Helium balloons float so we know helium is lighter than air. Helium is, in fact, 1/7th the density of air, therefore reducing the amount of drag on the spin of the platters, causing a reduction in the amount of energy required for the disks to spin. However, this was actually a secondary feature, the primary feature of helium was to allow for packing 7 platters in the same form factor that would typically only hold 5. When trying to attempt this with air filled drives, it would cause turbulence. If we remember back to our airplane wing analogy from earlier this ties in perfectly. Since helium reduces drag, this eliminates the turbulence.

What we also know about balloons is that after a few days, helium filled balloons start to sink because helium is escaping the balloons. The same could be said for these drives. It took years before manufacturers had created a container that prevented the helium from escaping the form factor for the life of the drive. Backblaze experimented and found that while helium hard drives had a lower annualized error rate of 1.03%, while standard hard drives resulted in 1.06%. Of course, that is so small a difference it is hard to conclude much from it46.

A helium filled form factor can have a hard disk drive encapsulated that uses PMR, which we went over above, or could contain a microwave-assisted magnetic recording (MAMR) or heat-assisted magnetic recording (HAMR) drive. You can pair any magnetic storage technology with helium instead of air. In 2014, HGST combined two cutting edge technologies into their 10TB helium hard disk that used host-managed shingled magnetic recording, or SMR. Let’s go over a bit about SMR then we can cover MAMR and HAMR.

Shingled magnetic recording (SMR) We went over perpendicular magnetic recording (PMR) earlier which was SMR’s predecessor. In contrast to PMR, SMR writes new tracks that overlap part of the previously written magnetic track, which in turn makes the previous track narrower, allowing for higher track density. The technology’s namesake stems from the fact that the overlapping tracks are much like that of roof shingles.

SMR results in a much more complex writing process since writing to one track winds up overwriting an adjacent track. This doesn’t come into play when a disk platter is empty and data is sequential. But once you are writing to a series of tracks that already contain data, this process is destructive to existing adjacent data. If an adjacent track contains valid data it must be rewritten. This is quite similar to NAND flash as we covered earlier.

Device-managed SMR devices hide this complexity by having the device firmware manage it resulting in an interface like any other hard disk you might encounter. On the other hand, host-managed SMR devices rely on the operating system to know how to handle the complexity of the drive.

Seagate started shipping SMR drives in 2013 claiming a 25% greater density than PMR47.

Microwave-assisted magnetic recording (MAMR) MAMR is an energy-assisted magnetic storage technology, like HAMR which we will cover next, that uses 20-40GHz frequencies to bombard the disk platter with a circular microwave field, lowering the its coercivity, meaning the platter has a lower resistance of its magnetic material to changes in magnetization. We learned above that changes in magnetization of a region of the platter are used to denote a 0 or a 1 so this allows the data to be written much more densely on the disk since it has a lower resistance to changes in magnetization. The core of this new technology is the spin torque oscillator used to generate the microwave field without sacrificing reliability.

Western Digital, also known as WD, unveiled this technology in 201748. Toshiba followed shortly after in 201849. While WD and Toshiba are busy pursuing MAMR, Seagate is betting on HAMR.

Heat-assisted magnetic recording (HAMR) HAMR is an energy-assisted magnetic storage technology for greatly increasing the amount of data that can be stored on a magnetic device, such as a hard disk drive, by using heat delivered by a laser to help write data onto the surface of a hard disk platter. The heat causes the data bits to be much closer together on the disk platter, which allows greater data density and capacity.

This technology is quite difficult to achieve. A 200mW laser heats a teeny area of the region to 750 °F (400 °C) quickly before writing the data, while also not interfering with or corrupting the rest of the data on the disk50. The process of heating, writing the data, and cooling must be completed in less than a nanosecond. These challenges required the development of nano-scale surface plasmons, also known as a surface guided laser, instead of direct laser-based heating, as well as new types of glass platters and heat-control coatings to tolerate rapid spot-heating without damaging the recording head or any nearby data, and various other technical challenges that needed to be overcome51.

Seagate first demonstrated this technology, despite many skeptics, in 201352. They started shipping the first drives in 201853.

End of tape, rewind We started this article in 1951 and are concluding after looking at the future of storage technology. Storage has changed a lot over time, from paper tape, to metal tape, magnetic tape, rope memory, spinning disks, optical disks, flash, and others. Progress has led to faster, smaller, and more performant devices for storing data.

If we compare NVMe to the 1951 UNISERVO metal tape, NVMe can read 486,111% more digits per second. If we compare NVMe to my childhood favorite in 1994, Zip disks, NVMe can read 213,623% more digits per second.

One thing that remains true is the storing of 0s and 1s. The means by which we do that vary greatly. I hope the next time you burn a CD-RW with a mix of songs for a friend, or store home videos in an Optical Disc Archive54, you think about how the non-reflective bumps translate to a 0 and the reflective lands of the disk translate to a 1. Or if you are creating a mixtape on a cassette, remember that those are very closely related to the Datasette used in the Commodore PET. Lastly, remember to be kind and rewind55.

Thank you to Robert Mustacchi and Rick Altherr for tidbits (I can’t help myself) throughout this article!

1. http://archive.computerhistory.org/resources/text/IBM/Stretch/pdfs/06-07/102632284.pdf [return]
2. https://archive.org/stream/byte-magazine-1977-02/1977_02_BYTE_02-02_Usable_Systems#page/n145/mode/2up [return]
3. https://web.archive.org/web/20170403130829/http://www.bobbemer.com/BYTE.HTM [return]
4. https://www.computerhistory.org/storageengine/tape-unit-developed-for-data-storage/ [return]
5. https://www.ibm.com/ibm/history/exhibits/701/701_1415bx26.html [return]
6. https://www.ibm.com/ibm/history/exhibits/storage/storage_fifty.html [return]
7. https://spectrum.ieee.org/computing/hardware/why-the-future-of-data-storage-is-still-magnetic-tape [return]
8. https://www.ibm.com/ibm/history/exhibits/650/650_pr2.html [return]
9. https://www.youtube.com/watch?v=zOD1umMX2s8 [return]
10. https://www.ibm.com/ibm/history/ibm100/us/en/icons/ramac/ [return]
11. https://www.pdp8.net/tu56/tu56.shtml [return]
12. https://www.ibm.com/ibm/history/ibm100/us/en/icons/floppy/ [return]
13. https://archive.org/details/ibms360early370s0000pugh/page/513 [return]
14. https://web.archive.org/web/20100707221048/http://archive.computerhistory.org/resources/access/text/Oral_History/102657926.05.01.acc.pdf [return]
15. https://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/apollo/public/visual3.htm [return]
16. http://wav-prg.sourceforge.net/tape.html [return]
17. https://www.c64-wiki.com/wiki/Datassette [return]
18. You will be rick rolled by a still photo. [return]
19. https://tools.ietf.org/html/rfc4856#page-17 [return]
20. https://www.pcmag.com/encyclopedia/term/st506#:~:text=ST506,using%20the%20MFM%20encoding%20method. [return]
21. You will be rick rolled. [return]
22. https://www.jstor.org/stable/24530873?seq=1 [return]
23. https://www.americanradiohistory.com/hd2/IDX-Consumer/Archive-Byte-IDX/IDX/80s/82-83/Byte-1983-09-OCR-Page-0169.pdf [return]
24. https://www.wired.com/2010/12/1216beethoven-birthday-cd-length/ [return]
25. https://books.google.co.uk/books?id=RTwQAQAAMAAJ [return]
26. https://www.economist.com/technology-quarterly/2006/03/11/not-just-a-flash-in-the-pan [return]
27. https://ieeexplore.ieee.org/document/1487443 [return]
28. http://aturing.umcs.maine.edu/~meadow/courses/cos335/Toshiba%20NAND_vs_NOR_Flash_Memory_Technology_Overviewt.pdf [return]
29. https://www.computerhistory.org/storageengine/solid-state-drive-module-demonstrated/ [return]
30. http://meseec.ce.rit.edu/551-projects/spring2017/2-6.pdf [return]
31. https://www.westerndigital.com/company/innovations/history [return]
32. https://www.compactflash.org/ [return]
33. https://computer.howstuffworks.com/cd-burner8.htm [return]
34. https://www.diffen.com/difference/USB_1.0_vs_USB_2.0 [return]
35. https://web.archive.org/web/20161220102924/http://www.usb.org/developers/presentations/USB_DevDays_Hong_Kong_2016_-_USB_Type-C.pdf [return]
36. https://www.eetimes.com/hard-drives-go-perpendicular/# [return]
37. https://books.google.com/books?id=S90OaKQ-IzMC&pg=PA590&lpg=PA590&dq=heads+disk+airfoils&source=bl&ots=7VVuhw6mgm&sig=ACfU3U0PXCehcs7dKI5IhDGbRMZvqsgeHg&hl=en&sa=X&ved=2ahUKEwi82fm9_onoAhUIr54KHR6-BtUQ6AEwAHoECAgQAQ#v=onepage&q=heads%20disk%20airfoils&f=false [return]
38. https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130813_A12_Onufryk.pdf [return]
39. https://nvmexpress.org/wp-content/uploads/2013/04/NVM_whitepaper.pdf [return]
40. https://www.pcgamer.com/best-nvme-ssd/ [return]
41. https://ieeexplore.ieee.org/document/5388605 [return]
42. https://arxiv.org/pdf/1909.12221.pdf [return]
43. https://ieeexplore.ieee.org/document/5388621 [return]
44. https://arxiv.org/pdf/1909.12221.pdf [return]
45. https://www.anandtech.com/show/9541/intel-announces-optane-storage-brand-for-3d-xpoint-products [return]
46. https://www.backblaze.com/blog/helium-filled-hard-drive-failure-rates/ [return]
47. https://www.anandtech.com/show/7290/seagate-to-ship-5tb-hdd-in-2014-using-shingled-magnetic-recording/ [return]
48. https://www.storagereview.com/news/wd-unveils-its-microwave-assisted-magnetic-recording-technology [return]
49. https://www.theregister.co.uk/2018/12/07/toshiba_goes_to_mamr/ [return]
50. https://fstoppers.com/originals/hamr-and-mamr-technologies-will-unlock-hard-drive-capacity-year-326328 [return]
51. https://www.seagate.com/www-content/ti-dm/tech-insights/en-us/docs/TP707-1-1712US_HAMR.pdf [return]
52. https://www.computerworld.com/article/2485341/seagate--tdk-show-off-hamr-to-jam-more-data-into-hard-drives.html [return]
53. https://blog.seagate.com/craftsman-ship/hamr-next-leap-forward-now/ [return]
54. Yup, you heard that right: https://pro.sony/en_GB/products/optical-disc. [return]
55. This is a tribute to Blockbuster but there are still open formats for using tape today: https://en.wikipedia.org/wiki/Linear_Tape-Open. [return]

View Details

When you upload photos to Instagram, back up your phone to “the cloud”, send an email through GMail, or save a document in a storage application like Dropbox or Google Drive, your data is being saved in a data center. These data centers are airplane hangar-sized warehouses, packed to the brim with racks of servers and cooling mechanisms. Depending on the application you are using you are likely hitting one of Facebook’s, Google’s, Amazon’s, or Microsoft’s data centers. Aside from those major players, which we will call the “hyperscalers”, many other companies run their own data centers or rent space from a colocation center to house their server racks.

Most of the hyperscalers have made massive strides to get a “carbon neutral” footprint for their data centers. Google, Amazon, and Microsoft have all pledged to decarbonize completely, however none has succeeded in completely ditching fossil fuels as of yet. If a company claims to be “carbon neutral” it means they are offsetting their use of fossil fuels with renewable energy credits, also known as RECs1. A REC represents one megawatt-hour (MWh) of electricity that is generated and delivered to the electricity grid from a renewable energy resource such as solar or wind power. Essentially by purchasing RECs, “carbon neutral” companies are giving back clean energy to prevent someone else from emitting carbon. Most companies become “carbon neutral” by investing in offsets that primarily avoid emissions, such as paying folks to not cut down trees or buying RECs. These offsets do not actually remove the carbon that they are emitting.

A “net zero” company actually has to remove as much carbon as it emits. This is referred to as “net zero” since a company is still creating carbon emissions, however their emissions are equal to the amount of carbon removed. This differs from “carbon neutral” since a “carbon neutral” company takes a look at their carbon footprint and has to prevent enough other folks from emitting that much carbon, through RECs or otherwise. Whereas a “net zero” company has to find a way to remove the amount of carbon they emit.

Lastly, if a company calls themselves “carbon negative” it means they are removing more carbon than they emit each year. This should be the gold standard for how companies operate. None of the FAANG (Facebook, Apple, Amazon, Netflix and Google)2 today claim to be “carbon negative”, but Microsoft issued a press release stating they are going to be carbon negative by 20303.

Power usage efficiency, also known as PUE, is defined as the total energy required to power a data center (including lights and cooling) divided by the energy used for servers. 1.0 would be perfect PUE since 100% of electricity consumption is used on computation. Conventional data centers have a PUE of about 2.0, while hyperscalers have gotten theirs down to about 1.2. According to a 2019 study from the Uptime Institute, which surveyed 1,600 data centers, the average PUE was 1.674.

PUE as a method of measurement is a point of contention. PUE does not account for the location of a data center, which means a data center that is located in a part of the world that can benefit from free cooling from outside air will have a lower PUE than one in a very hot temperature climate. It is most ideal to measure PUE as an annual average since seasons change and affect the cooling needs of a data center over the course of a year. According to a study from the University of Leeds, “comparing a PUE value of data centres is somewhat meaningless unless it is known whether it is operating at full capacity or not.”5

Google claims a PUE of 1.1 on average, yearly, for all its data centers, while individually, some are as low as 1.086. One of the actions Google has taken for lowering their PUE is using machine learning to cool data centers with inputs from local weather and other factors7, such as if the weather outside is cool enough they can use it without modification as free cold air. They can also predict wind farm output up to 36 hours in advance8. Google took all the data they had from sensors in their facilities monitoring temperature, power, pressure, and other resources to create neural networks to predict future PUE, temperature, and pressure in their data centers. This way they can automate and recommend actions for keeping their data centers operating efficiently from the predictions9. Google also sets the temperature of its data centers to 80°F, versus the usual 68-70°F, saving a lot of power for cooling. Weather local to the data center is a huge factor. For example, Google’s Singapore data center has the highest PUE and is the least efficient of its sites because Singapore is hot and humid year-round.

Wired conducted an analysis10 of how Google, Microsoft, and Amazon stack up when it comes to the carbon footprint of their data centers. Google claims to be “net zero” for carbon emissions11 and also publishes a transparency report of their PUE every year12. While Microsoft claims to be “carbon negative” by 2030, they are still “carbon neutral” today13. They also claim to be pursuing 100% renewable energy by 2025.

On the other hand, Amazon is in the worst position of large tech companies when it comes to carbon footprints. As we went over above, the location of the data center matters and some Amazon regions might be greener than others due to the weather conditions in those areas or having more access to solar or wind energy14. Bezos has pledged to get to “net zero” by 204015. Greenpeace seems to believe otherwise, claiming that Amazon is not dedicated to that pledge since its Virginia data centers were only at 12% renewable energy16. It’s hard to know, of course, until 2040 comes and either Amazon succeeds in their pledge or doesn’t.

In 2018, Apple claimed 100% of their energy was from renewable sources17. Facebook claims they will be at 100% renewable energy by the end of 202018. While US companies have followed suit on pledging to lower their carbon footprint, Chinese Internet giants such as Baidu, Tencent, and Alibaba have not.

What is using power in a data center? According to a study from Procedia Environmental Sciences, 48% of power in a data center goes to equipment like servers and racks, 33% to heating, ventilation, and air conditioning (HVAC), 8% to uninterrupted power supply (UPS) losses, 3% lighting, and 10% to everything else19.

HVAC for data centers is a delicate process of making sure hot air from server exhaust doesn’t mix with cool air and raise the temperature of the entire data center. This is why most data centers have hot and cold aisles. The goal is to have the cold air flow into one side of racks while the hot air exhaust comes out the other side of the racks. Optimizing air flow throughout your racks and servers is essential for helping with HVAC efficiency.

Power comes off the grid as AC power. This can be single-phase power which has two wires, a power wire and a neutral wire, or three-phase power which has three wires, each 120 electrical degrees out of phase with each other. The key difference being that three-phase can handle higher loads than single-phase. The frequency of the power off the grid can be either 50 or 60Hz. Voltage is any of: 208, 240, 277, 400, 415, 480, or 600V20.

Since most equipment in a data center uses DC power, the AC power needs to be converted which results in power losses and wasted energy adding up to around 21-27% of power. Let’s break this down. There is a 2% loss when utility medium voltage, defined as voltage greater than 1000V and less than 100 kV, is transformed to 480VAC. There is a 6-12% loss within a centralized UPS due to conversions from AC-to-DC and DC back to AC. There is a 3% power loss at the power distribution unit (PDU) level due to the transformation from 480VAC to 208VAC. Standard power supplies for servers convert 208VAC to the required DC voltage resulting in a 10% loss, assuming the power supply is 90% efficient21. This is all to say that power is wasted all throughout traditional data centers in transformations and conversions.

To try to lessen the amount of wasted power from conversions, some folks rely on high-voltage DC power distribution. Lawrence Berkeley National Labs conducted a study in 200822 comparing the use of 380VDC power distribution for a facility to a traditional 480VAC power distribution system. The results showed that the facility using DC power eliminated multiple conversion stages to result in a 7% decrease in energy consumption compared to a typical facility with AC power distribution. However, this is rarely done at hyperscale. Hyperscalers tend to have three-phase AC to the rack, then convert to DC at the rack or server level.

More power efficient compute Other than RECs and using 100% renewable energy, there are other ways hyperscalers have made their data centers more power efficient. In 2011, the Open Compute Project started out of a basement lab in Facebook’s Palo Alto headquarters23. Their mission was to design from a clean slate the most efficient and economical way to run compute at scale. This led to using a 480VAC24 electrical distribution system to reduce energy loss, removing anything in their servers that didn’t contribute to efficiency, reusing hot aisle air in winter to heat the offices and the outside air flowing into the data center, and removing the need for a central power supply. The Facebook team went ahead and installed the newly designed servers in their Prineville data center which resulted in 38% less energy to do the same work as their existing data centers. It also cost 24% less.

Let’s dive into some of the details of the Open Compute designs that allow for power efficiency. The Open Rack design25 includes a power bus bar with either 12VDC or 48VDC of distributed power to the nodes. The bus bar runs along the back of the rack vertically. It transmits power from the rack level power supply units (PSUs) to the servers in the rack. The bus bar allows the servers to plug in directly to the rack for power so when you are servicing an Open Rack you do not need to unplug power cords, you can just pull out the server from the front of the rack. With the Open Compute designs, network connections to servers are at the front of the rack so the technician never has to go to the back of the rack, i.e. the hot aisle.

Redundancy Conventional servers have PSUs in every server. The Open Rack design has centralized PSUs for the rack, which allow for N+M redundancy for the rack, the most common deployment being N+1 redundancy. This means there is an extra PSU per rack of servers. In a conventional system this would be 1+1 since there is one extra PSU in every individual server. By keeping the PSUs centralized to the rack, this results in a reduction in power converting components which increases the efficiency of the system.

Right-sized PSUs Server designers tend to choose PSUs that have enough headroom to deliver power for the maximum configuration. Server vendors would rather carry a small number of power supply SKUs that are oversized, than carry a large number of power supply SKUs that are right-sized to purpose since the economies of scale prefer the former. This leads to an oversizing factor of at least 2-3 times26 the required capacity for conventional power supplies. In comparison, a rack-level PSU will be less oversized since it is right-sized for purpose. The hyperscalers also have the advantage of economies of scale for their hardware. The typical Open Rack compliant power supply is oversized at only 1.2 times the required capacity, if even that.

Optimal efficiency Every power supply has a sweet spot for load versus efficiency. 80 Plus is a certification program for PSUs to measure efficiency. There are a few different grades: Bronze, Silver, Gold, Platinum, and Titanium. The most power efficient grade of the 80 Plus standard is Titanium. The most common grade of PSU used in data centers is 80 Plus Silver, which has a maximum efficiency of 88%. This means it wastes 12% electric energy as heat at the various load levels. In comparison, the 12V and 48VDC PSUs have data showing maximum efficiencies at 95%27 and 98%28, respectively. This means the rack-level PSUs only waste between 5 and 2% of energy.

While the efficiency of the rack-level PSU is important, we still need to weigh the cost of the number of conversions being made to get the power to each server. For every unnecessary power conversion, you are paying an efficiency cost. For example with a 48VDC rack-level power supply, the server might need to convert the rack provided 48VDC to 12VDC then that 12VDC to VCORE. VCORE is the voltage supplied to the CPU, GPU, or other processing core. With Google’s 48VDC power supply, they advocate for using 48V to point of load (PoL)29 to deliver power to the servers. This means placing a DC-to-DC or linear power supply regulator going from the rack-level PSU to the server which would reduce the number of conversions needed to get the power to the processing cores. However, the 48VDC to DC regulators required for Google’s implementation are not common and come at a premium cost. It is likely their motivation for opening the specs for the 48VDC rack is to drive more volume to those parts and drive down costs. In contrast, 12VDC to DC regulators are quite common and low cost.

Reading a power efficiency graph Below is an example of a power efficiency graph for a power supply.

Source for image: https://e2e.ti.com/blogs_/b/industrial_strength/archive/2019/04/04/three-considerations-for-achieving-high-efficiency-and-reliability-in-industrial-ac-dc-power-supplies

We can see the peak of the graph is where the PSU is the most efficient. We divide the output power by the input power to calculate efficiency. The x-axis of the graph measures the load of the power supply in Watts, while the y-axis measures efficiency.

Let’s go through an example of choosing the right power supply for the load. For the example graph above if we know our peak load is 120W and idle is 60W, this power supply would be more than we need since it can handle up to 150W. At our peak load of 120W with 230VAC, this power supply would have a maximum efficiency of around 94% and a minimum efficiency at idle of around 92% with 230VAC. We now know the losses of this specific power supply and can compare it to other supplies to see if they are more efficient for our load.

Open Compute servers without a bus bar Not all Open Compute servers include a power bus bar. Microsoft’s Olympus servers require on AC power30. The Olympus power supply has three 340W power supply modules, one for each phase, with a total maximum output of 1000W31. Therefore, these power supplies assume all deployments are three-phase power. The minimum efficiency of the PSU is 89-94% depending on the load32. This places the grade of the Olympus power supply around an 80 Plus Platinum33.

Like all technical decisions, using per server AC power supplies versus rack level DC is a trade-off. By having separate power supplies, different workloads can balance the power they are consuming individually rather than at a rack level. In turn though, Microsoft needs to build and manufacture multiple power supplies to ensure they are right-sized to run at maximum efficiency for each server configuration. Serviceability also requires technicians to unplug power cables and go to the back of the rack.

At the time Microsoft made the decision to use individual AC power supplies per server, the Open Rack design was at v1 not v2 like it is today, the cost of the copper for the power bus bar was higher and the loss of efficiency to resistance was a factor. The Open Rack v1 design had an efficiency concern with the power loss due to heating the copper in the bus bar. If a rack holds 24kW of equipment, a 12VDC power bus bar must deliver 2kA of current. This requires a very thick piece of copper which has an amount of power loss that is not insignificant due to resistance in the bus bar.

Let’s break down how to measure the relationship of power to resistance. Ohm’s law declares electric current (I) is proportional to voltage (V) and inversely proportional to resistance (R), so V=IR. To see the relationship of power to resistance, we combine Ohm’s law (V=IR) with P = IV, which translates to power (P) is the product of current (I) and voltage (V). Substituting I = V/R gives P = (V/R)V = V2/R. Then, substituting V = IR gives P = I(IR) = I2R. So P = I2R is how we can calculate the power loss due to resistance in the bus bar.

For their decision, Microsoft balanced the conversion efficiency against the material cost of the bus bar and the resistive loss. However, Open Rack v2 changes the trade-offs of their original decision. With a 48VDC bus bar, a rack that holds 24kW of equipment only requires 500A, as opposed to the 2kA required by the 12VDC power bus bar from the v1 spec. This translates into a much cheaper bus bar and lower losses due to resistance. The bus bar still has more loss than 208VAC cables but there is an improved efficiency from the power supply unit at the rack-level, which makes it compelling. However, as we stated earlier you need to be mindful of the number of conversions getting the power to the components on the motherboard. If your existing equipment is 12VDC, you would want to avoid any extra conversions using that with a 48VDC bus bar. Save the 48VDC bus bar for new equipment that has 48V to point of load to avoid any extra conversions.

The main difference between Microsoft’s design with individual power supplies and the 24VDC and 48VDC Open Rack designs is the way the initial set of power is delivered to the servers. For Microsoft’s design, they distribute three-phase power to the servers individually through power supplies while the 24VDC and 48VDC power bus bar distributes the power delivery to the servers. Once power is delivered to the server, the power is sent through typically a DC-to-DC power supply regulator which in turn powers the components on the motherboard. This step is shared whether the power is coming from a single power bus bar or individual power supplies.

There is another interesting bit that comes into play with uninterrupted power supplies (UPSes). We talked a bit earlier about the losses in efficiency due to UPSes. Let’s go over a bit about what this means in terms of a DC bus bar or individual AC PSUs. When AC power is going into each individual server you have two choices: a UPS on the AC before it gets distributed to the individual servers or a UPS per server integrated into each server’s PSU. Deploying and servicing individual batteries per server is a nightmare for maintenance. Because of this, most facilities that use AC power to the servers wind up using rack-wide or building-wide UPSes. Since the batteries in a UPS are DC, an AC UPS has an AC-to-DC converter for charging the batteries and a DC-to-AC inverter to provide AC power from the battery. For online UPSes, meaning the battery is always connected, this requires two extra conversions from AC-to-DC and DC back to AC with power efficiency losses for both.

With a DC rack-level design, battery packs can be attached directly to the bus bar. The rack-level PSUs are the first AC-to-DC conversion state so there is not a need for another conversion since everything from there runs on DC. The downside is that the rack-level PSU needs to adjust the voltage level to act as a battery charger. This means the servers need to accept a fairly wide tolerance on the 48V target, around +/-10V so 40-56V isn’t unreasonable. Because DC-to-DC converters are fairly tolerant about input voltage ranges, this is fairly straightforward to deal with without any significant loss in power efficiency. It’s important to note that for hyperscalers UPSes are only present to allow for a generator to kick in which is a few seconds versus around 10-15 minutes for a traditional data center.

With commodity servers, like Dell34 or Supermicro35, the cost of individual power supplies is much higher on power efficiency since those PSUs are not as high of a grade and have much more oversizing. They also tend to lack power supply regulators that minimize power conversion losses in supplying power to the components on the board. This would lead to around an 8-12% gain in power efficiency by moving from a bunch of commodity servers in a rack to an OCP design. Not to mention, the serviceability ease of the bus bar would benefit technicians as well.

By designing rack level architectures, huge improvements can be made for power efficiency over conventional servers since PSUs will be less oversized, more consolidated, and redundant for the rack versus per server. While the hyperscalers have benefitted from these gains in power efficiency, most of the industry is still waiting. The Open Compute project was started as an effort to allow other companies running data centers to benefit from the power efficiencies as well. If more organizations run rack-scale architectures in their data centers, we can lessen the wasted carbon emissions caused by conventional servers.

Huge thanks to Rick Altherr, Amir Michael, Kenneth Finnegan, Arjen Roodselaar, and Scott Andreas for their help with the nuances in this article.

1. https://www.epa.gov/greenpower/renewable-energy-certificates-recs [return]
2. https://en.wikipedia.org/wiki/Facebook,_Apple,_Amazon,_Netflix_and_Google [return]
3. https://blogs.microsoft.com/blog/2020/01/16/microsoft-will-be-carbon-negative-by-2030/ [return]
4. https://uptimeinstitute.com/resources/asset/2019-data-center-industry-survey [return]
5. http://eprints.whiterose.ac.uk/79352/1/GBrady%20Case%20Study%20of%20PUE.pdf [return]
6. https://www.google.com/about/datacenters/efficiency/ [return]
7. https://deepmind.com/blog/article/safety-first-ai-autonomous-data-centre-cooling-and-industrial-control [return]
8. https://deepmind.com/blog/article/machine-learning-can-boost-value-wind-energy [return]
9. https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40 [return]
10. https://www.wired.com/story/amazon-google-microsoft-green-clouds-and-hyperscale-data-centers/ [return]
11. https://cloud.google.com/blog/topics/google-cloud-next/our-heads-in-the-cloud-but-were-keeping-the-earth-in-mind [return]
12. https://www.google.com/about/datacenters/efficiency/ [return]
13. https://www.microsoft.com/en-us/corporate-responsibility/sustainability/operations [return]
14. https://aws.amazon.com/about-aws/sustainability/ [return]
15. https://blog.aboutamazon.com/sustainability/the-climate-pledge [return]
16. https://www.greenpeace.org/usa/news/greenpeace-finds-amazon-breaking-commitment-to-power-cloud-with-100-renewable-energy/ [return]
17. https://www.apple.com/newsroom/2018/04/apple-now-globally-powered-by-100-percent-renewable-energy/ [return]
18. https://about.fb.com/news/2018/08/renewable-energy/ [return]
19. https://www.sciencedirect.com/science/article/pii/S1878029617300956 [return]
20. https://ctlsys.com/support/electrical_service_types_and_voltages/ [return]
21. https://www.facebook.com/notes/facebook-engineering/designing-a-very-efficient-data-center/10150148003778920/ [return]
22. https://eln.lbl.gov/sites/default/files/lbnl-2001006.pdf [return]
23. https://www.facebook.com/notes/facebook-engineering/building-efficient-data-centers-with-the-open-compute-project/10150144039563920/ [return]
24. https://www.facebook.com/notes/facebook-engineering/designing-a-very-efficient-data-center/10150148003778920/ [return]
25. https://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns [return]
26. https://blog.se.com/datacenter/2018/05/24/12v-vs-48v-the-rack-power-architecture-efficiency-calculator-illustrates-energy-savings-of-ocp-style-psus/ [return]
27. https://www.opencompute.org/files/External-2018-OCP-Summit-Google-48V-Update-Flatbed-and-STC-20180321.pdf [return]
28. http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Plenary/APEC%20Plenary%20Google.pdf?ver=2017-04-24-091315-930×tamp=1495563027516 [return]
29. http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Plenary/APEC%20Plenary%20Google.pdf?ver=2017-04-24-091315-930×tamp=1495563027516 [return]
30. https://www.datacenterdynamics.com/en/news/dcd-zettastructure-why-project-olympus-relies-on-ac-power/ [return]
31. http://files.opencompute.org/oc/public.php?service=files&t=2247ac812c026ea8fa15d29622779fa7&download [return]
32. http://files.opencompute.org/oc/public.php?service=files&t=2247ac812c026ea8fa15d29622779fa7&download [return]
33. https://en.wikipedia.org/wiki/80_Plus [return]
34. https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/power-and-cooling-innovations_030216.pdf [return]
35. https://www.supermicro.com/en/support/resources/pws [return]

View Details

I had a lot of fun writing blog posts in the past about my home lab and some of my personal infrastructure so I thought I would do the same as we built out our office. Much like moving into a new place, the first thing I always plan to have setup on move-in day is internet. We did the same with our office as well. Before we even had any real furniture, we made sure that we had a network connection.

You may recognize that furniture from the garage.

For the office I really wanted our network infrastructure to be off-the-charts good. Everyone knows shitty internet is a productivity killer. Since I use UniFi for my network setup at home, we used the same for the office.

Here’s what we got:

• 48 Port Switch
• Dream Machine
• 2 Wifi APs
• A couple of cameras
• Amplifi Alien

The Switch We have a hard line coming from the 48 port switch to each section of desks. I cabled all this myself. As we grow we will likely segment this off to each desk having its own little 4- or 8-port network switch but for now this works.

The Cables All the cables running to the desks are Cat7s from Monoprice. Every type of cable has a maximum distance. For ethernet cables, the maximum distance is the maximum upload/download speed. Cat7 gets praised for its 100 Gbps speed, but that will only work for distances up to 15 meters (slightly over 49 feet). From 15 meters up to 50 meters, a Cat7 cable downgrades to 40 Gbps. Beyond that, it drops to the same 10 Gbps speed of Cat6 and Cat6a, however it still retains its superior 600 Mhz bandwidth. We use 100ft cables to the desks and 50ft cables wherever we can reach to maximize speed.

The Router The Dream Machine is acting as our gateway and controller. Before we got the other 2 APs, it was our only access point and did a great job of that.

The Access Points We have a large warehouse with a lot of square feet, while the Dream Machine does have coverage to every corner, it’s nice to have a strong signal from anywhere in the office. As we grow we will have more and more devices on our network, so having some other APs to handle that load is necessary.

The Cameras The cameras will be installed outside our office so we can see what is going on when we are not there. This is mainly for security.

The Isolated Network Router Lastly, is the AmpliFi Alien. Since AmpliFi is not a part of the rest of the UniFi fleet, it exposes its own network. I foresee this becoming the network for our lab equipment or anything we don’t want on the main network. It’s a very, very nice secondary network that is fully isolated with Wi-Fi 6 capabilities and a max speed of 4804 Mbps. If only all devices supported Wi-Fi 6!

It’s been fun to build out the network infrastructure in our office and make sure it scales while we scale out the team. We have hired some of the brightest folks that I am happy to call coworkers. This is just one very small detail of our startup journey, but I am glad I got to share it. Our previously empty office is now one with 20 desks, 2 kitchen tables and a large, cozy couch area with whiteboards for brainstorming. Can’t wait to see what the future brings!

View Details

WE STARTED A COMPUTER COMPANY!! You have no idea how long I’ve been waiting to say that! I guess some context would help… Steve Tuck, Bryan Cantrill, and I officially started the Oxide Computer Company. Since then, we’ve been working on closing up fundraising, getting an awesome office, and hiring!

You are probably thinking “a computer company? that’s outrageous!”.. well it is and it isn’t. Over the last year, I had the opportunity to spend a lot of time talking with folks who are currently running workloads on premises. The consensus from all my conversations has been that everyone setting up infrastructure themselves is in a great deal of pain and they have been largely neglected by any existing vendor. All these folks have very good reasons for running on premises that include security, strategic reasons like latency, specialized workloads, or the reality that the unit economics of running at their scale in the cloud are unsustainable.

I’ve had the privilege of working on projects in my career that have had a positive impact on a lot of people – from Docker, to the Go programming language, to Kubernetes. Over the course of talking to folks, I soon realized that working on solving the pain for those running on premises would have a huge amount of impact. Hyperscalers like Facebook, Google, and Microsoft have what I like to call “infrastructure privilege” since they long ago decided they could build their own hardware and software to fulfill their needs better than commodity vendors. We are working to bring that same infrastructure privilege to everyone else! This leads to better integration between the hardware and software stacks, better power distribution, and better density. It’s even better for the environment due to the energy consumption wins!

I can’t think of a better problem space and group of folks to work with to build a company. I’ve known both Bryan and Steve from the container conference circuit and couldn’t pass up an opportunity to work with both of them. I’ve also been in love with computers since I was a child… as proof of this (not that it’s needed) here is a very early diary entry of mine.

In typical cliche computer company fashion we have been working out of my garage. Bryan brought over his collection of computer manuals and even added to my collection of floppy disks. It’s basically been a computer nerd’s nirvana!

Since we now have funding, we will be moving to a bigger space. I can’t wait; I fell in love the minute we saw it. It’s perfect for a nascent computer company to grow.

We will still be using the original garage to record episodes of On the Metal, our podcast. You will not want to miss episodes of that! We have been lucky to have some amazing conversations with technologists, the first being Jeff Rothschild, and I can’t wait to share them with you!

The past few months have been some of the most fun and rewarding of my entire career and we are only getting started. If you want to read more about some of the deep technical problems we will be solving check out my ACM Queue articles: Open Source Firmware and Opening up the Baseboard Management Controller.

We would love to have you join us in our mission to “Kick butt, have fun, don’t cheat, love our customers, change computing forever” – head over to our careers page or if you are a designer who codes send us a pull request! If you are currently running on premises and are interested in what we are building, join our mailing list and I will be in touch!

Thank you to all of our friends and family for their support of our endeavor. I, personally, could not have done this without your advice, guidance, and positivity during our fundraise and after. I’ve been waiting for the day I could share with everyone what we’ve been up to and I am beyond excited to build a company and a product people will love! Stay tuned!

View Details

Last week I attended the Open Source Firmware Conference. It was amazing! The talks, people, and overall feel of the conference really left me feeling inspired and lucky to attend.

Having been pushed to attend vendor conferences and trade shows through my career for various jobs, it was so refreshing to have the chance to hang out with folks from such a genuine community that really just want to help one another.

When the talks hit YouTube you should be sure to check them out (I also tweeted about a few of them). What I will focus on in this post was the last two days of the conference that were devoted to the hackathon.

I had bought a X10SLM-F Supermicro board off of eBay a few months ago that I wanted to run CoreBoot on. If you are interested in finding a board that will work with CoreBoot, you should check its status on the status page. I had been talking to Zaolin about wanting a board to hack on and he recommended this one.

At the hackathon, we decided to start with the BMC instead of the CPU BIOS. This made for some fun problems and definitely a lot of lessons learned. I had a Dediprog SF100 flash programmer I brought to the hackathon as well. Some people use RaspberryPis as their flash programmer but the Dediprog was recommended to me and definitely came in handy. However, if you want a cheaper alternative there are a bunch of ways you can skin that cat.

To get started, we read the original binary off the SPI flash… this worked fairly simply. We used the opensource dpcmd tool from dediprog to do it. But you could also use flashrom.

While inspecting the original binary, we found the string linux a few times… as well as a MAC adress, boot commands, IP address, and some other interesting strings.

Before flashing on new firmware we also made sure the board actually booted the BMC. We didn’t have access to any console so we made due with an IPMI LAN port and dnsmasq to work with DHCP. It worked and we got into the BMC user interface over the web. If you’ve ever used a Supermicro server I probably don’t need to tell you that it’s a piece of shit running a 2.6 linux kernel on the BMC. Getting to the UI proved the board actually booted with the original BMC firmware so we began to break it by trying to run OpenBMC.

Our board has a ASPEED 2400 BMC. We chose a OpenBMC configuration that would give us a kernel supporting that chip. Thanks Joel Stanley for all your work on the kernel patches for all the BMCs. We flashed the SPI flash with our new BMC firmware image and attempted to power on the board.

I am going to interrupt the story here for a second to explain the pain involved with this development cycle of writing to firmware to SPI flash. The SPI flash is 16MB but requires erasing the previous contents (4KB per sector) before you can even write. A delete cycle of a sector is 120ms per sector at the worst. So that’s definitely not ideal and anything you can do to make this faster is very much so ideal. Most flash programmers will not rewrite a sector if its contents have not changed which helps, but still super painful coming from the workflow of a software developer.

Back to our board… our OpenBMC image we flashed didn’t work. Again, a lot of this would have been easier to debug with a serial console but we didn’t have one and we didn’t have the spec to get a UART. Our assumption from this failure was that the IPMI LAN port we were using was not the same port configured for that specific configuration.

So we went to build a custom kernel…

With the help of Joel we built a custom kernel completely separate from OpenBMC, however we flashed the kernel directly to the SPI flash without even u-boot, LOL… obviously this didn’t work.

Then we decided to try something easier and had a hunch a different configuration in the OpenBMC project would have the right port enabled. We built the image for that and flashed it onto the SPI flash. This was arguably faster than making our own OpenBMC configuration with our new kernel.

It also didn’t work, but here we got into a bit more trouble. After this point we could no longer write to the SPI flash. The problem was the BMC was interfacing with the SPI flash and we couldn’t take over the ability to write to it. The SPI flash only allows one device to interact with it at a time. We also could not flash the SPI flash without the board powered on because the entire board was pulling power which was too much for our flash programmer to handle. This is a huge pain in the ass. It turns out it is such a pain in the ass that people have made solutions for it.

Fortunately for us, Felix Held had just given a talk on this pain the day before and he was also in the room. He had one more prototype of his tool, qspimux, and we got to use it on our board.

Qspimux allows for the access to a real SPI flash chip to be multiplexed between the target and a programmer that also controls the multiplexer. This way we could flash the SPI flash with the board powered off.

To get his tool installed we had to de-solder the SPI flash and solder it back on after getting the qspimux parts attached. Props to Edwin Peer for his awesome soldering skills here. Here is a live action shot…

It's been a journey, desoldered the flash for the BMC now using Felix Held's qspimux… so the BMC doesn't interfere with the flash, so we can actually flash it! https://t.co/M2mezEMeLa pic.twitter.com/iL1xBQzAwh

— jessie frazelle 👩🏼‍🚀 (@jessfraz) September 6, 2019

After finishing this, we could write to the SPI flash again. At this point we were trying to re-flash the original Supermicro flash onto the board, just to make sure we didn’t mess anything up along the way. This proved to be more difficult than we thought. We got the firmware to write to the flash but the board still wasn’t working. We verified with the oscilliscope that data was indeed leaving the MOSI (master-out-slave-in) pin and the clock was working on the flash.

Then I tried to read the firmware back from the SPI flash chip to make sure it was indeed our original flash. We suspected that maybe we were writing to the device too quickly. This was indeed the case. The two firmware images did not match. I then wrote the firmware to the SPI flash on the slowest setting just to be sure. Then I could actually verify the image we wrote and the image we read back matched our original firmware image. At this point everything was kosher and we knew the image on the SPI flash chip was indeed the same as the original we pulled off the board the day before.

At this point the board was still not booting the original firmware image. This is when we had to go home and firmware camp was over. Overall, this was a great learning experience. I would have been sad had everything gone smoothly because we would not have learned as much about how to debug all the components of the SPI flash and board. I definitely have not given up on this board and will continue down this rabbit hole until it has open source firmware on the BMC and open source BIOS for the CPU.

I would like to thank everyone at the Open Source Firmware Conference for making this a truly amazing week and specifically those who helped with the crazy hackathon project: Rick Altherr, Edwin Peer, Joel Stanley, Felix Held, Bryan Cantrill, Jacob Yundt (who I can’t seem to find online), Joshua M. Clulow, and everyone else I am forgetting who gave us wires, clips, cords, and whatever else we needed to get this thing going! It truly takes a village.

I cannot wait for the next OSFC, but until then I will work on playing with a logic analyzer to see if what the BMC is reading from the SPI flash is even the right data ;)

View Details

At lunch today I learned about Transactional Synchronization Extensions (TSX) which is an implementation of transactional memory. The conversation started as a rant about why transactional memory is bad but then it evolved into how this concept even came to be and how it even got implemented if it’s such a terrible idea.

What is transactional memory? First let’s start by going over what transactional memory is.

You might be familiar with a deadlock. A deadlock occurs when a process or thread is waiting for a specific resource, which is also waiting on a different resource that is being held by another waiting process. You can think of this as P1 needs R1 and has R2, while in turn P2 needs R2 and has R1. That is a deadlock.

Transactional memory removes the possibility of getting a deadlock and replaces it with what is known as a livelock. A livelock happens when processes are constantly changing with regard to one another but neither of them move forward or progress in anyway. Imagine you are walking down the street while another person is heading towards you. You move to the right to avoid running into them as they also move in that direction to avoid running into you. You both then move to the other side so as to not run into each other. This repeats over and over again with no progress forward since both people are moving in the same direction. That is a livelock. With transactional memory you no longer have deadlocks but livelocks.

Why is this? Well, transactional memory works very similarly to database transactions. A transaction is a group of operations that can execute and commit changes as long as there are no conflicts. If there is a conflict, it will start from state zero and try to run again until there are no conflicts. Therefore, until there is a successful commit of a run, the outcome of any operation is speculative.

Intel’s implementation of TSX behaves in such a way that when a transaction aborts due to a hardware exception, it does not fire typical exceptions. Instead, it invokes a user-specified abort handler without informing the underlying OS. This seems like it might lead to some really bad behavior… we should probably know wtf is going on in our system at any given point in time.

Side-Channel Attacks So we know the outcome of any operation in a transaction is speculative. Hmmm speculative you say… I am reminded of spectre and meltdown. The solution in the kernel for defending against spectre and meltdown was Kernel Page Table Isolation (KPTI). Instead let’s focus on what you can break with Spectre and meltdown which is Kernel Address Space Layout Randomization (KASLR). KASLR randomizes the address layout per each boot. This raises the bar for an exploit forcing an attacker to guess where the code and data are located in the address space. The probability of an attack then becomes the probability of an information leak multiplied by the probability of a memory corruption vulnerability.

However, this can be exploited without an information leak but instead using a translation lookaside buffer (TLB) and a timing attack. A TLB is a memory cache that reduces the time taken to access a user memory location. It keeps recent translations of virtual memory to physical memory.

In the DrK paper, the authors describe an attack that uses the behavior of TSX as a feature of the exploit. As described above, TSX has the behavior of aborting a commit without leaving any trace as to why it was aborted. So in DrK, the authors use TSX to create a bunch of access violations of the privileged address space inside transactions and turn that into knowledge of mapping and executable status of the address space without even generating a page fault.

The point I am making with this example is that transactional memory and it’s implementation TSX are a bad idea.

But who could have possibly seen this as a bad idea?

Rewind to 2008 Concurrency is the biggest hype in town. This comes from a lot of different things but can be found in an article, Technical perspective: Transactions are tomorrow’s loads and stores, in Communications of the ACM (CACM). It seems at the time, this craze was started out of academia. Some practitioners, Bryan Cantrill and Jeff Bonwick, wrote rebuttles in the name of “please dear god do not make transactional memory A Thing”. That can be seen in Bryan’s blog post, Concurrency’s Shysters, and the follow-up ACM Queue article, Real-world Concurrency.

Clearly, in 2008 there was a division between academia and practitioners.

Fastforward to 2012 Intel shipped TSX in February 2012.

EDIT: It was pointed out that Azul shipped transactional memory in 2006. Thanks @davidcrawshaw!

Why is this interesting? Hype cycles come and go and if you spend anytime in our industry you tend to become pretty numb to them. Seeing through the hype has always been a joy of mine and I find it interesting the vectors through which hype travels have changed drastically over time.

With transactional memory, the hype began in academia through academic conferences and articles in journals. Before the 2000s even, hype might have spread through magazines like Byte. Today, we have multiple channels for hype through social networks: Twitter, Reddit, blogging, YouTube, GitHub, Hacker News (slashdot before that), and others.

Hype seems to travel through the unconscious need of people to connect to others. Being a part of movements, like open source projects and a shared sense of need, allows people to be a part of something bigger than just themselves.

Twitter is fascinating due to the way it hosts so many subcultures. One of my favorite examples of this is Canadian twitter where everyone is polite and nice to each other. There are also vehement subcultures around the latest technology trends. The way technology can spread has turned from a place where very few people have a voice (through getting papers accepted at conferences and in journals) to social networks where everyone has a voice. My hope is that the loudest of the voices are the ones used to build technology for the best causes.

I’ll leave you with that, hope you enjoyed and learned something from my rather weird example of a technology hype wave.

View Details

Hello!

I thought it would be fun to write a post aimed towards business leaders making technology decisions for their organizations. There is a lot of hype in our field and little truth behind the hype.

Like most things I write about, this started from an idea I had on Twitter:

has anyone ever done technical breakdowns of these products in Gartner reports that are actually just trash, is this something you'd read..?

— jessie frazelle 👩🏼‍🚀 (@jessfraz) July 24, 2019

This post will cover some hard truths of Kubernetes and what it means for your organization and business. You might have heard the term “Kubernetes” and you might have been led to believe that this will solve all the infrastructure pain for your organization. There is some truth to that, which will not be the focus of this post. To get to the state of enlightenment with Kubernetes, you need to first go through some hard challenges. Let’s dive in to some of these hard truths.

Stateful Data is Hard Kubernetes is not to be used for stateful data. There has been a lot of work done in this area but it is still not sufficent. For the more technical members of our audience I direct you to exhibit A. The linked issue goes over problems when a “StatefulSet” gets into an error during deploying or upgrading. This can lead to data loss or corruption since Kubernetes will need manual intervention to fix the state of the deployment. This could even lead to the point where the only recommended fix is you delete the state. What does this mean for your business? Well, if you lose or corrupt your data it could mean a lot of different things depending on what the data was. If the data was your customer database of new account signups, well you might have just lost the data for your new customers. If you are an ecommerce site, it might have been your latest sale. If you are in banking or investments, it might have been data accounting for the movement of capital.

Databases holding valuable information like the examples above should always have mechanisms for replication which is not something Kubernetes is going to solve for you. While you might choose to use Kubernetes for stateful data, you should always remember to handle replicating that data in case there is a failure.

Exposed Dashboards A lot of organizations are dipping their toes into Kubernetes but forgetting to disable or secure the dashboard for the control plane from the rest of the internet. The control plane dashboard is a website you can navigate to that controls your cluster. Leaving the dashboard exposed to the public can have huge implications on your business. If your dashboard is exposed, anyone could find your dashboard and then control it. Finding an exposed dashboard is not that difficult if you know what you are looking for and have access to a site like shodan.

What would the finder of the dashboard control? Everything running in Kubernetes. If your website is running in Kubernetes, it means someone else could make your website go offline, someone else could replicate your website but send all sales and monetary transactions to their own bank account, someone else can breach your customers’ data, or someone else could hold your infrastructure up for ransom and not give you back control of your website unless you pay what they demand. This is just a few things I thought of off the top of my head but you could probably think of more.

There is a whole other aspect of this in that if this breach goes public, then you have a huge public relations problem on your hands. Which for a public company might even have implications on your stock price if shareholders end up losing trust from the news of your company’s technical incompetence and they decide to sell their shares.

If it’s not the dashboard being exposed it might be your API server or another service. There’s a few options for this particular failure mode.

Upgrading your Kubernetes version seems to always break something I’ve heard from a bunch of people that whenever they need to upgrade their production environment of Kubernetes it always leads to something breaking. It’s recommended that you have more than one cluster in production for this very reason. Then, if one cluster in production is broken from being upgraded, the other cluster that has not been upgraded is still running the technical parts of your business. This is very good from a reliability point of view. It means reaching your website has a “plan B” where if the “plan A” infrastructure has a problem, everything will be redirected to “plan B” and your customers will not even know the difference. As a downside, your operations teams now have to figure out ways for managing and maintaining two clusters (more work for them) but your business is in a better place for it.

The other option is you just don’t upgrade. However, if you don’t upgrade, your infrastructure might be vulnerable to security threats and then we are back in the situation above where you might have data breached by hackers, a hostile takeover of your website, and then a huge public relations scandal leading to investors and shareholders selling their stock.

Steep learning curve, complexity is king, and operational pain A lot of the criticism I hear about Kubernetes is how complex it is. For your organization, this means your staff are going to have to surmount this very steep learning curve. As with learning anything, things only get worse before they get better. So get ready for a lot of production outages and failovers as your team starts to learn the ins and outs of this overly complex system. What does this mean for your website and customers? Availability will be spotty for awhile but we hope eventually it will even out. Lastly, to quote someone very wise (send a pull request if you know who!), “Hope is not a strategy.”

Managed Kubernetes Now you are probably thinking, “my cloud provider said they’d take away all the pain you just described by selling me their managed Kubernetes.” That is indeed the dream. However, it is not reality. Having worked for some cloud providers, I have seen the pain customers still go through trying to learn the patterns Kubernetes implements and applying those patterns to their existing applications. This means your teams will still have to handle the steep learning curve. Just because it’s managed does not mean that your application’s uptime and availability are covered. That is still on your team. Customers being able to use your website on the internet is your team’s responsibility and understanding Kubernetes is still required for that. For every line of YAML written and debugged to get your website running, it is time that is being taken away from building on what your business actually does. Unless of course you are a business of selling Kubernetes, then if so, carry on.

You will also want to be sure your cloud provider did not fall prey to the pitfalls I outlined above as well. You should make sure your cluster is fully isolated from other customer’s clusters. The way the managed Kubernetes offerings work is by the cloud provider managing the “master” for your cluster. This means all the data for your cluster is managed by your cloud provider. If your data is not properly isolated from all the other customer’s data, it means that if the cloud provider gets breached by means of a different customer’s cluster then your data has been breached as well. Then, we are in the scenario where a hacker owns your website, can hold it for ransom, or cause a very public incident for your company that you will need to handle.

This was just a brief overview and I am not trying to throw shade. I merely wanted to phrase some of these prevalent problems in a way that people running a business might be more aware of the impact adopting this technology might have. It should not be understated, if your organization does tackle these difficulties (and others I didn’t mention), then you will possibly see great impact on developer productivity, faster feature releases and deployments (among all the other wins Kubernetes can provide). Just be aware that with the good, comes some bad.

View Details

Below is the foreward for the new book on Linux Observability with BPF by two of my favorite programmers, David Calavera and Lorenzo Fontana! I was pretty stoked about getting to write the foreward, I asked O’Reilly if I could publish it on my blog as well and they said yes. I hope you all check out this book and share what you’ve built after!

As a programmer (and a self confessed dweeb) I like to stay up to date on the latest additions to various kernels and research in computing. When I first played around with Berkeley Packet Filters (BPF) and Express Data Path (XDP) in Linux I was in love. This is such a NICE THING and I am glad this book is putting BPF and XDP on the center stage so more people can start using it in their projects.

Let me go into detail about my background and why I fell in love with these kernel interfaces… I worked as a Docker core maintainer, along with David (one of the brilliant authors of this book). Docker, if you are not familiar, shells out to iptables for a lot of the filtering and routing logic for containers. The first patch I ever made to Docker was fixing a problem where a version of iptables on CentOS didn’t have the same command-line flags so writing to iptables was failing. There were a lot of weird issues like this and anyone who has ever shelled out to a tool in their software can likely commiserate. Not only that, having thousands of rules on a host is not what iptables was built for and has performance side effects because of it.

Then I heard about BPF and XDP. This was like music to my ears. No longer would my scars from iptables bleed with another bug! The kernel community is even working on replacing iptables with BPF! Halleluyah! Cilium, container networking, is using BPF and XDP for the internals of their project as well.

But that’s not all! BPF can do so much more than just fulfilling the iptables use case. With BPF, you can trace any syscall or kernel function as well as any user-space program. bpftrace gives users dtrace-like abilities in Linux from their command line. You can trace all the files that are being opened and the process calling the open, count the syscalls by the program calling them, trace the OOM killer, and more… the world is your oyster! XDP and BPF are also used in Cloudflare and Facebook’s load balancer to prevent DDoS attacks. I won’t spoil why XDP is so great at dropping packets because you will learn about that in the XDP and networking chapters of this book (cough you don’t even allocate a kernel struct cough)!

Lorenzo, another of the authors, I have had the privilege of knowing each other through the Kubernetes community. His tool, kubectl-trace, allows users to run their custom tracing programs easily inside their kubernetes clusters.

Personally, my favorite use case for BPF has been writing custom tracers to prove to other folks that the performance of their software was not up to par or making really expensive amounts of calls to syscalls. Never underestimate the power of proving someone wrong with hard data. Don’t fret, this book will walk you through writing your first tracing program so you can do the same ;). The beauty of BPF lies in the fact that before now other tools used lossy queues to send sample sets to user space for aggregation whereas, BPF is great for production since it allows for constructing histograms and filtering right at the source of events.

I have spent half of my career working on tools for developers. The best tools allow autonomy in their interfaces for developers like you to use them for things even the authors never imagined. To quote Richard Feynman, “I learned very early the difference between knowing the name of something and knowing something.” Until now you might have only known the name BPF and that it might be useful to you. What I love about this book is that it gives you the knowledge you need to be able to create all new tools using BPF.

The best books don’t confine readers into a box and that is why I love this one in particular. After reading and following the exercises, you will be empowered to use BPF like a super power. You can use this in your toolkit to use on demand when it’s most needed and most useful. You won’t just learn BPF you will understand it. This book is a path to open your mind to the possibilities of what you can build with BPF.

This developing ecosystem is very exciting! I hope it will grow even larger as more people start wielding BPF’s power. I am excited to learn about what the readers of this book end up building, whether it’s a script to track down a crazy software bug or a custom firewall or even infrared decoding! Be sure to let us all know what you built!

View Details

“Can I get an encore, do you want more” - Jay-Z

I recently read Ben Horowitz’s book, The Hard Thing about Hard Things. It’s really eye opening and creates a level of empathy in the reader for leaders that make hard decisions every day. It covers everything from how to know your company is toxic to how to do layoffs. Ben starts each chapter with a rap quote so as did I above ;) obviously I chose Jay-Z but I also love Tupac, as is shown by my first blog post ever.

I have a corollary to this: power dynamics. I, personally, have seen and experienced what it is like being a leader when no one really has a full view of who you are as a person. I try to always be authentic and personable, but the fact of the matter is: we are all humans and we all have off days.

Most people only get a view of who I am through Twitter, but that is not fully who I am. I think that is the case for most people on that website. For executives of companies or leaders of large teams, the same holds true: you only see a small subset, through very limited communication, of who they really are.

At work, I like to move fast and get things done. This may result in abrupt communications which is not typical of how I am on the internet. Even more so, if I was to give feedback or an opinion on something, someone might feel it with the heat of a thousand suns and think it is aggressive, even if that is not how I intended it. The best we can do is apologize and grow when we fuck up.

Another example would be if someone in a position of power asks someone to do something. The person without the power might think they have to do it a certain way and can’t push back. We can try to solve this by always making an effort to ask for other’s opinions and feedback.

I really do not enjoy when people hero worship me and I do not think people should hero worship anyone. We are all humans and we are all flawed in our own ways. Anyone who believes someone to be perfect will soon find that they are not. This holds true for anyone: executives of companies, senior engineers, tennis champions, and hollywood stars.

Leave room for people to make mistakes, because they will. What truly matters is how a person grows after making a mistake. It helps to make it very clear that you will make mistakes and welcome feedback. When someone discovers a mistake you’ve made try to treat it as a gift. Allow for failure and growth from failure in others and they will do the same for you as well.

If you are a leader and you empathize with this, I think this problem can also be solved with time. You need time for people to understand how you work and time to grow trust. As long as you continue to be transparent about mistakes over time and grow from them, trust will follow.

It’s hard to see a power dynamic at play if you are in it and hold the power. Power dynamics are in the eye of the beholder. We can all try to be conscious of this and patient as the vines of trust grow around us.

View Details

I gave a talk recently at GoTo Chicago on Why open source firmware is important and I thought it would be nice to also write a blog post with my findings. This post will focus on why open source firmware is important for security.

Privilege Levels In your typical “stack” today you have the various levels of privileges.

• Ring 3 - Userspace: has the least amount of privileges, short of there being a sandbox in userspace that is restricted further.
• Ring 0 - Kernel: The operating system kernel, for open source operating systems you get visibility into the code behind this.
• Ring -1 - Hypervisor: The virtual machine monitor (VMM) that creates and runs virtual machines. For open source hypervisors like Xen, KVM, bhyve, etc you have visibility into the code behind this.
• Ring -2 - System Management Mode (SMM), UEFI kernel: Proprietary code, more on this below.
• Ring -3 - Management Engine: Proprietary code, more on this below.

The negative rings were made up because there was no other way to express something with more privileges.

From the above, it’s pretty clear that for Rings -1 to 3, we have the option to use open source software and have a large amount of visibility and control over the software we run. For the privilege levels under Ring -1, we have less control but it is getting better with the open source firmware community and projects.

It’s counter-intuitive that the code that we have the least visibility into has the most privileges. This is what open source firmware is aiming to fix.

Ring -2: SMM, UEFI kernel This ring controls all CPU resources.

System management mode (SMM) is invisible to the rest of the stack on top of it. It has half a kernel. It was originally used for power management and system hardware control. It holds a lot of the proprietary designed code and is a place for vendors to add new proprietary features. It handles system events like memory or chipset errors as well as a bunch of other logic.

The UEFI Kernel is extremely complex. It has millions of lines of code. UEFI applications are active after boot. It was built with security from obscurity. The specification is absolutely insane if you want to dig in.

Ring -3: Management Engine This is the most privileged ring. In the case of Intel (x86) this is the Intel Management Engine. It can turn on nodes and re-image disks invisibly. It has a kernel that runs Minix 3 as well as a web server and entire networking stack. It turns out Minix is the most widely used operating system because of this. There is a lot of functionality in the Management Engine, it would probably take me all day to list it off but there are many resources for digging into more detail, should you want to.

Between Ring -2 and Ring -3 we have at least 2 and a half other kernels in our stack as well as a bunch of proprietary and unnecessary complexity. Each of these kernels have their own networking stacks and web servers. The code can also modify itself and persist across power cycles and re-installs. We have very little visibility into what the code in these rings is actually doing, which is horrifying considering these rings have the most privileges.

They all have exploits It should be of no surprise to anyone that Rings -2 and -3 have their fair share of vulnerabilities. They are horrifying when they happen though. Just to use one as an example although I will let you find others on your own, there was a bug in the web server of the Intel Management Engine that was there for seven years without them realizing.

How can we make it better? NERF: Non-Extensible Reduced Firmware NERF is what the open source firmware community is working towards. The goals are to make firmware less capable of doing harm and make its actions more visible. They aim to remove all runtime components but currently with the Intel Management Engine, they cannot remove all but they can take away the web server and IP stack. They also remove UEFI IP stack and other drivers, as well as the Intel Management/UEFI self-reflash capability.

me_cleaner This is the project used to clean the Intel Management Engine to the smallest necessary capabilities. You can check it out on GitHub: github.com/corna/me_cleaner.

u-boot and coreboot u-boot and coreboot are open source firmware. They handle silicon and DRAM initialization. Chromebooks use both, coreboot on x86, and u-boot for the rest. This is one part of how they verify boot.

Coreboot’s design philosophy is to “do the bare minimum necessary to ensure that hardware is usable and then pass control to a different program called the payload.” The payload in this case is linuxboot.

linuxboot Linuxboot handles device drivers, network stack, and gives the user a multi-user, multi-tasking environment. It is built with Linux so that a single kernel can work for several boards. Linux is already quite vetted and has a lot of eyes on it since it is used quite extensively. Better to use a open kernel with a lot of eyes on it, than the 2½ other kernels that were all different and closed off. This means that we are lessening the attack surface by using less variations of code and we are making an effort to rely on code that is open source. Linux improves boot reliability by replacing lightly-tested firmware drivers with hardened Linux drivers.

By using a kernel we already have tooling around firmware devs can build in tools they already know. When they need to write logic for signature verification, disk decryption, etc it’s in a language that is modern, easily auditable, maintainable, and readable.

u-root u-root is a set of golang userspace tools and bootloader. It is then used as the initramfs for the Linux kernel from linuxboot.

Through using the NERF stack they saw boot times were 20x faster. But this blog post is on security so let’s get back to that….

The NERF stack helps improve the visibility into a lot of the components that were previously very proprietary. There is still a lot of other firmware on devices.

What about all the other firmware? We need open source firmware for the network interface controller (NIC), solid state drives (SSD), and base management controller (BMC).

For the NIC, there is some work being done in the open compute project on NIC 3.0. It should be interesting to see where that goes.

For the BMC, there is both OpenBMC and u-bmc. I had written a little about them in a previous blog post.

We need to have all open source firmware to have all the visibility into the stack but also to actually verify the state of software on a machine.

Roots of Trust The goal of the root of trust should be to verify that the software installed in every component of the hardware is the software that was intended. This way you can know without a doubt and verify if hardware has been hacked. Since we have very little to no visibility into the code running in a lot of places in our hardware it is hard to do this. How do we really know that the firmware in a component is not vulnerable or that is doesn’t have any backdoors? Well we can’t. Not unless it was all open source.

Every cloud and vendor seems to have their own way of doing a root of trust. Microsoft has Cerberus, Google has Titan, and Amazon has Nitro. These seem to assume an explicit amount of trust in the proprietary code (the code we cannot see). This leaves me with not a great feeling. Wouldn’t it be better to be able to use all open source code? Then we could verify without a doubt that the code you can read and build yourself is the same code running on hardware for all the various places we have firmware. We could then verify that a machine was in a correct state without a doubt of it being vulnerable or with a backdoor.

It makes me wonder what the smaller cloud providers like DigitalOcean or Packet have for a root of trust. Often times we only hear of these projects from the big three or five. I asked this on twitter and didn’t get any good answers…

I’m surprised how many people are responding that they love DigitalOcean but seem entirely unconcerned there’s no answer here. You should be concerned.

— jessie frazelle 👩🏼‍🚀 (@jessfraz) May 8, 2019

There is a great talk by Paul McMillan and Matt King on Securing Hardware at Scale. It covers in great detail how to secure bare metal while also giving customers access to the bare metal. When they get back the hardware from customers they need to ensure with consistency and reliability that there is nothing from the customer hiding in any component of the hardware.

All clouds need to ensure that the hardware they are running has not been compromised after a customer has run compute on it.

Platform Firmware Resiliency As far as chip vendors go, they seem to have a different offering. Intel has Platform Firmware Resilience and Lattice has Platform Firmware Resiliency. These seem to be more focused on the NIST guidelines for Platform Firmware Resiliency.

I tried to ask the internet who was using this and heard very little back, so if you are using Platform Firmware Resiliency can you let me know!

It seems that Intel has some effort called Platform Firmware Resiliency (anyone using this one?!) https://t.co/fQq2gdLNOm

— jessie frazelle 👩🏼‍🚀 (@jessfraz) May 8, 2019

From the OCP talk on Intel’s firmware innovations, it seems Intel’s Platform Firmware Resilience (PFR) and Cerberus go hand in hand. Intel is using PFR to deliver Cerberus’ attestation priniciples. Thanks @msw for the clarification.

It would be nice if there were not so many tools to do this job. I also wish the code was open source so we could verify for ourselves.

How to help I hope this gave you some insight into what’s being built with open source firmware and how making firmware open source is important! If you would like to help with this effort, please help spread the word. Please try and use platforms that value open source firmware components. Chromebooks are a great example of this, as well as Purism computers. You can ask your providers what they are doing for open source firmware or ensuring hardware security with roots of trust. Happy nerding! :)

Huge thanks to the open source firmware community for helping me along this journey! Shout out to Ron Minnich, Trammel Hudson, Chris Koch, Rick Altherr, and Zaolin. And shout out to Bridget Kromhout for always finding time to review my posts!

View Details

Last week, I had the pleasure of meeting with the Transposit team in San Francisco. Tech is a super small world and it turns out the two founders and I are separated by one-degree through several different people we know. In meeting them I closed many loops without even realizing it, but I digress…

Their product is really cool, it exposes a SQL interface for interacting with numerous APIs at once. For someone like myself who deploys a lot of bots, this is great. Usually when I have a complex bot I end up writing a lot of “glue code” to combine a few different APIs and get the information I want. Most of my bots have some sort of pagination logic and all have the N+1 problem where I don’t really optimize my queries or use anything fancy like graphQL. Many APIs don’t even have graphQL interfaces but also I am old school and I don’t really want to learn something new. This is why I was super intrigued by Transposit’s SQL interface, because hey, I know SQL!

Adam, the CEO, challenged me to try it out, give them feedback, and see if I could break it with something complex. I am not one to back down from a challenge and I have some super weird ass bots, so I decided to start with the weirdest.

Gitable Gitable is a bot I made for sending all my open issues and PRs on GitHub to a table in Airtable. I fucking love Airtable. It’s design just feels right and works the way my brain works.

I set out to make this bot work in Transposit because I know it has some super weird loops and has the N+1 problem where I loop over all my repos, then make another API call after.

To reiterate, the goal of the bot is to iterate through all my repos on GitHub and sync the list of issue and PRs with a table in Airtable.

Query all the user’s repos First, I need to get all my repos that are not forks. So I need a SQL query for this, in Transposit it looks like this:

``` SELECT name, full_name FROM github.list_repos_for_user WHERE username=@owner AND type='owner' AND fork=false

```

The github.list_repos_for_user table is a built in to Transposit and they handle all your API keys and authorizations when you choose “Github” as a data connection in the UI. It also caches the response which is a huge win because I am the queen of being rate limited.

I named that query: list_repos_for_user so when I want to use it elsewhere in another query, I can call it by this.list_repos_for_user.

Query all the issues in all the user’s repos To get all the issues in all my repos I can use a join on that table I just created. It ends up looking like this:

``` SELECT A.created_at AS created, A.updated_at AS updated, B.full_name, A.number, A.html_url AS url, A.state, A.title, A.user.login AS author, A.labels, B.name, A.closed_at AS completed, A.comments FROM github.list_issues_for_repo AS A JOIN this.list_repos_for_user AS B ON A.repo = B.name WHERE A.owner=@owner AND B.owner=@owner

```

Okay so I didn’t break anything yet and I just joined my table with all my repos, this.list_repos_for_user, with the built-in table in Trasnposit github.list_issues_for_repo. This has now replaced my N+1 code with just this one SQL query and Transposit does all the optimizations on their end.

I called this table list_issues_for_user and @owner is a parameter, so anyone else can fork this app and change it to their own username.

Query all the records in an Airtable table Now I need to get all the existing airtable records in my table so I can know later on down the road if I need to create a row or update a row with the new information from the GitHub API.

In my Airtable table I have a column called “reference” which stores information about the issue or PR as owner/repo#num so for example it looks like jessfraz/.vim#1. This is a column defined by me, but I also know it to be unique. So I want to get the reference of every column and it’s airtable record ID so I can use that to update the record.

``` SELECT id, fields.Reference as reference FROM airtable.get_records WHERE baseId=@baseID AND table=@table

```

That winds up looking like the query above. @baseID and @table are parameters so anyone can replace those with their own for their table in Airtable.

I named this query get_airtable_records so when I call it later I can do so with this.get_airtable_records.

Update and create rows in Airtable for each of the issues in user’s repos Okay so now’s the part where I am thinking… I’m going to break this thing. (Narrator: I didn’t.)

Transposit has both SQL and Javascript operations and since the next part was where a lot of the logic was I used Javascript. I haven’t written Javascript in a long time so mind my shitty code. Honestly, SQL is turing complete so I considered using SQL but I wanted to get this done in an hour. (I will leave it as an exercise for the reader to fork my app and make it all in SQL.)

What I needed to do was take our earlier table to list_issues_for_user, iterate over them, and update or create an Airtable record for each of them. This ends up looking like the following:

``` function run(params) { var results = api.run("this.list_issues_for_user", {owner: params.owner});

for (var i = 0; i < results.length; i++) {   
    // Build the reference for the issue with the full name and number.
    // Winds up looking like "jessfraz/.vim#1"
    var reference = results[i].full_name + "#" + results[i].number;

    // Get the Airtable recordID for the reference if it exists.
    var id = api.query("select id from this.get_airtable_records where reference='"+reference+"'", {baseID: params.baseID, table: params.table});

    // Define the object params for create and update.
    var obj = {
        baseID: params.baseID, 
        table: params.table,
        reference: reference,
        title: results[i].title,
        state: results[i].state,
        author: results[i].author,
        type: 'issue',
        comments: results[i].comments,
        url: results[i].url,
        updated: results[i].updated,
        created: results[i].created,
        completed: results[i].completed,
        repo: results[i].name,
    };

    if (id.length > 0) {
        results[i].airtable_id = id[0].id;
        obj.recordID = id[0].id;

        // Update the result in the table.
        var r = api.run("this.update_record", obj);
        api.log(r);
    } else {
        // Create record in the table.
        results[i].airtable_id = 0;
        var r = api.run("this.create_record", obj);
        api.log(r);
    }

    results[i].reference = reference;
}
return {
    results
};

}

```

You might be wondering what this.create_record and this.update_record look like. These are just helper operations so I can use all the fields for the records as parameters.

Create an Airtable record create_record calls the built-in airtable.create_record which looks like the following:

``` SELECT * FROM airtable.create_record AND baseId=@baseID AND table=@table AND $body=(SELECT { 'fields' : { 'Reference': @reference, 'Title': @title, 'State': @state, 'Author': @author, 'Type': @type, 'Comments': @comments, 'URL': @url, 'Updated': @updated, 'Created': @created, 'Completed': @completed, 'Repository': @repo, } })

```

Everything starting with an @ is a parameter we can change on the fly in our Javascript function like you saw above.

Update an Airtable record update_record is very similar, it calls the Transposit built-in airtable.update_record:

``` SELECT * FROM airtable.update_record WHERE recordId=@recordID AND baseId=@baseID AND table=@table AND $body=(SELECT { 'fields' : { 'Reference': @reference, 'Title': @title, 'State': @state, 'Author': @author, 'Type': @type, 'Comments': @comments, 'URL': @url, 'Updated': @updated, 'Created': @created, 'Completed': @completed, 'Repository': @repo, } })

```

Doing the above with pull requests rather than issues is the exact same code but you swap out the query for issues with pull requests. You can schedule your operations to run at certain times like cron or when you call an API endpoint.

Sadly, I failed at breaking the thing with one of my most complex bots. But maybe you will have better luck trying ;) You can fork my app or look at the queries here: console.transposit.com/t/jessfraz/gitable.

View Details

I came up with a list of questions I would ask my cloud provider if I was buying a product. They are as follows:

1. What problem is this solving? I would ask this to make sure I even need this product. So many people tend to buy into the hype for “shiny”, they miss if they even needed the thing in the first place.

2. How did you implement this? What is your threat model? So much of the cloud is built on popsicle sticks and glue. Does that make you feel safe at night knowing your customer data is being stored in a proof of concept that was shipped before it should have been? Best to get your security team to assess if the product is actually built on the providers side up to standard. This does not mean what you see as a customer, it means the proprietary bits you cannot see.

What does the service license agreement say for what happens if the provider themselves is hacked? Do they have to tell you or can they just sweep it under the rug? What if a vulnerability comes out on the open source project they are using, do they have to give you a risk assessment as to if you were hacked?

What if they don’t know if they were hacked after a vulnerability is public? Red flag…

If they themselves do not know their own threat model, that should be a huge warning sign.

Bonus points if their implementation is open source; but I will let you in on a secret, most aren’t. The exception is Joyent :)

1. What customers did you speak to before building this feature? Ties back to number one, what problem is this solving? So often these features seem to be built for fun or based off a feeling a product manager had.

Hope this helps! I will probably update over time. :)

View Details

This post is co-authored by Kathy Simpson.

“understanding the true nature of instinctive decision making requires us to be forgiving of those people trapped in circumstances where good judgment is imperiled.” ― Malcolm Gladwell, Blink: The Power of Thinking Without Thinking

As leaders, setting up a structure that helps us navigate decisions under pressure is of the utmost importance. When writing and delivering software we rely on our continuous integration (CI) infrastructure and test suites to tell us when a test is failing and code should not be merged.

As leaders, before acting or making decisions it would be nice to have a set of tests and checks, established ahead of time, to make sure we are in the right headspace to think, behave and make decisions that are in the best interest of everyone and our company. There are devastating consequences to taking actions based on fear and pride; we hope this set of questions enables taking action based on growth, humility, inclusion, and soulful reflection.

The following are the sets of questions we brainstormed, but expect them to change over time as we experience and deal with new problems. These were started in a gist and are copied below. The diff of this post and the gist will serve as the evolution of this thought process.

It’s important to note that in some instances answering all the questions might take too much time. Perhaps prioritizing the most important ones in the moment would be more effective.

Answering all the questions may be a luxury at times, so we suggest breaking them down based on the situation you find yourself in: prioritize the most important ones to your role, have a few ‘go to’ questions, or categorize them based on the situations you find yourself in more often. The important part of this list is to help us navigate a difficult situation while still maintaining the integrity we intend for ourselves as leaders.

1. Do I want to die on this hill?

Pass: This is morally good and if not handled has long term consequences.

Fail: This is self serving. 2. Am I including everyone?

Pass: My ego is not driving this conversation.

Fail: The people in this conversation will only tell me I’m right and not push back. 3. Am I hiding something?

Pass: The information, though painful, is known to all.

Fail: Yes. 4. Is there transparency here?

Pass: The team agrees on context and can repeat it back to me.

Fail: Hidden misalignment (test: what do we align on). 5. Am I being curious?

Pass: I’m asking questions that make me uncomfortable, and I’m comfortable being wrong.

Fail: I want my way. 6. Is my team afraid to tell me things?

Pass: They freely and continually come to me with answers and information that they know I will not like.

Fail: They go to each other or people outside the team with the information, and telling me what they think I want to hear. 7. Am I only communicating with the same people over and over?

Pass: My sphere of influence is diverse. I feel comfortable talking with anyone on the team.

Fail: I continually consult the same individuals (test: do I have entourage?). 8. Do I feel insecure?

Pass: I feel empowered and am willing to take feedback and risks regardless of the outcome as it’s good for the company and the customer.

Fail: I retreat, I am not comfortable, I am not giving up the information because I am scared of what people will think. 9. Can my team do the job I hired them to do? Is the job they are hired to do the job that needs to be done?

Pass: The team ships outcomes efficiently.

Fail: The team is not empowered and often stalls (test: do I often have to intervene?). 10. Are you scratching an itch?

Pass: This is a problem that’s bigger than myself.

Fail: It may feel good to solve this problem but only for myself and temporarily. 11. Am I being judgmental?

Pass: Do I trust my team and their decisions?

Fail: Is someone speaking up and telling me that I’m being judgmental? 12. Am I taking risks?

Pass: I feel comfortable and confident that this decision will lead to positive and fruitful outcomes.

Fail: I am being a pushover, and I am compromising in the wrong ways. 13. Am I being manipulative?

Pass: I’m being honest, real, straightforward and I’m OK with the outcome and hearing ‘no’.

Fail: I’m intentionally using words that aren’t representative of what I’m trying to communicate. 14. Am I speaking for people or letting them speak for themselves?

Pass: I am doing the minority of the speaking and people are disagreeing with my opinions.

Fail: I am being quoted back to myself. I am talking the majority of the time.

Be sure to keep up with the original gist as well to see how this list evolves!

View Details

Last week I got to see what it was like to be an investigative journalist for a day. It was thrilling. I will get into what I learned but first I waned to give some background on why I was doing this.

I have a general curiosity for people. It’s interesting to me to uncover what people are motivated by. Humans are individual snowflakes and no one is exactly like the next. It is our unique experiences that form the way we think and behave, as well as what drives us.

It is in my nature to learn and absorb information. I also recently learned, although I should have realized this throughout my life, I am well attuned to absorbing others emotions. I think my deep drive for understanding others and value of the truth is somewhat perfect for the role of “investigative journalism”.

Researching things for investigative journalism is very similar to that of research for academia. Investigative journalism seems to be driven by intuition, while academia might be more driven by novel research.

I got to see what Jeff Kao’s job was like for a few hours and I learned a lot.

One of the more interesting things we discussed was diffs. I brought up if diffs (as in those used by a source control tool) could work as a line of truth. With a diff, the history of a document is fully transparent, anyone can see any and all changes to it (of course taking into account, tracking force pushes as well).

Jeff pointed out that there is past history of journalism using “diffs”. One example was from an article that uncovered bills and laws being copied and influenced by corporations. They compared the text of the bills to others and showed the changes, similarities, and motivations behind them.

I then realized that Jeff was the author of the amazing article from a couple years ago on how net neutrality comments were likely faked. He used natural language processing to find the similarities in the comments.

Both these articles use comparisons of text to uncover falsifications or motivations. This is super similar to diffs, which is also a comparison of text! I also started thinking about how in my previous article I mentioned it would be cool if laws were versioned with git. By doing that, we would get the diff and history of changes to the laws. Changes to laws or language used over time could be visualized quite easily with the tools for source control.

Overall, the day was fascinating. Investigative journalism was really aligned with my joy of learning new things from a variety of different perspectives and using intuition and research to try to find truth.

Another thought I have been thinking on is: how can we separate emotion from the truth? So much of the news today is trying to trigger an emotional response for clicks. Or in the worst case, it is trying to trigger an emotional response for influencing an election. How can we promote the news sources that focus on the truth versus triggering a reaction? The truth itself should be enough of a trigger.

View Details

I’ve been talking to a lot of people in different layers of the stack during my funemployment. I wanted to share one of the problems I’ve been thinking about and maybe you can think of some clever solutions to solve it.

Conway’s Law states “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”

If you were to apply Conway’s Law to all the layers of the software stack and open source software you’d see a problem: There is not sufficient communication between the various layers of software.

Let’s dive in a bit to make the problem super clear.

I’ve met a bunch of hardware engineers and I’ve made a point about asking each of them how they feel about using a single chip for multiple users. This is, of course, the use case of the cloud. All of the hardware engineers either laugh or are horrified and the resounding reaction is “you’d be crazy to think hardware was ever intended to be used for isolating multiple users safely.” Spectre and Meltdown proved this was true as well. Speculative execution was a feature intended to make processors faster but was never thought about in terms of the vector of hacking something running multi-tenant compute, like a cloud provider. Seems like the software and hardware layers should better communicate…

That’s just one example, let’s reverse the interaction. I’ve talked to a bunch of firmware and kernel engineers and they’d all love if the firmware from chip vendors did less complexity. For instance, it seems like a unanimous vote among firmware and kernel engineers that CPU vendors should not include runtime services or SMM with their firmware. Open source firmware and kernel developers would rather handle those problems at their layer of the stack. All the complexity in the firmware leads to overlooked bugs and odd behavior that can’t be controlled or debugged from the kernel developers layer and/or user space. Not to mention, a lot of CPU vendors firmware is proprietary so it’s really hard to know if a bug is truly a firmware bug.

Another example would be the hack of SoftLayer. Hackers modified the firmware on the BMC from a bare metal host the cloud provider was offering. This shows another mistake in having blinders on and not being conscious of the other layers of the stack and the entire system.

Let’s move up the stack a bit to something I personally have experienced. I worked a lot on container runtimes. I also have worked on kubernetes. I was horrified to find people are running multi-tenant kubernetes clusters with multiple customers processes, aka for isolating untrusted processes. The architecture of kubernetes is just not designed for this.

A common miscommunication is the “window dressing.” For example, there is a feature in kubernetes that prevents exec-ing into containers. This is implemented by merely preventing the API call in kubernetes. If a person has access to a cluster there are about 4 dozen different ways I can think of to exec into a container and bypass this “feature” and kubernetes entirely. Using said “security feature” in kubernetes alone is not sufficient for security in any respect. This is a common pattern.

All these problems are not small by any means. They are miscommunications at various layers of the stack. They are people thinking an interface or feature is secure when it is merely a window dressing that can be bypassed with just a bit more knowledge about the stack. I really like the advice Lea Kissner gave: “take the long view, not just the broad view.” We should do this more often when building systems.

The thought I’ve been noodling on is: how do we solve this? Is this something a code hosting provider like GitHub should fix? But, that excludes all the projects that are not on that platform. How do we promote better communication between layers of the stack? How can we automate some of this away? Or is the answer simply, own all the layers of the stack yourself?

View Details

I recently have started researching and playing around with RISC-V for fun. I thought it might be nice to combine some of what I’ve learned into a blog post. However, I don’t just want to highlight what I learned. I want to use this as an example of how to go about learning something new.

Recently, Erik St. Martin, Shubheksha Jalan, and I were discussing how we learn new things and we all thought it might be beneficial to have a way to document this process for others. What better way to document this then by example with my recent research into RISC-V?

I’ve said it before and I will say it again, I think anyone is capable of doing or learning anything, they just need the right motivation and to believe in themselves. I also made a point of including the book Super Brain on my list of recommended books, because it confirms with science that if you set your sights high you can accomplish great things, but if you set your expectations low it becomes a self-fulfilling prophecy. To put it more bluntly, believe in yourself!

I became fascinated by what is happening in the RISC-V space just by seeing it pop up every now and then in my Twitter feed. Since I am currently unemployed I have a lot of time and autonomy to dig into whatever I wish.

RISC-V is a new instruction set architecture. To understand RISC-V, we must first dig into what an instruction set architecture is. This is my learning technique. I bounce from one thing to another, recursively digging deeper as I learn more.

What is an instruction set architecture (ISA)? An instruction set architecture is the interface between the hardware and the software.

Models of processors can implement the same instruction set but have different internal designs for implementing the interface. This leads to various processors having the same instruction set but differing in performance, physical size, and monetary cost. For example, Intel and AMD have processors that both implement the same x86 instruction set but have very different internal designs.

In order to dig deeper, we should look into what some of the various types of instruction set architectures are.

What are the types of instruction set architectures? Most commonly these are described and classified by their complexity.

Reduced Instruction Set Computer (RISC) This only implements frequently used instructions, less common operations are implemented as subroutines. By using subroutines, there is a trade-off of performance, however it’s only applied to the least common operations.

RISC uses a load/store architecture; meaning it divides instructions into ones that access memory and ones that perform arithmetic logic unit (ALU) operations.

RISC, the name, came out of Berkeley in the 1980s (from a project led by David Patterson) around the same time MIPS (a project led by John L. Hennessy) was going on at Stanford. RISC became commercialized as SPARC by Sun Microsystems and MIPS became commercialized by MIPS Computer Systems. Both are RISC architectures. You might also be familiar with more modern implementations like ARM or PowerPC which are commercialized as well. There are many RISC implementations other than just these, I implore you all to dig further if you so choose.

RISC architectures can also be traced back to before the name existed as well. Examples include Alan Turing’s Automatic Computing Engine (ACE) from 1946 and the CDC 6600 designed by Seymour Cray in 1964.

Complex Instruction Set Computer (CISC) This has many very specific, specialized instructions, some may never be used in most programs. In CISC, one instruction can denote an execution of several low-level operations or one instruction is capable of multi-step operations and/or addressing modes.

The term was coined after RISC, so everything that is not RISC tends to get lumped here. It’s become somewhat of a contentious point since some modern CISC designs are in fact less complex than some RISC designs. The main difference is that CISC architectures have arithmetic/computation instructions also perform memory accesses.

Most architectures were classified after the fact since the term wasn’t around at the time of their birth. Some examples include IBM’s System/360 and System Z, the PDP-11, the VAX architecture, and Data General’s Nova.

Very Long Instruction Word (VLIW) and Explicitly Parallel Instruction Computing (EPIC) These were designed to exploit instruction level parallelism, executing multiple instructions in parallel. This requires less hardware than CISC or RISC and leaves the complexity for the compiler.

Traditionally, processors use a few different ways to improve performance, let’s dig into these.

• Pipelining divides instructions into substeps so the instructions can be executed partly at the same time.
• Superscalar architectures dispatch individual instructions to be executed independently in different parts of the processor.
• Out-of-order execution executes instructions in an order different from the program.

The methods above all complicate hardware by requiring the hardware to perform all this logic. In contrast, VLIW leaves this complexity to the program. As a trade-off the compiler becomes a lot more complex while the hardware is simplified and still performs well computationally.

VLIW is most commonly found in embedded media processors and graphics processing units (GPU). However, Nvidia and AMD have moved to RISC architectures to improve performance for non-graphics workloads. You can also find VLIW in system-on-a-chip (SoC) designs where customizing a processor for an application is popular.

EPIC architecture was based on VLIW but made a few changes. One of which allows for groups of instructions, called bundles, to be executed in parallel if they do not depend on any subsequent group of instructions. You can often distinguish EPIC from VLIW because of EPICs focus on full instruction predication. This is used to decrease the occurrence of branches and to increase the speculative execution of instructions. Speculative execution loads data before we know whether or not it will be used.

You might be familiar with speculative execution from the Spectre and Meltdown attacks. The Spectre and Meltdown attacks are a whole different rabbit hole I won’t go down in this post, but I hope you can understand how your own learning is almost like a choose your own adventure game. You can choose to go further down any path at any time.

Minimal Instruction Set Computer (MISC) This is more minimal than RISC. It includes a very small number of basic operations and corresponding opcodes. Commonly these are categorized as MISC if they are stack based rather than register based, but can also be defined by the number of instructions (fewer than 32 but greater than one).

Quite a few of the first computers can be classified as MISC. These include (but are not limited to) the ORDVAC (1951) and the ILLIAC (1952) from the University of Illinois and the EDSAC (1949) from the University of Cambridge.

One Instruction Set Computer (OISC) This describes an abstract machine that uses only one instruction. It removes the necessity for a machine language opcode. For example, “mov” is turing complete which means it’s capable of being an OISC, as well as other instructions using subtract.

This has not been commercialized, as far as I know, but it is very popular for teaching computer science.

This leads down a few paths, some can get into all the nitty gritty details of each instruction set and their differences. For the sake of learning more about RISC-V, let’s dig more into that specific design.

RISC-V Design There is a great paper on the RISC-V design from Berkeley. Chapter 2, “Why Develop a New Instruction Set?”, is my favorite. It goes over the pros and cons of a lot of prior instruction sets, why the authors decided to create a new instruction set, and what lessons they learned and brought over from their knowledge of the past. I will summarize what I thought was interesting but I urge you to dig in for yourself and read the entire paper.

For one, the authors state the importance of the fact that RISC-V is a completely free and open instruction set architecture. In contrast, all the most widely adopted instruction set architectures are proprietary. They are all also immensely complex. For example, you cannot get a hard copy of the x86 manual anymore and even in PDF form it’s ~5,000 pages and that doesn’t include the extensions. Who has time to read all of that? Although there is no exact number, it’s estimated there are around 2,500 instructions in x86, which is just unwieldy.

Props to Sun Microsystems for the fact that SPARC V8 is an open standard, but the design decisions are highly reflective of the other instruction sets from that time, leaving it unsuitable as a modern instruction set. “It was designed to be implemented in a single-issue, in-order, five-stage pipeline, and the ISA reflects this assumption.”

Alpha came out of Digital Equipment Corporation (DEC) in the 1990s so it got to be built with some learning from the earlier eras. However it seems like they over-engineered it. Most interestingly, they also did not think to create any room for extra opcode space for extensions. The authors also point out that ISAs can die and Alpha is a great example of an ISA being pretty obsolete outside of owning an old DEC computer, other than the last implementation by HP in 2004 when the IP changed hands again.

ARMv7 is widely used and the authors seriously considered it due to the fact of its popularity and ubiquity. However ARMv7 is a closed standard and cannot be extended making it unsuitable for the authors. They also found some technical problems as well, but the biggest determent to me was the fact it has over 600 instructions making it quite complex.

The authors go over a few more instruction sets but I think you get the point that none of them were suitable for their needs. Of course you are more than welcome to dig in further yourself, I am just not going to take the time to reiterate their work here.

Recapping how I learn The paper continues into the details of the design of the RISC-V architecture. Some of this I will cover in my DotGo EU talk. For the sake of showing how I learn things I urge you to read the paper yourself and when you hit a term or concept you don’t know: research that concept. Continue this until you get a general understanding then jump back up into the paper where you left off. This cycle is how I dive into new things.

At the beginning of this post I said I would take you down the path of how I dug into RISC-V, yet I have not even begun to describe the actual design or features of RISC-V. I did this to make a point (and because I was tired, maybe mostly because I was tired). Look how much I dug into the fundamentals of instruction sets before even digging into the thing I set out to learn. This is commonly what I find happens and I wanted to show an example of my process. Now you can go and continue the rest of the process yourself by continuing to read the RISC-V design paper, watching other RISC-V talks, getting some RISC-V books, or finding other RISC-V papers and learning from those.

Then, buy a board and start playing with it. I got the HiFive Unleashed and it’s awesome!

I hope this helps open your mind to learning and digging deeper on any topics that interest you. Happy learning!

View Details

I learned a lot about myself and the way big companies are organized over the past year or so. I had mentioned a bit in a previous blog post and podcast about “the N + 1 shithead problem” (from Bryan Cantrill’s talk on leadership). To reiterate, the “N +1 shithead problem” occurs when you are demotivated by seeing people who are a level above you behave poorly, or more bluntly when they behave like a shithead. I know from experience what a huge demotivator this is and after talking to several other folks I realized this is quite common.

When faced with this demotivator, I found myself thinking “why would I want to be at their level, when once I get there I’ll just be one amongst the dipshits.” It’s a horrible feeling to have and I’d love to have a model that resembles what I think of as a distinguished engineer or technical fellow.

In this post I will define what it means to me to be a distinguished engineer or technical fellow and maybe others that agree will modify their ladders to incentivize people to resemble these qualities.

Technical Leader The first thing people think of when they think of a distinguished engineer is that they are a technical leader. I fully agree. A technical leader can understand all parts of a system. They can also be dropped into a new system and pick up the way it is architected and designed with relative ease. I think this is an important distinction to make. It’s good to be an expert in a field, but only being an expert is limiting. It’s also important to understand the full picture and that takes general knowledge. I think having a general knowledge of things outside your area of expertise is key if you choose to gain expertise in something.

Value learning A technical leader can always realize that there is more to learn. One cannot be an expert in everything and you can have a general knowledge of most things without fully understanding the details within. A technical leader can always strive to continue to learn and persuade others to continue to learn as well.

Empower others A technical leader can build up others and empower their colleagues to do things that are more challenging than what they might think they are capable of. This is key for growing other members of an organization. I personally believe you don’t need a high title to take on a hard task, you just need the support and faith that you are capable of handling it. That support can come from the distinguished engineer and be reflected in their behavior towards others.

A technical leader can also make time for growing and mentoring others. They can be approachable and communicate with their peers and colleagues in a way that makes them approachable. They can welcome newcomers to the team and treat them as peers from day one.

Give constructive technical criticism A distinguished engineer can never tear others down but they can be capable of giving constructive criticism on technical work. This does not mean finding something wrong just to prove their brilliance; no, that would make them the brilliant jerk. Constructive criticism means teaching others to make their work better when there are problems, while also encouraging them to iterate and empowering them to succeed.

Have opinions loosely held A technical leader can be able to have opinions loosely held on designs and architecture. Making an active effort not to say “strong opinions, loosely held” because with a power dynamic that could over power the rest of the voices. Technical leaders can make sure all voices are heard and they can fully articulate the “why” of their opinion for others.

They do not need to have opinions on everything, that would be pedantic. Technical leaders can be able to use their experience to help others succeed, while also empowering others to own solutions. Technical leaders can not pass down solutions to problems but allow others to learn by letting others come up with solutions themselves. This is where good constructive criticism (from above) can come into play.

Great communicator and bridge A technical leader can have strong communication skills and be able to articulate the “why” of a problem as well as articulate the technical details of designs. They can never communicate in a derogatory manner. They can always communicate to others as peers and colleagues.

At times, technical leaders will need to act as a bridge between teams. It is really important to be able to clearly communicate then as well as always.

Humility and empathy A technical leader can not be driven by ego but by a constant urge to learn and grow both themselves and their colleagues. They can have empathy for others and portray kindness towards their peers and colleagues.

Prioritize shipping and decisiveness A technical leader can value shipping and decisiveness. They can not be susceptible to analysis paralysis. At the end of the day most people have jobs to get things out the door and this can be a priority. Of course, shipping can not come with the trade off of burning out a team or setting the company on fire.

Customer focused Technical leaders can always seek feedback from their customers. This might be the internal customers of their infrastructure or external customers if they are on a product team. The best technical leaders are capable of empathizing with customers and iterating quickly on customer feedback.

Build resilient systems A part of being a technical leader is having the experience of building multiple systems in the past. Distinguished engineers can be able to anticipate various failures from their past experiences and build systems that will not create the same failures. Of course no system is perfect so they can be able to learn from the failures they cannot anticipate as well. This is a cycle that they can then use when building the next system.

Value quality, performance, and security Great technical leaders value quality, performance, and security in what they build. They stay up to date on advancements and research in technology so that they might be able to use new techniques for bettering their solutions. Technical leaders can also build with respect for users and their privacy.

Value maintainability Technical leaders can value writing code that is easy to maintain and easy to understand. They can value unit and integration tests as well as making sure if a bug is fixed it has a test to make sure there is not a regression. Technical leaders can use code comments, not as a garnish, but to denote things a reader would need to know. This could be details of a code section that fixes a specific bug or maybe reasoning behind why something is written a certain way. Documenting context is super valuable and helpful for maintainability.

Community Good technical leaders are also leaders in the outside communities. This can include giving talks on various things they have built as well as mentoring others in the community or the workplace.

Learn from external community If you silo yourself to only learning within your company, you are missing out on a world of experiences and expertise different than yours from the external community. Technical leaders realize this and place importance on learning from the larger world of computing than just their silo.

Value listening and be open to feedback By gaining feedback and making yourself visible to an external community, leaders avoid a dunning-kruger like effect of only growing inside an echo chamber. It is always valuable to see where the rest of the industry is focusing and how technical leaders at other companies are solving problems. Technical leaders realize that there is much to learn from people with different experiences than their own. They can always be open to listening to others.

Humility Technical leaders can always remain humble and modest. The best technical leaders know that it’s not possible for them to know everything and will prioritize keeping an open mind to always be learning.

Call upon other experts The best technical leaders know when they need to call on experts in specific areas for help or feedback on certain designs or architecture. By participating in the external community, leaders form strong networks and bonds with fellow engineers they can call on when they need them. Technical leaders can always be eager to use these relationships when they need them or introduce others to these folks if they could use their expertise.

Value research Along with being able to call upon other experts, technical leaders can value well researched solutions. They can strive to learn from prior art.

Have fun Always make sure to have fun and not take yourself too seriously!

Take the long view, not just the broad view.

These are just a few of the things I think define a strong technical leader and engineer. I am sure I will grow this list as I personally grow myself every day.

Most importantly you must actually do these things. Actions speak louder than words.

View Details

I have written a bit about how I am spending my time while being unemployed and I thought I would continue.

There was one thing I had left out of my previous post on my visit to the Pentagon. THEY HAVE A REAL ENIGMA MACHINE THERE. Okay, moving on…

QCon and University of Cambridge I gave a talk at QCon on SGX and ended up giving the same talk to some really awesome folks at University of Cambridge. Each time I gave the talk provoked some really interesting conversations. One of the topics that came up a couple of times was if RISC-V was going to be supported by any major cloud provider anytime soon. My honest opinion, which some might disagree with, is this is years away BUT it would certainly help adoption and integration into projects if it was backed by a company with a lot of time to develop integrations. Also I got a bit nerd sniped by some ARM folks and researchers to look more into TrustZone (which is the ARM secure enclave). I haven’t dug in yet but it’s on my list.

It was awesome spending a day in Cambridge (thanks Anil for the tour!) and learning about all the awesome things they are doing. The MirageOS team is booting unikernels on baremetal RISC-V!

🎉OCaml boots on bare-metal @ShaktiProcessor @risc_v! 🎉 An important milestone towards building safer apps using @OpenMirage on open source hardware. pic.twitter.com/XFosAxPROR

— KC Sivaramakrishnan (@kc_srk) March 1, 2019

They use this on boards to power light bulbs (at the University!) super securely since it removes the need for all the shitty firmware most other things ship and has a super minimal environment. I’m sure you can think of a number of different other use cases as well. Honestly, unikernels replacing all the crap firmware in the world would be a huge win.

Open Compute Summit Just this past week I spent a day at the Open Compute Summit. What is happening there in the open firmware space is truly awesome. They had demos of hardware they are booting with LinuxBoot and Coreboot. Facebook runs this on their infrastructure as well as with OpenBMC to replace the traditional, proprietary BMC firmware. Trammel Hudson has some great posts on LinuxBoot, which include links to some really great talks by him and Ron Minnich.

😍 the open systems firmware community is awesome pic.twitter.com/DAqudm6M4Z

— jessie frazelle 👩🏼‍🚀 (@jessfraz) March 14, 2019

Facebook’s server racks are gorgeous. They have a power bus which runs down the center and everything gets power from that, with the main power coming out of the power unit towards the middle of the rack (in the first picture below).

The Facebook rack and node designs are seriously gorgeous, simple. The power bar chef kiss pic.twitter.com/pGphy9uLLl

— jessie frazelle 👩🏼‍🚀 (@jessfraz) March 14, 2019

Boot Guard One thing I learned that I found fascinating was about Boot Guard for Intel processors and the equivalents on ARM and AMD. Boot Guard is supposed to verify the firmware signatures for the processor. The problem with this, in Intel’s case, is only Intel has the keys for signing firmware packages. This makes it impossible for you to then use Coreboot and LinuxBoot or equivalents as firmware on those processors. If you tried, the firmware would not be signed with Intel’s key and would brick the board. Matthew Garrett wrote a great post about this as well.

If a person owns the hardware, they have a right to own the firmware as well. Boot Guard prevents this. In another great talk by Trammel, he found a vulnerability to bypass BootGuard.

CVE-2018-12169 also potentially allows a developer to "jailbreak" their BootGuard protected laptop since the UEFI DXE volume can be replaced with a user provided LinuxBoot ROM image. https://t.co/yHwwMOTyx7 pic.twitter.com/MeWI0DGUBf

— Trammell Hudson ⚙ (@qrs) September 24, 2018

This “feature” from hardware vendors is preventing the innovation of this community and preventing pushing technology to a safer place. If you are in a position to push back on these hardware vendors, please do so. They need all the help they can get.

Server rack encased in liquid Lastly, I saw something bat shit crazy at Open Compute Summit. It was something I saw in the Expo Hall. One vendor has encased an entire server rack in liquid for liquid cooling. I’m not sure I could sleep at night using this. The funniest part about this though was the demo at their booth still had fans in the rack! I mean… why would you need fans if you had liquid cooling… they claimed it was just “left over” and you wouldn’t need that. But at a conference where everyone is showing off their custom hardware, you’d think they would have left the fans at home ;).

That’s the end of this update of my adventures. Hope you all enjoyed it. I know I enjoyed living it!

View Details

I stated in my first post on my reflections of leadership in other industries that I would write a follow up post after having hung out in the world of finance for a day. This is pretty easy to do when you live in NYC. Originally for college, I was a finance major at NYU Stern School of Business before transferring out, so I have always had a bit of affinity for it.

I consider myself pretty good at reading people. This, of course, was not always the case. I became better at reading people after having a few really bad experiences where I should have known better than to trust someone. I’ve read a bunch of books on how to tell when people are lying and my favorite I called out in my books post. This is not something I wish that I had to learn but it does protect you from people who might not have the best intentions.

Most people will tell you to always assume good intentions, and this is true to an extent. However, having been through some really bad experiences where I did “assume good intentions” and should not have, I tend to be less and less willing to do that.

I am saying this, not because I think people in finance are shady, they aren’t, but because I believe it is important in any field. I, personally, place a lot of value on trust and integrity.

I’m not really going to focus this post on what an investment bankers job is like because honestly it wasn’t really anything to write home about. What I did find interesting was the lack of trust in the workplace. Trust is a huge thing for me, like I said, and I think having transparency goes hand-in-hand with that.

To gain trust, I believe a leader must also have integrity and a track record of doing the right thing. I liked this response to a tweet of mine about using “trust tokens” in the case leadership needs to keep something private.

They are. It gets hard with legal things like SEC filings and acquisitions but that’s where an already good leadership team can use existing trust tokens.

— Silvia Botros (@dbsmasher) February 21, 2019

I think people tend to under estimate how important it is to be transparent about things that don’t need to be private. I’ve seen a lot of people in positions of power, use their power of keeping information private against those under them. They don’t fully disclose the “why” and it leads to people they manage not fully being able to help solve the problem as well as not fully understanding the problem. It also doesn’t build trust.

Leaders should try to be cognisant of when something needs to be private and when they can be transparent about information. I also really enjoyed this insightful tweet as well:

Unlike respect, which can start from a positive value and go up or down depending on behavior, trust starts at 0. You have to earn the trust of your colleagues and reports before you can take loans out on it. https://t.co/aWRpdjAtBR

— julia ferraioli (@juliaferraioli) March 1, 2019

Just thought I would put my thoughts in writing since I said I would. This experience seeing how other industries work has been super fun for me. I might try to find some other jobs to check out as well in the future.

View Details

I’ve had a bit of a crazy week. Tuesday, I got a tour of the Pentagon from a friend that is in the US Digital Service (USDS) for the Department of Defense (DoD), called the Defense Digital Service (DDS). Wednesday (the day of writing this), I shadowed a friend who is a surgical resident during their shift in a hospital. Friday, I have plans to shadow a friend who is an investment banker at a private equity firm and will do a follow up post. You can consider this like “Eat. Pray. Love.” except it’s “Government. Medicine. Capitalism?”

First, I would like to thank everyone for sharing a bit of their life with me and now I get to share what I learned from these experiences with you. When I went into this, I didn’t think much of it. I wanted to go to DC to see some museums and ended up texting my friend on the way down so we made a day of it. My other friend, who is a surgical resident, and I had once gotten into a pretty deep discussion about how weird tech’s culture is compared to theirs so I always had an open offer to see how they work. Then I posted on twitter what I was doing and I guess there was a sort of pattern so it became a thing…

Sweet, it’s on, Friday I’m going to be a douchey investment banker at Lehman Brothers, no just kidding some private equity firm, but I nailed the joke I’ll fit right in ;)https://t.co/yz3EgKD2Ib

— jessie frazelle 👩🏼‍🚀 (@jessfraz) February 27, 2019

Let’s dive into what I’ve learned and observed, then I will try to put a nice ribbon on it and tie it all together for you.

Government. Let me start by saying, if you ever have a chance to do a stint at the US Digital Service it seems absolutely amazing. The program is great for tech people who want to have an impact on modernizing technology for the government. Having just left a job at Microsoft, I was quite familiar with a very large organizational structure and the power dynamics that exist in people with titles. It was interesting to me to see the parallel between that and the setup of the government.

When you see someone who has served time in the military they usually have a set of badges showing their accomplishments. This is cool because it is accomplishment based. I love accomplishment based incentive systems since you have to do something in order to get rewarded. There are badges for everything, one I really liked was the “ranger tab” which effectively means if that person was ever dropped in the middle of nowhere they could fend for themselves and survive. WOW, what a meaningful item. I found this system really cool since I like decoding things and I personally hate titles that mean nothing. This badge system holds a lot of meaning and defines what the person has done.

Before going to the pentagon, I had watched this Bryan Cantrill talk on leadership where he brought up the “N+1 shithead problem”. The “N+1 shithead problem” happens when there is a person (who acts like a shithead) in a title bump above you and how it is a huge demotivator. What helped Bryan get over this was: instead of looking at the shithead a title above him, he focused on the best person in the title above him and used them for motivation. This works to an extent. I know from experience how demotivating it is seeing a shithead consistently fail up. I believe that most titles are bullshit, climbing a career ladder is bullshit, what really matters is what you do and what impact you have. The talk also covers how Bryan set up his team to only have one title, Software Engineer. And that he motivated his team with a purpose and a mission, not with climbing a ladder.

My friend and I ended up getting into an interesting conversation about this and how they handle authority and titles at the DDS and within the government. The way the DDS program is set up, the individuals who join are at a colonel level rank, which is one below a general, which means they are pretty high up in the pecking order. They also have orders from the Secretary of State to override any authority if need be. They end up not needing to escalate to using those orders though, since just the threat of using it is enough to get bad actors to listen to them.

The Defense Digital Service (DDS) also recruits internally from inside the government. If there is a truly exceptional individual technically in another role they will recruit them to the DDS. They have had people from the Army, Navy, and other parts join. Since the structure and dynamic of the organizations where they came from is so different, joining the Defense Digital Service (DDS) ends up having an effect on them. Before being a part of the DDS, they typically could not fight back from those with authority over them and the DDS is all about fighting for the truth, and fixing what is broken, even if they are the only ones that wear hoodies and not a uniform.

When a general or authority comes into the office of the Defense Digital Service they aren’t greeted with coffee and have their feet kissed, they are just asked to sit on the couch side-by-side their peers (the DDS) and to talk about things like colleagues. This is how leadership should work, not with power over someone else but working with others as peers and colleagues.

Another thing I learned was that for non-confidential code, they use GitHub and try to modernize agencies they work with to do the same and use modern languages and tools. It also reminded me of this awesome article about how someone had changed the law via a GitHub pull request because the District of Columbia’s legal code is hosted on GitHub. Wouldn’t it be cool if that’s how every part of the government worked? Then, if you wanted to change a process or law you would just send a pull request…

This is just a few of the things I learned in my day at the Pentagon, again I highly recommend applying for the USDS if this is interesting to you. Let’s move on to medicine…

Medicine My friend had a shift in a hospital here in New York City and I asked to follow along. I got to wear scrubs and everything. I also had to wake up at the crack of dawn for this (5am). This is the fourth year of my friends surgical residency so he’s considered pretty senior since usually that’s a five year thing.

The “junior residents” report to “senior residents” (my friend) who then report to the “attending physicians”. George Clooney was not their attending physician, I was disappointed since I like the show ER ;). Back to reality, we did rounds and checked on all the patients and then I got to watch a surgery. That was super cool, also not my first surgery since I had shadowed a friends dad who was an anesthesiologist in high school.

What I really took away from the day was the respect that the attending physicians had towards the senior residents. There was a lot of respect from the attendings and the senior residents seemed to have a lot of autonomy. In the operating room, the surgery was lead by a different senior resident and the attending was mostly passing tools. I thought this was super cute. I even called it “super cute” out loud after…. to which my friend rolled their eyes. But really, the surgery was done “as a team” with no one calling out orders to someone like a “code monkey.”

I thought this was great and in stark contrast to what I see when technical people are promoted to manager. Ill-trained managers pass down technical work without telling the “why” and already arriving at a solution. In contrast, it is actually the team’s goal to come to a solution, not the managers. That is not actually a part of being a manager. I wish more managers would focus on managing and growing the people on their team versus using it as a position of power over the technical work. If they still need to do technical work in that role it should be like that of “passing the scalpel” when someone needs it.

There was also a point in the day where my friend helped a junior resident with some assignments. It was super interesting and I wondered if an open source mindset could help here. I remembered a talk I saw at Linux Conf Australia in 2018 on open source pharma. The talk focused on how the open sharing of research is leading to innovation in biomedical research.

What I love about open source is the ability to share knowledge and “ping” an expert when you need it. We did this with docker a couple times, when we “pinged” the kernel namespaces maintainer on features to make sure we had implemented it correctly. It would be pretty cool to be able to learn and collaborate with the best easily in any field.

Tying it all together In both Government and Medicine I found hierarchical structures to learn from. The badges in the military as a system of tracking accomplishments and not power really spoke to me. As well as the attending physician passing the tools to the senior residents and working as a team versus the attending taking charge completely.

I think both Government and Medicine could be changed in a way that would also change the world if more laws and open knowledge wound up on GitHub.

We live in a world where Reddit, Twitter, and other social sites are littered with hate and fake news. I wish there was a place for a source of intelligence and knowledge. I think that would change the world while also allowing the world (or a law) to be changed with just a pull request.

View Details

You can find my goodreads account at goodreads.com/jessfraz.

Romanticized Tech I call this genre of books “romanticized tech” because of the way tech is portrayed in them in a very idealistic and whimsical way. It’s nice to pick up one of these if you are feeling very “Black Mirror” to remember why you might have even started in this field.

• Soul of a New Machine: Bryan Cantrill recommended this to me and it’s amazing. It’s about Data General building a new computer and the passion the team building it put into it. I wrote a post with some thoughts on it and so did he.
• A People’s History of Computing in the United States: This book has a bunch of short stories that chronologically take you through some of the most important moments of computing history. I loved reading this after “Soul of a New Machine” because it really tied in nicely to the references of Data General and DEC.
• The Big Machine: I found this book at the infamous Bell’s books in Palo Alto. It was all worn like someone had previously loved it a lot and it makes me love it even more. It’s about CERN and how it came to be and has the same sort of romanticized view of tech as “Soul of a New Machine” and “A People’s History of Computing in the United States”.
• On the Edge: The story of the Commodore computer company. Goes through the entire history. Super desnse, but over all interesting.
• Where Wizards Stay Up Late: This book details the story of how the Internet came to be!

Non-Fiction * Spy the Lie: Former CIA Officers Teach You How to Detect Deception: I have now read this book twice. It is amazing if you want to be able to read when people are lying to you. It’s a good read backed by a lot of experience from the CIA. Honestly, after reading it the world will be a much different place. * Super Brain: I loved this book. It uses science to describe how the brain processes different emotions and what that does to your overall health. It will leave you with all sorts of good feelings after as well as teaching you quite a bit about misconceptions on how the brain works. * “Surely You’re Joking, Mr. Feynman!”: Adventures of a Curious Character: A witty book taken from short stories the notorious professor used to tell. Awesome read, flows quite quickly and is fun. It’s filled with fun little physics and life lessons. * “What Do You Care What Other People Think?”: Further Adventures of a Curious Character: More Feynman stories just like the above. * Quiet: The Power of Introverts in a World That Can’t Stop Talking: This is an awesome book and you should watch her TED talk as well. It’s about the power of introverts and how being an introvert should not be something to be ashamed of but rather proud of. * Brief Answers to the Big Questions: If you have read “A Brief History of Time”, you will like this follow up answering some of the larger questions of the universe. * Ego is the Enemy: This book is a great reminder in staying modest and humble. Ego so often gets in the way of great leadership and success and I greatly enjoyed reading a book that focused on self-confidence without ego. * The Datacenter As a Computer: Designing Warehouse-scale Machines (Synthesis Lectures on Computer Architecture): This is a overview of how Google designs their datacenters. Overall, super valuable if you work in the space of high-scale compute. I only wish it disclosed more of the reasoning behind certain technical decisions. * A Programmer’s Introduction to Mathematics: I was a math major so I have a huge fondness for mathematics. This is a great book about math from the point of view of programming. * Switch: How to Change Things When Change Is Hard: I got this book as a recommendation from Lara Hogan. It is a great read if you are trying to change something in a culture that does not embrace change. It really details a great approach for doing so that feels like it could almost be weaponized :). * The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change: Every single book list should include Camille’s book. It is a great read for managers and non-managers and has given me the tools for knowing what is normal and what is not. * Accelerate: I cannot believe I forgot this book the first time around, great for high-performance teams who want to ship software, based on real data, and written by the badass Nicole Forsgren. * Dear Founder: This book takes you through the evolution and stages of starting a business. Its in the form of letters and a good and fast read. Also seems like an effective reference to flip back to when you need specific advice in a pinch. * Good Strategy, Bad Strategy: Nicole recommended this book to me and it is a very good approach to strategy, real. It really focuses on keeping things transparent and real. * The Last Days of Night: A story about this guy that gets sued by Thomas Edison. Super fun to read, has a lot of history interwoven in. Tesla makes an appearance and there’s lots of drama with false motives. Great book :) * The Hard Thing about Hard Things: This book is great when it comes to management and also having empathy for those faced with hard decisions. It puts an emphasis on transparency and saying things “like they are” and I really appreciate that. Also there are rap quotes. * Finite and Infinte Games: This book is kinda a mind trip in the best ways. I’ll leave it at that. * Deep Down Things: The Breathtaking Beauty of Particle Physics: A great introduction to particle physics. * The Character of Physical Law: Richard Feynman’s breif overview of the laws of physics. * The Feynman Lectures on Physics: The complete set of Feynman lectures on physics. * The Particle Odyssey: A Journey to the Heart of Matter: Great introduction to particle physics with tons of illustrations. * The Telomere Effect: Science behind aging and how you can change your lifestyle to help you live longer.

Bookshelves If you are interested in books and/or bookshelves I started a thread with pictures of bookshelves and there are some great find in here:

These are my favorite two shelves of my bookshelf, and yes that’s a slug Jerry. Show me your bookshelves, doesn’t just have to be tech books :)

(most of these are from my grandpa :) pic.twitter.com/0qiqytAYuL

— jessie frazelle 👩🏼‍🚀 (@jessfraz) February 23, 2019

View Details

A few of you, thank you, have reached out to me saying that you love my writing style. It means a lot to me because I like to think that I write how I speak. This was not always taken well, however. I tend to be a bit of a sarcastic troll.

The following post is meant to show others who may be like me and hesitant towards their writing style due to feedback they’ve gotten. I’d love to empower them to be comfortable with themselves.

In high school, I got my first D on an assignment ever in AP English. It was on my senior thesis. Now, if you haven’t been able to tell already, I am a bit of a troll and I also value hearing all perspectives and then finding the truth somewhere in the middle. In this sense, I wrote my senior thesis on viewing the moon landing from the other side and trying to prove it was fake. Today, this is probably equivalent to trying to prove flat-earthers are correct. I had some shady sources as you can imagine and the whole thing was written with a large dose of satire. My English professor, on the other hand, was not one for jokes I soon learned because I landed myself a big fat D on the assignment. Luckily for me, I had already been accepted to NYU early admission so other than a large dose of feeling like shit (I am a perfectionist) I decided to not give any fucks.

I tried to find that paper and couldn’t, but instead I found my college entrance essay, which is in the same style. I worked at a pharmacy all through high school and on breaks from college and some pretty weird shit happened. My mom thought this was a terrible idea for a college essay and I’d never be accepted anywhere and my dad loved it. Never be ashamed to be yourself and think differently. So here it is…

Pharmy Tales I work in a pharmacy, which sounds like a pretty normal job where nothing of great importance happens. Don’t jump to conclusions, however. Here is a bundle of short stories-using no real names; I like to call them “Pharmy Tales.”

……

Every time Mrs. H calls the pharmacy she complains, either about her bill or her latest delivery of medications. Unfortunately, I usually answer the phone.

“Camelback Village Pharmacy,” I squeaked.

“Is Dan there?” Mrs. H asked sending a chill down my back. Dan is the owner. Just the sight of him causes even the angriest customer to give up their battle.

“Tuesday is his day off. Would you like to speak to Laurie?” I answer.

“No, I would not. I just received my bill for July and it has two delivery charges on it. How do you explain that?”

“Did you get two deliveries?”

“That’s not the issue. Every time I get my bill something is wrong. I shouldn’t have to always second check your billing statements.”

“I’m sorry.”

“Well ‘sorry’ doesn’t cut it. Make sure Dan calls me tomorrow.” The receiver clicks.

I felt like I had just run a marathon.

“I’ll pick up the phone in case she calls again,” offers Ross, another employee.

The phone rings about 10 minutes later and Ross answers. As it turns out, it’s a call from a different customer thanking us because he received his mail-out order and it was perfect.

Just my luck.

……

Ross hates Mr. W, an infamous customer known for his inappropriate language. One Saturday, he strolls in drunk with his dog, Bella. I love Bella. She looks like Beethoven. Ross immediately ducks down behind the counter.

“We just had a scotch on the rocks and we’re really feeling it right now,” says Mr. W, referring to himself and his dog. “Did you order my cane, Dan?”

“Yes, I have it right here,” Dan holds up the cane.

“That’s not it, that looks like a turd. Send it back or even better, throw it away,” Mr. W yells. By now Ross’s back is killing him so he is forced to stand up.

“Is that the sexy boy from Germany?” exclaims Mr. W, pointing at Ross. He then spends the next thirty minutes interrogating Ross about the World Cup and whether or not he lost his virginity. By the end of their conversation, Bella was asleep on the pharmacy floor and drooling.

……

In the middle of one routine day at the pharmacy, a woman passed me a prescription, I passed it to the Pharmacist, and I continued into “La-La-Land” for the next twenty minutes.

The pharmacist realized that the prescription was fraudulent and inconspicuously called the police.

Ross stalled her with a conversation. She decided to leave and come back when it was ready. Five minutes later, Ross met the police outside the pharmacy and told them what the woman had looked like.

On her way back to pick up the prescription she identified Ross with the Police and she rolled under numerous cars in the parking lot to escape. The police soon caught and arrested her.

One officer came into the pharmacy to talk to the rest of us. As the very attractive police officer glided down the allergy and laxative aisle, I snapped out of “La-La-Land” and questioned what was going on. I had no idea all of this action was taking place just yards away from our neighborhood pharmacy.

……

Don’t be fooled by the misconception that working in a pharmacy is dull. Everyday, I come home with a new story to tell.

View Details

I like to consider all the variables in a problem space before coming to a conclusion. As humans we have a tendency to jump to conclusions rather quickly. I try not to do this but everyone makes mistakes.

More information about Intel SGX was brought to my attention after my initial blog post on it. I’d like to take the time to go through that information and my current thoughts on the technology after having this extended context.

Trammel Hudson (@qrs) pointed out to me yesterday that SGX was originally built for the use case of DRM for Netflix, Microsoft, etc. Having this context makes the problems that arise when you try to do code execution inside an enclave seem like a forgivable sin. It was not until the HAVEN paper that people even considered using enclaves as an execution environment. In that regard, the HAVEN paper was truly novel. I may disagree with shoving an entire operating system in there, but the idea of executing code in an environment with encrypted memory as a way to use the cloud without trusting the cloud is a respectable feat.

Another person who I truly respect and admire for the thought they put into what they build is Joanna Rutkowska (@rootkovska). She recently started working at golem a shared compute providing company focused on security and privacy. She wrote an awesome blog post considering all the tradeoffs of a technology such as SGX. The post links to other posts where she really weighs the pros and cons of the technology. This is why I really respect her thoughts on the matter. The solution is pretty cool in that you can run docker containers inside the enclave. It’s better than the SCONE paper, which also runs containers, in my opinion, because it doesn’t do the crazy syscall toss outside the enclave. It’s more aligned with the HAVEN paper in that it includes all the code inside the enclave. Her post is great; it really goes into detail on their thought process and what they designed their solution to prioritize.

Considering SGX was not built as an execution environment, I think it will be interesting to see where Intel takes this technology in the future now that people are using it as such. It will also be interesting to see how they solve the problems with side-channel attacks. Computing is all about tradeoffs. I learned from experience with everything I’ve worked on that people will use it for things it was not built for. This happened a lot with Docker. It’s always fun to see the new ways people use what you build and then to iterate considering the new use cases.

I value taking all contexts into consideration when thinking about a problem. I hope you all do the same. Hope you enjoyed my additional learnings and thoughts. Always be learning and open to new thoughts.

I’ll be giving a talk on SGX at QCon London the first week of March :) hope to see some of you there.

View Details

I’m a huge, HUGE, fan of LD_PRELOAD let me tell you… oh wait it’s my blog so I’m going to. Where do I begin…

About three years ago, I wrote a blog post about the 10 LDFLAGS I love. After writing the post, I realized I should have made the number odd because I think that is part of BuzzFeed’s “click algorithm.” But more seriously, I realized just how many people on the internet you can upset when you don’t include LD_PRELOAD in your favorite LDFLAGS post. I am going to take the time right now to make one thing very clear, VERY CLEAR, listen closely: LD_PRELOAD IS NOT A FLAG. It is an environment variable. Wake up sheeple! Phew!

Now that’s out of the way, we can continue… I love LD_PRELOAD. I love it so much I am devoting this entire blog post to professing my undying love for it. So here we go…

Background For those who don’t know what LD_PRELOAD is: TODAY IS YOUR LUCKY DAY! LD_PRELOAD allows you to override symbols in any library by specifying your new function in a shared object.

When you run LD_PRELOAD=/path/to/my/free.so /bin/mybinary, /path/to/my/free.so is loaded before any other library, including libc. When mybinary is executed, it uses your custom function for free. PRETTY FREAKING AWESOME RIGHT!

FEEL THE POWER! Okay, so moving on…

Fun Times on the Internet One night, I’m just hanging around in my apartment, laying on my couch, and I think “oh I’m going to ask the Internet what they’ve done with LD_PRELOAD.” This is how most of my tweets start for what it’s worth. So I asked…

yo internet nerds, tell me all the ways you've done dirty things with LD_PRELOAD…. I need them…. for… science…

— jessie frazelle 👩🏼‍🚀 (@jessfraz) January 21, 2019

This tweet blew up in THE BEST WAY! I got some really cool responses I will highlight below.

Not mine but my favorite: https://t.co/zljcn70pmh

— ダデイさま (@leifwalsh) January 21, 2019

$ FORCE_PID=42 LD_PRELOAD=./getpid.so bash -c 'echo $$'
42

For forcing specific bad ssh key generation when the RNG was busted…

— 𝙺𝚎𝚎𝚜 𝙲𝚘𝚘𝚔 (@kees_cook) February 10, 2019

i didn't use this but dropbox recently stopped working on non-ext4 filesystems and there's this LD_PRELOAD hack to make it work anyway https://t.co/DqRL12FNMk

— 🔎Julia Evans🔍 (@b0rk) January 21, 2019

Intercept readline calls to add undo to any interpreter that uses readlinehttps://t.co/M44lDMaeFyhttps://t.co/aoeldkK4X6 pic.twitter.com/w84O715eQG

— Thomas Ballinger (@ballingt) January 21, 2019

We actually mention this in an academic paper! https://t.co/qg5ac6vXx7 We used LD_PRELOAD to interpose on the OnStar software modem audio interface.

— Karl (@supersat) January 21, 2019

I wrote a silly hack that let you mount an app’s objc runtime as a filesystem so you could easily browse the class hierarchy. It could be inserted via dyld. Here is a screenshot of the Finder browsing the runtime. https://t.co/zyYxSsGaoS

— Bill Bumgarner (@bbum) January 22, 2019

enabling rapid-fire railguns in quake3 rocket arena by hooking gettimeofday() via LD_PRELOAD, enable/disable by hooking strstr() and using console commands

— HD Moore (@hdmoore) January 21, 2019

I made a thing to disable SSL certificate verification in a bunch of popular applications/libraries 😈https://t.co/jMWQtbl0Kb

— Dаvіd Вucһаnаn (@David3141593) January 21, 2019

This isn’t all of them but isn’t the internet utterly awesome! You can poke through the thread more and find ones you love as well. But let’s move on to some mad science…

SCIENCE No, not the Incubus album… but my science experiment that I did with LD_PRELOAD. My friends, Greg (@grepory), Aditya (@chimeracoder), and I came up with this absolutely insane idea for “kernelless”. Yeah, it’s a joke making fun of all the other “-less”s. But ours was special, m’kay. Greg even made a dope website for it, kernelless.cloud.

So the way we were going to implement this in a mad science way would be as “Cloud Native Syscalls.” Let me tell you about the “Cloud Native Syscalls”…

Cloud Native Syscalls The first part of the “Cloud Native Syscalls” architecture consists of a daemon on a cloud VM which has a network endpoint accepting incoming syscalls and their arguments. The daemon then performs these syscalls, almost in a code execution as a service type way.

To use “Cloud Native Syscalls”, you compile your binary with the library as follows: LD_PRELOAD=/path/to/my/cloudnativesyscalls.so /bin/ls. This ensures that all your syscalls when you run ls on your host are actually performed in the cloud and sent to the daemon described above.

F’king nuts right… I know. We are working on our A-round don’t worry. It’s truly revolutionary.

Anyways, that was our little science experiment. Hope you liked it, or at least enjoyed all the other people’s fun hacks. :) Keep LD_PRELOADing.

View Details

From the Intel x86 Manual:

In the mid-1960s, Intel cofounder and Chairman Emeritus Gordon Moore had this observation: “… the number of transistors that would be incorporated on a silicon die would double every 18 months for the next several years.” Over the past three and half decades, this prediction known as “Moore’s Law” has continued to hold true.

Moore’s Law is coming up a lot lately in the context of coming to an end. It’s kind of been a running joke for quite some time though so I think there is still a bit of skepticism around claiming it’s ending. However, Moore’s Law ending can mean a lot of different things for the future of computing.

Golden Age of Garage Computer Builders Personally, I look back on the golden age of computers as the time when people were building the first personal computers in their garage. There is a certain whimsy of that time fueled with a mix of hard work and passion for building something crazy with a very small team. In today’s age, at large companies, most engineers take jobs where they work on one teeny aspect of a machine or website or app. Sometimes they are not even aware of the larger goal or vision but just their own little world.

Back in the garage computer building era (or so I will call it), a very small group of people aligned on a mission could create something bigger than themselves and have immense impact. This is more aligned with how startups work, in my opinion, in that small groups of people with the same end goal build something together.

Soul and Passion This break I read The Soul of a New Machine, thanks @bcantrill for the recommendation. (He also wrote an amazing blog post on it.) In the book, a small team built an entire machine.

The book really hit home for me on so many levels. The team wasn’t driven by power or greed, but by accomplishment and self-fulfillment. They put a part of themselves in the machine therefore producing a machine with a soul.

The chapter 15 of The Soul of a new Machine is so powerful. For the engineers it wasn’t about recognition it was about accomplishment and self-fulfillment. And putting a part of them in the machine. pic.twitter.com/Lr1T5OVamM

— jessie frazelle 👩🏼‍🚀 (@jessfraz) January 29, 2019

Not only did the team have a very strong bond, but it was built on trust. The team was made up of programmers with utmost expertise and experience and also with new programmers. I love this detail. One of the stories from the book is about how they thought about creating a simulator for the machine to iterate more quickly. Well West, the most senior, wrote it off as impossible in the given time, but one of the new programmers brought it to life and wrote it. It’s amazing what a person can do when they don’t know something is impossible and are empowered to take on a task.

I loved this book in the same way I love Halt and Catch Fire. It’s a TV show based in the same time about building computers and gaming software. It’s amazing if you haven’t seen it. Highly recommend. Thanks @dynamicwebpaige for introducing me to it. It showcases all the same passion and idealism for building as The Soul of a New Machine.

I think there is a different class of programmer like those in The Soul of a New Machine & Halt and Catch Fire… the idealists_dreamers? Those who build things w soul, value accomplishments & being a part of something bigger than themselves. I feel like we’ve lost some of that.

— jessie frazelle 👩🏼‍🚀 (@jessfraz) January 29, 2019

I also love thinking about that era in computing because my grandpa was a computer programmer. After attending college, my mom and dad stayed with him for a bit. My mom likes to tell this story about how he had made his computer talk. It was something he had been working on for a long time at his office, but he was also working on it at home. She came to his house after work one day and as she walked in the door the computer said “Hi Debbie” and his face lit up.

I love the passion of building and to me that time was the golden age of passionate building. So now you might be wondering where I’m going with this and how this fits in with Moore’s Law and today….

New Golden Age In their Turing lecture, Hennessey and Patterson call today’s age “A New Golden Age for Computer Architecture”.

“The end of Dennard scaling and Moore’s Law and the deceleration of performance gains for standard microprocessors are not problems that must be solved but facts that, recognized, offer breathtaking opportunities.”

I love this and I believe it. There are so many opportunities today because of the circumstances of computing changing that will be awesome to see unfold.

I’m not going to play hand-wavy, armchair, “here is the future” with you all. Instead, I will give you a few quotes from an article on sigarch of the ACM that I really loved and let you come to your own conclusions and theories.

“Prediction #1: Technology scaling will continue to deliver benefits to certain markets

2: Beloved computing abstractions will fail, opening new opportunities for innovation

3: Democratization of technology will result in a golden age for computer architecture”

“By 2030, the rise of open source cores, IP, and CAD flows targeting these advanced nodes will mean that designing and fabricating complex chips will be possible by smaller players.”

“Hardware startups will flourish for the reasons that the open source software ecosystem paired with commoditized cloud compute has unleashed software startups over the past decade.”

Thanks for reading my cheese ball post! I truly believe it’s a great time to be alive and a passionate builder!

View Details

I started dipping into some firmware and hardware things on my vacation and unemployment and I figured I would take you down my journey as well.

Baseboard management controller The first thing I dipped into was openbmc. This is pretty cool. At face value it has support for a lot of different boards. It uses IPMI (Intelligent Platform Management Interface) to perform tasks for monitoring and operating the components of a computer. The IPMI interface has been around for a super long time. RedFish is kind of the successor. It’s an HTTP API and is more modern as a thoughtful approach to hardware deployment in a datacenter. The standard doesn’t include every sensor that IPMI has but it does allow for someone to add more sensors types to their implementation.

So I dug into the openbmc project a bit and tried to lick my wounds of dbus, seeing that was what it was using. I thought hmmm I wonder if there are more projects like this…

It turns out there are! u-bmc from the same folks that made u-root seemed like a more simple, opinionated solution. However, it only has the support of one board currently, although others seem planned. I thought it was a kinda neat and interesting detail that u-bmc used gRPC instead of IPMI, seems like a cool choice to modernize but I had some naive questions so I headed to the internet for answers.

Anyone know what the memory overhead for using gRPC for this is… I would think it’s not insignificant, or you’d want to use one of the “tiny grpc” replacements, or maybe something that didn’t reinvent its own HTTP server perhaps…? https://t.co/gIpW97r7Xw

— jessie frazelle 👩🏼‍🚀 (@jessfraz) February 5, 2019

That thread is awesome. Thanks to some super awesome and smart friends from the internet I learned a lot more about these two projects. I will let you read the thread and form opinions of your own but there’s a lot of experience and knowledge in there.

Currently, I’m feeling a bit nerd sniped by the idea of a BMC implemented in Rust to solve some of the problems mentioned in the thread. A girl can dream right? :)

That was a bit of a rabbit hole so I decided to move on, mostly because of ADHD and my ever growing curiosity about all things computers.

Intel Management Engine I started looking into the Intel Management System… boy does that do a lot of stuff.

[enters weird rabbit hole]
“wow there’s a lot of tunnels in here” pic.twitter.com/oHslyJ0TuF

— jessie frazelle 👩🏼‍🚀 (@jessfraz) February 5, 2019

The craziest part that I found were all the security vulnerabilities and theories of backdoors. I live for researching things like this so I was intrigued. Intel gave people a way to disable the ME, and vendors have, as well as Dell even selling computers to government contracts with it disabled. I stumbled across this super dope laptop company, Purism (thanks @bcantrill), that sells laptops using coreboot with the ME memory erased. Their approach and blog is super neat and interesting. Also coreboot looks just lovely, I need to play around with it more.

Intermission So in between bouncing back and forth reading about various forms of firmware and how shitty and sketchy closed source firmware is, I read the book Bad Blood. The book details the absolute cluster-fuck that was the startup Theranos, so everything from here on out is with “paranoid as fuck” goggles on because I was shook.

Reading Bad Blood pic.twitter.com/C1SN7CF91B

— jessie frazelle 👩🏼‍🚀 (@jessfraz) January 31, 2019

Keep that in mind as we head into the next section.

SGX Intel’s SGX (Software Guard Extension) is just utterly bananas. I went down this tunnel next. Oh it’s a doozy of a tunnel let me tell you.

In short, SGX provides what is known as a Secure Enclave. You can put keys in here for safe keeping because the memory is isolated and encrypted from everything else in the computer. (Or so they say, but we will get to that.) This creates a way to store data that you don’t want the host computer user to know about. Some cloud providers are using SGX as a way for customers to use the cloud without trusting the cloud provider, only trusting the hardware provider, in this case Intel.

Existing Knowledge I had done a Papers We Love talk on the SCONE paper over a year ago. This paper was an experiment in running docker containers in an enclave. You can watch the talk, but the short version is I wasn’t really sold. While being a technological feat, it was slow and it required a bunch of code. Basically you need to reinvent all of computing inside the enclave (the HAVEN paper approach put bluntly). Or if you do what they did in the SCONE paper, run syscalls outside the enclave. If you toss syscalls outside the enclave, you need to deal with encrypting all of I/O and a bunch of other surface area since you are now running both inside and outside the enclave. In that case, your boundary is more like a blurred line.

My opinion, which I’m sure the readers on Hacker News will call me all sorts of names for, I question what is the point if you need to trust so much base code just to run a damn thing in the enclave and when you run your process it’s slow. AND it won’t even protect you from side channel attacks or timing attacks.

Anyways, that was my background knowledge going into this rabbit hole once again. But there I was going back for round two thinking I wonder wtf is up in the SGX world…. TURNS OUT A LOT.

Round Two Thanks to the awesome internet I stumbled upon a 118 page run down of the technology.

Here for this shade, thanks @msw for the link https://t.co/WJtgf9vZBc pic.twitter.com/DaoZQunloJ

— jessie frazelle 👩🏼‍🚀 (@jessfraz) February 8, 2019

This is a great paper, if you really want to learn about the internals of not only SGX but computer architecture as well, I strongly suggest reading it. It’s wonderfully written and very detail oriented.

The paper is on the second generation of the technology and outlines the side-channel attacks making the hardware insecure. The interesting thing I took away from the paper, other than a fuck ton of nuance, was the licensing of SGX.

Launch Control SGX has this feature called “launch control”. Launch control is the gatekeeper for launching enclaves it requires an Intel license and provides launch tokens for launching other enclaves. You use what’s called a “launch enclave” to create a “launch token”. Anyways, it wasn’t really documented at this time, and the paper makes interesting insights about it. While SGX from the outside is a feature to secure computing, it also has this hidden feature of securing the market for Intel perhaps?

Well Intel responded and made “Flexible Launch Control.” This allows a different party, other than Intel, to handle the launch control process. That’s nice, seems like a shit ton of work though and sadly, making the UX better around this got me thinking. Cloud providers couldn’t do launch control for people since that then defeats the purpose of only trusting the hardware vendor and not the cloud. So this is up to the customer and in my opinion it seems like a lot to land on them. Also it seems like the cloud provider would have to somehow even enable this feature…

Honestly, I dunno, I’m not an expert here.

Okay so I was basically over launch control at this point and ready to go deeper. Thanks twitter for all the paper links :)

Attacks Foreshadow is fucking nuts. It uses the same type of attack as Meltdown but the fixes for Meltdown didn’t prevent the attack since KPTI (kernel page table isolation) doesn’t cover the enclave address space. In the paper they steal secrets from inside an enclave, which honestly would be the end game of a lot of hackers. The authors take it further by getting the private keys for the enclave and creating fake enclaves that appear perfectly fine and attestations. Wow!

But that’s not all. Foreshadow-NG took it a step further, from the paper:

At a high level, whereas previous generation Meltdown-type attacks are limited to reading privileged supervisor data within the attacker’s virtual address space, Foreshadow-NG attacks completely bypass the virtual memory abstraction by directly exposing cached physical memory contents to unprivileged applications and guest virtual machines.

With Foreshadow-NG, the hacker can access all cached memory, not just their own virtual memory. Bananas… right. But there’s more…

Do you need a new feature set for your malware? Because you can use SGX to conceal cache attacks and amplify them!!!

Here’s a quote from the second paper linked above:

Our attack tool named CacheZoom is able to virtually track all memory accesses of SGX enclaves with high spatial and temporal precision.

If enclave malware interests you, there’s another paper that just went out detailing that yesterday.

I am forgetting a bunch of other details and papers but this should paint a pretty good picture of the state of the SGX world.

I wrote more about SGX in my Reflections on SGX post.

Thank You Thank you to everyone for linking me to awesome papers and engaging in my nerdery with these things. I’m not done at all with this rabbit hole but I thought I’d sum it up for now.

Shout out to @msw, @bcantrill, @anliguori, @iancoldwater, @hugelgupf, @bascule, @kc8apf, @nasamuffin, and everyone else I apologize if I forgot.

View Details

I thought it would be fun to start a blog post series containing design docs from my personal archive that never saw the light of day. This will be the first of the series. It contains what I thought about in detail for a general multi-tenant secured container orchestrator. The use case would be for running third party code securely isolated from each other. If you would like to see this in google doc form it also lives here.

Requirements Base * API to run docker images in such a way that each process is isolated entirely from all the others. * Abusive actions can be terminated immediately. * The agent should be auto-updateable to handle security issues as they arise. * Ability to use the entire syscall interface for the processes being run. * This all assumes that you have some sort of software and hardware level root of trust you can use to ensure security as well.

Other Features * Disallow and kill any and all bitcoin miners from using the infrastructure, BPF tracers * Firewall off any existing network endpoints * Firewall off the container running the process from everything around it on the local links and any reachable internal IP * If one layer of isolation is compromised, rely on another layer of isolation entirely. If two layers are compromised then we at least tried our best…

Design The host OS and up needs to be secure.

Overview We require the following per container running:

• Block/io cgroups so that disk does not have noisy neighbors
• CPU limit
• Memory limit
• Network/bandwidth limiting
• Isolated network from everything else on the network (BPF or iptables)

Host OS The host OS should be a reduced operating system, minimal distribution (though possibly shared with the OS used inside containers). This is for reasons of security in locking down the available weaknesses in the host environment and lessening the control plane attack surface.

Operating Systems Examples of these Operating Systems include:

• CoreOS Container Linux
• Container Optimized OS
• Intel Clear Linux
• LinuxKit

Features CoreOS Container Linux and Container Optimized OS both have the following features:

• Verified boot
• Read-only /usr
  • Container Optimized OS has root filesystem (/) mounted as read-only with some portions of it re-mounted as writable, as follows:
  • /tmp, /run, /media, /mnt/disks and /var/lib/cloud are all mounted using tmpfs and, while they are writable, their contents are not preserved between reboots.
  • Directories /mnt/stateful/partition, /var and /home are mounted from a stateful disk partition, which means these locations can be used to store data that persists across reboots. For example, Docker’s working directory /var/lib/docker is stateful across reboots.
  • Among the writable locations, only /var/lib/docker and /var/lib/cloud are mounted as “executable” (i.e. without the noexec mount flag)
  • CoreOS Container Linux has root filesystem (/) mounted as read_write and /usr is read-only.

All of the operating systems allow seamless upgrades for security issues.

Container Runtime The container runtime should be a hypervisor to make sure that user configurations of Linux containers do not diminish the security of the cluster.

Why not containers? It should not go without saying that it is possible to have multi-tenancy with containers as is proven with contained.af that no one has managed to break out of.

To be allowed to use the entire syscall interface though (my ACM Queue Research for Practice article), Firecracker seems like the right fit.

Just using containerd out of the box as a base and building on that should be perfect :)

Network The network should be locked down by default with a deny all policy for ingress and egress. This will create a form of security that makes sure all networking between pods or to the rest of the world is explicit.

This could be done with iptables or directly with BPF (which in my opinion is way more clean).

DNS Do not allow any inter-cluster DNS.

No Scheduling on Master and System Nodes Make sure that the master and system nodes in the cluster cannot be scheduled on.

This allows a separation of concerns from system processes to anything else.

The Scheduler The scheduler should not do bin packing. Seen this fail in a lot of scenarios with transient workloads where the first few nodes get burned out while all the other nodes are not being used. Because the workloads are constantly completing freeing up resources on those first few nodes (in the case of batch jobs).

There is knowledge in: kube-batch scheduler. It is built on years of experience from HPC clusters at IBM. We can use the same type of logic. This is more meant for batch jobs though, so if we plan on supporting long term applications we would need to modify.

We should also account for proximity to the docker image being pulled. The largest constraint on time for running a container is pulling an image so let’s optimize for making that as short as possible.

If we are running on bare metal we need to account for power management, BIOS updates, hardware failures and more. These are all things the orchestration tools of today completely ignore.

Resource Constraints Manage resources and set limits with cgroups.

• Disk IO
• Network Bandwidth
• Memory
• CPU

Preventing Miners * CPU Tracers with eBPF: monitor cpu usage so if it’s not fluctuating it might be a miner, most other processes fluctuate * Binary tracers: look for binaries/ processes with a certain name, miners can rename but block the lazy ones * Network tracers: look for processes reaching out to known miner endpoints

Other Why not kubernetes? I’m super pragmatic about these things and don’t want to reinvent the world for nothing but I have now seen this go terribly wrong, as in people turning off firewalls accidentally…. And I don’t want the security of something that allows arbitrary code execution to have only one layer of security which someone might inadvertently turn off.

We don’t need 90% of the features of kubernetes.

Kubernetes is hard to secure… there are a lot of components and there is no isolation between etcd, and the kubelet to apiserver communication cannot be isolated either.

I wrote a blog post on secure k8s and we are a long ways off. It’s too complex and has too many third party drivers. (my hard multi-tenancy in kubernetes blog post). All in all the surface area is just too big and we don’t need all the feature set anyways.

By keeping our implementation more simple it is easier to keep track of the components’ communication and ensure it is secure. The surface area is WAYYY smaller. The only downside is we operational knowledge of k8s but the concepts and patterns are the same.

The biggest Kubernetes cluster is 5000 nodes and they hit a lot of issues: blog.openai.com/scaling-kubernetes-to-2500-nodes/. We might need a different key value store and multiple clusters. And, like I noted above, I would not be confident considering it “secure”.

Kubernetes will by default schedule at most 110 pods per node. This is something you can change but it is also important to note that the default scheduler in kubernetes is extremely not resource aware and we would have to fix that as well. See above in “The Scheduler.” And the first few nodes in a cluster get burned through quickly due to the logic of the default scheduler.

Even Google doesn’t use Kubernetes internally to schedule VMs, that is a whole separate thing.

Kubernetes inserts a bunch of extra env variables into the containers we would have to take care of as well… as seen here: Kubernetes Hard Multi-Tenancy Design Doc.

What do we do if there is a kernel 0day that effects the isolation? For one, update the kernel, but if that is not possible we can trap the kernel function that is vulnerable using eBPF and kill any container trying to exploit the vulnerability. This has a trade off of jobs failing but we can try to get it as close as possible to have no false positives.

This assumes we have systems in place to continually build kernels and apply patches.

How secure is this? Well let’s think about the threat model. Mostly it would be someone attacking our infrastructure itself so we should make sure all these servers are isolated on the network from the rest of the stack.

The next threat would be the users’ code and secrets that we are running. After breaking out of a container it would leave the hacker still in the firecracker VM so they will still need to break out of the VM. This would be the case in the event of a container runtime bug.

Monitoring, monitoring, monitoring. We should detect using eBPF or otherwise any rogue process on the host that is not that of our containers or of our agents_infrastructure and kill_alert immediately.

Any file that is touched that is outside the scope of the given container should have the container killed and alerted on.

Additionally we can even hide the fact that it is running in a specific container runtime etc. So there is less knowledge of the environment, unless of course they read this doc.

View Details

My top used shell command is |. This is called a pipe.

In brief, the | allows for the output of one program (on the left) to become the input of another program (on the right). It is a way of connecting two commands together.

For example, if I were to run the following:

``` echo "hello"

```

I get the output hello.

But if I run:

``` echo "hello" | figlet

```

The figlet program, changes the letters in hello to look all bubbly and cartoony.

This is a really blunt way of describing something that, in my opinion, is brilliant software design, but I will get into that in a second.

Let’s go back to the origin of pipes.

According to doc.cat-v.org/unix/pipes/, the origin of pipes came long before Unix. Pipes can be traced back to this note from Doug McIlroy in 1964:

``` - 10 - Summary--what's most important.

To put my strongest concerns into a nutshell:

1. We should have some ways of coupling programs like garden hose--screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also.

2. Our loader should be able to do link-loading and controlled establishment.

3. Our library filing scheme should allow for rather general indexing, responsibility, generations, data path switching.

4. It should be possible to get private system components (all routines are system components) for buggering around with.

                                           M. D. McIlroy
                                           October 11, 1964
   

```

The Unix philosophy is documented by Doug McIlroy as:

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.
2. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.

From the Bell Systems Technical Journal

What I love about Unix is the philosophy of “do one thing well” and “expect the output of every program to become the input to another”. This philosophy is built on the use of tools. These tools can be used separately or combined to get a job done. This is in stark contrast to monolithic programs that do everything or one-off programs used to solve a specific problem.

System programs and commands like echo, which we saw above, output information to your terminal by default. For example, cat will “concatenate” (its namesake) files and print the result to your terminal. While reading Program design in Unix, I realized that printing the output of the tool to the user’s terminal was actually the special case.

“Perhaps surprisingly, in practice it turns out that the special case is the main use of the program.”

When a user redirects the output of cat via a | to some other program, cat becomes so much more than what the original author intended. This is one of the most brilliant design patterns, in my opinion. For one, programs being simple and doing one thing well makes them easy to grok. The beautiful part, though, is the fact that in combination with a operator like | the program becomes one step in a much larger plan. The original author of cat does not even need to know about the larger plan. That is the beauty of the | it allows for solving problems by combining small, simple programs together.

I love software design that enables creativity, values simplicity, and doesn’t put users in a box. The pipe, is a key element for keeping programs simple while enabling extensibility. A simple program in combination with a | becomes so much more than what the original author could have dreamed of.

I hope this post helped you learn something, if not, just pipe it to /dev/null.

View Details

I thought it might be fun to write a blog post on “The Life of a GitHub Action.” When you go through orientation at Google they walk you through “The Life of a Query” and it was one of my favorite things. So I am re-applying the same for a GitHub Action.

For those unfamiliar Actions was a feature launched at GitHub’s conference Universe last year. You can sign up for the beta here.

The overall idea is scriptable GitHub but rather than do all that hand-wavy crap to try and explain I will take you through what happens when you run an Action.

The Problem Here is a typical workflow:

• I create a pull request on a repository.
• The pull request is merged.
• The branch lingers around until the end of time and eats away at the part of my soul that likes everything to be clean.

Let’s focus on my pain of the lingering branches. This is totally a problem right? So let’s solve it by creating an Action to delete branches after the pull request has been merged.

All the code for this action lives here if you want to skip ahead.

The Workflow File You can create actions from the UI or you can write the Workflow file yourself. In this post, I am just going to use a file.

Here is what it ends up looking like and I will explain what everything means in comments on the file. This lives in .github/main.workflow in your repository.

```

Workflow defines what we want to call a set of actions.

workflow "on pull request merge, delete the branch" { ## On pull_request defines that whenever a pull request event is fired this ## workflow will be run. on = "pull_request"

## What is the ending action (or set of actions) that we are running. ## Since we can set what actions "need" in our definition of an action, ## we only care about the last actions run here. resolves = ["branch cleanup"] }

This is our action, you can have more than one but we just have this one for

our example.

I named it branch cleanup, and since it is our last action run it matches

the name in the resolves section above.

action "branch cleanup" { ## Uses defines what we are running, you can point to a repository like below ## OR you can define a docker image. uses = "jessfraz/branch-cleanup-action@master"

## We need a github token so that when we call the github api from our ## scripts in the above repository we can authenticate and have permission ## to delete a branch. secrets = ["GITHUB_TOKEN"] }

```

The Event Okay so since this post is called “The Life of an Action” let’s start on wtf actually happens. All actions get triggered on a GitHub event. For the list of events supported see here.

Above we chose the pull_request event. This is triggered when a pull request is assigned, unassigned, labeled, unlabeled, opened, edited, closed, reopened, synchronized, a pull request review is requested, or a review request is removed.

Okay let’s assume we triggered this event.

“Something” happened on a pull request…. Now, GitHub is like “oh holy shit, something happened on a pull request, let me fire all ze missiles of things that happen on a pull request.”

Going back to our Workflow file above, GitHub says “I am going to run the workflow ‘on pull request merge, delete the branch’”.

What does this resolve? Oh it’s “branch cleanup”. Let me order all the Actions branch cleanup requires (in this case none) and run them in order/parallel so we end on “branch cleanup.”

The Action At this point GitHub is like ‘yo you guys, I need to run the “branch cleanup” Action. let me get what it is using.’

This takes us back to the uses section of our file. We are pointing to a repository: jessfraz/branch-cleanup-action@master.

In this repository is a Dockerfile. This Dockerfile defines the environment our action will run in.

Dockerfile Let’s take a look at that and I will add comments to try and explain.

```

FROM defines what Docker image we are starting at. A docker image is a bunch

of files combined in a tarball.

This image is all the files we need for an Alpine OS environment.

FROM alpine:latest

This label defines our action name, we could have named it butts but

I decided to be an adult.

LABEL "com.github.actions.name"="Branch Cleanup"

This label defines the description for our action.

LABEL "com.github.actions.description"="Delete the branch after a pull request has been merged"

We can pick from a variety of icons for our action.

The list of icons is here: https://developer.github.com/actions/creating-github-actions/creating-a-docker-container/#supported-feather-icons

LABEL "com.github.actions.icon"="activity"

This is the color for the action icon that shows up in the UI when it's run.

LABEL "com.github.actions.color"="red"

These are the packages we are installing. Since I just wrote a shitty bash

script for our Action we don't really need all that much. We need bash,

CA certificates and curl so we can send a request to the GitHub API

and jq so I can easily muck with JSON from bash.

RUN apk add --no-cache \ bash \ ca-certificates \ curl \ jq

Now I am going to copy my shitty bash script into the image.

COPY cleanup-pr-branch /usr/bin/cleanup-pr-branch

The cmd for the container defines what arguments should be executed when

it is run.

We are just going to call back to my shitty script.

CMD ["cleanup-pr-branch"]

```

The Script Below is the contents of the bash script I am executing.

```

!/bin/bash

set -e set -o pipefail

This is populated by our secret from the Workflow file.

if [[ -z "$GITHUB_TOKEN" ]]; then echo "Set the GITHUB_TOKEN env variable." exit 1 fi

This one is populated by GitHub for free :)

if [[ -z "$GITHUB_REPOSITORY" ]]; then echo "Set the GITHUB_REPOSITORY env variable." exit 1 fi

URI=https://api.github.com API_VERSION=v3 API_HEADER="Accept: application/vnd.github.${API_VERSION}+json" AUTH_HEADER="Authorization: token ${GITHUB_TOKEN}"

main(){ # In every runtime environment for an Action you have the GITHUB_EVENT_PATH # populated. This file holds the JSON data for the event that was triggered. # From that we can get the status of the pull request and if it was merged. # In this case we only care if it was closed and it was merged. action=$(jq --raw-output .action "$GITHUB_EVENT_PATH") merged=$(jq --raw-output .pull_request.merged "$GITHUB_EVENT_PATH")

echo "DEBUG -> action: $action merged: $merged"

if [[ "$action" == "closed" ]] && [[ "$merged" == "true" ]]; then
    # We only care about the closed event and if it was merged.
    # If so, delete the branch.
    ref=$(jq --raw-output .pull_request.head.ref "$GITHUB_EVENT_PATH")
    owner=$(jq --raw-output .pull_request.head.repo.owner.login "$GITHUB_EVENT_PATH")
    repo=$(jq --raw-output .pull_request.head.repo.name "$GITHUB_EVENT_PATH")
    default_branch=$(
        curl -XGET -sSL \
            -H "${AUTH_HEADER}" \
            -H "${API_HEADER}" \
            "${URI}/repos/${owner}/${repo}" | jq .default_branch
    )

    if [[ "$ref" == "$default_branch" ]]; then
        # Never delete the default branch.
        echo "Will not delete default branch (${default_branch}) for ${owner}/${repo}, exiting."
        exit 0
    fi

    echo "Deleting branch ref $ref for owner ${owner}/${repo}..."
    curl -XDELETE -sSL \
        -H "${AUTH_HEADER}" \
        -H "${API_HEADER}" \
        "${URI}/repos/${owner}/${repo}/git/refs/heads/${ref}"

    echo "Branch delete success!"
fi

}

main "$@"

```

So at this point GitHub has executed our script in our runtime environment.

GitHub will post the status of the action back to the UI and you can see it from the Actions tab.

Hopefully this has made some clarity as to how things are run in GitHub Actions. I can’t wait to see what you all build.

View Details

I have realized recently that a lot of people think I am just a shill for Kubernetes and I am not. What I have done is write a few blog posts on some interesting problems to be solved in Kubernetes. But I would like to emphasize that those problems are pretty exclusive to the way Kubernetes was designed and you could easily build your own orchestrator without them.

Use Containerd If you need an example of a custom, minimal orchestrator with containerd you should checkout stellar.

Or see my design doc for a multi-tenant orchestrator.

I’ll let you dive into that in your own time though. Let’s take a new look at a blog post I wrote about Building images securely on Kubernetes.

I feel like I should have more clearly stated how this problem is pretty exclusive to Kubernetes. It’s also not really a hard problem. The hard problem I was solving in that post was not how to build images on Kubernetes but how to build images as an unprivileged user in Linux. That is a hard problem. And a serious problem for companies who don’t allow root on their machines.

The easier choice if all you need to do is build an image and you are already using containerd, is to run buildkit on the same machine and then you can use the buildkit API library to build your dockerfiles.

Or just run docker-in-docker, I have done this for years on my CI with absolutely no problems.

Anyways, the point I am trying to make is you should use whatever is the easiest thing for your use case and not just what is popular on the internet. With complexity comes a steep learning curve and with a massive number of pluggable layers comes yak shaves until the end of time.

Think for yourselves, don’t be sheep.

View Details

Wireguard is the hip, new way to VPN :P

No, but seriously I wanted to try it out because it is super interesting and I think the direction it is going is awesome. Read about it on their website if you have not already.

What is cool about Wireguard is it integrates into the Linux networking stack so you have a lot of power over interactions with it. In other words, it is very easy to clone the interface into specific containers. Or just use it on your host.

If you are new to my blog, I HATEEEE installing things on my host. I run everything in containers. Wireguard is a kernel module. BUT guess what, literally anything can be run in a container. This post is going to go over how to install the Wireguard module by using a container and how to run the tools from a container as well.

UPDATE (April 2020): You might want to use Tailscale. It is simple to install and cross platform since it uses the go implementation of wireguard. Then you don’t have to mess with the kernel!

I will never forget this thread from 2017 ;) so glad to see the go implementation happen!

Soooooo waiting for the userspace portable Go implementation.

— Filippo Valsorda 🇮🇹 (@FiloSottile) June 21, 2017

Installing I wrote a Dockerfile for installing the kernel module.

You can run it with:

``` $ docker run --rm -it \ --name wireguard \ -v /lib/modules:/lib/modules \ -v /usr/src:/usr/src:ro \ r.j3ss.co/wireguard:install

```

This only works if you have your kernel headers installed in /usr/src and your kernel allows kernel modules (CONFIG_MODULES=y). This will change your kernel modules on your host since you are mounting that directory.

If you are like me and set CONFIG_MODULES=n then you can use my kernel-builder Dockerfile to build a custom kernel.

``` $ docker run --rm -it \ -v /usr/src:/usr/src \ -v /lib/modules:/lib/modules \ -v /boot:/boot \ --name kernel-builder \ r.j3ss.co/kernel-builder

```

That will pop you into a bash shell where you can run the following build script to build a specific kernel version.

```

build_kernel 4.17.1

```

That saves the vmlinuz to /boot (on your host, since you mounted that directory) where you can then update your initramfs for the new image and add it to your bootloader if needed.

Using the tools wg is the command for interacting with Wireguard. You can learn more about it in their docs.

I put the tools in a container and added a bash alias for them:

``` $ type wg wg is a function wg () { docker run -it --rm --log-driver none -v /tmp:/tmp --cap-add NET_ADMIN --net host --name wg r.j3ss.co/wg "$@" }

```

Then you can run the following commands to try sending some packets through Wireguard. The below steps come from the following script which is Copyright (C) 2015-2018 Jason A. Donenfeld. All Rights Reserved. GPL-2.0. I merely added comments for the steps.

``` $ export WG_PRIVATE_KEY="$(wg genkey)"

open file descriptior 3 with some initial details

this is just wireguard's demo server, you can use the container to spin up

your own, don't actually use this as your server

$ exec 3<>/dev/tcp/demo.wireguard.com/42912

$ wg pubkey <<<"$WG_PRIVATE_KEY" >&3

$ IFS=: read -r status server_pubkey server_port internal_ip <&3

make sure the status is "OK"

$ echo $status OK

delete the link if you already had one

$ sudo ip link del dev wg0 || true

create the wireguard link

$ sudo ip link add dev wg0 type wireguard

save the private key to a temporary file

obviously in a real world scenario we wouldn't be throwing these around

all willy nilly

$ echo "$WG_PRIVATE_KEY" > /tmp/wg-privatekey

configure the interface

$ wg set wg0 private-key /tmp/wg-privatekey peer "$server_pubkey" allowed-ips 0.0.0.0/0 endpoint "demo.wireguard.com:$server_port" persistent-keepalive 25

assign an ip address/peer

$ sudo -E ip address add "$internal_ip"/24 dev wg0

bring the interface up

$ sudo ip link set up dev wg0

make it the default route

$ export WG_HOST="$(wg show wg0 endpoints | sed -n 's/.\t(.):.*/\1/p')"

add the routes

$ sudo -E ip route add $(ip route get $WG_HOST | sed '/ via [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/{s/^(. via [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})./\1/}' | head -n 1) $ sudo ip route add 0/1 dev wg0 $ sudo ip route add 128/1 dev wg0

```

Test it is routing!

``` $ curl https://httpbin.j3ss.co/ip {"origin":"163.172.161.0"}

```

And that’s all. Just thought it was kinda fun using this and now it is very easy to install :)

View Details

I figured it would be nice to have one canonical place for talks I have given. So here it is…

2019 CERN - Why Open Source Firmware is Important This talk will dive into some of the problems of running servers at scale, including data from surveys about physical infrastructure and firmware concerns. In this talk, we’ll understand how open source firmware will solve some of these common problems. Why is open source firmware important for security and root of trust? We’ll discuss that as well, and cover the state of open source firmware today.

QCon London - Panel: Secure Isolation of Applications Co-Speakers: Justin Cormack, Per Buer, Allison Randal, Kenton Varda

Applications have been isolated by lots of different means: processes, virtual machines, containers, and new methods are appearing such as SGX and in-process isolates. What is secure? Have Spectre and Meltdown changed the landscape? What should be used?

QCon London - A Journey into Intel’s SGX This talk takes a deep dive into Intel’s SGX technology. It covers an overview of computer architecture as background and walks the audience through one version of the hardware and its flaws, as well as what changed in the next version.

2018 re:Invent - Container Power Hour Co-Speakers: Clare Liguori and Abby Fuller

This talk goes over using containers on AWS.

ChaosConf - Breaking Containers Chaos engineering and stories of bugs about containers.

LinuxConfAu - Containers aka crazy user space fun Like the movie Plan 9 from outer space, this talk covers containers from user space. What are they? Where did they come from? How much koolaid is involved in adopting them into your life… watch for the jokes, learn from the interesting technical details.

2017 Google Cloud Next - Build user trust: running containers securely Co-Speaker: Alex Mohr

This talk covers all the ways you can secure your Kubernetes cluster using a Certificate Authority, Authentication, Secrets and more. We also describe and demonstrate the ways you can use Seccomp, AppArmor, SELinux and cgroups to make your application containers as secure as possible - so you can build organizational and customer trust.

CoreOS Fest - Container Linux on the Desktop! This talk covers how to build a secure desktop OS with only containers and CoreOS Container Linux. It also describes the benefits gained from using Container Linux as a base OS and how to go about running it on the desktop.

Kubecon - Dance Madly on the Lip of a Volcano Co-Speaker: Brandon Philips

This talk covers how we designed an awesome security release process for Kubernetes and all it’s sub-projects.

Open source projects strive to be transparent in everything they do, but when it comes to fixing security patches they need to find the right balance of “open” and “responsible.” This means vulnerabilities should be reported in a safe way as well as patches tested and reviewed with a limited audience. The companies that rely on Kubernetes should have time to patch their systems before a public announcement.

Various sets of infrastructure and collaboration are needed to make this a reality. The design we used could also be applied to other projects and even internally in your company.

2016 Container Summit - Building Containers in Pure Bash and C This talk demonstrates how to build containers from the Primitives in Linux without using a container runtime. Learn about the objects that make up containers themselves.

Arrested DevOps - Exciting Topics like Containers & Security Ben Hughes and I chat with Bridget Kromhout about everyone’s favorite topic, security.

Github Universe - Blurry lines between individual contributor & corporate backers When working on open source projects, your contributions and opinions on the project and its motives are usually very personal. This talk covers intricacies of “choosing your battles” and how personal passion for a project might conflict with corporate motives.

Container Camp - Application Sandboxes vs. Containers This talk covers the differences between application sandboxes and containers. The most well known sandbox is Chrome, for providing “hard guarantees about what ultimately a piece of code can or cannot do no matter what its inputs are”.

At its core, the Linux Chrome sandbox uses namespaces along with seccomp and other native features to provide these guarantees. Containers are composed of the same primitives. What is needed for containers to provide this promise? Can it be done by default? What steps are already being made to get towards containers that actually “contain”? What challenges will be faced?

2015 Dockercon EU - The Latest in Docker Engine Co-Speaker: Arnaud Porterie

Learn about the latest capabilities in Docker Engine and how to use them in your application. This session also covers best practices for using Engine, troubleshooting tips, and cool lesser known features.

This video has the first ever demo of Seccomp in Docker as well as a fun story about trying to save a docker image to a floppy disk.

DockerCon - Container Hacks and Fun Images This talk is a 100% live demo of running desktop applications in containers. Everything from Spotify to Skype. Explore some of the more interesting things you can containerize on Linux. View first hand different workflows for how to run/build different apps in containers. This talk covers desktop apps as well as some other apps you would have never thought could run in a container.

Container Camp - Willy Wonka of Containers This talk has live demos of desktop applications in containers including Steam.

HashiConf - Dockerizing all the Things This talk goes over the way the Docker project uses containers for their testing infrastructure as well as internal infrastructure. Find out about real pain points solved by running things in containers as well as some different hurdles uncovered along the way.

DotGo - The Docker Trail This talk recounts stories from the trenches of developing Docker, explaining 3 odd things her team stumbled upon in their Go code and how they fixed them. One of which is very odd and gets into the depths of dlopen-ing yourself.

Google Cloud Platform Podcast - Containers Francesc Campoy and I talk all about Dockercon EU and containers.

View Details

There seems to be some confusion around sandboxing containers as of late, mostly because of the recent launch of gvisor. Before I get into the body of this post I would like to make one thing clear. I have no problem with gvisor itself. I think it is very technically “cool.” I do have a problem with the messaging around it and marketing.

There is a large amount of ignorance towards the existing defaults to make containers secure. Which is crazy since I have written many blog posts on it and given many talks on the subject. But I digress, let’s focus on the part of the README that mentions sandboxing with SELinux, Seccomp, and Apparmor. It says: “However, in practice it can be extremely difficult (if not impossible) to reliably define a policy for arbitrary, previously unknown applications, making this approach challenging to apply universally.”

Greetings. Reporting for duty. Literally I am the person who can do that. I was the person who did do that. I added the default Seccomp profile to Docker and maintained the default Apparmor profile. I have also done A LOT of research with regard to Linux kernel isolation and making containers secure. I also literally reported for duty, two years ago and made the patch to add the Seccomp annotation to Kubernetes… with the hopes of eventually turning on a default filter.

@nathanmccauley @brendandburns @kelseyhightower @thockin I already offered to help

— jessie frazelle (@jessfraz) April 5, 2016

All big organizations have problems with “not invented here.” I tried my very best to inform everyone how these sandboxing mechanisms work but I am going to try one last time here.

More than One Layer of Security Required In my last blog post, Hard Multi-Tenancy in Kubernetes, I mentioned this as well. It is also a good read if you want to learn about the thought process for secure isolation. To be truly secure you need more than one layer of security so that when there is a vulnerability in one layer, the attacker also needs a vulnerability in another layer to bypass the isolation mechanism.

In Docker, we worked really hard to create secure defaults for the container isolation itself. I then tried to bring all those up the stack into orchestrators.

Container runtimes have security layers defined by Seccomp, Apparmor, kernel namespaces, cgroups, capabilities, and an unprivileged Linux user. All the layers don’t perfectly overlap, but a few do.

Let’s go over some of the ones that do overlap. I could do them all, but I would be here all day. The mount syscall is prevented by the default Apparmor profile, default Seccomp profile, and CAP_SYS_ADMIN. This is a neat example as it is literally three layers. Wow.

Everyone’s favorite thing to complain about in containers or to prove that they know something is creating a fork bomb. Well this is actually easily preventable. With the PID cgroup you can set a max number of processes per container.

What about things that are not namespaced by the linux kernel..? CAP_SYS_TIME prevents people from changing the time inside containers. And the default Seccomp profile prevents modifications or interacting with the kernel keyring.

If you would like a list of all the syscalls prevented by the default Seccomp profile, I behoove you to read the list here. It also has descriptions of each.

Two years ago, there was a great Whitepaper from NCC Group about hardening linux containers. Still to this day I get all the good feels when I see all the mentions of my work in it. But if you have any hesitations towards the defaults in Docker or otherwise I suggest you educate yourself first.

I will call out my favorite chart here though. Below shows the defaults from various container runtimes as of two years ago. Note the strong defaults in Docker. The paper also explains at length the defaults and would be a less biased version than me explaining myself.

The non-events are also an interesting read.

Breaking Changes A lot of the push back I got from the default Seccomp profile was related to it being a breaking change.

I get that this is very scary. No really I get it. When we added it to Docker, guess who got paged when the Docker apt repo was down and it was on the front page of hacker news with tech bros crying: me. So I was absolutely horrified at the thought of making a breaking change that might land on the front page of hacker news as well.

The last thing I ever wanted to do was cause a breaking change. That shit was terrifying. I lost sleep over weeks worrying about it. I tested every single Dockerfile on GitHub with the default profile. I ran strace on each for EPERMS and sent them all to elastic search. I made a project just for it: strace2elastic. It’s super dumb but was fun.

By the time we released I knew I had done at least everything in my power to make sure we didn’t break anyone. The release actually went really well too. However, when you try to explain this to other projects they of course have their doubts, which I do not blame them for. I wish there was a better way to trust the genuine people who just want to help in open source.

So why all the confusion and FUD? Well, it’s simple really. Marketing. The tech never sells itself. It’s all about marketing.

When you work at a large organization you are surrounded by an echo chamber. So if everyone in the org is saying “containers are not secure,” you are bound to believe it and not research actual facts. To be clear I am not saying containers are secure, literally nothing is secure. Spreading FUD while ignorant or not doing proper research is harmful to the facts and hard work many people put in to making containers at least decently isolated by default.

Operability There is another problem I have with gvisor. In my opinion, I think it would be quite hard to operate. People enjoy debugging with certain workflows and reinventing syscalls is going to be quite hard to debug. Just look up one of Bryan Cantrill’s rants on unikernels which are harder to debug as well.

I believe it is putting a lot of extra burden on the operator to learn how to operate. At the end of the day you are left with a decision to trust or research the container security defaults or use a new runtime that re-implements all the syscalls in user-space and has poorer performance because of that. I also have yet to see a report on the fact that running in user-space is actually more secure. The implementation could be closely related to that of user mode linux and even user mode linux was never fully vetted for multi-tenancy so what are you really gaining. I truly believe it cannot be possibly more secure than the defaults for containers are today and surely it is not as secure as a real hypervisor. But, again, nothing is actually secure.

I am not trying to throw shade at gvisor but merely clear up some FUD in the world of open source marketing. I truly believe that people choosing projects to use should research into them and not just choose something shiny that came out of Big Corp. I also believe that people at Big Corp should embrace the work and ideas of people outside their echo chamber. Sometimes they even work in the echo chamber but just don’t abide by the echo chamber beliefs.

Open your minds and hearts to the ideas of other people and you might just create something you never thought was possible in the first place.

Update: See James Bottomley’s research on Horizontal Attack Profile which shows gVisor uses more syscalls than a standard docker container.

View Details

EDIT: See my post on a design doc for a multi-tenant orchestrator instead. I wrote this when an internal requirement was to use Kubernetes but I do not personally think you should use Kubernetes for this use case.

Kubernetes is the new kernel. We can refer to it as a “cluster kernel” versus the typical operating system kernel. This means a lot of great things for users trying to deploy applications. It also leads to a lot of the same challenges we have already faced with operating system kernels. One of which being privilege isolation. In Kubernetes, we refer to this as multi-tenancy, or the dream of being able to isolate tenants of a cluster.

The models for multi-tenancy have been discussed at length in the community’s multi-tenancy working group. NOTE: to view most of these Google docs you need to be a member of the kubernetes-wg-multitenancy Google group. There have also been some proposals offered to solve each model. The current model of tenancy in Kubernetes assumes the cluster is the security boundary. You can build a SaaS on top of Kubernetes but you need to bring your own trusted API and not just use the Kubernetes API. Of course, with that comes a lot of considerations you must also think about when building your cluster securely for a SaaS even.

The model I am going to be focusing on for this post is “hard multi-tenancy.” This implies that tenants do not trust each other and are assumed to be actively malicious and untrustworthy. Hard multi-tenancy means multiple tenants in the same cluster should not have access to anything from other tenants. In this model, the goal is to have the security boundary be the Kubernetes namespace object.

The hard multi-tenancy model has not been solved yet, but there have been a few proposals. All systems have weaknesses and nothing is perfect. With a system as complex and large as Kubernetes it is hard to trust the entire system to not be vulnerable. In this regard and in the regard of the existing proposals, one single exploit in Kubernetes leads to full supervisor privileges and then it’s game over.

This is not an acceptable way to secure a system and guarantee isolation between tenants. I will cover in this post why having more than one layer of security is so important.

The attack surface with the highest risk of logical vulnerabilities is the Kubernetes API. This must be isolated between tenants. The attack surface with the highest risk of remote code execution are the services running in containers. These must also be isolated between tenants.

If you take one look at the open source repository and the speed to which Kubernetes is growing, it is already taking on a lot of the same aspects of the monolithic kernels of Windows, Mac OS X, Linux, and FreeBSD. Fortunately, there have already been a lot of solutions to privilege separation in monolithic kernels researched and implemented.

The solution I am going to focus on is Nested Kernel: Intra-Kernel Isolation. This paper solves the problem of privilege isolation in monolithic kernels by nesting a small kernel inside the monolithic kernel.

More than One Layer of Security Required What we know of today as “sandboxes” are defined as having multiple layers of security. For example, the sandbox I made for the contained.af playground has security layers defined by seccomp, apparmor, kernel namespaces, cgroups, capabilities, and an unprivileged Linux user. All those layers don’t necessarily overlap, but a few do. If a user was to have an apparmor or seccomp bypass and they tried to call mount inside the container, the Linux capability of CAP_SYS_ADMIN would still prevent them from executing mount.

These layers ensure that one vulnerability in the system does not take out the entire security of the system. We need this for hard multi-tenancy in Kubernetes as well. This is why all the existing proposals are insufficient. We need at least two layers and these comprise only one.

With intra-kernel isolation applied to Kubernetes, we get two layers. Let me dive in a bit deeper into how this would work.

Isolation via Namespaces The existing proposals for hard multi-tenancy assume that the security boundary for multiple users on Kubernetes would be the namespace. “Namespace” in this regard being those defined by Kubernetes. The proposals all have the weakness that if you exploit one part of Kubernetes you can then have privileges to transverse namespaces and therefore transverse the tenants.

With Intra-Kernel Isolation, the namespace would still be the security boundary. However, instead of all sharing the main Kubernetes system services, each namespace would have it’s own “nested” Kubernetes system services. Meaning the api-server, kube-proxy, etc would all be running individually in a pod in that namespace. The tenant who deploys to that namespace would then have no access to the actual root-level Kubernetes system services but merely the ones running in their namespace. An exploit in Kubernetes would not be game over for the whole system, but only game-over within that namespace.

Another security boundary would also be the container isolation itself. These pods could be further locked down by the existing resources like PodSecurityPolicy and NetworkPolicy. With the ever growing innovation in the ecosystem, you could even run VMs (katacontainers) for hardware-isolation between containers giving you the highest level of security between the services in your cluster.

For those familiar with Linux namespaces you can think of this as a clone for Kubernetes. The design is roughly similar.

For example on linux cloning new namespaces looks like:

``` clone(CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_NEWNET | CLONE_NEWPID… )

```

So when you create a new Kubernetes namespace with intra-kernel isolation this roughly translates to, purely example not to be taken literally:

``` clone(CLONE_NEWAPISERVER | CLONE_NEWKVSTORE | CLONE_NEWKUBEPROXY | CLONE_NEWKUBEDNS…)

```

In Linux, namespaces control what a process can see. This holds true for users designated to a namespace in Kubernetes. Since each namespace would have its own system services that would be all they could see.

Unlike the pseudo code above, the Kubernetes namespace will automatically get new components of each system service. This is more in line with the design of Solaris Zones or FreeBSD Jails.

In my blog post Setting the Record Straight: containers vs. Zones vs. Jails vs. VMs, I go over the differences between those various isolation techniques. In this design, we are more inline with that of Zones or Jails. Containers come with all the parts. The namespaces in Kubernetes should automatically set up a well isolated world, just like that of Zones or Jails without the user having to worry about if they configured it correctly.

Another problem with namespaces in Linux is that not everything is namespaced. This design ensures that every part of Kubernetes is isolated per tenant.

Isolation via Resource Control There are still a few unanswered questions just with the design above alone. Let’s take a look at another control mechanism in Linux: cgroups. Cgroups control what a process can use. They are the masters of resource control.

This concept would need to be applied to Kubernetes namespaces as well. Rather than controlling resources like memory consumption and CPU, it would apply to nodes. The tenant within a namespace would only be able to access certain nodes designated to it. All the namespace services would be isolated at the machine level as well. No services from different tenants would run on the same machine. This could always be a setting in the future but the default should be that nodes are not shared.

This model allows for designating to our nested API server a set of kubelets on various nodes to use.

At this point we have isolation of what a tenant can see (Kubernetes namespace) and what they can use (nodes designated to a namespace).

Alternatively, if the system services for the namespaces were isolated with nested VM containers (katacontainers) and you considered all the other variables outlined in this design doc. Then those services could share nodes. This would give you a bit better resource utilization than above. It is illustrated below.

Taking it even a step further for even better resource utilization, if you isolated the whole system and containers into fully sandboxed or VM containers as per this design doc, then all services could share nodes. This is illustrated below.

Tenants that Span Multiple Namespaces A few times it has been brought up in the working group that tenants might need to span multiple namespaces. While I don’t believe this should be a default, I don’t see a problem with it.

Let’s take a look again at how namespaces work in Linux and how we use them for containers. Each namespace is a file descriptor. You can share a namespace between containers by designating the file descriptor for the namespace you want to share and calling setns.

In Kubernetes, we could implement the same sort of design. A superuser can delegate a namespace is to be shared between tenants with access to that namespace.

Overall this design uses the expertise from past art of kernel isolation techniques. It is also designed with the lessons learned from past kernel isolation techniques.

With the growing ecosystem and core of Kubernetes it’s important to have more than one layer of security between tenants. Security techniques such as failsafe defaults, complete mediation, least privilege, and least common mechanism are very popular but hard to apply to monolithic kernels. Kubernetes by default shares everything and has many different, sometimes very broken, drivers and plugins just like that of an operating system kernel. Applying the same isolation techniques of kernels to Kubernetes will allow for a better privilege isolation solution.

Where does this leave us? We have fully isolated and solved our threat model in a very strong way. The attack surface with the highest risk of logical vulnerabilities, the Kubernetes API, has full logical separation in that each tenant has their own. The attack surface with the highest risk of remote code execution, containers themselves, have full virtualized separation from other tenants. This isolation either comes from isolating via designated nodes themselves to tenants or by running containers that use hardware isolation. The only viable path to other tenants is getting remote code execution in some service, then breaking out of the container (and/or VM).

The first diagram of intra-kernel isolation via node resource control illustrates close to the same as having two fully separate clusters operated by one superuser. Since nodes are designated to each tenant, you do not really gain more efficient resource utilization either.

The model with the highest gain of resource control comes from securely setting up your cluster to use nested virtual machines as containers or fully sandboxing the containers themselves so that the boundary is the container not the node. This eases the operators pain of running more than one cluster and allows resources to be used more effectively while also sustaining more than one layer of security.

None of this is set in stone. This is my idea for solving this problem. If you are interested in discussing this or other aspects of tenancy in Kubernetes please join the working group. I look forward to discussing this there. Thanks!

View Details

A lot of people seem to want to be able to build container images in Kubernetes without mounting in the docker socket or doing anything to compromise the security of their cluster.

This all was brought to my attention when my awesome coworker at Gabe Monroy and I were chatting with Michelle Noorali over pizza at Kubecon in Austin last December.

Here is pretty much how it went down:

``` Gabe: I’d would love to switch our clusters to a lightweight runtime like containerd, but we need those docker build apis right now. I wish someone would come up with an unprivileged container image builder..

Me: Oh that’s easy

Gabe: Bullshit, if it was easy someone would have done it already. I’ve wanted this for years. Please pass the ranch dressing.

Me: I’m telling you you’re wrong. I’ll prove it to you. It’s easy.

Judgy Four Seasons Staff: Excuse me, can I help you?

Me: Nah we’re good. Actually if you could grab me a slice of that Papa John's jalapano & pineapple that would be great.

.. next morning ..

100 lines of bash shaming in Gabe's inbox proving it could be done.

```

Prior Art A few years ago when I worked at Docker, Stephen Day and Michael Crosby did a POC demo of a standalone image builder.

It still actually exists today in a fork of docker/distribution on Stephen’s github. It consisted of a dist command line tool for interacting with the registry and runc. Combined together with the awesome powers of bash like so (nsinit was runc before runc was A Thing):

```

!/bin/bash

function FROM () { mkdir rootfs dist pull "$1" rootfs }

function USERNS() { export nsinituserns="$1" }

function CWD() { export nsinitcwd="$1" }

function MEM() { export nsinitmem="$1" }

function EXEC() { nsinit exec \ --tty \ --rootfs "$(pwd)/rootfs" \ --create \ --cwd="$nsinitcwd" \ --memory-limit="$nsinitmem" \ --memory-swap -1 \ --userns-root-uid="$nsinituserns" \ -- $@ }

function RUN() { t="\"$@\"" EXEC sh -c "$t" }

```

So in their demo, you would source the above bash script and then execute your Dockerfile like it was also a bash script. Pretty cool right.

So that is what I sent to Gabe’s inbox to prove it was possible but also: “Look, I will make you something nice.”

Designing Something Nice So I went out on my mission to make them something nice, which lead me through a sea of existing tools. I collected all my findings in a design doc if you are curious as to what I think about the other existing tools.

I didn’t want to reinvent the world I just wanted to make it unprivileged and a single binary with a simple user interface that could easily be switched out with docker.

Not all of my ideas are good. I first started on a FUSE snapshotter. Turns out FUSE kinda sucks…

so fuse calls getxattr 2x the amount it calls lookup even if the damn inodes have no xattrs…. and it has to go back and forth from kernel to userspace to do it… I need a drink.

— jessie frazelle (@jessfraz) February 8, 2018

I started playing with buildkit. It’s an awesome project. Tõnis Tiigi did a really stellar job on it and I thought to myself, “I definitely want to use this as the backend.”

Buildkit is more cache-efficient than Docker because it can execute multiple build stages concurrently with its internal DAG.

Then I stumbled upon Akihiro Suda’s patches for an unprivileged Buildkit. This was perfect for my use case.

I owe all these fine folks so much for the great work I got to build on top of. :)

And thus came img.

So that was all fine and dandy and it works great as unprivileged… on my host. Now I’m a huge fan of desktop tools and this actually filled a large void in my tooling that now I can build as unprivileged on my host without Docker.

But I still have to make this work in Kubernetes so I can make Gabe happy and fulfill my dreams of eating more pineapple and jalapeno pizzas at Kubecons.

Why is this problem so hard? Let me go over in detail some of the patches needed to even make this work as unprivileged on my host.

For one, we need subuid and subgid maps. See @AkihiroSuda’s patch. We also need to setgroups. See @AkihiroSuda’s patch for that as well. Those allow us to use apt in unprivileged user namespaces.

Then if we want to use the containerd snapshotter backends and actually mount the filesystems as we diff them, then we need unprivileged mounting. Which can only be done from inside a user and mount namespace. So we need to do this at the start of our binary before we even do anything else.

Granted mounting is not a requirement of building docker images. You can always go the route of orca-build and umoci and not mount at all. umoci is also an unprivileged image builder and was made long before I even made mine by the talented Aleksa Sarai who is also responsible for a lot of the rootless containers work upstream in runc.

Getting this to work in containers… img works on my host which is all fine and dandy but I gotta help my k8s pals do their builds…

Enter the next problem. For the record, all these problems apply to any builder that is using runc to launch containers as an unprivileged user.

The next issue involved not being able to mount proc inside a Docker container.

My first thought was “well it must be something Docker is doing”. So I isolated the problem, put it in a container and ten minutes after I dove into the rabbit hole I realized it was the fact that Docker sets paths inside /proc to be masked and readonly by default, preventing me from mounting.

Duh I thought to myself. Remember that thing we never thought we’d need… well we need it.

"We'll never need this"

"Fuck, we need that"

— julia ferraioli (@juliaferraioli) March 4, 2018

You can find all the fun details on opencontainers/runc#1658.

Well this blows, I could obviously just run the container as --privileged but thats really stupid and defeats the whole point of this exercise. I did not want to add any extra capabilities or any host devices which is exactly what privileged does… gross.

So I opened an issue on Docker and made a patch.

Okay so problem solved. Wait… no… now I gotta pull that option through to kubernetes…

So I opened a proposal there: kubernetes/community#1934.

And I made a patch just for playing with it on my fork: jessfraz/kubernetes#rawproc.

Okay now I want to try it in a cluster… enter acs-engine. I made a branch there as well for easily combining together all my patches for testing: jessfraz/acs-engine#rawaccess.

Here is a yaml file you can use to deploy and try it:

``` apiVersion: v1 kind: Pod metadata: labels: run: img name: img annotations: container.apparmor.security.beta.kubernetes.io/img: unconfined spec: securityContext: runAsUser: 1000 initContainers: # This container clones the desired git repo to the EmptyDir volume. - name: git-clone image: r.j3ss.co/jq args: - git - clone - --single-branch - -- - https://github.com/jessfraz/dockerfiles - /repo # Put it in the volume securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true volumeMounts: - name: git-repo mountPath: /repo containers: - image: r.j3ss.co/img imagePullPolicy: Always name: img resources: {} workingDir: /repo command: - img - build - -t - irssi - irssi/ securityContext: rawProc: true volumeMounts: - name: cache-volume mountPath: /tmp - name: git-repo mountPath: /repo volumes: - name: cache-volume emptyDir: {} - name: git-repo emptyDir: {} restartPolicy: Never

```

So is this secure? Well I am running that pod as user 1000. Granted it does have access to a raw proc without masks… the nested containers do not. The nested containers have /proc set as read-only and masked paths. The nested containers also use a default seccomp profile denying privileged operations that should not be allowed.

Your main concern here is my code and the code in buildkit and runc. Personally I think that’s fine because I obviously trust myself, but you are more than welcome to audit it and open bugs and/or patches.

If you randomly generate different users for all your pod builds to run under then you are relying on the user isolation of linux itself.

If you are running a cluster inside your organization, it’s unlikely someone is going waste a kernel 0day popping your cluster from within your org.

This is much better than the current situation where people are mounting the docker socket into containers and everything is running as root.

You can even use a Pod Security Policy and set MustRunAs to make sure all your pods are being run as users within a certain range of uids.

You are effectively as safe as any other non-root user running on a shared machine.

If you are running random builds from users off the internet I would suggest using VMs. You can use my patches to acs-engine to run all your pods in Intel’s Clear Containers and you would then have hardware isolation for your little builders :) You just need to use this config.

And that ends the most epic yak shave ever, minus the patches all being merged upstream. Thanks for playing. Feel free to try it out on Azure with my branch to acs-engine. That was a lot of patching and I’m tired. Peace.

View Details

This is a story about how I got nerd sniped by a blog post from Cloudflare Engineering. The TLDR on their post is that you can script in Go if you use BINFMT_MISC in the kernel.

BINFMT_MISC is really well documented and awesome. In the end, all they had to do to script in Go was to mount the filesystem:

``` $ mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

```

Then, register the Go script binary format:

``` $ echo ':golang:E::go::/usr/local/bin/gorun:OC' | sudo tee /proc/sys/fs/binfmt_misc/register :golang:E::go::/usr/local/bin/gorun:OC

```

Then you can ./ any go file on your host:

``` $ chmod u+x helloscript.go $ ./helloscript.go Hello, world!

```

They go through all the extraordinary details of exit codes for the shell and blah blah blah. It’s a great post you should really read it. Do it, go read it, then come back here and I will take it to 11.

…

Okay, cool, you are back. That post was dope right?

I kinda want to do this with all languages. Because I LOVE SCRIPTING. Have you seen my cloud native dotfiles? My bash scripts smell like roses.

Right, so I want to do this with all languages… but what I also hate is installing shit on my host. Ew, we have containers for those silly things. Luckily, I know a thing or two about containers…

A few years ago I made a project called binctr. It creates fully static, unprivileged, self-contained, containers as executable binaries. (Wow that was a lot of words, let’s break it down.) What binctr does is embed an entire container image (aka rootfs) into a fully static binary and when you execute the binary it will unpack the image and run it as a container. So you get containers without a daemon or privileges and without even having the image for the rootfs of the container. You just need this one binary.

(Huge thanks to @lordcyphar who got rootless containers into runc so I could actually archive my gross hack for binctr.)

Kinda seems like the perfect match for trying to use all languages with BINFMT_MISC. So I tried it.

(Preface: this post should not be tried at home, which is why I did not unarchive binctr, I am merely showing a different, very crazy abstraction).

I put common lisp in a container. Why common lisp? Well I could do this with any language and I’m a bit insane haven’t you noticed…

Then I embedded the image into a binary with binctr. I made one slight modification to the spec in binctr that allowed me to use local files, basically so I could get the script into the container after the executable is run pointing to the file.

Then I registered my common lisp binary format with BINFMT_MISC…

``` $ echo ':clisp:E::lisp::/usr/local/bin/clisp:OC' | sudo tee /proc/sys/fs/binfmt_misc/register :clisp:E::lisp::/usr/local/bin/clisp:OC

```

/usr/local/bin/clisp is just my binctr generated binary with common lisp.

And boom, now I can “dot slash” any .lisp file and it will run in my common lisp container.

Obviously, my container needed to be packaged with any dependencies and packages I needed but I didn’t need to install any of that shit on my host so I consider it a win.

Imagine if an entire OS had all the languages packaged this way so that everything could be “dot slashed” and executed but without actually installing the language to your host operating system.

I think it would be dope.

Thanks for tuning in for this crazy blog post. Catch ya later. Hacker news, you can shove your comments right up your

View Details

This post is kind of like “part two” on my series on all the weird things I do for my personal infrastructure. If you missed “part one”, you should check out Home Lab is the Dopest Lab.

I run a lot of little things to make my life easier, like a CI, some bots, and a bunch of services just for the lolz. This post will go over all of those. These run scattered across my NUCs and the cloud.

Let’s start with the most useful.

Continuous Integration I host my own continuous integration server. Yes, you guessed it… it’s Jenkins. I use the Jenkins DSL plugin to keep everything in sync. You can find all my DSLs in my repo github.com/jessfraz/jenkins-dsl. This has all the configurations for views, keeps forks up to date, mirrors all my repositories to private git (more on this in git), builds all Dockerfiles to push to Docker Hub and my private registry (more on this in private docker registry) and a bunch of maintenance scripts.

The Makefile in this repo calls out to bash scripts which generate new DSLs for any new GitHub repos I create. Yep I even generate the automation…

There’s a bunch of other fun things in there as well that you can discover by poking around yourself.

I host my own postfix server alongside Jenkins. You can find the postfix docker image at r.j3ss.co/postfix or the Dockerfile. It’s super minimal and less gross than literally every other postfix image in existence.

You can run it with:

``` $ docker run --restart always -d \ --name postfix \ --net container:jenkins \ -e "ROOT_ALIAS=root@blah.com" \ -e "RELAY=[smtp-relay.gmail.com]:587" \ -e "TLS=1" \ -e "MY_DESTINATION=...., localhost" \ -e "MAILNAME=blah.com" \ r.j3ss.co/postfix

```

Private Docker Registry I host my own private docker registry with my own notary server and authentication server. Why? Well because about 4 years ago when I started using docker, Docker Hub was super slow and I came to love having my own super fast one.

I still push all the images to both Docker Hub and my registry and both are signed so it’s really like I am using Docker Hub as my backup. Yay, highly available… just kidding.

I made a pretty shitty UI for it. You can play with it at r.j3ss.co. The UI is from my reg project but the server component lives in the server subdirectory.

The really nice thing about both the reg command line and server is that you can get a list of CVEs on an image.

I do this by hosting my own instance of CoreOS’s Clair.

Most of my Dockerfiles live at github.com/jessfraz/dockerfiles if you are curious.

I also went over all of this on my talk on Over Engineering my Laptop / Container Linux on the Desktop. This includes all the reasons why I have continuous integration as well.

I have a script to cleanup the registry of old images clean-registry. This deletes old registry blobs that are not used in the latest version of the tag. I don’t really care about old images and I don’t want to have a huge registry filled with old shit. There is a jenkins DSL to run this.

Git Server I host my own git server. You can find the gitserver docker image at r.j3ss.co/gitserver or the Dockerfile.

You can run it with:

``` $ docker run --restart always -d \ --name gitserver \ -p 127.0.0.1:22:22 \ -e "PUBKEY=$(cat ~/.ssh/authorized_keys)" \ -v "/mnt/disks/gitserver:/home/git" \ r.j3ss.co/gitserver

```

It has it’s own UI that is run with Gitiles. You can find the Gitiles docker image at r.j3ss.co/gitiles or the Dockerfile.

You can run it with:

``` $ docker run --restart always -d \ --name gitiles \ -p 127.0.0.1:8080:8080 \ -e BASE_GIT_URL="git@git.blah.com" \ -e SITE_TITLE="git.blah.com" \ -v "/mnt/disks/gitserver:/home/git" \ -w /home/git \ r.j3ss.co/gitiles

```

ghb0t This is one of my most useful things. It’s a GitHub Bot to automatically delete your fork’s branches after a pull request has been merged.

I am so OCD about keeping git repos clean and this is my little helper.

Check out the repo: github.com/jessfraz/ghb0t.

I go to fork your thing and there is like 300 branches my face is like pic.twitter.com/JpdpO447KS

— jessie frazelle (@jessfraz) January 23, 2017

IRC Bouncer I host my own IRC Bouncer with ZNC. You can find the ZNC docker image at r.j3ss.co/znc or the Dockerfile.

You can run it with:

``` $ docker run --restart always -d \ --name znc \ -p 6697:6697 \ -v "/mnt/disks/znc:/home/user/.znc" \ r.j3ss.co/znc

```

upmail This service provides email notifications for sourcegraph/checkup. If you are unfamiliar with checkup… it’s distributed, lock-free, self-hosted health checks and status pages, written in Go.

I wrote a small little server to send email alerts for it and it lives at github.com/jessfraz/upmail.

iPython Not really all that novel but I also run an iPython server for doing little script things in. I just use the jupyter/minimal-notebook Docker image for that.

Conclusion I run a lot of little shitty services for a personal pastebin and other things but those are all really less cool. My attention span for blog posts is about 5 minutes and we have runneth over so I am going to call it a day with this… until next time. Peace out.

View Details

I always have some random side project I am working on, whether it is making the world’s most over engineered desktop OS all running in containers or updating all my Makefiles to be the definition of glittering beauty.

This post is going to go over I how I recently redid all my home networking and ultimately how I got to here:

ssh-ed into my dev NUC from a Pixelbook 39,000 feet, authenticated from an ssh key on a yubikey, the future is dope AF

— jessie frazelle (@jessfraz) November 22, 2017

I used Unifi for everything and this is what I got:

• Access Point: AP AC SHD
• Switch: Switch 16-150W
• Router: Security Gateway

It was so good looking when it arrived.

My network is about to get real… fast!!!

This switch is (dare I say it) sexy as hell. pic.twitter.com/fmaLkW2AFB

— jessie frazelle (@jessfraz) November 16, 2017

I love fun side projects so obviously I set it all up right away. You need a “controller” to have the nice Unifi UI. You can buy a cloud key but I wanted to run the controller in container just like Dustin Kirkland. So I set about writing a Dockerfile for the controller and it is now at r.j3ss.co/unifi.

You can run it with:

``` docker run -d --restart always \ -v /etc/localtime:/etc/localtime:ro \ --name unifi \ --volume path/to/where/you/want/your/data:/config \ -p 3478:3478/udp \ -p 10001:10001/udp \ -p 8080:8080 \ -p 8081:8081 \ -p 8443:8443 \ -p 8843:8843 \ -p 8880:8880 \ r.j3ss.co/unifi

```

The web UI is at https://{ip}:8443. To adopt an access point, and get it to show up in the software you will need to ssh into the AP and run:

``` ssh ubnt@$AP-IP mca-cli set-inform http://$address:8080/inform

```

Then I went crazy and made sure everything that needed to talk to each other was on the same subnet and everything else was isolated into it’s own subnet. I used VLANs to do this.

Also be careful not to subnet yourself into a hole ;)

me just now: "this was my fear! sub-netting myself into a hole!"

— jessie frazelle (@jessfraz) November 30, 2017

The best thing about these APs are they are Power over Ethernet! One cord, one cord!!!

You down wit' PoE?

— Dan McDonald (@kebesays) November 16, 2017

NUCs I have a bunch of Intel NUCs thanks to Carolyn Van Slyck and Joe Beda for their thought leadership… my wallet is not happy with you two. Also check out Carolyn’s post on her NUC setup.

They have LEDs on the front that change color. There is a kernel driver for them.

— Joe Beda (@jbeda) October 18, 2017

I hooked them all into my Switch (glorious) and into their own subnet. Then I went about setting up SSH for all of them.

I use Yubikeys for authentication to GitHub and literally everything else where that is possible so I made a bot to sync any new ssh keys added to my GitHub to the authorized keys on my server. It lives at github.com/jessfraz/sshb0t.

I would ONLY recommend doing that if you have two factor auth turned on so you ensure no one else but you can access your account. And honestly if someone gets into my GitHub account I am going to have wayyyy worse issues that them getting into my NUCs.

I have ssh keys on Yubikeys that I set up. There is a really great guide to doing this on GitHub so I am not going to repeat it.

I have dockerfiles for all the Yubikey tools you need to set it up in my dockerfiles repo.

For example you can jump into a container with ykman with:

``` docker run --rm -it \ -v /etc/localtime:/etc/localtime:ro \ --device /dev/usb \ --device /dev/bus/usb \ --name ykman \ r.j3ss.co/ykman bash

```

This works for all the other docker images like ykpersonalize etc. If you get stuck all the commands are in my dotfile aliases at github.com/jessfraz/dotfiles.

I like to require “touch to authenticate”. You can do this with:

```

for every ssh connection

ykman openpgp touch aut on

for signing

ykman openpgp touch sig on

for encrypting

ykman openpgp touch enc on

```

For the Chromebook Pixelbook ssh client authentication you just need the Smart Card reader extension and you are good to go! You can find the guide on that from the Chromium Docs.

Let me just answer the most common question I get… No, I don’t use Crouton on my Chromebooks I just ssh to the cloud or to my home lab. I like things clean and minimal if you have not noticed already.

Okay so that’s all for now. I’ll do another deep dive into the rest of my infrastructure when I’m not overwhelmed with how much there is…

There’s so much:
- scripts for setting up ssh on yubikeys
- unifi setup
- nuc provisioning
- auto updates & maintenance
- build infrastructure for all my images etc
- security of all the things
- cameras
- keeping all laptops up to date

— jessie frazelle (@jessfraz) November 29, 2017

View Details

I recently started a job at Microsoft. In my first week I have already learned so much about Windows, I figured I would try to put it all into writing. This post is coming to you from a Windows Subsystem for Linux console!

I'm headed to Seattle because I'M JOINING MICROSOFT, at the airport wearing this awesome shirt from @listonb & @Taylorb_msft ���� pic.twitter.com/8rnAg1dsPd

— jessie frazelle (@jessfraz) September 4, 2017

New job and I got a Windows computer and a Linux computer! If you are new to my blog, let me tell you: I love setting up a perfect desktop experience. I’ve written a few posts on it (for Linux), you should check them out. Setting up a Windows computer is something I have not done in quite some time. I will describe a bit how to set up a windows machine in a reproducible way at the end of this post.

I would like to thank Rich Turner, John Starks, Taylor Brown, and Sarah Cooley for taking the time to explain a lot of the following to me. :)

Windows Subsystem for Linux (WSL) Let’s start with Windows Subsystem for Linux, aka WSL. Even @monkchips wrote that since I joined Microsoft “Linux Subsystem for Windows will definitely be getting a workout.” I am super excited about Windows Subsystem for Linux. It is one of the coolest pieces of tech I’ve seen since I started using Docker.

First, a little background on how WSL works…

You can learn a lot more about this from the Windows Subsystem for Linux Overview. I will go over some of the parts I found to be the most interesting.

The Windows NT kernel was designed from the beginning to support running POSIX, OS/2, and other subsystems. In the early days, these were just user-mode programs that would interact with ntdll to perform system calls. Since the Windows NT kernel supported POSIX there was already a fork system call implemented in the kernel. However, the Windows NT call for fork, NtCreateProcess, is not directly compatible with the Linux syscall so it has some special handling you can read about more under System Calls.

There are both user and kernel mode parts to WSL. Below is a diagram showing the basic Windows kernel and user modes alongside the WSL user and kernel modes.

The blue boxes represent kernel components and the green boxes are Pico Processes. The LX Session Manager Service handles the life cycle of Linux instances. LXCore and lxsys, lxcore.sys and lxss.sys respectively, translate the Linux syscalls into NT APIs.

Pico Processes As you can see in the diagram above, init and /bin/bash are Pico processes. Pico processes work by having system calls and user mode exceptions dispatched to a paired driver. Pico processes and drivers allow Windows Subsystem for Linux to load executable ELF binaries into a Pico process’ address space and execute them on top of a Linux-compatible layer of system calls.

You can read even more in depth on this from the MSDN Pico Processes post.

System Calls One of the first things I did in WSL was run a syscall fuzzer. I knew it would break but it was interesting for the purposes of figuring out which syscalls had been implemented without looking at the source. This was how I realized PID and mount namespaces were already implemented into clone and unshare!

The WSL kernel drivers, lxss.sys and lxcore.sys, handle the Linux system call requests and translate them to the Windows NT kernel. None of this code came from the Linux kernel, it was all re-implemented by Windows engineers. This is truly mind blowing.

When a syscall is made from a Linux executable it gets passed to lxcore.sys which will translate it into the equivalent Windows NT call. For example, open to NtOpenFile and kill to NTTerminateProcess. If there is no mapping then the Windows kernel mode driver will handle the request directly. This was the case for fork, which has lxcore.sys prepare the process to be copied and then call the appropriate Windows NT kernel APIs to create and copy the process.

You can learn more from the MSDN System Calls post.

Launching Windows Executables Since WSL allows for running Linux binaries natively (without a VM), this allows for some really fun interactions.

You can actually spawn Windows binaries from WSL. Linux ELF binaries get handled by lxcore.sys and lxss.sys as described above and Windows binaries go through the typical Windows userspace.

You can even launch Windows GUI apps as well this way! Imagine a Linux setup where you can launch PowerPoint without a VM…. well this is it!!

Launching X Applications You can also run X Applications in WSL. You just need an X server. I used vcxsrv to try it out. I run i3 on all my Linux machines and tried it out in WSL like my awesome coworker Brian Ketelsen did in his blog post.

The hidpi is a little gross but if you play with the settings for the X server you can get it to a tolerable place. While I think this is neat for running whatever X applications you love, personally I am going to stick to using tmux as my entrypoint for WSL and using the Windows GUI apps I need vs. Linux X applications. This just feels less heavy (remember, I love minimal) and I haven’t come across an X application I can not live without for the time being. It’s nice to know X applications can work when I do need something though. :)

Pain Points There are still quite a few pain points with using Windows Subsystem for Linux, but it’s important to remember it is still in the beginnings. So that you all have an idea of what to expect I will list them here and we can watch how they improve in future builds. Each item links to the respective GitHub issue.

Keep in mind, I am using the default Windows console for everything. It has improved significantly since I played with it 2 years ago while we were working on porting the Docker client and daemon to Windows. :)

• Copy/Paste: I am used to using ctrl-shift-v and ctrl-shift-c for copy paste in a terminal and of course those don’t work. From what I can tell enter is copy… supa weird… and ctrl-v says it’s paste. Of course it doesn’t work for me. I can get paste to work by two-finger clicking in the term, but that does not work in vim and it’s a pretty weird interaction.
• Scroll: This might just be a huge pet peeve of mine but the scroll should not be able to scroll down to nothing. This happens all the time by accident for me with the mouse and I have no idea why the terminal is rendering more space down there. Also typing after I have scrolled should return me back to the console place where I am typing. It unfortunately does not.
• Files Slow: Saving a lot of files to disk is super slow. This applies for example to git clones, unpacking tarballs and more. Windows is not used to applications that save a lot of files so this is being worked on to be more performant. Obviously the unix way of “everything is a file” does not scale well when saving a lot of small files is super slow.
• Sharing Files between Windows and WSL: Right now, like I pointed out, your Windows filesystem is mounted as /mnt/c in WSL. But you can’t quite yet have a git repo cloned in WSL and then also edit from Windows. The VolFS file system, all file paths that don’t begin with /mnt, such as /home, is much closer to Linux standards. If you need to access files in VolFS, you can use bash.exe to copy them somewhere under /mnt/c, use Windows to do whatever on it, then use bash.exe to copy them back when you are done. You can also all Visual Studio code on the file from WSL and that will work. :)

Setting Up a Windows Machine in a Reproducible Way This was super important to me since I am used to Linux where everything is scriptable and I have scripts for starting from a blank machine to my exact perfect setup. A few people mentioned I should check out boxstarter.org for making this possible on Windows.

Turns out it works super well! My gist for my machine lives on github. There is another powershell script there for uninstalling a few programs. I love all things minimal so I like to uninstall applications I will never use. I also learned some cool powershell commands for listing all your installed applications.

```

--- List all installed programs --

Get-ItemProperty HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall* | Select-Object DisplayName, DisplayVersion, Publisher, InstallDate |Format-Table -AutoSize

--- List all store-installed programs --

Get-AppxPackage | Select-Object Name, PackageFullName, Version |Format-Table -AutoSize

```

I am going to be scripting more of this out in the future with regard to pinning applications to the taskbar in powershell and a bunch of other settings. Stay tuned.

Overall, I hope you now understand some basics around Windows Subsystem for Linux and are as excited as I am to see it grow and evolve in the future!

View Details

I recently gave a talk at DevOps Days (slides) and it had a pretty great response. I’m still pretty care-mad about the topics it covered so I figured I would turn some key points from it into a blog post.

The overall outline of the talk covered the past, present, and future of usable security. Let’s start with the past.

The Past A lot of the security tooling of the past (that we still use today) require users to jump through a lot of hoops or learn a hard to grok interface. One of the examples I used was GPG. Contrary to popular opinion, I actually don’t find GPG entirely unusable. I obviously agree that it could be easier to use, rotate keys, revoke keys blah blah blah. While I find it not exactly terrible, I can see and completely understand why the majority of criticism I hear about GPG is that it is hard to use.

There is a point at which better security comes at the expense of convenience. This needs to stop happening. Stop compromising convenience for security. Instead find the right balance between the two. Doing this takes collaboration from both security engineers and software engineers.

Dave Cheney recently had a great tweet.

Why is all software shit? Today I discovered the @duosec API returns 200 even if someone denies the 2fa request.

— Dαve Cheney (@davecheney) July 25, 2017

I love this tweet because it reeks of the stench that only security engineers built this API. Most software engineers I know would decide to use an HTTP status code… I mean that’s what they are for. ;)

When you combine expertise in different areas you build better products. This is not rocket science. However egos tend to get in the way as well as biases towards people who know and like the same things you do. I assure you, though, when security and software engineers work together truly usable security will be the outcome.

The Present A lot of the content for this portion of the talk focused on how containers make securing your infrastructure easier. I will touch on some of that but if you wish to know more you should checkout the slides or some of my other blog posts on container security.

Two key features in Docker are the default AppArmor and Seccomp profiles. AppArmor and Seccomp are Linux Security Modules that are not exactly usable by someone who is unfamiliar with either.

AppArmor can control and audit various process actions such as file (read, write, execute, etc) and system functions (mount, network tcp, etc). It has its own meta language, so to speak, and I actually have a repo that changes the docs for it to more a readable format via a cron job: github.com/jessfraz/apparmor-docs. The default profile for AppArmor does super sane things like preventing writing to /proc/{num}, /proc/sys, /sys and preventing mount to name a few.

Syscall filters allow an application to define what syscalls it allows or denies. The default in Docker is a whitelist that I initially wrote. Some of the key things it blocks are:

• add_key, keyctl, request_key: Prevent containers from using the kernel keyring, which is not namespaced. I wrote a blog post on Two Objects not Namespaced by the Linux Kernel and the keyring was one I mentioned.
• clone, unshare: Deny cloning new namespaces. Also gated by CAP_SYS_ADMIN for CLONE_* flags, except CLONE_USERNS. I specifically wanted to block cloning new user namespaces inside containers because they are notorious for being points of entry for kernel bugs.

There also is an entire document that I started in the docker repo that outlines what we block and why.

Having written the default seccomp profile for Docker I am pretty familiar with how hard this would be for other people. It requires a deep knowledge of the application being contained and the syscalls it requires. This was also a quite terrifying feature to add to Docker. When I added it, Docker was already very popular and if anything would break in a big way it would be on the front page of hacker news and all the maintainers would have a very bad day. So turning on something that will EPERM by default if we left out any important syscall is terrifying. I had stress nightmares for weeks. In the end everything went much smoother than I feared but that was also after HEAVY HEAVY testing. Luckily I run super obscure things in containers so I even caught that we left out send and recv right before the release by running Skype (a 32 bit application) in a container.

By making a default for all containers, we can secure a very large amount of users without them even realizing it’s happening. This leads perfectly into my ideas for the future and continuing this motion of making security on by default and invisible to users.

The Future I tend to have pretty weird brain child ideas and this is one of them. I started thinking about where else a kernel feature like seccomp could easily be integrated and used by a large number of people. The answer is… programming languages. I do work with the Go team and as a full content warning none of this crazy that follows is in any way endorsed by them. ;)

The idea I had is to do build-time generated seccomp filters that will be applied on run.

Why generate seccomp filters at build-time? Generating security filters/profiles at runtime has been done in the past & failed… over and over and over again. Something is always missed while profiling the application. You cannot guarantee that everything that your application will do will be called while in this profiling phase. Unless of course you have 100% test coverage, which if you do: Good For You. When the “thing that was missed” is called and blocked, users will just turn off the “security.” This happens all the time with things like SELinux and AppArmor.

By generating filters at build-time we can ensure ALL code is included in the filter. I wrote a POC of this and I showed it at Kiwicon.

There are three problems though.

1. Executing other binaries. I can’t know what syscalls the binary being called is going to use so we are back at square one.

``` package main

import ( "fmt" "log" "os/exec" )

func main() { cmd := exec.Command("myprogram") out, err := cmd.CombinedOutput() if err != nil { log.Fatal(err) } fmt.Printf("%s\n", out) }

``` 2. Plugins. This problem is solvable in that if this feature was to exist we could export at the plugin build time the seccomp filters to a field in the ELF binary or something similar.

``` func main() { p, err := plugin.Open("plugin_name.so") if err != nil { log.Fatal(err) } v, err := p.Lookup("V") if err != nil { log.Fatal(err) } fmt.Printf("%#v\n", v) }

`` 3. Sending arbitrary arguments tosyscall.RawSyscall` and similar.

``` func main() { if len(os.Args) <= 3 { log.Fatal("must pass 4 arguments to syscall.RawSyscall") } r1, r2, errno := syscall.RawSyscall(strToUintptr(os.Args[0]), strToUintptr(os.Args[1]), strToUintptr(os.Args[2]), strToUintptr(os.Args[3])) if errno != 0 { log.Fatalf("errno: %#v", errno) } fmt.Printf("r1: %#v\nr2: %#v\n", r1, r2) } func strToUintptr(s string) uintptr { return (uintptr)(unsafe.Pointer(&s)) }

```

While this is not perfect by any stretch of the imagination I believe it should open your mind to what could be possible in the future. Hopefully my dream of making binaries sandbox themselves will eventually get there. I know I won’t stop until it does. ;) Overall, I would like you to remember to find the right balance between secure AND usable. Don’t break users and get security engineering and software engineering working together!

View Details

If you are new to my blog then you might be new to the concept of Linux kernel namespaces. I suggest first reading Getting Towards Real Sandbox Containers and Setting the Record Straight: containers vs. Zones vs. Jails vs. VMs.

Linux namespaces are one of the primitives that make up what is known as a “container.” They control what a process can see. Cgroups, the other main ingredient of “containers”, control what a process can use. But let’s focus for this post on namespaces. The current set of namespaces in the kernel are: mount, pid, uts, ipc, net, user, and cgroup. These all cover basically exactly what they are named after. But what is not covered? Well, let’s go over two of the things not namespaced by the Linux kernel.

Time First, and my favorite to nerd out about, is time. Now, it should go without saying that if you want to set the time in Linux you need CAP_SYS_TIME. By default you do not get this capability in Docker containers. The settimeofday, etc syscalls are also blocked by the default seccomp profile in Docker as well.

What happens if you do change the time in a container?

Well, it’s not namespaced so the time on the host would change as well. “But whaaaaa? I thought containers were just like a VM”, you ask. Again, you should read my post Setting the Record Straight: containers vs. Zones vs. Jails vs. VMs.

One of my favorite questions I have been asked at a conference is “If you could add any new namespace to Linux what would it be?” Obviously this is an awesome question, totally up my alley, and not even a statement from someone trying to prove to me “they know things.” But I digress, I always answer with “Time.” There is no production use case for this, other than making more NTP hell for yourself. I do believe there is a development use case: if you want to change the time for a test running in one container but not mess with the other tests running in other containers. What a fun way to make a chaos monkey for NTP! :P

Kernel Keyring The kernel keyring is another item not namespaced. There have been recent efforts to fix this for user namespaces. Again, the default Docker seccomp profile blocks these syscalls so you don’t shoot yourself in the foot.

What happens if you use the kernel keyring from within in a container?

Well if root in one container stores keys in the keyring, any other containers on that same host can see it in their keyring, which is really just the same exact keyring.

All in all, I hope this proves once again that you need more than just namespaces and cgroups to get any sort of “real” isolation with containers. Please, please don’t disable seccomp or add extra capabilities you don’t need. Happy containering! I must leave you with this gif… :D

View Details

I’m tired of having the same conversation over and over again with people so I figured I would put it into a blog post.

Many people ask me if I have tried or what I think of Solaris Zones / BSD Jails. The answer is simply: I have tried them and I definitely like them. The conversation then heads towards them telling me how Zones and Jails are far superior to containers and that I should basically just give up with Linux containers and use VMs.

Which to be honest is a bit forward to someone who has spent a large portion of her career working with containers and trying to make containers more secure. Here is what I tell them:

The Design of Solaris Zones, BSD Jails, VMs and containers are very different. Solaris Zones, BSD Jails, and VMs are first class concepts. This is clear from the Solaris Zone Design Spec and the BSD Jails Handbook. I hope it can go without saying that VMs are very much a first class object without me having to link you somewhere :P.

Containers on the other hand are not real things. I have said this in many talks and I’m saying it again now.

CONTAINERS ARE NOT A REAL THING!!! @jessfraz talking containers #GoogleNext17 pic.twitter.com/gzxjNnSk2n

— Jorge Silva (@thejsj) March 10, 2017

A “container” is just a term people use to describe a combination of Linux namespaces and cgroups. Linux namespaces and cgroups ARE first class objects. NOT containers.

I am trying to make this distinction very clear to make a point. The designs are different. PERIOD.

Let’s go over some of the things you can do with containers that you CANNOT do with Jails or Zones or VMs.

Sharing Namespaces

Since containers are made with specific building blocks of namespaces this allows for doing some super neat things like sharing namespaces.

There are many different namespaces but I will give a couple examples.

This specific example can be seen in a demo by Arnaud Porterie from our talk at Dockercon EU in 2015. You can have your application running in one container, then in a different container sharing a net namespace you can run wireshark and inspect the packets from the first container.

You could also do the same with sharing a pid namespace, except instead of running wireshark you can run strace and debug your application from an entirely different container.

Sharing X socket

I assume if you are on my blog you are familiar with my posts on running containers on your desktop.

Legos To really drive home a point I’m going to make an analogy describing each of these things in terms of legos.

VMs, Jails, and Zones are if you bought the legos already put together AND glued. So it’s basically the Death Star and you don’t have to do any work you get it pre-assembled out of the box. You can’t even take it apart.

Containers come with just the pieces so while the box says to build the Death Star, you are not tied to that. You can build two boats connected by a flipping ocean and no one is going to stop you.

This kind of flexibility allows for super awesome things but of course comes at a price.

Complexity == Bugs Now is the point where the person I would be having the conversation with starts yelling at me that containers are not secure. Hello, thank you, I am aware. Also if anyone gives a shit about actually fixing this, it’s me.

Again, containers were not a top level design, they are something we build from Linux primitives. Zones, Jails, and VMs are designed as top level isolation.

The cool things I expressed above allow for a level of flexibility and control that Zones, Jails, and VMs do not. By design.

This extra complexity leads to bugs that lead to container escapes. Don’t get me wrong you could also escape a VM, Jail or Zone, but the design is not as complicated as that of the primitives that make up containers. Less is more, and the less complexity you have the less likely you will have odd, edge case bugs.

The point I am trying to make is that Jails, Zones, VMs and containers were designed and built in different ways. Containers are not a Linux isolation primitive, they merely consume Linux primitives which allow for some interesting interactions. They are not perfect; Nothing is.

We can make them better by reducing some of the complexity and building hardening features around them which is a goal I have been trying and will continue trying to do.

You can get a sandbox level of isolation with containers, which I wrote in more detail about here. But this requires doing the work of building the Death Star from your pieces of Seccomp, AppArmor, and SELinux profiles.

I personally love Zones, Jails, and VMs and I think they all have a particular use case. The confusion with containers primarily lies in assuming they fulfill the same use case as the others; which they do not. Containers allow for a flexibility and control that is not possible with Jails, Zones, or VMs. And THAT IS A FEATURE.

</rant>

View Details

Over the past couple of years I have set out to create the ultimate Linux on the desktop experience for myself. Obviously everyone who runs Linux has their own opinions on things. What this post will outline is my ultimate Linux on the desktop experience. So just remember that before you get your panties in a knot on HackerNews because you live and die by Xmonad (I live and die by i3, fight me).

First, you should already know that I run everything on my laptop in containers. I outlined this in my posts about Docker Containers on the Desktop and Runc Containers on the Desktop.

Base OS I used to use Debian as my base OS but I recently decided to try and run CoreOS’ Container Linux on the desktop. Container Linux is made for servers, so obviously it doesn’t have graphics drivers. I added them and made a few other horrible tweaks that I’m sure would make some people at CoreOS cringe. I am not proud of these things but overall it worked!

Mostly the changes for graphics drivers were the same exact changes you would make installing Gentoo on your host: setting VIDEO_CARDS="intel i915" in /etc/portage/make.conf, emerge-ing sys-kernel/linux-firmware etc, etc.

Then I cut out the things I don’t need that only pertain to if you are using Container Linux on your server, cluster management tools, support request tools (lolz) etc. These were all pretty simple changes that I made to a new ebuild that I cloned from the coreos-base/coreos ebuild.

I need to clean up the mess I’ve made of my forks of the coreos build scripts, init, ebuilds, manifest, and base layout. But you can checkout the desktop branch at each of those.

Let me go over some of the benefits I get from using CoreOS’ Container Linux as my base OS.

1. I can build my own images, it’s Gentoo and I know emerge so I can customize the base anyway I want.
2. CoreOS’ Container Linux uses the same auto update system as ChromeOS, which is all based on Google’s Omaha. So all I need to do to have auto updates for my OS is continuously release the modded version of Container Linux to an Omaha server, which I will host. (Yes I know I am insane to go through this much effort for my own laptop but whatever.)
3. The filesystem is setup perfectly for running containers, there is a read-only /usr and a stateful read/write /. The data stored on / will never be manipulated by the update process. Plus since /usr is read-only it really forces you to run everything in containers.
4. My hardware has TPM capabilities so I get Trusted Computing through Container Linux.

X11 & Wayland Currently this setup is using X11 but that is not the goal in the future. I plan to move it over to Wayland after the port of i3, sway, is feature compatible with i3. It’s really close to done so I can try it out currently.

This would eliminate all the problems with X being the worst, something something keylogging blah blah blah. I’m not going to go into more detail now because this is not meant to be a rant.

Everything in Containers I already mentioned my two other blog posts on running desktop apps with Docker and Runc, but on this laptop I wanted something better.

You see the problem with both Docker and Runc as they are today is that they must be run as root. And I’m not talking about the process in the container. I’m talking about the container spawner itself.

I outlined the future of Sandbox Containers and there are patches to Runc to enable rootless containers. If you want to know more you should also watch Aleksa Sarai’s talk. On this laptop I am only using rootless containers.

So overall everything runs in containers, I can automatically update my operating system, and the containers are NOT running as root on my host. This is the dream and reality.

If you want to know all the stuff about what laptop I use you should checkout my uses this interview.

I gave a talk on this at CoreOS Fest 2017, check out the video and slides.

View Details

It all started innocently enough. I had “jfrazelle” as my GitHub handle for years, but my Twitter, IRC and other handles are all “jessfraz”. No one on GitHub was actually using “jessfraz” so I sat on it waiting to make my move.

I’m currently on vacation this week so of course I was looking to break all the things. One thing you must know about me is that at no point was I thinking I hate this. I actually love stuff like this, I live for pain. Why else would I run Linux on the desktop? But back to the story.

I polled the twitterverse…

Over/Under how many links you think I will break if I change my github username? (Yes, I know there are redirects, but still.)

— Jess Frazelle (@jessfraz) September 30, 2016

And then I made my move…

It is done. Let's watch the world burn together. https://t.co/YpLqpP1X38 pic.twitter.com/4MX1tTthHO

— Jess Frazelle (@jessfraz) September 30, 2016

Everything was fine for a few minutes. Another thing you must know about me is: I have a private Jenkins instance for continuous builds and testing. Yes, I am this much of a nerd, but it is essential for building all the Dockerfiles for my publicly readable private docker registry at r.j3ss.co. I will save all that for another blog post, but the jobs started triggering. Immediately I got a bunch of emails about failed builds because Jenkins could not clone the repos.

oh noe pic.twitter.com/ZRQnwWNR5L

— Jess Frazelle (@jessfraz) September 30, 2016

“This is fine” I thought to myself. It’s all configured with Jenkins DSLs and I can just do a sed on those files and it will work again.

I do this.

The “apply-dsl” job is still red, oh duh because it cannot clone the repo where the DSLs live to even fix the problem. So I change it manually.

This is fine.

The builds all start again. Except now all the Go builds are failing because importing “jfrazelle/…” is not working. Vendor your crap kids!!!

So I fix all these repos with the best vim command ever argdo. argdo will apply the script you run to all the open buffers, so just open the buffers of all the go files and run this:

``` argdo %s/jfrazelle/jessfraz/g | update

```

The | update makes sure it saves the buffer when it’s done editing.

After ~50 repos of this I am tired but it’s fine. It’s all fine. Things are working again.

Now I’m wondering who else I have broken… I search GitHub to see…

I'm going to need some more tires for this fire. pic.twitter.com/YGgMWmaETt

— Jess Frazelle (@jessfraz) September 30, 2016

I am for sure going to hell for this. What have I done?

I made/am making some pull requests to various repos. A few of those in the query above are actually forks of my repos that don’t show up in GitHub as forks because of the way the person forked it so they can be ignored.

Overall, I think I really f*cked this entire situation by having an account for “jfrazelle” and an account for “jessfraz” and swapping them. I think this is why the git clone/fetch/etc redirects that should happen when you change your username are broken. So let me just make this clear, none of this is GitHub’s fault. I pretty much did this super wrong. Also I have a deep fear of someone taking my old username and making fake repos to try and trick imports in Go so I figured I will squat on it forever to avoid this. Maybe someone from GitHub can alleviate my probably irrational fear.

Amazingly all the Travis CI builds transferred seamlessly. People have been all up in my mentions on Twitter saying they did this and all their autobuilds for Docker Hub broke. This honestly doesn’t affect me because I host my own registry that continuously builds AND I allow the general public to pull images from it.

In conclusion, I actually think everything is fine now. :)

View Details

Last week, I gave a talk at Github Universe and afterwards several people suggested I write a blog post on it. Here it is. This post will cover intricacies of “choosing your battle” and how personal passion for a project might conflict with corporate motives.

I have experienced open source from the side of the contributor, the side of the maintainer, and the side of the corporate-backed maintainer and contributor. The latter is what really comes into play here but a lot of the passion I talk about is obviously present in the former and of course important for empathy for those on the other side.

Passion Passion is a driving force behind involvement in open source software. People who believe in a project and use it are the ones who contribute and give back most heavily. Open source is this rare opportunity to work with people who want to achieve the same things as you.

If you have contributed to an open source project before, you know that feeling when your first pull request to a project is merged. It is magical. In that moment you have become a part of something so much bigger than yourself.

What happens to this fiery passion when it is fueled by a paycheck from a company? This is where things get complicated. When should you fight? When should you compromise?

The goal of this post is to show that you can stand up for what you believe in and keep your job. Getting paid to work on open source is a rare and wonderful opportunity, but you should not have to give up your passion in the process. Your passion should be why companies want to pay you.

Lessons Learned In my talk (which I will link to the video when it comes out), I told a brief history of how we evolved the Docker core team during my time there. We even had three different names: core, meta, engine (yet kept the same team members the entire time). I’m not going to go over all those stories so I suggest you check out the video, but these are the lessons we learned.

Hire from the community. Previous to joining Docker I had used Docker, given talks on Docker, and contributed to the project. The startup I worked at previously built its infrastructure around Docker. All members of the Docker core team were a part of the community before joining. This is important.

When you are a member of a community you feel surrounded by your peers being a part of it. And this includes all members outside the company. Your passion for the project will and should always come first. You may get paid by the company but you will protect the project at all costs, because at the end of the day, the project and the community were what you were a part of first.

You cannot hire everyone from the community. Then the trust for the project and any further growth to the community is effectively ruined.

Maintainership for a project must be earned. When an employee joins your company they should not automatically get push access to the project. I almost feel like I should repeat this because it is SO important to building trust with the community. Everyone must play by the same rules. EVERYONE.

The Docker project collects stats on just about everything using github.com/icecrime/vossibility-stack. Whether you are contributing code, contributing documentation, commenting on issues, or doing code reviews you are eligible to become a maintainer after regular activity.

This is key because this eliminates the “it’s all about who you know” scenario. Without hard data of contributions there is no way to be sure you are not overlooking some amazing gem in your project that should be rewarded for their hard work.

Allow saying NO. A very common conflict that will occur is one between other teams in the company and your “core” team. The company will have a feature they want to push, which perhaps lands as a patch bomb right before a release, has no tests, and has pockets of code relying on a service not even in production yet. They will just expect this to be merged.

Now of course, your open source team will fight it. They will stand up for the project. Maybe some members will eventually cave but others will still keep on fighting. It will cause stress and fear of being terminated. It will also cause turmoil between these teams internally which creates an awkward work environment.

There are a few things you can do to avoid this, the first one being: allow saying NO. Even from external maintainers, since as I said EVERYONE plays by the same rules, allow saying NO.

Create explicit guidelines for acceptable patches & release cutoffs. By creating explicit guidelines you now make sure that everyone plays by these same rules. People outside the company cannot send a patch bomb without tests right before a release and neither can those internally. Of course you can make whatever rules you want, as long as everyone plays by them.

When everyone plays by the same rules, your community will trust you the most.

LGTM lasts forever. When you LGTM a pull request it is there forever, publicly. You can of course change your mind and revert. But the second that feature gets into a release, it will take a very long time to deprecate it out if you feel like you made the wrong decision. Someone will wind up relying on it and removing it will become a bikeshed only leading to community disagreement.

LGTM is tied to the individual who said it. You cannot get a LGTM from a corporation; you get it from an individual. I have never seen a company with a GitHub account going around doing code reviews. People do code reviews.

If someone comes back to some feature down the road, they can see who approved it. It reflects on that person, not on their company. They will make sure they really mean it before they say it.

Collaboration and compromise is key. Do not isolate your “core” team from the rest of the company. Isolation will only create a non-inviting atmosphere to work in.

You are all on the same team; you can find a way to work together and compromise to benefit both the company AND the community.

Go and succeed! If you are thinking about open sourcing a project at your company, try to keep these things in mind! It’s never easy and there will always be some friction, but the benefits of creating a great open source project and community will pay off! Most of all LISTEN to the people at your company with the passion for the project.

View Details

I was inspired last night by Cate Huston’s post, The Day I Leave the Tech Industry. I decided to write my own, except I’m not as eloquent a writer as Cate so before I go any further please, please, please read her post and not mine.

Mine is going to be a bit different. Lately I’ve been thinking more and more about this. It seems imminent. I’m only 27 and let me repeat: it seems imminent.

I’m going to tell you all the fantasy that plays in my brain for when this happens.

The day I leave the tech industry will feel like a giant weight has finally been lifted. It will be freeing. There are a few scenarios I’ve played out for what I will do after.

• teach math in a third world country
• write a book
• play professional poker

I could do all three. One thing is for sure though, the day I leave the tech industry will be the day I contribute my last piece of code to open source software.

Today, I am not quite ready to give up this thing I have such a “hate/love” relationship with. Today, I want to get more women contributing so that maybe in the distant future we will feel welcome. Maybe we won’t have to fight so hard just to be heard; to have our opinions matter.

Today is not my last day in the tech industry. But it is comforting to me to plan out this very real future. I am not just “the container girl”. I am a human being with feelings, a limit, and a future outside of tech.

View Details

I really enjoyed Felipe Hoffa’s post on Analyzing GitHub issues and comments with BigQuery .

Which got me wondering about my favorite subject ever, The Art of Closing. I wonder what the stats are for the top 15 projects on GitHub in terms of pull requests opened vs. pull requests closed. This post will use the GitHub Archive dataset.

Top 15 repositories with the most pull requests First let’s find the top 15 repos with the most pull requests from 2015. Let’s make sure to check the payload action is ”opened”.

``` SELECT repo.name, COUNT(*) c FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"opened"') GROUP BY repo.name ORDER BY c DESC LIMIT 15

```

| repo_name | c | | --- | --- | | openmicroscopy/snoopys-sandbox | 11656 | | brianchandotcom/liferay-portal | 10803 | | Homebrew/homebrew | 9519 | | caskroom/homebrew-cask | 6833 | | apache/spark | 6667 | | saltstack/salt | 6636 | | mozilla-b2g/gaia | 6609 | | jlord/patchwork | 6155 | | GoogleCloudPlatform/kubernetes | 5937 | | jsdelivr/jsdelivr | 5747 | | rust-lang/rust | 5559 | | cms-sw/cmssw | 5507 | | code-dot-org/code-dot-org | 5267 | | docker/docker | 5083 | | NixOS/nixpkgs | 4873 |

Okay that’s a lot of pull requests. Let’s find the projects will the most unique number of pull request authors.

``` SELECT repo.name, COUNT(*) c, COUNT(DISTINCT actor.id) authors, FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"opened"') GROUP BY repo.name ORDER BY authors DESC LIMIT 15

```

| repo_name | c | authors | | --- | --- | --- | | jlord/patchwork | 6155 | 5396 | | octocat/Spoon-Knife | 3966 | 3741 | | deadlyvipers/dojo_rules | 4847 | 3076 | | Homebrew/homebrew | 9519 | 2186 | | udacity/create-your-own-adventure | 2709 | 2167 | | caskroom/homebrew-cask | 6833 | 1517 | | borisyankov/DefinitelyTyped | 2694 | 1127 | | rails/rails | 3100 | 1012 | | LarryMad/recipes | 1086 | 989 | | laravel/framework | 2736 | 891 | | docker/docker | 5083 | 882 | | rdpeng/ProgrammingAssignment2 | 922 | 866 | | apache/spark | 6667 | 851 | | JetBrains/swot | 951 | 836 | | rust-lang/rust | 5559 | 835 |

Now let’s see what the merge vs. close numbers look like for those projects.

``` SELECT repo.name, COUNT(*) c, COUNT(DISTINCT actor.id) authors, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END) AS merged, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN 1 ELSE 0 END) AS closed, FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"closed"') GROUP BY repo.name ORDER BY authors DESC LIMIT 15

```

| repo_name | c | authors | merged | closed | | --- | --- | --- | --- | --- | | deadlyvipers/dojo_rules | 1636 | 1022 | 0 | 1636 | | octocat/Spoon-Knife | 1103 | 944 | 0 | 1103 | | jlord/patchwork | 6595 | 705 | 4905 | 1690 | | LarryMad/recipes | 588 | 532 | 0 | 588 | | apache/spark | 6653 | 468 | 0 | 6653 | | Homebrew/homebrew | 9548 | 451 | 5 | 9543 | | udacity/create-your-own-adventure | 2765 | 301 | 1946 | 819 | | rdpeng/ProgrammingAssignment2 | 341 | 284 | 0 | 341 | | docker/docker | 5250 | 254 | 3979 | 1271 | | NixOS/nixpkgs | 4707 | 249 | 3438 | 1269 | | odoo/odoo | 3412 | 233 | 712 | 2700 | | borisyankov/DefinitelyTyped | 2529 | 221 | 2173 | 356 | | mozilla-b2g/gaia | 7197 | 215 | 5251 | 1946 | | rails/rails | 3254 | 212 | 2090 | 1164 | | caskroom/homebrew-cask | 6928 | 210 | 3044 | 3884 |

Oh that is super weird. After looking into a few of the repos with 0 merged, it seems they aren’t really using GitHub for merges.

Calculating the merge ratio So let’s exclude those and try again, this time we can even calculate the merge ratio.

``` SELECT repo.name, COUNT() c, COUNT(DISTINCT actor.id) authors, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END) AS merged, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN 1 ELSE 0 END) AS closed, ROUND(100SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END)/COUNT(*),2) AS merge_ratio FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"closed"') GROUP BY repo.name HAVING merged > 10 ORDER BY authors DESC LIMIT 15

```

| repo_name | c | authors | merged | closed | merge_ratio | | --- | --- | --- | --- | --- | --- | | jlord/patchwork | 6595 | 705 | 4905 | 1690 | 74.37 | | udacity/create-your-own-adventure | 2765 | 301 | 1946 | 819 | 70.38 | | docker/docker | 5250 | 254 | 3979 | 1271 | 75.79 | | NixOS/nixpkgs | 4707 | 249 | 3438 | 1269 | 73.04 | | odoo/odoo | 3412 | 233 | 712 | 2700 | 20.87 | | borisyankov/DefinitelyTyped | 2529 | 221 | 2173 | 356 | 85.92 | | mozilla-b2g/gaia | 7197 | 215 | 5251 | 1946 | 72.96 | | rails/rails | 3254 | 212 | 2090 | 1164 | 64.23 | | caskroom/homebrew-cask | 6928 | 210 | 3044 | 3884 | 43.94 | | cms-sw/cmssw | 5475 | 205 | 4312 | 1163 | 78.76 | | symfony/symfony | 2587 | 185 | 1387 | 1200 | 53.61 | | facebook/react-native | 1563 | 185 | 494 | 1069 | 31.61 | | robbyrussell/oh-my-zsh | 731 | 185 | 307 | 424 | 42.0 | | githubteacher/github-for-developers-sept-2015 | 404 | 181 | 301 | 103 | 74.5 | | nightscout/cgm-remote-monitor | 1096 | 178 | 419 | 677 | 38.23 |

Using the diff data Sweet now let’s see on average what the size of the diffs are for these projects’ pull requests.

``` SELECT repo.name, COUNT() c, COUNT(DISTINCT actor.id) authors, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END) AS merged, SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN 1 ELSE 0 END) AS closed, ROUND(100SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END)/COUNT(*),2) AS merge_ratio, AVG(JSON_EXTRACT(payload, '$.pull_request.additions')) AS avg_additions, AVG(JSON_EXTRACT(payload, '$.pull_request.deletions')) AS avg_deletions, AVG(JSON_EXTRACT(payload, '$.pull_request.changed_files')) AS avg_changed_files, FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"closed"') GROUP BY repo.name HAVING merged > 10 ORDER BY authors DESC LIMIT 15

```

| repo_name | c | authors | merged | closed | merge_ratio | avg_additions | avg_deletions | avg_changed_files | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | jlord/patchwork | 6595 | 705 | 4905 | 1690 | 74.37 | 47.45595147839272 | 172.14268385140258 | 175.20257771038666 | | udacity/create-your-own-adventure | 2765 | 301 | 1946 | 819 | 70.38 | 30.39746835443038 | 13.116455696202532 | 6.742133815551537 | | docker/docker | 5250 | 254 | 3979 | 1271 | 75.79 | 214.36685714285716 | 115.88342857142857 | 8.139619047619048 | | NixOS/nixpkgs | 4707 | 249 | 3438 | 1269 | 73.04 | 339.9751434034417 | 40.72678988740174 | 5.380072232844699 | | odoo/odoo | 3412 | 233 | 712 | 2700 | 20.87 | 1626.0741500586166 | 1907.4182297772568 | 128.01992966002345 | | borisyankov/DefinitelyTyped | 2529 | 221 | 2173 | 356 | 85.92 | 887.0581257413997 | 864.4827995255041 | 2.8730723606168445 | | mozilla-b2g/gaia | 7197 | 215 | 5251 | 1946 | 72.96 | 415.85396693066554 | 138.59233013755733 | 10.55578713352786 | | rails/rails | 3254 | 212 | 2090 | 1164 | 64.23 | 54.88414259373079 | 29.18561770129072 | 6.880762138905962 | | caskroom/homebrew-cask | 6928 | 210 | 3044 | 3884 | 43.94 | 8.448469976905312 | 4.0329099307159355 | 3.315675519630485 | | cms-sw/cmssw | 5475 | 205 | 4312 | 1163 | 78.76 | 2160.7702283105023 | 713.1713242009132 | 37.51086757990868 | | facebook/react-native | 1563 | 185 | 494 | 1069 | 31.61 | 189.86756238003838 | 86.54638515674984 | 10.595649392194497 | | robbyrussell/oh-my-zsh | 731 | 185 | 307 | 424 | 42.0 | 54.0328317373461 | 11.285909712722297 | 1.987688098495212 | | symfony/symfony | 2587 | 185 | 1387 | 1200 | 53.61 | 142.36722071897952 | 168.96366447622728 | 28.32006184770004 | | githubteacher/github-for-developers-sept-2015 | 404 | 181 | 301 | 103 | 74.5 | 18.217821782178216 | 0.7574257425742574 | 2.517326732673267 | | nightscout/cgm-remote-monitor | 1096 | 178 | 419 | 677 | 38.23 | 519.7043795620438 | 246.9434306569343 | 8.777372262773723 |

Well that’s not all that interesting…

Can we prove you should always keep your pull requests small? We know that it is always better to make a small pull request to have it merged. Let’s see if we can prove that with data!

``` SELECT repo.name, COUNT() c, COUNT(DISTINCT actor.id) authors, ROUND(100SUM(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN 1 ELSE 0 END)/COUNT(*),2) AS merge_ratio, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN JSON_EXTRACT(payload, '$.pull_request.additions') END) AS merged_avg_additions, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN JSON_EXTRACT(payload, '$.pull_request.deletions') END) AS merged_avg_deletions, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('true') THEN JSON_EXTRACT(payload, '$.pull_request.changed_files') END) AS merged_avg_changed_files, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN JSON_EXTRACT(payload, '$.pull_request.additions') END) AS closed_avg_additions, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN JSON_EXTRACT(payload, '$.pull_request.deletions') END) AS closed_avg_deletions, AVG(CASE WHEN JSON_EXTRACT(payload, '$.pull_request.merged') IN ('false') THEN JSON_EXTRACT(payload, '$.pull_request.changed_files') END) AS closed_avg_changed_files, FROM [githubarchive:year.2015] WHERE type IN ( 'PullRequestEvent') AND JSON_EXTRACT(payload, '$.action') IN ('"closed"') GROUP BY repo.name HAVING merge_ratio > 5 ORDER BY authors DESC LIMIT 15

```

| repo_name | c | authors | merge_ratio | merged_avg_additions | merged_avg_deletions | merged_avg_changed_files | closed_avg_additions | closed_avg_deletions | closed_avg_changed_files | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | jlord/patchwork | 6595 | 705 | 74.37 | 9.3565749235474 | 0.033231396534148826 | 1.0014271151885832 | 158.03431952662723 | 671.6674556213018 | 680.798224852071 | | udacity/create-your-own-adventure | 2765 | 301 | 70.38 | 7.863309352517986 | 0.6747173689619733 | 1.8144912641315518 | 83.94017094017094 | 42.67887667887668 | 18.45054945054945 | | docker/docker | 5250 | 254 | 75.79 | 176.0874591605931 | 90.80949987434029 | 5.965317919075145 | 334.2045633359559 | 194.38001573564122 | 14.946498819826909 | | NixOS/nixpkgs | 4707 | 249 | 73.04 | 137.50581733566025 | 31.841768470040723 | 2.91564863292612 | 888.5090622537431 | 64.79826635145784 | 12.056737588652481 | | odoo/odoo | 3412 | 233 | 20.87 | 200.9129213483146 | 195.0870786516854 | 7.095505617977528 | 2001.8944444444444 | 2358.966296296296 | 159.90814814814814 | | borisyankov/DefinitelyTyped | 2529 | 221 | 85.92 | 390.8085595950299 | 482.45467096180397 | 2.1339162448228257 | 3916.13202247191 | 3196.3567415730336 | 7.384831460674158 | | mozilla-b2g/gaia | 7197 | 215 | 72.96 | 398.6246429251571 | 86.15311369262997 | 6.51628261283565 | 462.3448098663926 | 280.09198355601234 | 21.45580678314491 | | rails/rails | 3254 | 212 | 64.23 | 23.657416267942583 | 11.615789473684211 | 2.6382775119617223 | 110.95274914089347 | 60.732817869415804 | 14.49828178694158 | | caskroom/homebrew-cask | 6928 | 210 | 43.94 | 8.201708278580815 | 5.042706964520368 | 3.9244415243101183 | 8.641864057672503 | 3.241503604531411 | 2.8385684860968072 | | cms-sw/cmssw | 5475 | 205 | 78.76 | 994.2810760667903 | 619.9148886827459 | 8.133812615955472 | 6485.7067927773005 | 1058.9337919174548 | 146.43078245915734 | | symfony/symfony | 2587 | 185 | 53.61 | 63.3914924297044 | 72.16582552271089 | 8.235760634462869 | 233.65 | 280.84583333333336 | 51.534166666666664 | | facebook/react-native | 1563 | 185 | 31.61 | 204.29757085020242 | 88.43522267206478 | 8.024291497975709 | 183.19925163704397 | 85.67352666043031 | 11.783910196445277 | | robbyrussell/oh-my-zsh | 731 | 185 | 42.0 | 49.74267100977199 | 10.824104234527688 | 1.6612377850162867 | 57.139150943396224 | 11.620283018867925 | 2.224056603773585 | | githubteacher/github-for-developers-sept-2015 | 404 | 181 | 74.5 | 4.700996677740863 | 0.4186046511627907 | 1.1727574750830565 | 57.71844660194175 | 1.7475728155339805 | 6.446601941747573 | | nightscout/cgm-remote-monitor | 1096 | 178 | 38.23 | 173.24582338902147 | 58.885441527446304 | 4.985680190930788 | 734.1299852289512 | 363.3338257016248 | 11.124076809453472 |

IT IS PROVEN!!!

View Details

This blog post is going to be a bit different. After watching Stranger Things, my friend and I started discussing scary movies from our childhood. I couldn’t help but remember a very specific strange thing that happened to me growing up. I thought, hey, this would be a kinda weird blog post. So here it is. The events following are factual.

It was a hot, dry summer in July of 1995 in Phoenix, Arizona. We were getting our house repainted. For those of you unfamiliar with the dry summers in Arizona it gets to be around 120°F, which is around 49°C. The painters had left their varnish rags on top of the trash cans outside our house. Which directly lined up with the side of the house.

A diagram is below and will come in handy later in the story.

Varnish rags are flammable and in combination with the Arizona heat this caused spontaneous combustion. I kid you not.

The fire started at the trash cans and reached all the way to my parent’s bedroom. Which at the time had this horrendous green carpet.

Fortunately no one was home at the time. My sister and I were spending the night at our friends house. My mom was busy elsewhere. My dad had the interesting circumstances of driving home while all the fire trucks kept passing him heading the same direction. He was the first to find out.

The next morning, he came to our sleepover. The very sight of him at our friends breakfast table was unusual to say the least. My sister and I knew something was wrong. He explained what happened and that we needed to go shopping for new clothes. It was surreal. And the only thing I could think about was how I left my favorite teddy bears with an “I owe you” note that they could come to the next sleepover. Which of course would never happen. I had abandoned them in my room which was so unfortunately placed right next to the exterior wall where the trash cans were.

This story is interesting for, of course, the moral lesson I learned at a young age that material objects don’t matter. It’s the relationships with people that do. But also there was something rather creepy that happened with regard to the fire.

The map is important here.

Growing up my parents always made us clean out our closets over summer. My sister and I had just done that. In this massive purge of items that had fallen into closet abyss throughout the year we also threw away our Sunday school books.

Now I’m not a religious person, on my Facebook page my religion is literally “hugs” so take this as you want. Those books were in the trash cans that spontaneously combusted. The pages were left burned all over the house. But one page made its way all the way to the front door of the house. It was the Ten Commandments. My parents framed the perfectly fire scorched page and it’s in their house today.

View Details

Hello and welcome to what will become the most sarcastic post on my blog. This is going to be a series of “buzzfeed” style programming articles and after this post I very happily pass the baton to Filippo Valsorda to continue. And I urge you to write your own as well.

@jessfraz "We asked Jess for her top 10 ldflags; you won't believe what happened next"

— adg (@enneff) July 17, 2016

So here they are:

1. -static

I would be an embarassment to myself if I didn’t start with the flag that tells the linker to not link against shared libraries. This is the best flag. STATIC BINARIES FTW. 2. --export-dynamic

This flag tells the linker to add all the symbols to the dynamic symbol table. This is especially important if you want to do “The Macgyver of Dlopening” and dlopen yourself. 3. --whole-archive

This is another flag that comes in handy when you want to dlopen yourself. See most linkers will only take into account the things it knows it needs. But with this flag, you tell it “YOLO, I want it all” so that later you can dlopen yourself with that symbol that was never actually used until runtime. FUN! 4. --no-whole-archive

This flag un-sets the --whole-archive flag which is nice for when you only want the whole archive of one library but not all the others you are linking to. 5. --print-map

This flag is just dope. It prints a link map to stdout. This gives you information about object files, common symbols, and the values assigned to symbols. 6. --strip-all

This flag strips all the symbol information from the artifact produced. If say you are a few KB/MB off from your binary fitting on a floppy disk, this flag is your friend. 7. --strip-debug

This flag is very similar to --strip-all except it only strips the debug symbol information. This all really depends on how much you need to shave off to fit that binary on a floppy disk. 8. --trace

This flag is great for debugging. It prints the names of the input files as ld processes them. 9. -nostdlib

This flag forces the linker to only search the libraries you specify with --library-path or -L. This is nice when someone completely messes with your library path and the world is burning and you just want to link to those things you put in some random directory somewhere. 10. --unresolved-symbols=ignore-all

This flag is helpful when telling the linker you DGAF about unresolved symbols and to stop yelling at you.

View Details

Being an open source software maintainer is hard. The following post is geared towards maintainers and not contributors. If you are a new contributor to open source I would stop reading now because I don’t want you to get the wrong idea or discourage you. Tons of patch requests get merged per day, but this is going to focus on the ones that don’t.

I’ve talked to maintainers from several different open source projects, mesos, kubernetes, chromium, and they all agree one of the hardest parts of being a maintainer is saying “No” to patches you don’t want.

To quote some very smart people I’ve worked with in the past:

One of the numerous examples of information asymmetry in open source: contributors put effort in a pet PR, but maintainers manage cattle. 🕒

— Arnaud Porterie (@icecrime) May 20, 2016

Rule #1 of open-source: no is temporary, yes is forever.

— Solomon Hykes (@solomonstre) March 30, 2016

To make this rather unpleasant experience of closing someone’s patch request easier I have a few ways of going about it. Now of course I am no expert in this area, but on the Docker project we have stats for just about everything. I might have used this data to make a “Ultimate Dream Killers” chart with the maintainers who closed (without merging) the most pull requests, AND I might have been #1 on this chart for some time.

None of the suggestions below are going to save you from that person hate mailing you since you didn’t merge their patch. But hey anything helps.

1. The ego stroke and close.

People love hearing how awesome they are. They also love hearing how awesome their code is. In this option you use this to your advantage. Here’s an example:

“Thanks so much for spending time on this amazing patch. We really appreciate it. However I do not think this is something we want to add right now, because of yadda yadda but in the future this can change. Thanks so much!”

AAAANNNDD close. 2. Close early.

No one wants to have to do 300 rebases before learning the design of their patch isn’t approved. If you know there is no way you will ever accept their patch, close it right then. Making someone wait and/or do more work while waiting will just make the situation worse when you do close it. 3. The “I kinda like this but it’s just not right”.

If someone creates a new feature that you might like if it was done differently but the current implementation has no way of being merged (maybe for design flaws etc.) I believe it’s best to close with opportunity for the person open another patch with the desired design. Here’s an example:

Hi X, We really appreciate you taking the time to make this patch. However the design was not discussed prior to writing it. We do see potential in what you are trying to build, but we think it would be more effective as blah, blah, and blah. We are going to close this but would love to see you open a patch that takes the above direction. Thanks, this could really be an awesome feature!

See how the ego stroke comes in handy here too :). AAAAAAND close. 4. The carry.

Carrying a patch is when a maintainer will take a user’s patch and add edits on top of it so it is mergable. On the docker project we do this every so often and for various reasons:

* Contributor disappeared but the patch is viable just needs some edits.
* Patch is like #3 above but it would be easier if we just did the
implementation itself.It’s important to note if you are going to carry a patch, DO NOT close the

original patch request until you have opened your carry patch. You obviously need to let the contributor know before hand you will be carrying it so they don’t waste their time. Also be sure to keep their original commit’s and add yours on top so the right people get credit :)

Here’s an example:

Hi X, we really like your patch, but since there hasn’t been a response in Y days we are going to carry this patch and make the edits ourselves. We will link to the new pull request here when it’s ready.

Maintainer works on patch… opens new patch… then you can close the original patch request. See if you close it before opening the new one, the contributor will assume you are lying and never going to do it.

These are just a few of the techniques we’ve used in the past. I hope if you are a maintainer of a project they are helpful for you, but I would love to know your tips as well.

Happy Maintaining and always be closing!

View Details

Containers are all the rage right now.

At the very core of containers are the same Linux primitives that are also used to create application sandboxes. The most common sandbox you may be familiar with is the Chrome sandbox. You can read in detail about the Chrome sandbox here: chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md. The relevant aspect for this article is the fact it uses user namespaces and seccomp. Other deprecated features include AppArmor and SELinux. Sound familiar? That’s because containers, as you’ve come to know them today, share the same features.

Why are containers not currently being considered a “sandbox”? One of the key differences between how you run Chrome and how you run a container are the privileges used. Chrome runs as your own unprivileged user. Most containers (be it docker, runc, or rkt) run as root.

Yes, we all know that containers run unprivileged processes; but creating and running the containers themselves requires root privileges at some point.

How can we run containers as an unprivileged user? Easy! With user namespaces, you might say. But it’s not exactly that simple. One of the main differences between the Chrome sandbox and containers is cgroups. Cgroups control what a process can use. Whereas namespaces control what a process can see. Containers have cgroup resource management built in. Creating cgroups from an unprivileged user is a bit difficult, especially device control groups.

If we ignore, for the time being, this huge tire fire that is creating cgroups as an unprivileged user, then unprivileged containers are easy. User namespaces allow us to create all the namespaces without any further privileges. The one key caveat being that the {uid,gid}_map must have the current host user mapped to the container uid that the process will be run as. The size of the {uid,gid}_map can also only be 1. For example if you are running as uid 1000 to spawn the container, your {uid,gid}_map for the process would be 0 1000 1 for uid 0 in the container. The 1 there refers to the size.

How is this different than the user namespace support currently in Docker? This is quite different, but for very good reason. In Docker, by default, when the remapped user is created, the /etc/subuid and /etc/subgid files are populated with a contiguous 65536 length range of subordinate user and group IDs, starting at an offset based on prior entries in those files. Docker’s implementation has a larger range of users that can exist in the container as well as having a more “anonymous” mapped host user. If you want to read more about the user namespace implementation in Docker I would checkout @estesp’s blog or the the docker docs.

POC or GTFO As a proof of concept of unprivileged containers without cgroups I made binctr. Which spawned a mailing list thread for implementing this in runc/libcontainer. Aleksa Sarai has started on a few patches and this might actually be a reality pretty soon!

Update: it took almost a year, but this was added to runc in Mar 2017.

Where does this put us in the “sandbox” landscape? With this implementation we get:

• namespaces
• apparmor
• selinux
• seccomp
• capabilities limiting

all created by an unprivileged user!

Sandboxes should be very application-specific, using custom AppArmor profiles, Seccomp profiles and the like. A generic container will never be equivalent to a sandbox because it’s too universal to really lock down the application.

Containers are not going to be the answer to preventing your application from being compromised, but they can limit the damage from a compromise. The world an attacker might see from inside a very strict container with custom AppArmor/Seccomp profiles greatly differs than that without the use of containers. With namespaces we limit the application from seeing various things such as network, mounts, processes, etc. And with cgroups we can further limit what the attacker can use, be it a large amount of memory, cpu, or even a fork bomb.

But what about cgroups? We can set up cgroups for memory, blkio, cpu, and pids with an unprivileged user as long as the cgroup subsystem has been chowned to the correct user. Devices are a different story though. Considering the fact you cannot mknod in a user namespace it is not the worst thing in the world.

Let’s not completely rule out the devices cgroup. In the future this might be entirely possible. In kernels 4.6+, there is a new cgroup namespace. For now all this does is mask the cgroups path inside the container so it is not entirely useful for unprivileged containers at all. But in the future maybe it could be (if we ask nice enough?).

What is the awesome sauce we all gain from this? Well judging by the original GitHub issue about unprivileged runc containers, the largest group of commenters is from the scientific community who are restricted to not run certain programs as root.

But there is so much more that this can be used for. One of my most anticipated use cases is the work being done by Alex Larsson on xdg-app to run applications in sandboxes. Definitely checkout bubblewrap if you are interested in this.

Also subgraph, the container based OS which specializes in security and privacy, have this same idea in mind.

I am a huge fan of running desktop applications in containers as well as solving multi-tenancy for running containers. I definitely hope to help evolve containers into real sandboxes in the future.

View Details

Sup, let me give you fair warning here. Everything contained in this post is my opinion so don’t go getting your panties all in a knot on Hacker News because you don’t agree with me. I could honestly care less, because that’s the thing about my opinion, it’s mine.

I am going to give you my honest and dare I say it “blunt” opinion about each of the Docker graphdrivers so you can decide for yourself which one is the best one for you. None are perfect each has it’s flaws and I will be laying those out. Let’s begin.

Overlay Overlayfs was added in the 3.18 kernel. This is important to note because if you are running overlay on an older kernel than 3.18 you are either:

1. Not running the same overlay.
2. Running a kernel with overlayfs backported onto it, which is what we call a “frankenkernel”. Frankenkernels are not to be trusted. This is not to say it won’t work, hey it might work great, but it’s not to be trusted.

Overlay is great but you need a recent kernel. There are also some super obscure kernel bugs with regard to sockets or certain python packages docker/docker#12080. But I will say personally I use overlay, I have not hit these bugs recently and I have all my 100+ dockerfiles running as continuous builds on my server with overlay and they all work.

Aufs Aufs is another great one. But it is not in the kernel by default which blows. On Ubuntu/Debian distros this is as easy as installing the kernel extras package but other distros it might not be as simple.

Btrfs Btrfs is great too but you need to partition the disk you will use for /var/lib/docker as btrfs first. This is kinda a hurdle to jump that I don’t think a lot of people are willing to do.

Zfs Zfs is another good one, of course, like btrfs it takes some setup and installing the zfs.ko on your system. But this driver might become a whole lot more popular if Ubuntu 16.04 ships with zfs support.

Devicemapper Honestly it makes me super disappointed to say this, but buyer beware. Hey on the plus side…. it’s in the kernel. You must must must have all the devicemapper options set up perfectly or you will find yourself only being able to launch ~2 containers.

Let me tell you a story.

My mom once asked her friend for her famous chicken enchilada recipe so she could make it herself. The friend gave the recipe but left out one key ingredient so that my mom’s never tasted just right. There was always something off about it.

This is how I think of devicemapper.

It works on RedHat.

Vfs I sure hope to hell you are just testing something or clinically insane.

That’s about all. Thanks for reading my opinion if you even made it this far.

View Details

This is so cool I can hardly stand it.

In Docker 1.10, the awesome libnetwork team added the ability to specify a specific IP for a container. If you want to see the pull request it’s here: docker/docker#19001.

I have a IP Block on OVH for my server with 16 extra public IPs. I totally use these for good and not for evil.

But to use these previously with Docker containers meant hackery with the awesome pipework. Or even worse some homegrown, Jess bash scripts.

But now MY LIFE JUST GOT SO MUCH EASIER. Let me show you how:

```

create a new bridge network with your subnet and gateway for your ip block

$ docker network create --subnet 203.0.113.0/24 --gateway 203.0.113.254 iptastic

run a nginx container with a specific ip in that block

$ docker run --rm -it --net iptastic --ip 203.0.113.2 nginx

curl the ip from any other place (assuming this is a public ip block duh)

$ curl 203.0.113.2

BOOM golden

```

It’s so amazing I can rewrite tupperwarewithspears to use this :D

View Details

Almost exactly a year ago, I wrote a post about running Docker Containers on the Desktop. Well it is a new year, and I have ended up converting all my docker containers to runc configs, so it’s the perfect time for a new blog post.

For those of you unfamiliar with the Open Container Initiative you should check out opencontainers.org.

Why the switch? you ask… well let me explain.

Our fellow Docker maintainer and pal Phil Estes made an awesome patch to add user namespaces to Docker.

Now me, being the completely insane containerizer that I am, desperately wanted to run all my crazy sound/video device mounting containers in user namespaces.

Well the way this could work is by having a custom gid_map for the audio and video groups to map to the host groups so we can have permission to access these devices in the container. In layman’s terms, I basically wanted to poke a teeny tiny map in the user namespace to be able to have permission to use my sound and video devices.

Obviously this was not the design of the feature, but since runc exposes the uidMappings and gidMappings, I knew I could have the power to do as I please. This is the awesome thing about runc. You, the user, have all the control.

So for chrome, this is what you get for mappings: github.com/jessfraz/containers:chrome/config.json#L223. If you look closely, or know what you are looking at, you can see group 29 and 44 are mapped to the same group ids as the host.

Then you can do cool things like listen to Taylor Swift in a container with a user namespace.

Pretty cool right. So I went all OCD on this, like most things I encounter, and I converted all my containers. Obviously I found a way to generate them.

Riddler Introducing github.com/jessfraz/riddler! riddler will take a running/stopped docker container and convert the inspect information into the oci spec (which can be run by runc, or any other oci compatible tool).

It has some opinionated features in that it will always try to set up a gid_map that works with your devices. You can also pass custom hooks to automatically add to the runc config as prestart, poststart, or poststop hooks. Which leads me to the next tool I built.

Netns Say hello to github.com/jessfraz/netns! So you want your runc containers to have networking, eh? How about something super simple like a bridge? netns does just that. It sets up a bridge network for all your runc containers when added via the prestart hook.

It’s actually super simple code as well thanks to the awesome netlink pkg from vishvananda.

netns even saves the ip for the container in a .ip file in the directory with your config. Then other hooks can use this to do other things. For instance I use the hostess cli to then add an entry to my host’s /etc/hosts file, so I don’t have to remember the ip for the container when I want to reach it.

You can find all my hook scripts in github.com/jessfraz/containers:hack/scripts.

Magneto The last tool I made was a copy of docker stats for runc. But what I really wanted was the new pids cgroup stats, that Aleksa Sarai added to the kernel and runc (and soon docker ;).

runc has a command runc events which outputs json stats in an interval. All you have to do is pipe that to magneto to get the awesome ux.

The following is for my chrome container:

``` $ sudo runc events | magneto

```

All the configs If you are interested in all the configs for my containers, checkout github.com/jessfraz/containers.

I even included a systemd service file that can easily run any container (without a tty) in this directory via:

``` $ sudo systemctl start runc@foldername

for example:

$ sudo systemctl start runc@chrome

```

NOTE: Keep in mind since these are generated a lot of the filepaths are hardcoded for things on my host. So if you try to run these and aren’t me I don’t want to hear any whining.

Happy namespacing!

View Details

In case you missed it, we recently merged a default seccomp profile for Docker containers. I urge you to try out the default seccomp profile, mostly so we can rest easy knowing the defaults are sane and your containers work as before. You can download the master version of Docker Engine from master.dockerproject.org or experimental.docker.com.

We even have a doc describing the syscalls we purposely block and security vulnerabilities the profile blocked.

But that’s not what this blog post is about. This post is about how you can create your own custom seccomp profiles for your containers. And how to debug when your profile is missing a syscall.

So this is not the most sane thing in the world, I even tried in the process to create a bash script that takes the output from strace, collects the syscalls, and generates a profile. But like all tools of this sort (eg. aa-genprof) it missed some, well to be exact it missed 6. Which is no small feat to debug, so this post is in the format: learn by example. I am going to take you step by step through what I did.

1. Wake up go to starbucks… just kidding… not that specific.

I wanted to make a custom profile for my chrome container. I decided to get the syscalls it used by changing the entrypoint for my chrome/Dockerfile to ENTRYPOINT [ "strace", "-ff", "google-chrome" ]. So the only things that changed was wrapping the command in strace and of course installing strace in the container. The -ff option makes sure strace follows forks. Which is essential for chrome because they fork a bunch of processes (fun fact: each tab is a process with it’s own PID namespace).

Cool beans, moving on.

So I used chrome the entire day like this to create the most verbose strace output so I wouldn’t miss any syscalls.

At the end of the day I saved this output into a file by running docker logs chrome > $HOME/chrome-strace.log 2>&1.

Then I used the world’s most janky bash script to generate a profile:

```

!/bin/bash

set -e set -o pipefail

main(){ local file=$1 local name=$(basename "$0")

if [[ -z "$file" ]]; then
    cat >&2 <<-EOF
    ${name} [strace-output-filename]

    You must pass a filename that has the strace output.
    EOF
fi

# get just the syscalls
local IFS=$'\n'
raw=( $(perl -lne 'print $1 if /([a-zA-Z_]+\()/' "$file" | sort -u) )
unset IFS


syscalls=( )

tmpfile=$(mktemp /tmp/seccomp-strace.XXXXXX)

curl -sSL -o "$tmpfile" https://raw.githubusercontent.com/torvalds/linux/master/arch/x86/entry/syscalls/syscall_64.tbl

for syscall in "${raw[@]}"; do
    # clean the trailing (
    syscall=${syscall%(}

    if grep -R -q -w $syscall "$tmpfile"; then
        syscalls+=( $syscall )
    fi
done

# start the seccomp profile
cat <<-EOF > "$tmpfile"
{
    "defaultAction": "SCMP_ACT_ERRNO",
    "syscalls": [
    EOF

    for syscall in "${syscalls[@]}"; do
        cat <<-EOF
        {
            "name": "${syscall}",
            "action": "SCMP_ACT_ALLOW",
            "args": null
        },
        EOF
    done >> "$tmpfile"

    # remove trailing comma
    sed -i '$s/,$//' "$tmpfile"

    cat <<-EOF >> "$tmpfile"
    ]
}
EOF

cat "$tmpfile"
rm "$tmpfile"

}

main $@

```

You use this script like so:

``` $ ./shitty-seccomp-profile-generator.sh chrome-strace.log

```

Now you have a whitelist generated from your strace output. But it’s super bad and when you try to run your container with it you get a vague error and Operation not permitted.

Just for this example the error was: [1:1:0104/214046:ERROR:nacl_fork_delegate_linux.cc(314)] Bad NaCl helper startup ack (0 bytes).

So now we have to use our brains. WHAT!? NOOOOO!

So I opened the generated profile and took a look at what it was allowing.

Now I know a little bit about how chrome uses namespaces/seccomp to create a sandbox, so my first thought was let’s make sure we allow unshare, clone, seccomp and setns. Sure enough, unshare and setns were missing… thanks strace you really sucked that one up, even I know chrome calls those.

After further thought I realized it was also missing setgid and exit/exit_group.

This all took a super long time of guessing and checking but I ended up with this profile.

Obviously noone else is going to do this, debug for hours the syscalls that are missing. This is why the default profile is so important, we wanted to create sane defaults that would protect people but also not cause all this pain.

So please, please, please try it out and open an issue if you find your container that used to run perfectly is now giving Operation not permitted.

If you are curious about syscalls or are trying to track down what you are missing, this is a great syscall table: filippo.io/linux-syscall-table.

Also, things are going to get better. We are working on sane security profiles for containers that don’t make you want to pull your hair out. You can read up on the proposal at docker/docker#17142.

View Details

Okay so this is part 2.5 in my series of posts combining my two favorite things, Docker & Tor. If you are just starting here, to catch you up, the first post was “How to Route all Traffic through a Tor Docker container”. The second was on “Running a Tor relay with Docker”. I thought it only made sense to show how to set up a Tor socks5 proxy in a container, for routing some traffic through Tor; in contrast to the first post, where I explained how to route all your traffic.

Tor Socks5 Proxy I have made a Docker image for this which lives at jess/tor-proxy on the Docker hub. But I will go over the details so you can build one yourself.

The Dockerfile looks like the following:

``` FROM alpine:latest

Note: Tor is only in testing repo -> http://pkgs.alpinelinux.org/packages?package=emacs&repo=all&arch=x86_64

RUN apk update && apk add \ tor \ --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ \ && rm -rf /var/cache/apk/*

expose socks port

EXPOSE 9050

copy in our torrc file

COPY torrc.default /etc/tor/torrc.default

make sure files are owned by tor user

RUN chown -R tor /etc/tor

USER tor

ENTRYPOINT [ "tor" ] CMD [ "-f", "/etc/tor/torrc.default" ]

```

Which looks a lot like the Dockerfile for a relay, if you recall. But the key difference is the torrc. Now the only thing I have changed from the default torrc is the following line:

``` SocksPort 0.0.0.0:9050

```

This is so that it can bind correctly to the network namespace the container is using.

This image weighs in at only 11.51 MB!

To run the image:

``` $ docker run -d \ --restart always \ -v /etc/localtime:/etc/localtime:ro \ # i like this for all my containers, but it's optional -p 9050:9050 \ # publish the port --name torproxy \ jess/tor-proxy

```

Okay, awesome, now you have the socks5 proxy running on port 9050. Let’s test it:

```

get your current ip

$ curl -L http://ifconfig.me

get your ip through the tor socks proxy

$ curl --socks http://localhost:9050 -L http://ifconfig.me

obviously they should be different ;)

you can even curl the check.torproject.org api

$ curl --socks http://localhost:9050 -L https://check.torproject.org/api/ip

```

If you are like me and use @ioerror’s gpg.conf you can uncomment the line:

``` keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050

```

Now you can import and search for keys on a key server with improved anonymity. Obviously there are a bunch of other things you can use the socks proxy for, but I wanted to give this as an example.

You could even run chrome in a container through the proxy…

Can we take this even further? Yes.

Privoxy HTTP Proxy The socks proxy is awesome, but if you want to additionally have an http proxy it is super easy!

What we can do is link a Privoxy container to our Tor proxy container.

NOTE: I have seen people have a Tor socks proxy and Privoxy in the same container. But I prefer my approach of 2 different containers, because it is cleaner, maybe sometimes you do not need both, and you completely eliminate the need for having an init system starting 2 processes in one container. Not that there is anything wrong with that, but it is not my personal preference.

So on to the Dockerfile, which also lives at jess/privoxy:

``` FROM alpine:latest

RUN apk update && apk add \ privoxy \ && rm -rf /var/cache/apk/*

expose http port

EXPOSE 8118

copy in our privoxy config file

COPY privoxy.conf /etc/privoxy/config

make sure files are owned by privoxy user

RUN chown -R privoxy /etc/privoxy

USER privoxy

ENTRYPOINT [ "privoxy", "--no-daemon" ] CMD [ "/etc/privoxy/config" ]

```

This image is a whopping 6.473 MB :D

The only change I made to the default privoxy config was the following:

``` forward-socks5 / torproxy:9050 .

```

This is so that when we link our torproxy container to the privoxy container, privoxy can communicate with the sock.

Let’s run it:

``` $ docker run -d \ --restart always \ -v /etc/localtime:/etc/localtime:ro \ # again a personal preference --link torproxy:torproxy \ # link to our torproxy container -p 8118:8118 \ # publish the port --name privoxy \ jess/privoxy

```

Awesome, now to test the proxy:

```

get your current ip

$ curl -L http://ifconfig.me

get your ip through the http proxy

$ curl -x http://localhost:8118 -L http://ifconfig.me

obviously again, they should be different ;)

curl the check.torproject.org api

$ curl -x http://localhost:8118 -L https://check.torproject.org/api/ip

```

That’s all for now! Stay anonymous on the interwebs :p

View Details

This post is part two of what will be a three part series. If you missed it part one was How to Route Traffic through a Tor Docker container. I figured it was important, if you are going to be a tor user, to document how you can help the Tor community by hosting a Tor relay. And guess what? You can use Docker to do this!

There are three types of relays you can host, a bridge relay, a middle relay, and an exit relay. Exit relays tend to be the ones recieving take down notices because the IP is the one the public sees traffic from Tor as. A great reference for hosting an exit node can be found here blog.torproject.org/blog/tips-running-exit-node-minimal-harassment. But I will go over how to host each from a Docker container. My example will have a reduced exit policy and limit which ports you are willing to route traffic through.

If you don’t want to host an exit node, host a middle relay instead! And if you want your relay not publically listed in the network then host a bridge.

Creating the base image I have created a Docker image jess/tor-relay from this Dockerfile. Feel free to create your own image with the following Dockerfile:

``` FROM alpine:latest

Note: Tor is only in testing repo

RUN apk update && apk add \ tor \ --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ \ && rm -rf /var/cache/apk/*

default port to used for incoming Tor connections

can be changed by changing 'ORPort' in torrc

EXPOSE 9001

copy in our torrc files

COPY torrc.bridge /etc/tor/torrc.bridge COPY torrc.middle /etc/tor/torrc.middle COPY torrc.exit /etc/tor/torrc.exit

make sure files are owned by tor user

RUN chown -R tor /etc/tor

USER tor

ENTRYPOINT [ "tor" ]

```

As you can see we are copying 3 different torrc’s into the container. One for each a bridge, middle, and exit relay.

I used alpine linux because it is super minimal. The size of the image is 11.52MB! Crazyyyyyyy!

Running a bridge relay A bridge relay is not publically listed as part of the Tor network. This is helpful in places that block all the IPs of publically listed Tor relays.

The torrc.bridge file for the bridge relay looks like the following:

``` ORPort 9001

A handle for your relay, so people don't have to refer to it by key.

Nickname hacktheplanet ContactInfo ${CONTACT_GPG_FINGERPRINT} ${CONTACT_NAME} ${CONTACT_EMAIL} BridgeRelay 1

```

To run the image for a bridge relay:

``` $ docker run -d \ -v /etc/localtime:/etc/localtime \ # so time is synced --restart always \ # why not? -p 9001:9001 \ # expose/publish the port --name tor-relay \ jess/tor-relay -f /etc/tor/torrc.bridge

```

And now you are helping the tor network by running a bridge relay! Yayyy \o/

Running a middle relay A middle relay is one of the first few relays traffic flows through. Traffic will always pass through at least 3 relays. The last relay being an exit node and all relays before that a middle relay.

The torrc.middle file for the middle relay looks like the following:

``` ORPort 9001

A handle for your relay, so people don't have to refer to it by key.

Nickname hacktheplanet ContactInfo ${CONTACT_GPG_FINGERPRINT} ${CONTACT_NAME} ${CONTACT_EMAIL} ExitPolicy reject :

```

To run the image for a middle relay:

``` $ docker run -d \ -v /etc/localtime:/etc/localtime \ # so time is synced --restart always \ # why not? -p 9001:9001 \ # expose/publish the port --name tor-relay \ jess/tor-relay -f /etc/tor/torrc.middle

```

And now you are helping the tor network by running a middle relay!

Running an exit relay The exit relay is the last relay traffic is filtered through.

The torrc.exit file for the exit node looks like the following:

``` ORPort 9001

A handle for your relay, so people don't have to refer to it by key.

Nickname hacktheplanet ContactInfo ${CONTACT_GPG_FINGERPRINT} ${CONTACT_NAME} ${CONTACT_EMAIL}

Reduced exit policy from

https://trac.torproject.org/projects/tor/wiki/doc/ReducedExitPolicy

ExitPolicy accept :20-23 # FTP, SSH, telnet ExitPolicy accept :43 # WHOIS ExitPolicy accept :53 # DNS ExitPolicy accept :79-81 # finger, HTTP ExitPolicy accept :88 # kerberos ExitPolicy accept :110 # POP3 ExitPolicy accept :143 # IMAP ExitPolicy accept :194 # IRC ExitPolicy accept :220 # IMAP3 ExitPolicy accept :389 # LDAP ExitPolicy accept :443 # HTTPS ExitPolicy accept :464 # kpasswd ExitPolicy accept :465 # URD for SSM (more often: an alternative SUBMISSION port, see 587) ExitPolicy accept :531 # IRC/AIM ExitPolicy accept :543-544 # Kerberos ExitPolicy accept :554 # RTSP ExitPolicy accept :563 # NNTP over SSL ExitPolicy accept :587 # SUBMISSION (authenticated clients [MUA's like Thunderbird] send mail over STARTTLS SMTP here) ExitPolicy accept :636 # LDAP over SSL ExitPolicy accept :706 # SILC ExitPolicy accept :749 # kerberos ExitPolicy accept :873 # rsync ExitPolicy accept :902-904 # VMware ExitPolicy accept :981 # Remote HTTPS management for firewall ExitPolicy accept :989-995 # FTP over SSL, Netnews Administration System, telnets, IMAP over SSL, ircs, POP3 over SSL ExitPolicy accept :1194 # OpenVPN ExitPolicy accept :1220 # QT Server Admin ExitPolicy accept :1293 # PKT-KRB-IPSec ExitPolicy accept :1500 # VLSI License Manager ExitPolicy accept :1533 # Sametime ExitPolicy accept :1677 # GroupWise ExitPolicy accept :1723 # PPTP ExitPolicy accept :1755 # RTSP ExitPolicy accept :1863 # MSNP ExitPolicy accept :2082 # Infowave Mobility Server ExitPolicy accept :2083 # Secure Radius Service (radsec) ExitPolicy accept :2086-2087 # GNUnet, ELI ExitPolicy accept :2095-2096 # NBX ExitPolicy accept :2102-2104 # Zephyr ExitPolicy accept :3128 # SQUID ExitPolicy accept :3389 # MS WBT ExitPolicy accept :3690 # SVN ExitPolicy accept :4321 # RWHOIS ExitPolicy accept :4643 # Virtuozzo ExitPolicy accept :5050 # MMCC ExitPolicy accept :5190 # ICQ ExitPolicy accept :5222-5223 # XMPP, XMPP over SSL ExitPolicy accept :5228 # Android Market ExitPolicy accept :5900 # VNC ExitPolicy accept :6660-6669 # IRC ExitPolicy accept :6679 # IRC SSL ExitPolicy accept :6697 # IRC SSL ExitPolicy accept :8000 # iRDMI ExitPolicy accept :8008 # HTTP alternate ExitPolicy accept :8074 # Gadu-Gadu ExitPolicy accept :8080 # HTTP Proxies ExitPolicy accept :8082 # HTTPS Electrum Bitcoin port ExitPolicy accept :8087-8088 # Simplify Media SPP Protocol, Radan HTTP ExitPolicy accept :8332-8333 # Bitcoin ExitPolicy accept :8443 # PCsync HTTPS ExitPolicy accept :8888 # HTTP Proxies, NewsEDGE ExitPolicy accept :9418 # git ExitPolicy accept :9999 # distinct ExitPolicy accept :10000 # Network Data Management Protocol ExitPolicy accept :11371 # OpenPGP hkp (http keyserver protocol) ExitPolicy accept :19294 # Google Voice TCP ExitPolicy accept :19638 # Ensim control panel ExitPolicy accept :50002 # Electrum Bitcoin SSL ExitPolicy accept :64738 # Mumble ExitPolicy reject :*

```

To run the image for an exit node:

``` $ docker run -d \ -v /etc/localtime:/etc/localtime \ # so time is synced --restart always \ # why not? -p 9001:9001 \ # expose/publish the port --name tor-relay \ jess/tor-relay -f /etc/tor/torrc.exit

```

And now you are helping the tor network by running an exit relay!

After running for a couple hours, giving time to propogate, you can check atlas.torproject.org to check if your node has successfully registered in the network.

Stay tuned for part three of the series where I go over how to run Docker containers with a Tor networking plugin I am working with Docker’s new networking plugins. But of course if you are going to use the plugin or route all your traffic through a Tor Docker container (from my first post), you should really consider hosting a relay. The more people who run relays, the faster the Tor network will be.

View Details

My least favorite topic in the world is ‘Women in Tech’, so I am going to make this short but I think it’s something that needs to be said.

This industry is fucked.

Ever since I started speaking at conferences and contributing to open source projects I have been endlessly harassed. I’ve gotten hundreds of private messages on IRC and emails about sex, rape, and death threats. People emailing me saying they jerked off to my conference talk video (you’re welcome btw) is mild in comparison to sending photoshopped pictures of me covered in blood.

I wish I could do my job, something I very obviously love doing, without any of this bullshit. However that seems impossible at this point.

But I’m not leaving and I’m not going to stop being me. So this is me saying ‘Fuck You.’

View Details

So it turns out I’m pretty bad at vacation. I had this idea for a blog post and one thing lead to another and here we are…

You probably know by now I hate installing things on my host. At my previous job we did a lot of work with using Python and R for data science. I still love plotting data with ggplot and my favorite R package, wes anderson color palette.

Here’s a fast intro into how to do this with an R Docker image.

Now everyone loves their share of different packages, without a doubt I bet most of them are written by Hadley Wickham ;). Can you imagine if the percentage of packages contributed by Hadley to CRAN was mirrored by someone to NPM or pip? It would be crazy.

We are going to start with an R base and build our ideal (aka you can make yours different, chill…) R data science container, with the following Dockerfile:

```

our R base image

FROM r-base

install packages

these are ones I like

RUN echo 'install.packages(c("ggplot2", "plyr", "reshape2", "RColorBrewer", "scales","grid", "wesanderson"), repos="http://cran.us.r-project.org", dependencies=TRUE)' > /tmp/packages.R \ && Rscript /tmp/packages.R

create an R user

ENV HOME /home/user RUN useradd --create-home --home-dir $HOME user \ && chown -R user:user $HOME

WORKDIR $HOME USER user

set the command

CMD ["R"]

```

Build the image:

``` $ docker build --rm --force-rm -t jess/r-custom .

```

Run and use the image:

```

we need X11 for the graph to display, alternatively

you can save to a file that is in a bind-mounted dir

or you can docker cp the file to the host :)

$ docker run -it --name analytics \ -v /tmp/.X11-unix:/tmp/.X11-unix \ -e DISPLAY=unix$DISPLAY \ jess/r-custom

bind mount your data

$ docker run -v $(pwd)/data:/home/user/data \ -it --name analytics \ -v /tmp/.X11-unix:/tmp/.X11-unix \ -e DISPLAY=unix$DISPLAY \ jess/r-custom

```

Now plot something:

``` library(wesanderson)

library(ggplot2) ggplot(iris, aes(Sepal.Length, Sepal.Width, color = Species)) + geom_point(size = 3) + scale_color_manual(values = wes_palette("Royal2")) + theme_gray()

```

See that was super easy, now I can go back to being on vacation and reading the latest Vogue.

Other resources for such things:

• Rocker Wiki: for R in Docker examples
• yhat Blog: for all things fun and data sciencey, I might be biased though I used to work here

View Details

This blog post is going to explain how to route traffic on your host through a Tor Docker container.

It’s actually a lot simplier than you would think. But it involves dealing with some unsavory things such as iptables.

Run the Image I have a fork of the tor source code and a branch with a Dockerfile. I have submitted upstream… we will see if they take it. The final result is the image jess/tor, but you can easily build locally from my repo jessfraz/tor.

So let’s run the image:

``` $ docker run -d \ --net host \ --restart always \ --name tor \ jess/tor

```

Easy right? I can already hear the haters, “blah blah blah net host”. Chill out, the point is to route all our traffic duhhhh so we may as well, otherwise would need to change / overwrite some of Docker’s iptables rules, and really who has time for that shit…

You do? Ok make a PR to this blog post.

Routing Traffic Contain yourselves, I am about to throw down some sick iptables rules.

```

!/bin/bash

Most of this is credited to

https://trac.torproject.org/projects/tor/wiki/doc/TransparentProxy

With a few minor edits

to run iptables commands you need to be root

if [ "$EUID" -ne 0 ]; then echo "Please run as root." return 1 fi

set variables

destinations you don't want routed through Tor

_non_tor="192.168.1.0/24 192.168.0.0/24"

get the UID that Tor runs as

_tor_uid=$(docker exec -u tor tor id -u)

Tor's TransPort

_trans_port="9040" _dns_port="5353"

set iptables *nat

iptables -t nat -A OUTPUT -m owner --uid-owner $_tor_uid -j RETURN iptables -t nat -A OUTPUT -p udp --dport 53 -j REDIRECT --to-ports $_dns_port

allow clearnet access for hosts in $_non_tor

for _clearnet in $_non_tor 127.0.0.0/9 127.128.0.0/10; do iptables -t nat -A OUTPUT -d $_clearnet -j RETURN done

redirect all other output to Tor's TransPort

iptables -t nat -A OUTPUT -p tcp --syn -j REDIRECT --to-ports $_trans_port

set iptables *filter

iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

allow clearnet access for hosts in $_non_tor

for _clearnet in $_non_tor 127.0.0.0/8; do iptables -A OUTPUT -d $_clearnet -j ACCEPT done

allow only Tor output

iptables -A OUTPUT -m owner --uid-owner $_tor_uid -j ACCEPT iptables -A OUTPUT -j REJECT

```

Check that we are routing via check.torproject.org.

Woooohoooo! Success.

View Details

This is a tale about how we use Docker to test Docker. Yes, I am familiar with the meme. Puhlease.

Many of you are familiar with the fact I work on the Docker core team. Which consists of fixing bugs, doing releases, reviewing PRs, hanging out on IRC, mailing lists etc etc etc. But what you may not know is that in addition to all these things I also manage our testing infrastructure. Now really this in itself could be a fulltime job. However it is not my fulltime job, nor would I ever want it to be. [insert gif about yak shaving here]

This blog post is going to be about how I manage ~50 servers but don’t do anything at all. Of course, I have my angry sysadmin moments when everything breaks and I could punch a whole in a wall… but who doesn’t?

Our CI First let me take a chance to familiarize you with how we test Docker. Docker’s tests run in a Docker container. We use Jenkins as our CI mostly because we needed a lot of flexibility and control.

Obviously everything in our infrastructure runs in Docker, so that even goes for Jenkins. We use the official image for our Jenkins container.

Docker itself has 6 different storage driver options. These are aufs, btrfs, devmapper, overlay, vfs, and zfs. We have servers that use each of these hooked up to our Jenkins instance for testing.

Along with all the storage driver options, Docker also runs on any linux distro and a world of different linux kernel versions. In order to be able to try and test all this differentiation, each server runs a different kernel and all major linux distros are accounted for.

With every push to master on the docker/docker repo, we run tests on the entire storage driver matrix. We also trigger builds to test the unsupported lxc execdriver for Docker. And we trigger builds to test the Docker Windows client. Right there is three different jobs running on 8 different servers just for 1 push to master.

Did I mention we have 9 Windows servers and 9 linux remote hosts paired with those servers for testing Docker on Windows?

With every pull request to Docker we kick off 3 builds on 3 different servers. We have 8 linux Docker nodes reserved exclusively for testing PRs. These run the Docker tests and the new “Experimental Tests”. The last of the 3 is the Windows client test.

Considering the Docker project gets over 100 pull requests a week, with multiple revision cycles you can only imagine the number of builds we process in a day.

The manager for the PR builds is a small service called leeroy which also makes sure every PR has been signed with the Docker DCO before it even triggers a build. This of course also runs in a container.

Now of course not every build is perfect, sometimes you have to rebuild. To make this easy for all maintainers of the project we have an IRC bot, named lovingly after Docker’s turtle Gordon. The gordonbot runs in a container duh, and can kick off a rebuild on any of our bajillion servers.

Now I know what you are thinking, thats a lot of servers, how do you manage to know when heaven forbid one of them goes down.

Consul We have consul running in a container on all 50 servers in our infrastructure. This is AMAZING. We use a sweet project, consul alerts, also running in a container, to let us know when a node or service on a node goes down.

I would honestly be lost without consul. It keeps track via tags of the kernel version, storage driver, linux distro, etc of the server. When a server goes down I can decifer if it is a bug with any of those things.

A great example of this is we recently merged the awesome changes to the container network stack via libnetwork. However, I noticed after the merge the servers with kernels 3.19.x and 3.18.x were acting funny. We were able to fix kernel bugs that were specific to those versions related to networking before an RC was even cut.

Github Hooks for the Github Hooks Throne We trigger a lot of cool things with every push to master. We use nsq to collect the hooks and then pass the messages to all the consumers. Oh and obviously nsq runs in a container, as well as the hooks service.

Master Binaries With every push to master we push new binaries to master.dockerproject.org. This way people can easily try out new features.

The docker-bb service is run in a container ;). Hopefully you are catching on to a theme here…

Master Docs What good would being able to try new features be, if you didn’t have docs for how to use them?

With every push to master, we deploy new docs to docs.master.dockerproject.org.

This is done with a nsqexec service, wait for it…. RUNNING IN A CONTAINER.

Always Testing The greatest thing about all these services, which I so subtly mentioned, running in containers is that we can always be dogfooding and testing Docker. Right now we are getting ready to release Docker v1.7.0 and with every RC that is built I upgrade the servers so we can catch bugs.

On the off season, I will randomly upgrade all the servers to the Docker master binaries mentioned previously. This way we can catch things long before they even hit an RC.

All this is so seamless and runs so well that I have time to do my actual job of being a Docker core maintainer, occasionally fix some servers, spin up new servers if we add storage drivers, upgrade servers’ kernels, and write this blog post.

Hope you enjoyed, also help us test RC’s and Docker master! Also thanks to DigitalOcean and Azure for hosting our infrastructure.

View Details

Hello!

If you are not familiar with Docker, it is the popular open source container engine.

Most people use Docker for containing applications to deploy into production or for building their applications in a contained environment. This is all fine & dandy, and saves developers & ops engineers huge headaches, but I like to use Docker in a not-so-typical way.

I use Docker to run all the desktop apps on my computers.

But why would I even want to run all these apps in containers? Well let me explain. I used to be an OS X user, and the great thing about OS X is the OS X App Sandbox.

App Sandbox is an access control technology provided in OS X, enforced at the kernel level. Its strategy is twofold:

App Sandbox enables you to describe how your app interacts with the system. The system then grants your app the access it needs to get its job done, and no more.

App Sandbox provides a last line of defense against the theft, corruption, or deletion of user data if an attacker successfully exploits security holes in your app or the frameworks it is linked against.

Apple About App Sandbox

I am using the Apple App Sandbox as an example so people can grasp the concept easily. I am not saying this is exactly like that and has all the features. This is not a sandbox. It is more like a cool hack.

I hate installing things on my host and the files getting everywhere. I wanted the ability to delete an app and know it is gone fully without some random file hanging around. This gave me that. Not only that, I can control how much CPU and Memory the app uses. Yes, the cpu/memory hungry chrome is now perfectly contained!

“What?!?!”, you say. Let me show you.

The following covers a few of my favorite applications I run in containers. Each of the commands written below is actually pulled directly from my bash aliases. So you can have the same user experience as running one command today.

TUIs (Text User Interface, pronounced too-eee) Let’s start with some easy text-based applications:

1. Irssi Dockerfile

Best IRC client.

``` $ docker run -it \ -v /etc/localtime:/etc/localtime \ -v $HOME/.irssi:/home/user/.irssi \ # mounts irssi config in container --read-only \ # cool new feature in 1.5 --name irssi \ jess/irssi

```

1. Mutt Dockerfile

The text based email client that rules!

``` $ docker run -it \ -v /etc/localtime:/etc/localtime \ -e GMAIL -e GMAIL_NAME \ # pass env variables to config -e GMAIL_PASS -e GMAIL_FROM \ -v $HOME/.gnupg:/home/user/.gnupg \ # so you can encrypt ;) --name mutt \ jess/mutt

```

1. Rainbowstream Dockerfile

Awesome text based twitter client.

``` $ docker run -it \ -v /etc/localtime:/etc/localtime \ -v $HOME/.rainbow_oauth:/root/.rainbow_oauth \ # mount config files -v $HOME/.rainbow_config.json:/root/.rainbow_config.json \ --name rainbowstream \ jess/rainbowstream

```

1. Lynx Dockerfile

The browser everyone loves (to hate). but secretly I love

``` $ docker run -it \ --name lynx \ jess/lynx

```

Yes I know my blog looks GREAT in lynx

Okay, those text based apps are fun and all but how about we spice things up a bit.

GUIs None of the images below use X11-Forwarding with ssh. Because why should you ever have to install ssh into a container? EWWW UNNECESSARY BLOAT!

The images work by mounting the X11 socket into the container! Yippeeeee!

The commands listed below are run on a linux machine. But Mac users, I have a special surprise for you. You can also do fun hacks with X11. Details are described here.

Note my patch was added for --device /dev/snd in Docker 1.8, before that you needed -v /dev/snd:/dev/snd --privileged.

1. Chrome Dockerfile

Pretty sure everyone knows what chrome is, but my image comes with flash and the google talk plugin so you can do hangouts.

``` $ docker run -it \ --net host \ # may as well YOLO --cpuset-cpus 0 \ # control the cpu --memory 512mb \ # max memory it can use -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display -v $HOME/Downloads:/root/Downloads \ # optional, but nice -v $HOME/.config/google-chrome/:/data \ # if you want to save state --device /dev/snd \ # so we have sound --name chrome \ jess/chrome

```

1. Spotify Dockerfile

All the 90s hits you ever wanted and more.

``` $ docker run -it \ -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display --device /dev/snd \ # sound --name spotify \ jess/spotify

```

1. Gparted Dockerfile

Partition your device in a container.

MIND BLOWN.

``` $ docker run -it \ -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display --device /dev/sda:/dev/sda \ # mount the device to partition --name gparted \ jess/gparted

```

1. Skype Dockerfile

The other video conferencer. This relies on running pulseaudio also in a container.

```

start pulseaudio

$ docker run -d \ -v /etc/localtime:/etc/localtime \ -p 4713:4713 \ # expose the port --device /dev/snd \ # sound --name pulseaudio \ jess/pulseaudio

```

```

start skype

$ docker run -it \ -v /etc/localtime:/etc/localtime \ -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display --device /dev/snd \ # sound --link pulseaudio:pulseaudio \ # link pulseaudio -e PULSE_SERVER=pulseaudio \ --device /dev/video0 \ # video --name skype \ jess/skype

```

1. Tor Browser Dockerfile

Because Tor, duh!

``` $ docker run -it \ -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display --device /dev/snd \ # sound --name tor-browser \ jess/tor-browser

```

1. Cathode Dockerfile

That super old school terminal.

``` $ docker run -it \ -v /tmp/.X11-unix:/tmp/.X11-unix \ # mount the X11 socket -e DISPLAY=unix$DISPLAY \ # pass the display --name cathode \ jess/1995

```

So that’s enough examples for now. But of course I have more. All my Dockerfiles live here: github.com/jessfraz/dockerfiles and all my docker images are on the hub: hub.docker.com/u/jess.

I gave a talk on this at Dockercon 2015, check out the video.

Happy Dockerizing!!!

View Details

Hello!

This blog post is going to go over how to create a Linux partition on your mac and have everything working successfully.

Okay so lets begin with: sudo rm -rf / && sudo kill -9 1.

Hold the phone.

That was a test. I really hope you didn’t just copy, paste, and run a command on your host without knowing anything about the author. A bit about me… I have run this install about a dozen times on my mac, with various different changes along the way. I can finally say I found the perfect way to install Linux, specifically Debian Jessie, on a mac.

So now let’s actually get started.

Hardware The below installation was done on my MacBook Pro Retina (15-inch, Late 2013).

You will also need one of these nifty ethernet to thunderbolt adapters.

rEFInd Boot Manager The majority of times I installed Linux I ran rEFInd on my mac, so I could keep my mac partition and have a separate Linux partition. This last time, however, I was so fed up with OSX and the fact I never used it, I nuked it entirely.I boot purely into the Debian Bootloader now. But I will save that doosey for another blog post if I think people are really as crazy as I. rEFInd is the lesser of two evils between the other popular rEFIt, you will probably see some pain points and reasons for my fuck it, nuke it attitude towards OSX.

Instructions for installing rEFInd can be found here, but I will go into detail about how I install since you can tell those are a bit hard to read.

If you don’t know how to open terminal just stop now, sorry this isn’t going to be one of those blog posts.

The following works for OSX Mountain Lion. If you are running Yosemite you are SOL (not really but read this and I wish you luck on your journey):

``` $ curl -O http://downloads.sourceforge.net/project/refind/0.8.3/refind-bin-0.8.3.zip $ unzip refind-bin-0.8.3.zip $ cd refind-bin-0.8.3/

we are going to install with all drivers

because you honestly never know what you

will need, better be safe vs. sorry

$ sudo ./install.sh --alldrivers

```

Okay now you need to edit /EFI/refind/refind.conf. The key differences you should make to the default config are as follows:

```

Enable the scan for file system drivers

scan_driver_dirs EFI/tools/drivers,drivers

Choose which drives to scan. This will only scan the internal hard drive.

scanfor internal

Load the Linux file system driver

fs0: load ext4_x64.efi

I used ext4 (duh)

if you want to use btrfs

comment out ext4 line

and uncomment the next line

fs0: load btrfs_x64.efi

fs0: map -r

```

Let’s check it’s working. Restart your computer and you should see a super 90’s looking screen like:

If not, there are various debugging tips per version of Mac OSX here.

High five! Hard part’s done. Really. That is the hardest part.

Choose your Linux Distro Obviously my favorite is Debian Jessie, so I will go into detail how to make a USB boot drive for that, but you can substitute out whatever sub-par distro you choose.

As of the writing of this article, Debian Jessie is on it’s Beta 2 release. You can download the netist image from here. But detailed instructions follow:

```

download the iso

$ curl -O http://cdimage.debian.org/cdimage/jessie_di_beta_2/amd64/iso-cd/debian-jessie-DI-b2-amd64-netinst.iso

convert the .iso file to .img

$ hdiutil convert -format UDRW -o debian-jessie.img debian-jessie-DI-b2-amd64-netinst.iso

osx will most likely add the .dmg extension, rename it

$ mv debian-jessie.img.dmg debian-jessie.img

view your mounted drives to find the usb device

$ diskutil list

/dev/disk0

#: TYPE NAME SIZE IDENTIFIER

0: GUID_partition_scheme *500.3 GB disk0

/dev/disk1

#: TYPE NAME SIZE IDENTIFIER

0: USB_DEVICE *100.1 GB disk1

unmount the usb device

$ diskutil unmountDisk /dev/disk1

create the boot drive

$ sudo dd if=debian-jessie.img of=/dev/disk1

eject the usb device

mac osx will probably yell at you before you

can even do this with a popup asking if you want

to eject the unsupported device, you can click the

eject button there, it's the same thing

$ diskutil eject /dev/disk1

```

Partition Your HD Next you need to partition your hard drive so there is enough space for your linux distro. Here are the steps:

1. Open Disk Utility
2. Select the disk on the left panel (for example “500GB APPLE SSD”)
3. On the partition scheme resize the “Macintosh HD” partition, drag the bottom right edge of the partition scheme up unless you have enough space for Debian. Apply the changes.

Honestly the smaller you make the “Macintosh HD” partition the better, but maybe I am biased.

Installing your Linux Distro Make sure your computer is off. Connect your Ethernet adapter and your USB drive we made earlier.

Turn on your computer and hold down the option/alt key.

Select the EFI Boot relative to your USB drive (It’s going to be the bright orange drive looking thing) and continue with to the installer screen.

If your linux distro has Advanced Options like Debian for installing a certain Desktop Environment (and its not Ubuntu or XUbuntu) don’t even bother setting those we will handle that after nvidia drivers.

Continue through your install.

NOTE: If you get a CD-ROM error, you need to mount the USB device to /cdrom, super annoying. The process will fail and you will be given some options, choose the shell and run mount /dev/sdc1 /cdrom. It might also be /dev/sda1 or /dev/sdb1. You will know it when you hit it because you won’t get a mount error, then return to the menu and continue where you left off on the “CD-ROM install”.

When the installer arrives at the partitioning step, you can use the auto partioning, that’s what I did with all free space, then in the review screen I used ext4. If you are going to be running Docker on your system I highly recommend ext4 with the overlay storage driver and you should trust me.

Complete the install and reboot.

You are in a term, it feels bleek Do not fret. I repeat do not fret.

Login as root, yes I know you just created an actual user in the installation steps but ROOT ACCESS OR DEATH. Really though we need to install sudo and build a new kernel. After all that is done, you can continue on your way as your user.

Ok so at this point I know you are not copy and pasting this shit into your terminal so I’ll try to keep it concise. Remember, I’ve been here. We will get through this.

View your /etc/apt/sources.list and it is probably messed up and pointing to a CD-ROM.

Change it to the following (or whatever your distro wants):

``` deb http://ftp.us.debian.org/debian jessie main contrib non-free deb-src http://ftp.us.debian.org/debian/ jessie main contrib non-free

deb http://ftp.debian.org/debian/ jessie-updates main contrib non-free

deb http://security.debian.org/ jessie/updates main contrib non-free

```

Now we can:

``` $ apt-get update $ apt-get upgrade

install sudo and add our other user to it

$ apt-get install sudo $ adduser your_username sudo

```

Let’s build a kernel from source wooooo Now here’s the thing. Debian Jessie comes with a 3.16.x kernel. 3.17.x is really where the awesome is at for Mac OS X, because it has hotpugging for thunderbolt. WHAAAAA? YES!!!

So if you are going to ride with me on the awesome thunderbolt train we need to build ourselves a kernel from source. Or if you reallllllyyy trust me you can download my .deb for kernel 3.17.3 here, but honestly I build my own everytime so take that as you will.

Usually, I do these builds in a container. But for the sake of this we can just do it on our host cringe.

```

install deps to build kernel

$ apt-get install curl kernel-package fakeroot

download the source

which at the time of writing this the latest is 3.17.4

$ cd /usr/src $ curl -O https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.17.4.tar.xz $ tar -xvf linux-3.17.4.tar.xz $ cd linux-3.17.4/

Options:

you can either use my kernel .config

which has thunderbolt and all modules enabled

$ curl -O https://misc.j3ss.co/kernels/3.17.3/.config

OR

you can use the menu to configure yourself

be sure to turn on thunderbolt, that's the whole point

$ apt-get install libncurses5-dev # install menu dependency $ make menuconfig

clean the source tree

$ make-kpkg clean

compile the kernel

this will take about 30 min

$ fakeroot make-kpkg --initrd --revision=3.17.4 kernel_image

install the new kernel

$ dpkg -i ../linux-image-3.17.4_3.17.4_amd64.deb

reboot the system

$ reboot

```

After restarting, depending on your refind.conf file you may see a new option in your rEFInd menu for the new kernel. DO NOT select that, select the option that corresponds to the linux GRUB (or whichever) bootloader you use. If you do not see one for GRUB or your flavor bootloader you may need to bless the bootloader file on the Mac OSX side. See these instructions on blessing. Do you understand now why rEFInd is the hardest part? It’s like iptables, change one thing and everything comes crashing down.

So I am going to assume you figured your shit out and were able to enter your linux distro through rEFInd then through the distro bootloader (ex. GRUB).

Let’s clean things up.

```

Make sure we have the right kernel

$ uname -a

Linux debian 3.17.4 #1 SMP Wed Nov 12 01:11:57 PST 2014 x86_64 GNU/Linux

uninstall the shit we don't need now

$ apt-get purge --auto-remove kernel-package fakeroot

you can even uninstall the kernel that came with

$ apt-get purge --auto-remove linux-image-3.16.*

```

To avoid random controller freeze you need to set a particular kernel boot option. Edit /etc/default/grub and add the option libata.force=noncq (es. GRUB_CMDLINE_LINUX_DEFAULT="quiet libata.force=noncq") then run update-grub and reboot your system. If you are going to be installing Docker you may as well add GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" while you are there as well.

Drivers Okay now we are to the important part, let’s get shit to work.

Wifi

``` $ apt-get install firmware-linux-nonfree broadcom-sta-dkms

```

Graphics

``` $ apt-get install nvidia-driver xorg xserver-xorg-video-intel

probably want to restart after

$ reboot

```

Reverse Scroll (like Mac) Touchpad

``` $ clickpad_settings="Section \"InputClass\" Identifier \"touchpad catchall\" Driver \"synaptics\" MatchIsTouchpad \"on\" Option \"VertScrollDelta\" \"-111\" Option \"HorizScrollDelta\" \"-111\" EndSection"

$ mkdir -p /etc/X11/xorg.conf.d/ $ printf %s "$clickpad_settings" > /etc/X11/xorg.conf.d/50-synaptics-clickpad.conf

```

Font Anti-Aliasing

``` $ config="<?xml version='1.0'?>

rgb true hintslight true lcddefault "

$ printf %s "$config" > /etc/fonts/local.conf

$ dpkg-reconfigure fontconfig-config

Choose:

Autohinter

Automatic

No

$ dpkg-reconfigure fontconfig

```

Desktop Environment

Now is the time to install whatever desktop environment you love. i3 is my personal flavor:

``` $ apt-get install dunst feh i3 i3lock i3status scrot suckless-tools

```

Screen Backlight

I have a bash script https://misc.j3ss.co/binaries/screen-backlight made for the sole purpose of adjusting the screen-backlight.

You will want to add to your sudoers file the following line, so password is not required for the script to run:

```

where your user is called user

and your host is called host

user host = (root) NOPASSWD: /usr/bin/local/screen-backlight

```

then for the example of i3 you can add the following to your config:

``` bindsym XF86MonBrightnessUp exec sudo screen-backlight up bindsym XF86MonBrightnessDown exec sudo screen-backlight down

```

Keyboard Backlight

The same goes for the keyboard backlight. I have a bash script https://misc.j3ss.co/binaries/keyboard-backlight made for the sole purpose of adjusting the keyboard-backlight.

You will want to add to your sudoers file the following line, so password is not required for the script to run:

```

where your user is called user

and your host is called host

user host = (root) NOPASSWD: /usr/bin/local/keyboard-backlight

```

then for the example of i3 you can add the following to your config:

``` bindsym XF86KbdBrightnessUp exec sudo keyboard-backlight up bindsym XF86KbdBrightnessDown exec sudo keyboard-backlight down

```

Things that won’t work in Debian

I have not gotten the iSight camera or Screen Brightness to work. Other than that, everything is perfect, and thunderbolt hotplugging is a dream. The retina resolution is absolutely stunning, it’s seriously hard for me to switch to my Thinkpad which has 32GB of memory (so I should want to switch).

Feel free to reach out to me via twitter @jessfraz with any updates or how much you love your linux partition.

View Details

I would just like to preface this by saying I do not condone cheating but I thought of this as a “challenge” and not so much as “cheating”.

A project I am working on required me to checkin to places on foursquare that I was not currently near (or even close to). Now the answer to this was pretty simple. Checkin through the API using the lat and long of the venue I was “supposedly” at. Boom. Worked without a flaw. Ok I will admit it I am kinda a competitive person and well, the foursquare badges are so pretty I immediately started thinking about how I could check in remotely and collect them all. But surely, surely foursquare must have some sort of catches in place that do not allow this. Because I was ever so curious to find out what they may be (…and how to get around them) I decided to try.

Authentication Let’s start with the auth. If a user has not authed your application or is not currently logged into foursquare (assuming you created an app in the foursquare for developers dashboard) redirect them as follows.

``` $clientId = "YOUR-CLIENT-ID"; $redirectUri = "YOUR-REDIRECT-URI"; header("Location:https://foursquare.com/oauth2/authenticate?client_id=" . $clientId ."&response_type=code&redirect_uri=" . $redirectUri);

```

After authenticating, grab the authentication code foursquare redirected the user with.

``` $code = $_REQUEST['code'];

```

Now start a session and save your access token to it. This way we can easily see if the user is an authenticated app user by checking the session variable. You could also save it as a cookie if you want it to last longer.

``` session_start();

if (!isset($_SESSION['access_token'])) { $app_token_url = "https://foursquare.com/oauth2/access_token?client_id=" . $client_id . "&client_secret=" . $client_secret . "&grant_type=authorization_code&redirect_uri=" . $redirect_uri . "&code=" . $code;

$ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $app_token_url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $foursquare_token = curl_exec($ch); curl_close($ch);

$array_token = json_decode($foursquare_token, true); $token = $array_token['access_token']; $_SESSION['access_token'] = $token; }

```

Get Location Data Ok now you have your token and we can get into the fun part, winning at foursquare! To check into a venue you need to post the following parameters to foursquare: venueId, ll (latitude, longitude), llAcc (accuracy of previous points), oauth_token, and v (version, which foursquare takes in as todays date in the form “Ymd”).

So to make checking into various different venues easier I decided the only thing I want to pass to this function is the venueId, v, and oauth_token. This requires making a function to return the lat and long of the venue from the foursquare api.

``` function getLatLong($venue_id, $v, $oauth_token) { $venue_url = 'https://api.foursquare.com/v2/venues/' . $venue_id . '?oauth_token=' . $oauth_token . '&v=' . $v;

$response = file_get_contents($venue_url); $venue = json_decode($response, true); $venue_response = $venue['response']; $location = $venue_response['venue']['location']; $lat = $location['lat']; $long = $location['lng'];

return $lat . ', ' . $long; }

```

Checkin Now we can send this value into the checkin function.

``` function checkin($venue_id, $v, $oauth_token, $latlong) { $checkin_url = "https://api.foursquare.com/v2/checkins/add";

parameters = array( 'venueId' => $venue_id, 'broadcast' => 'private', //now i set this private, but can be public 'll' => $latlong, 'llAcc' => '1', 'oauth_token' => $oauth_token, 'v' => $v );

$curl = curl_init($checkin_url); curl_setopt($curl, CURLOPT_POST, true); curl_setopt($curl, CURLOPT_POSTFIELDS, $parameters); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); $response = curl_exec($curl);

return $response; }

```

Response The response from this will be in the following format.

``` { meta: { code: 200 }, notifications: [ { type: "notificationTray", item: { unreadCount: 0 } } ], response: { checkin: { id: "4d627f6814963704dc28ff94", createdAt: 1298300776, type: "checkin", shout: "Another one of these days. #snow", timeZoneOffset: -300, user: { id: "32", firstName: "Dens", photo: { prefix: "https://irs0.4sqi.net/img/user/", suffix: "/32_1239135232.jpg",

            },

        },
        venue: {
            id: "408c5100f964a520c6f21ee3",
            name: "Tompkins Square Park",
            contact: {
                phone: "2123877685",
                formattedPhone: "(212) 387-7685",

            },
            location: {
                address: "E 7th St. to E 10th St.",
                crossStreet: "btwn Ave. A & B",
                lat: 40.72651075083395,
                lng: -73.98171901702881,
                postalCode: "10009",
                city: "New York",
                state: "NY",
                country: "United States",
                cc: "US",

            },
            categories: [
                {
                    id: "4bf58dd8d48988d163941735",
                    name: "Park",
                    pluralName: "Parks",
                    shortName: "Park",
                    icon: {
                        prefix: "https://foursquare.com/img/categories_v2/parks_outdoors/park_",
                        suffix: ".png",

                    },
                    primary: true,

                },

            ],
            verified: true,
            stats: {
                checkinsCount: 25523,
                usersCount: 8932,
                tipCount: 85,

            },
            url: "http://www.nycgovparks.org/parks/tompkinssquarepark",
            likes: {
                count: 0,
                groups: [

                ],

            },
            specials: {
                count: 0,

            },

        },
        source: {
            name: "foursquare for Web",
            url: "https://foursquare.com/"
        },
        photos: {
            count: 1,
            items: [
                {
                    id: "4d627f80d47328fd96bf3448",
                    createdAt: 1298300800,
                    prefix: "https://irs3.4sqi.net/img/general/",
                    suffix: "/UBTEFRRMLYOHHX4RWHFTGQKSDMY14A1JLHURUTG5VUJ02KQ0.jpg",
                    width: 720,
                    height: 540,
                    user: {
                        id: "32",
                        firstName: "Dens",
                        photo: {
                            prefix: "https://irs0.4sqi.net/img/user/",
                            suffix: "/32_1239135232.jpg",

                        },

                    },
                    visibility: "priviate"
                }
            ],

        },
        likes: {
            count: 0,
            groups: [

            ],

        },
        like: false,
        score: {
            total: 1,
            scores: [
                {
                    points: 1,
                    icon: "https://foursquare.com/img/points/defaultpointsicon2.png",
                    message: "Have fun out there!",

                },

            ],

        },

    },

},

}

```

Summary So what I found was this:

• You can check into 15-20 places in a loop without loosing points or disabling the chance to win badges, but then you have to take a break for a few hours
• When changing locations over a vast distance (ex. Los Angeles –> San Francisco), you must wait the amount of time it takes to reasonably cover that distance before checking in or else you will not be able to earn badges.

With these in mind this is how I approached earning as many badges in as little time as possible. Once I was “in” a location area, I looped through a set array of about 15 venues. I made these arrays based off the places most blogs said you needed to win a badge. The expertise badges are easy; checkin to 3 different venues categorized as BBQ Joints, earn the badge. The city badges all have lists in foursquare that house the venues you need to go, hit five and you get the badge.

``` $windy_city_badge = array( '4b876c65f964a520e2be31e3', '4b4e0d9ff964a520c0df26e3', '4e1e0e65aeb75f77be667547', '4e70c1aa814dd2cb962265cb', '49dce128f964a520b65f1fe3' );

```

I would recommend conquering the city badges first because you will probably earn all the expertise badges in the process.

Go get ‘em! Haters gonna hate, but you just made foursquare yo biotch.

View Details

I saw this sign outside a coffee shop. Most people would just walk by and laugh, but it got me thinking. What would 2PAC do? Seeing as 2PAC is one of my favorite artists and I was already walking with earbuds on, I started playing an oldie but goodie on my iPhone, “Changes”.

Now if you have never heard of rapgenius.com before, you should definitely check it out. It has translations of basically every rap song thats popular, and the artists can login and say what the lyrics to the song actually meant. Seeing as 2PAC is deceased (saddness), and I don’t think the holographic 2PAC is going to be logging into rap genius anytime soon… I pondered the meaning of the lyrics myself.

My conclusion was, if you are not happy with the way things are currently going, you should stand up and try to change it. “Some things will never change,” but what you do have control of changing is yourself. Just like Gandhi said, “If we could change ourselves, the tendencies in the world would also change. As a man changes his own nature, so does the attitude of the world change towards him… We need not wait to see what others do.”

Presently, I have made some rather large changes in my life. I have decided to take a new job in New York. The hardest part of this decision was leaving behind my job now. Over the past year, my co-workers have gone from being friends to being my family. But the world of web is constantly changing and I am excited to get to learn new things and expand my experience.

2PAC’s interpretation of change (from what I gather) is that it should be sparked by something, but the truth is change can come about for a variety of reasons. I chose change to grow more knowledge in the field I care so dearly about, not to mention my love for the city of New York.

So, yes the next time you are faced with a decision you can think ‘What would 2PAC do?’, but also trust yourself because when it comes down to it you are your own best advocate.