Here’s How The Amazing Twitter Infosec Team Helps DevOps

Tuesday, December 25, 2012

Gene Kim


Want to see how infosec integrates into a DevOps work stream?  Watch this fantastic talk by Justin Collins, Neil Matatall, and Alex Smolen from Twitter, called “Put Your Robots To Work: Security Automation at Twitter.”  Or read my full notes of their talk here.   

Watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations.   Of course, they mention what it’s like to be at an organization going through hyper-growth (i.e., the famous “Fail Whale” due to capacity issues).  But much to the amusement of all of us, they describe the birth of the Twitter infosec program, triggered by the hacking of the @barackobama account.    

That breach resulted in an FTC injunction, requiring Twitter to be secure for the next 15 years. And hence, Twitter infosec was born.  

This newly-formed department made some huge strides during the Twitter Hack Week, which occurs once every quarter, where they were able to focus on proactive work.  

During Hack Weeks, everyone “works on whatever they want, which they then demo to the rest of the company,” similar to what’s done at Facebook.  These projects often focus on work that reduces technical debt, automate manual work, improve processes, etc.     They wanted to focus on creating more automation, but anchored in the these framing principles:

  • Get the right information to the right people
  • Find and fix bugs as quickly as possible
  • Don’t repeat your mistakes
  • Analyze from many angles:
  • Let people prove you wrong
  • Help people help themselves
  • Automate dumb work
  • Keep it tailored

The great part about this talk is how they describe integrating into the daily work of Development and IT Operations by automating most of their infosec work.    

Automating Security

Justin Collins makes a fantastic point of the workflow around static code analysis. Even though the static code analysis step is “automated,” infosec still has to do a lot of waiting:  waiting for the scan to complete, get back the big stack of reports, interpret the reports, and then find the person responsible for fixing it.  And when the code changes, we have to do it all over again!  Even though we’re using ‘automated tools,’ we’re still doing a lot of manual work....  So we wanted to put our robots to work.  By doing this, we do less dumb button-pushing tasks, doing more stuff with creativity and judgement.”  

Back to that first Hack Week: they starting building a series of tools that integrated into the continuous integration process run by Jenkins.  Among this was building SADB, the Security Automation Dashboard.  The logo is, of course, a sad bee.   In their talk, they described all the amazing tools they built to put infosec into the daily work of Development and IT Operations:  

  • SADB, which takes input from brakeman, phantom gang, csp, threatdeck, roshambo, and the outputs include emails that go to developers and infosec.
  • Brakeman that does static code analysis for Ruby on Rails.  The primary author is Justin Collins (@presidentbeef).  It’s had 25 releases in last year.  In the ideal, Brakeman runs not only at every code commit, but every time the developer saves code!
  • Phantom Gang that does dynamic application security testing (DAST).  It complements Brakeman by looking for issues like mixed content, sensitive forms posting over non-HTTPS, old versions of jquery that often pop up when new microsites are created, forms without authenticity token which are prone to forgery, etc.  It’s a bunch of node.js processes which emulates Webkit browser sessions, which are spun up as headless browser instances to see what users see.  The output of Phantom Gang goes to JIRA, as opposed to directly to developers directly.  Why?  Often the issues found by Phantom Gang are more difficult to trace to an individual developer.
  • CSP: “Twitter is a big fan of CSP.  It’s great for enforcing policy and protecting web sites.”  Here’s an article on how Twitter uses CSP.  They’ve configured CSP to disallow Javascript on the page itself, and have Javascript only served from themselves.  When there’s a violation of that policy, “it’s almost assuredly a sign that there’s a valid XSS attack underway.”  They also use their Big Data capability to look for site spikes, which is often an indicator of XSS attack

Again, watching this talk should be required for any infosec person who wants to see how they can integrate into the daily work of Development and IT Operations.  You can read my full notes of this talk at here.  


Possibly Related Articles:
Information Security
Twitter Automation DevOps
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.