Bring AWS To Its Knees

2014-05-13 by Robbie Allen (Meetup)

Automated Story Writing

  • Combines writing and programming skills
  • Making millions of articles not sound automated is difficult

Workflow

  1. Client uploads data to S3 and creates an SQS message with file path
  2. System autoscales based on total messages in SQS
    • each instance launches 10-30 processes
  3. Each process reads SQS message, download data from file path, generate content, upload content back to S3, create new SQS message with file path (to notify client)

Breaking AWS

  • Requested 1000 instances, got ~850
    • Scale across regions to work-around
  • Thousands of process/threads writing to S3 at once using sequential keys overwhelms Amazon's ability to keep up
    • Randomize beginning of S3's key to keep from generating sequential keys
  • Very complex MySQL jobs crashes RDS
    • Disable automated backups to turn off binary logging