Triangle.rb Notes - AWS
Bring AWS To Its Knees
2014-05-13 by Robbie Allen (Meetup)
Automated Story Writing
- Combines writing and programming skills
- Making millions of articles not sound automated is difficult
Workflow
- Client uploads data to S3 and creates an SQS message with file path
- System autoscales based on total messages in SQS
- each instance launches 10-30 processes
- Each process reads SQS message, download data from file path, generate content, upload content back to S3, create new SQS message with file path (to notify client)
Breaking AWS
- Requested 1000 instances, got ~850
- Scale across regions to work-around
- Thousands of process/threads writing to S3 at once using sequential keys overwhelms Amazon's ability to keep up
- Randomize beginning of S3's key to keep from generating sequential keys
- Very complex MySQL jobs crashes RDS
- Disable automated backups to turn off binary logging