Dad of two; Java and web developer.
123 stories

Peak performance: How retailers used Google Cloud during Black Friday/Cyber Monday

1 Share

At Google Cloud, we work with businesses in a range of industries, and we’ve seen nearly every business experience peak events when their online traffic skyrockets. For retailers, their peak events are Black Friday and Cyber Monday (or BFCM)—the period right after Thanksgiving in the U.S., when holiday shopping starts. The weekend kicks off the all-important holiday shopping season of November and December, when an estimated20% of all annual retail sales occur.

During an average day, online retail sales in the U.S. total about $1.4 billion,CNET reports. In contrast, on Black Friday 2018, U.S. online sales totaled $6.22 billion (up 24% from 2017). Cyber Monday 2018 sales surged to $7.9 billion (up 19% from 2017)—the biggest online sales day ever in the U.S., according toAdobe Analytics.  

Traffic to retailers’ mobile and shopping apps surges to levels unmatched during the rest of the year, and availability or scalability issues can result in millions of dollars of lost sales. Every year, there are well-publicized retail website crashes, so avoiding downtime—along with the accompanying reputation damage, unhappy customers and stressed, overworked IT teams—is particularly important for retailers.

We know that a solid technology infrastructure is the foundation for retailers to stay ahead of demand and succeed during this busy season. Beyond that, though, support for that infrastructure is essential. Support isn’t just activated if something goes wrong. Support for an event like Black Friday and Cyber Monday involves preparation well ahead of time, and includes testing, architecture reviews, capacity planning, operational drills, and war rooms during the event itself. We took a prescriptive approach to BFCM support, setting expectations and ownership early (more than six months ahead), to understand what each retail customer needed, both on their side and from our team.  

We’ll go through the steps that helped our retail customers have a fruitful and disaster-free season. These steps can generally help you prepare for your own peak event. We’ll also describe how one large-scale retail platform in particular—Shopify—had a successful BFCM using Google Cloud.  

Preparing to support retailers on Black Friday/Cyber Monday

We started planning for Black Friday and Cyber Monday for our retail customers in the spring of 2018 to align with their typical preparation timeline. We formed a task force composed of representatives from Google Cloud’s Professional Services, Customer Engineering,Support,Customer Reliability Engineering (CRE), and Product and Engineering teams. We met regularly to strategize, develop tactics, and execute on those tactics with the goal of making sure Google team members and our GCP retail customers were well-prepared.

We focused on a few key technology areas where planning could help prevent any issues.

1. Early capacity planning

As early as May 2018, our account teams began reaching out to GCP retail customers. We discussed high-level planning, such as their particular holiday shopping objectives and the infrastructure capacity they might need to meet those goals.

We worked closely with retailers to review their architectures and advise on techniques to forecast and plan for increases in capacity before Black Friday, since scalability is essential when planning for traffic spikes. We conducted tests across teams and services, and stress-tested systems to uncover any constraints or weaknesses and remediate as needed. Those tailored preparations paid off across the board. With GCP capacity status firmly green—available—throughout Black Friday and Cyber Monday, shoppers visiting our retail customers’ sites could make their purchases without running into a slow or unresponsive site.

2. Reliability testing

Identifying potential reliability issues in a “pre-mortem” (an important component of CRE) was another preemptive step we took. Early on, our CRE team partnered with our retail customers to analyze the reliability of their infrastructures, and run through tabletop exercises to see how well-prepared the customer was in the face of a failure. In some cases, the Professional Services team helped perform load testing to make sure retailers’ platforms could handle expected levels of peak traffic, and in others we encouraged regular load testing and evaluation. And given how important mobile commerce has become, we also tested the performance and reliability of customers’ mobile apps. We also employed Apigee’s API monitoring tools to ensure API stability. We’ve seen APIs become more important in retail technology, since they allow more flexible, microservice-based e-commerce sites.

3. Operational war rooms

“What could possibly go wrong?”

That’s the million-dollar question to ask before a big IT event. We got together with our retail customers’ IT and engineering teams to explore and test for possible worst-case scenarios, like an entire site crash. We created a central Black Friday/Cyber Monday war room staffed with senior-level, experienced Googlers from the Professional Services, Support, and Site Reliability Engineering (SRE) teams. This team of first responders was prepared to use real-time communications to stay connected and address any problems as soon as they arose. This was in addition to understanding customer and vendor integrations and making sure escalation paths were defined ahead of time, so that customer expectations were clear for various channels.

During that weekend, we doubled the number of on-call support staff available to retail customers. In some cases, we placed account teams on-site at GCP and Apigee retail customer locations to help as needed. We monitored whether any retail customers were starting to have reliability or latency problems. If something needed to be triaged, the war room team kicked into action, tackling issues and advising on next steps. The Google war room team also had direct, open access to Google engineers and executives for additional support.

Apigee team members kept a close eye on API traffic during the Black Friday period. The number of API calls for Apigee’s customers (excluding those who host the platform on-premises) grew 95% compared to the same span of time in 2017. Peak API traffic running through Apigee more than doubled, from 48,000 transactions per second (TPS) to 108,000 TPS this year, and the platform remained 99.999% available.

How retailers sailed through Black Friday and Cyber Monday

One of our retail partners, Shopify, is an e-commerce platform supporting more than 600,000 independent retailers. The complexity of managing all those storefronts makes predicting holiday site traffic and sales spikes even more challenging. Shopify provides a platform with 99.98% uptime, and calls BFCM their annual “World Cup” event.

Shopify’s platform is made up of many internal services and interaction points with third-party providers, such as payment gateways and shipping carriers. Each of those dependencies has to be reliable and perform well for BFCM to go off without a hitch.

In 2017, on Black Friday and Cyber Monday, only about 10% of Shopify’s stores ran on GCP. The rest were hosted from their own data center. In 2018, Shopify went all-in on GCP as its infrastructure provider, with 100% of its retailers running on our platform.

Shopify was an early adopter of Docker containers and now usesGoogle Kubernetes Engine as itscontainer management system, along with theCloud Storage unified object storage service.

Shopify Production Engineers began working side-by-side with Google’s BFCM team months before the holiday shopping season. We collaborated on capacity planning so Shopify would have the right capacity buffer needed to accommodate an even bigger peak load than they had in 2017, and helped diagnose and fix potential performance problems, such as network latency.

During the rest of the year, our Shopify account team stayed highly engaged with Shopify engineers on Slack, Google Hangouts Chat, and other real-time communications tools. For Black Friday and Cyber Monday, we increased our communication further and dispatched Googlers to Shopify’s own war room in Toronto.

“As we went into BFCM 2018, we no longer had data center capacity to fall back on,” says Camilo Lopez, Director of Production Engineering at Shopify. “But we were confident that with Google Cloud, we had the extra support and strong technology foundation needed for a successful Black Friday and Cyber Monday. The big event came and went without incident. Our merchants collectively sold over$1.5 billion USD in merchandise that weekend,up from $1 billion in 2017.”

This BFCM weekend was a record breaker for Shopify, with a peak of nearly 11,000 orders created per minute and around 100,000 requests per second being served for extended periods during the weekend. Overall, most system metrics followed a pattern of 1.8 times what they were in 2017.

Cloud planning and support make for stress-free events  

By following the above strategies, you can be ready for whatever comes your way, whether it’s a huge, unanticipated traffic spike or a major uptick in sales you count on every year. And that brings benefits for customers and your IT teams. After this year’s successful BFCM, a staff member from one of our newer retailers sent us a note of thanks and remarked that 2018 was the first time in years that he was able to enjoy Thanksgiving dinner with his family.

To achieve your own low-stress peak events, plan and prepare before the event. Consider how your service might fail, how you’d detect these failures, and how you’d react to them. Perform tests to find potential weaknesses. Choose good measures of your customers’ experience, and closely monitor your infrastructure during the event. Do a post-mortem immediately afterwards to make the next big event is even smoother. Find out more here on adopting these strategies for your organization.

And of course, our GCP support team is here to help during these events, both planned and unplanned. If you have a large event where we can help, get in touch with your Technical Account Manager, or your Google Cloud account team.

Read the whole story
8 days ago
Christchurch, New Zealand
Share this story

The Gamification of Rhetoric

2 Comments and 15 Shares

I posted a thought earlier on Twitter today and I’ll repost it here in non-tweet form:

It’s really frustrating to me that more people don’t understand that racist/alt-right people have gamified their rhetoric; they’re not interested in discussion, they’re slapping down cards from a “Debate: The Gathering” stack, and the only goal is taking heads.

They gamify their rhetoric because essentially this shit is a low-stake game for them, whereas for other people it’s their actual lives. That’s an advantage they have. If they lose, they shuffle their cards and go on to the next thing. If others lose, their life takes a hit.

And because their rhetorical strategy is essentially card-based, actual knowledge of issues is unimportant and probably a hinderance. They don’t want or need to understand the issues that affect others, they just need you to play their game so they can win.

I don’t have time anymore to diddle about with children who think other people’s lives are some sort of turn-based game, especially when all they want is to hurt other people. And it bothers me more people, especially those with power, don’t understand this shit.

I’m not going to tell people not to engage with these chuckleheads. But don’t engage with them on their terms. Engage with them on your own. One, they hate that, and two, it exposes what they’re doing as a pointless, hateful exercise, and them as awful people.

In sum: Understand what these folks are doing. Refuse to play along. And if you choose, point out to others the hollowness of their game. Because their “game” is to hurt other people, and then go on to the next target. Their game is other people’s lives.

Read the whole story
167 days ago
Christchurch, New Zealand
167 days ago
Share this story
1 public comment
165 days ago
Also, gamified rhetoric is easier to automate, easier to scale, than honest/empathic/actual debate. You aren't going to get far in most FPSes if you spend so much time on individual one-on-one matches and ignoring the vast scale of the mob surrounding you. Get to higher ground first if you are going to snipe; learn when you sometimes should bring an automatic weapon to a large scale battle.
Louisville, Kentucky

The United States of Guns

2 Comments and 9 Shares

Like many of you, I read the news of a single person killing at least 10 people in Santa Fe, Texas today. While this is an outrageous and horrifying event, it isn’t surprising or shocking in any way in a country where more than 33,000 people die from gun violence each year.

America is a stuck in a Groundhog Day loop of gun violence. We’ll keep waking up, stuck in the same reality of oppression, carnage, and ruined lives until we can figure out how to effect meaningful change. I’ve collected some articles here about America’s dysfunctional relationship with guns, most of which I’ve shared before. Change is possible — there are good reasons to control the ownership of guns and control has a high likelihood of success — but how will our country find the political will to make it happen?

An armed society is not a free society:

Arendt offers two points that are salient to our thinking about guns: for one, they insert a hierarchy of some kind, but fundamental nonetheless, and thereby undermine equality. But furthermore, guns pose a monumental challenge to freedom, and particular, the liberty that is the hallmark of any democracy worthy of the name — that is, freedom of speech. Guns do communicate, after all, but in a way that is contrary to free speech aspirations: for, guns chasten speech.

This becomes clear if only you pry a little more deeply into the N.R.A.’s logic behind an armed society. An armed society is polite, by their thinking, precisely because guns would compel everyone to tamp down eccentric behavior, and refrain from actions that might seem threatening. The suggestion is that guns liberally interspersed throughout society would cause us all to walk gingerly — not make any sudden, unexpected moves — and watch what we say, how we act, whom we might offend.

We’re sacrificing America’s children to “our great god Gun”:

Read again those lines, with recent images seared into our brains — “besmeared with blood” and “parents’ tears.” They give the real meaning of what happened at Sandy Hook Elementary School Friday morning. That horror cannot be blamed just on one unhinged person. It was the sacrifice we as a culture made, and continually make, to our demonic god. We guarantee that crazed man after crazed man will have a flood of killing power readily supplied him. We have to make that offering, out of devotion to our Moloch, our god. The gun is our Moloch. We sacrifice children to him daily — sometimes, as at Sandy Hook, by directly throwing them into the fire-hose of bullets from our protected private killing machines, sometimes by blighting our children’s lives by the death of a parent, a schoolmate, a teacher, a protector. Sometimes this is done by mass killings (eight this year), sometimes by private offerings to the god (thousands this year).

The gun is not a mere tool, a bit of technology, a political issue, a point of debate. It is an object of reverence. Devotion to it precludes interruption with the sacrifices it entails. Like most gods, it does what it will, and cannot be questioned. Its acolytes think it is capable only of good things. It guarantees life and safety and freedom. It even guarantees law. Law grows from it. Then how can law question it?

Roger Ebert on the media’s coverage of mass shootings:

Let me tell you a story. The day after Columbine, I was interviewed for the Tom Brokaw news program. The reporter had been assigned a theory and was seeking sound bites to support it. “Wouldn’t you say,” she asked, “that killings like this are influenced by violent movies?” No, I said, I wouldn’t say that. “But what about ‘Basketball Diaries’?” she asked. “Doesn’t that have a scene of a boy walking into a school with a machine gun?” The obscure 1995 Leonardo Di Caprio movie did indeed have a brief fantasy scene of that nature, I said, but the movie failed at the box office (it grossed only $2.5 million), and it’s unlikely the Columbine killers saw it.

The reporter looked disappointed, so I offered her my theory. “Events like this,” I said, “if they are influenced by anything, are influenced by news programs like your own. When an unbalanced kid walks into a school and starts shooting, it becomes a major media event. Cable news drops ordinary programming and goes around the clock with it. The story is assigned a logo and a theme song; these two kids were packaged as the Trench Coat Mafia. The message is clear to other disturbed kids around the country: If I shoot up my school, I can be famous. The TV will talk about nothing else but me. Experts will try to figure out what I was thinking. The kids and teachers at school will see they shouldn’t have messed with me. I’ll go out in a blaze of glory.”

In short, I said, events like Columbine are influenced far less by violent movies than by CNN, the NBC Nightly News and all the other news media, who glorify the killers in the guise of “explaining” them. I commended the policy at the Sun-Times, where our editor said the paper would no longer feature school killings on Page 1. The reporter thanked me and turned off the camera. Of course the interview was never used. They found plenty of talking heads to condemn violent movies, and everybody was happy.

Jill Lepore on the United States of Guns:

There are nearly three hundred million privately owned firearms in the United States: a hundred and six million handguns, a hundred and five million rifles, and eighty-three million shotguns. That works out to about one gun for every American. The gun that T. J. Lane brought to Chardon High School belonged to his uncle, who had bought it in 2010, at a gun shop. Both of Lane’s parents had been arrested on charges of domestic violence over the years. Lane found the gun in his grandfather’s barn.

The United States is the country with the highest rate of civilian gun ownership in the world. (The second highest is Yemen, where the rate is nevertheless only half that of the U.S.) No civilian population is more powerfully armed. Most Americans do not, however, own guns, because three-quarters of people with guns own two or more. According to the General Social Survey, conducted by the National Policy Opinion Center at the University of Chicago, the prevalence of gun ownership has declined steadily in the past few decades. In 1973, there were guns in roughly one in two households in the United States; in 2010, one in three. In 1980, nearly one in three Americans owned a gun; in 2010, that figure had dropped to one in five.

A Land Without Guns: How Japan Has Virtually Eliminated Shooting Deaths:

The only guns that Japanese citizens can legally buy and use are shotguns and air rifles, and it’s not easy to do. The process is detailed in David Kopel’s landmark study on Japanese gun control, published in the 1993 Asia Pacific Law Review, still cited as current. (Kopel, no left-wing loony, is a member of the National Rifle Association and once wrote in National Review that looser gun control laws could have stopped Adolf Hitler.)

To get a gun in Japan, first, you have to attend an all-day class and pass a written test, which are held only once per month. You also must take and pass a shooting range class. Then, head over to a hospital for a mental test and drug test (Japan is unusual in that potential gun owners must affirmatively prove their mental fitness), which you’ll file with the police. Finally, pass a rigorous background check for any criminal record or association with criminal or extremist groups, and you will be the proud new owner of your shotgun or air rifle. Just don’t forget to provide police with documentation on the specific location of the gun in your home, as well as the ammo, both of which must be locked and stored separately. And remember to have the police inspect the gun once per year and to re-take the class and exam every three years.

Australia’s gun laws stopped mass shootings and reduced homicides, study finds:

From 1979 to 1996, the average annual rate of total non-firearm suicide and homicide deaths was rising at 2.1% per year. Since then, the average annual rate of total non-firearm suicide and homicide deaths has been declining by 1.4%, with the researchers concluding there was no evidence of murderers moving to other methods, and that the same was true for suicide.

The average decline in total firearm deaths accelerated significantly, from a 3% decline annually before the reforms to a 5% decline afterwards, the study found.

In the 18 years to 1996, Australia experienced 13 fatal mass shootings in which 104 victims were killed and at least another 52 were wounded. There have been no fatal mass shootings since that time, with the study defining a mass shooting as having at least five victims.

From The Onion, ‘No Way To Prevent This,’ Says Only Nation Where This Regularly Happens:

At press time, residents of the only economically advanced nation in the world where roughly two mass shootings have occurred every month for the past eight years were referring to themselves and their situation as “helpless.”

But America is not Australia or Japan. Dan Hodges said on Twitter a few years ago:

In retrospect Sandy Hook marked the end of the US gun control debate. Once America decided killing children was bearable, it was over.

This can’t be the last word on guns in America. We have to do better than this for our children and everyone else whose lives are torn apart by guns. But right now, we are failing them miserably, and Hodges’ words ring with the awful truth that all those lives and our diminished freedom & equality are somehow worth it to the United States as a society.

Tags: guns   USA
Read the whole story
243 days ago
Christchurch, New Zealand
243 days ago
Share this story
2 public comments
217 days ago
Collection of articles about gun violence in America. I find especially interesting the idea that gun rights effectively lower the value of freedom of speech.
Texas, USA
246 days ago

A link is a promise.

1 Share

A few years ago, I came across a "content curation" service that billed itself as the easiest way to share your "thought leadership." You'd enter a few search terms, hook up your Twitter account, and start sharing articles.

It was a very generous definition of "content curation." But describing it as thought leadership was laughable.

That said, leadership communications can absolutely include sharing and commenting on useful resources and thought-provoking perspectives you find. That's especially true when you share with care (and due attribution!) and offer your own reflections on what you're linking to.

But a lot of people don't. There's a school of thought out there that stresses post frequency, clicks likes and shares over things that may be less easy to measure but have far greater impact — like attention, trust and authority.

And some folks I would have thought would know better are churning out links to the most shameless clickbait in the name of "thought leadership." I wrote about this a few years ago — but if anything, the problem just seems to be getting worse.

So let's be clear about something: a link is a promise.

When you share a link, we're taking your word for it on what we'll find when we get there.

Share a link that turns out to be clickbait… or a mediocre regurgitation of advice we've all read a thousand times before… and you're breaking that promise.

Share a headline that promises an exciting story about one thing… but only deals with that one thing in passing… and you're breaking that promise.

Share a story that turns out to be a hoax, or wildly exaggerated, or just plain not worth a reader's time… and you're breaking that promise.

Break enough promises, and you'll start breaking relationships. And no number of clicks, likes or shares is worth that.

?Photo by JJ Ying on Unsplash

The post A link is a promise. appeared first on Rob Cottingham.

Read the whole story
317 days ago
Christchurch, New Zealand
Share this story

Memcrashed - Major amplification attacks from UDP port 11211

1 Share
Memcrashed - Major amplification attacks from UDP port 11211

Memcrashed - Major amplification attacks from UDP port 11211CC BY-SA 2.0 image by David Trawin

Over last couple of days we've seen a big increase in an obscure amplification attack vector - using the memcached protocol, coming from UDP port 11211.

In the past, we have talked a lot about amplification attacks happening on the internet. Our most recent two blog posts on this subject were:

The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

Memcrashed - Major amplification attacks from UDP port 11211

Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.


Obscure amplification attacks happen all the time. We often see "chargen" or "call of duty" packets hitting our servers.

A discovery of a new amplification vector though, allowing very great amplification, happens rarely. This new memcached UDP DDoS is definitely in this category.

The DDosMon from Qihoo 360 monitors amplification attack vectors and this chart shows recent memcached/11211 attacks:

Memcrashed - Major amplification attacks from UDP port 11211

The number of memcached attacks was relatively flat, until it started spiking just a couple days ago. Our charts also confirm this, here are attacks in packets per second over the last four days:

Memcrashed - Major amplification attacks from UDP port 11211

While the packets per second count is not that impressive, the bandwidth generated is:

Memcrashed - Major amplification attacks from UDP port 11211

At peak we've seen 260Gbps of inbound UDP memcached traffic. This is massive for a new amplification vector. But the numbers don't lie. It's possible because all the reflected packets are very large. This is how it looks in tcpdump:

$ tcpdump -n -t -r memcrashed.pcap udp and port 11211 -c 10
IP > UDP, length 13
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 1400
IP > UDP, length 13

The majority of packets are 1400 bytes in size. Doing the math 23Mpps x 1400 bytes gives 257Gbps of bandwidth, exactly what the chart shows.

Memcached does UDP?

I was surprised to learn that memcached does UDP, but there you go! The protocol specification shows that it's one of the best protocols to use for amplification ever! There are absolutely zero checks, and the data WILL be delivered to the client, with blazing speed! Furthermore, the request can be tiny and the response huge (up to 1MB).

Launching such an attack is easy. First let's implant a large payload on an exposed memcached server, let's call the key y. Then, just request this key with UDP:

$ echo -en "\x00\x00\x00\x00\x00\x01\x00\x00get y\r\n" | nc -q1 -u 11211 | wc -c

Tcpdump shows the traffic:

$ sudo tcpdump -ni eth0 port 11211 -t
IP > UDP, length 15
IP > UDP, length 1400
IP > UDP, length 1400
...(repeated hundreds times)...

15 bytes of request triggered 134KB of response. This is amplification factor of 10,000x! In practice we've seen a 15 byte request result in a 750kB response (that's a 51,200x amplification).

Source IPs

The vulnerable memcached servers are all around the globe, with higher concentration in North America and Europe. Here is a map of the source IPs we've seen in each of our 120+ points of presence:

Memcrashed - Major amplification attacks from UDP port 11211

Interestingly our datacenters in EWR, HAM and HKG see disproportionally large numbers of attacking IPs. This is because most of the vulnerable servers are located in major hosting providers. The AS numbers of the IPs that we've seen:

│ 578 │ AS16276 │ OVH                                          │
│ 468 │ AS14061 │ DIGITALOCEAN-ASN - DigitalOcean, LLC         │
│ 231 │ AS7684  │ SAKURA-A SAKURA Internet Inc.                │
│ 199 │ AS9370  │ SAKURA-B SAKURA Internet Inc.                │
│ 165 │ AS12876 │ AS12876                                      │
│ 119 │ AS9371  │ SAKURA-C SAKURA Internet Inc.                │
│ 104 │ AS16509 │ AMAZON-02 -, Inc.                 │
│ 102 │ AS24940 │ HETZNER-AS                                   │
│  81 │ AS26496 │ AS-26496-GO-DADDY-COM-LLC -, LLC │
│  74 │ AS36351 │ SOFTLAYER - SoftLayer Technologies Inc.      │
│  65 │ AS20473 │ AS-CHOOPA - Choopa, LLC                      │
│  49 │ AS49981 │ WORLDSTREAM                                  │
│  48 │ AS51167 │ CONTABO                                      │
│  48 │ AS33070 │ RMH-14 - Rackspace Hosting                   │
│  45 │ AS19994 │ RACKSPACE - Rackspace Hosting                │
│  44 │ AS60781 │ LEASEWEB-NL-AMS-01 Netherlands               │
│  42 │ AS45899 │ VNPT-AS-VN VNPT Corp                         │
│  41 │ AS2510  │ INFOWEB FUJITSU LIMITED                      │
│  40 │ AS7506  │ INTERQ GMO Internet,Inc                      │
│  35 │ AS62567 │ DIGITALOCEAN-ASN-NY2 - DigitalOcean, LLC     │
│  31 │ AS8100  │ ASN-QUADRANET-GLOBAL - QuadraNet, Inc        │
│  30 │ AS14618 │ AMAZON-AES -, Inc.                │
│  30 │ AS31034 │ ARUBA-ASN                                    │

Most of the memcached servers we've seen were coming from AS16276 - OVH, AS14061 - Digital Ocean and AS7684 - Sakura.

In total we've seen only 5,729 unique source IPs of memcached servers. We're expecting to see much larger attacks in future, as Shodan reports 88,000 open memcached servers:

Memcrashed - Major amplification attacks from UDP port 11211

Let's fix it up

It's necessary to fix this and prevent further attacks. Here is a list of things that should be done.

Memcached Users

If you are using memcached, please disable UDP support if you are not using it. On memcached startup you can specify --listen to listen only to localhost and -U 0 to disable UDP completely. By default memcached listens on INADDR_ANY and runs with UDP support ENABLED. Documentation:

You can easily test if your server is vulnerable by running:

$ echo -en "\x00\x00\x00\x00\x00\x01\x00\x00stats\r\n" | nc -q1 -u 11211
STAT pid 21357
STAT uptime 41557034
STAT time 1519734962

If you see non-empty response (like the one above), your server is vulnerable.

System administrators

Please ensure that your memcached servers are firewalled from the internet! To test whether they can be accessed using UDP I recommend the nc example above, to verify if TCP is closed run nmap:

$ nmap TARGET -p 11211 -sU -sS --script memcached-info
Starting Nmap 7.30 ( ) at 2018-02-27 12:44 UTC
Nmap scan report for xxxx
Host is up (0.011s latency).
11211/tcp open          memcache
| memcached-info:
|   Process ID           21357
|   Uptime               41557524 seconds
|   Server time          2018-02-27T12:44:12
|   Architecture         64 bit
|   Used CPU (user)      36235.480390
|   Used CPU (system)    285883.194512
|   Current connections  11
|   Total connections    107986559
|   Maximum connections  1024
|   TCP Port             11211
|   UDP Port             11211
|_  Authentication       no
11211/udp open|filtered memcache

Internet Service Providers

Memcrashed - Major amplification attacks from UDP port 11211

In order to defeat such attacks in future, we need to fix vulnerable protocols and also IP spoofing. As long as IP spoofing is permissible on the internet, we'll be in trouble.

Help us out by tracking who is behind these attacks. We must know not who has problematic memcached servers, but who sent them queries in the first place. We can't do this without your help!


Please please please: Stop using UDP. If you must, please don't enable it by default. If you do not know what an amplification attack is I hereby forbid you from ever typing SOCK_DGRAM into your editor.

We've been down this road so many times. DNS, NTP, Chargen, SSDP and now memcached. If you use UDP, you must always respond with strictly a smaller packet size then the request. Otherwise your protocol will be abused. Also remember that people do forget to set up a firewall. Be a nice citizen. Don't invent a UDP-based protocol that lacks authentication of any kind.

That's all

It's anyone's guess how large the memcached attacks will become before we clean the vulnerable servers up. There were already rumors of 0.5Tbps amplifications in the last few days, and this is just a start.

Finally, you are OK if you are a Cloudflare customer. Cloudflare's Anycast architecture works well to distribute the load in case of large amplification attacks, and unless your origin IP is exposed, you are safe behind Cloudflare.

Dealing with DDoS attacks sound interesting? Join our world famous team in London, Austin, San Francisco and our elite office in Warsaw, Poland.

Read the whole story
326 days ago
Christchurch, New Zealand
Share this story

RPGS, Veblen, and Isms

1 Share
Go read Theory of Leisure Class, though it may depress you

Back when I was working on my masters, I had to take some PhD classes in econometrics. To do that I had to go to the economics department, fill out paperwork and ask them to let me into the classes (Since I wasn’t a PhD student, the department was rightly worried I might find the classes highly challenging. I did eek out a passing grade though.) While waiting to be seen, one of the professors walking by struck up a conversation with me. After about 5 minutes, he said to me (I paraphrase): “I like you. You should read Veblen’s Theory of the Leisure Class. You’re just the right type of weird to enjoy it.” Turns out he was the professor I was going to have daily 6 hour long classes with that summer. He was pretty weird himself, but in a good way.

Wanting to be more into economics than I really was, and wanting to impress the professor, I picked up a copy of the book. (by the way, it’s public domain and you can find it for free online if you care to read it. PDF EPUB and other formats). Published in 1899, it’s a bit of a slog full of unfamiliar words and archaic structure, but that professor was right, I was just the type of weird to enjoy it. Oddly enough, for an economic treatise written seventy years before the creation of role playing games, it provides an interesting lens to view them through and insights into the nature of them.

(Veblen can also be used to explain a lot of human behavior both on the personal and macro scale. As this is not a political site, I leave you to draw your own conclusions on that front. I’m 100% certain that you can find multiple sites on the topic elsewhere on the internet if you want to read up on it.)

Veblen’s main thesis is that despite the complaints of the economists of his day, and non-economists still today, human behavior is more or less rational despite appearances to the contrary. Of course, keep in mind that humans are notoriously bad at evaluating small short term vs larger long term gains, and are often self benefit driven, both a legacy from when life was nasty, brutish, and short.  To explain seeming lapses in rationality, Veblen explains the concept of agency and status and posits that all so called irrational behavior are attempts to gather one or the other:

  • Agency: Veblen’s concept of agency is related to but distinct from the commonly discussed gaming concept of player agency. It consists of two parts- the ability to take powerful actions which hold the appearance of consciousness (appearance of consciousness means things like powerful storms have agency even though we now know they’re not conscious) and the ability to impose your will on others with agency. Thus the farmer does not have agency, or has very little, because he imposes his will on plants, which do not have agency; but the hunter, shaman, and warrior do because they impose their will on beasts, storms, and other warriors. Note that how one imposes this will is irrelevant. The warrior who imposes his will through brute violence and the rogue who imposes his will through clever action both have equal agency.
    In your typical game, there is a ladder of agency. PCs, foes and important NPCs have agency, players and GM influence them with their agency, and above all of that, playing the part of the capricious whimsical gods are the dice which, much like the storms of our ancestors, seem to have a will all their own.
  • Status: Status is the acknowledgement, deference, and special treatment we give to those with greater agency. In early or brutal civilizations it’s not mouthing off to the warriors because if you do they might just run you through. In more abstract examples, it’s not mouthing off to the king because even though you’re pretty sure you could take him in a fight, his soldiers will run you through and his bureaucrats will see to it that your taxes are doubled.

Veblen explains that the primary way of gathering agency is the heroic exploit, and that the primary currency used is courage and cunning. Enter the typical archetype of the warrior, hunter, and trickster. But over time he argues it becomes more difficult to show off just how heroic you are. Trophies and treasure gained in exploit can only take you so far. Enter two more ways to show how much agency you have: conspicuous consumption and conspicuous leisure. Eventually heroes have so much accumulated riches from exploit that they can afford to blow those resources on ever more extravagant goods and services and on long periods doing nothing of value. Remember back to the carousing rules from the early Conan RPG and the early DnD rules for building your own keep, crafting magic items and doing spell research. All these things signal to the rest of the world just how formidable you are. Different RPG archetypes focus on different mixes of these four building blocks of agency and status.

  • Courage: imposing agency via force of arms and brute strength
  • Cunning: imposing agency via cleverness or guile
  • Conspicuous Consumption: showing status via display of resources gathered in heroic exploit
  • Conspicuous Leisure: showing status via not performing otherwise productive work when not involved in exploit

Veblen says that the expression of these qualities has social value in that you get special treatment for having or displaying them. They are largely expressed in the form of what Veblen terms invidious comparison. I am stronger than that person, I am smarter than this other person, more moral than this person, etc. More abstractly, they can be expressed through trophies of exploits, what resources you have, what groups you associate with, who you work for etc. In the extreme, one who successfully gathers large amounts of resources via exploit can’t spend it all themselves, and may even hire retainers whose only job is to spend their lord’s money.  Interestingly, some of these expressions are cultural and change over time. Recent studies have shown that modern Americans are less inclined to give others status for having lots of leisure time and lots of expensive possessions, which were major status symbols in Veblen’s time and are still (apparently) status symbols in other places. (article 1 and article 2).

What in particular struck me while reading through the book was that a large part of RPGs was an attempt to emulate characters with far more agency, and thus status, than players generally have. That’s of course an oversimplification. Different people play RPGs for lots of different reasons, and you can probably find some that don’t hinge on playing with agency, but I can’t think of any . Feel free to correct me if you like. What this means is that in theory RPG experiences featuring the right balance of character agency (enough agency to impose their will on others but not so much that opponents don’t appear to have agency of their own) should be a more satisfying play experience, and giving the characters markers of agency and status should also lead to satisfying play. In fact, one could write an entire book digging through Theory Of The Leisure Class and pulling out elements to enhance games*.

As a side note, plenty of playstyles and even problem playstyles can be chalked up to Veblen’s agency. On the innocuous side is pushing for more PC vs NPC agency. On the destructive side is outright player vs player or player vs GM agency. If you’ve ever wondered what the player who enjoys making everyone else’s experience miserable is getting out of it, this is it: imposing their agency over that of the other players and GM, and if they can’t be convinced to pick a more suitable target, they should be shown the door before it gets worse.

However, even though Veblen explains RPG motivations very well it turns out that taken to extremes, pursuing agency and status also explains a variety of the more unsavory elements of humanity: Slavery, discrimination, sexism, racism etc. are all big markers for agency and status. Some of them, like slavery, are a direct application of agency. One person has the agency to literally own another. Isms are a little more abstract but boil down to an imbalance of status between two groups. If you belong to the group an imbalance favors, you benefit from that status boost. You can see the evidence of this in plenty of the fiction that inspired the RPG hobby. Conan, for example has a scene in at least every other story where he forces himself on a woman and halfway through the act, she stops struggling to get him off her and starts struggling to get her panties off, because he’s just so manly and good at kissing it changes her mind, and Robert E Howard doesn’t even hold a candle to John Norman in that regard. A lot of early space opera sci-fi was heroic American vs evil communist Asians . . . innnn spaaaaace! There are still plenty of vestiges of this era sticking around too. All dwarves are racist against elves, elves are racist against . . . mostly dwarves, but also everyone. Lots of sentient species are kill on sight morally repugnant, and every half-orc gets the one good half orc treatment.

So, TLDR: RPGs are big power fantasies about imposing your will on others and everyone fawning all over you for doing it, and taken too far the natural extension of this is some of the worst parts of the human experience. Lovely.

Let me put on my devil’s advocate hat for a moment and just take things too far. If the above is true, wouldn’t featuring all of those uncomfortable bits of bad human behavior and allowing PCs to wallow in them make your game a more powerful expression of what RPGs are all about, and wouldn’t your game be better for it? Shouldn’t I be saying: “All this stuff adds to the fantasy that is explicitly the entire point of the game, so put it in your game, lots of it!”

That would be an awfully weird position to take, wouldn’t it? Turns out it’s a tempting position, but not necessarily a good one. First, there’s more than enough opportunity for heroic exploit, agency and status in your average game. Dipping into the more problematic aspects of agency just isn’t necessary. Even if you wanted to, most are very sensitive subjects and should be handled with care and only with complete group buy-in. And of course if you don’t take sensitive issues seriously, like other problem play styles, you run the very real risk of trading in game status for out of game status (i.e.: you get labeled a jerk and no one wants to play with you because you don’t respect others’ boundaries.)

But, can dealing with these difficult extremes of agency and status in your game be done without issues? Yes, but it’s nothing groundbreaking. On the extreme end, with player buy-in you can absolutely play an evil game with PCs that push the envelope. In a more traditional game, you can include these issues but reserve them for villainous NPCs that heroic PCs put a stop to. You can include lesser versions of them (the faux racism of all dwarves and elves). You can reserve involuntary traits indicating a lesser status for non sentient or irredeemably evil beings (golems and demons are OK to pick on because even though their status is linked to race one is a mindless automaton, the other is literally made of evil. If your golems and demons don’t follow that mold, things get hazier. This is where the “all orcs are evil” problems come from. Are orcs inherently evil like in Lord of the Rings, or do they fall in a spectrum like most fantasy races?) In fact, these could lead to some interesting introspective role playing moments. Nothing groundbreaking. All common enough that this is not the first time you’ve heard of them.

So that’s it. That’s the short version of viewing role playing games through the lens of Veblen’s Theory Of The Leisure Class, and the problems that come with it. Hopefully it’s an interesting perspective.



Writing said book has been on my “someday projects” list for quite some time. Also feel free to let me know if that sounds lame and I should let it languish or if it sounds awesome and I should move it up the list

Read the whole story
367 days ago
Christchurch, New Zealand
Share this story
Next Page of Stories