Web Excursions 2021-12-12
Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region
We want to provide you with some additional information about the service disruption that occurred in the Northern Virginia (US-EAST-1) Region on December 7th, 2021.
Internals of the AWS network.
AWS makes use of an internal network to host foundational services including monitoring, internal DNS, authorization services, and parts of the EC2 control plane.
we connect this network with multiple geographically isolated networking devices
These networking devices provide additional routing and network address translation
that allow AWS services to communicate between the internal network and the main AWS network.
At 7:30 AM PST,
an automated activity to scale capacity of one of the AWS services hosted in the main AWS network
triggered an unexpected behavior from a large number of clients inside the internal network.
This resulted in a large surge of connection activity
that overwhelmed the networking devices between the internal network and the main AWS network,
resulting in delays for communication between these networks.
These delays increased latency and errors for services communicating between these networks,
resulting in even more connection attempts and retries.
This led to persistent congestion and performance issues on the devices connecting the two networks.
Operators instead relied on logs to understand what was happening and initially identified elevated internal DNS errors.
This change improved the availability of several impacted services by reducing load on the impacted networking devices,
but did not fully resolve the AWS service impact or eliminate the congestion.
[Analysis]
the impact on internal monitoring limited our ability to understand the problem.
our internal deployment systems, which run in our internal network, were impacted
because many AWS services on the main AWS network and AWS customer applications were still operating normally,
we wanted to be extremely deliberate while making changes to avoid impacting functioning workloads
[Measures]
We immediately disabled the scaling activities that triggered this event
and will not resume them until we have deployed all remediations.
a latent issue prevented these clients from adequately backing off during this event.
We have also deployed additional network configuration that protects potentially impacted networking devices even in the face of a similar congestion event.
As the impact to services during this event all stemmed from a single root cause,
we opted to provide updates via a global banner on the Service Health Dashboard,
which we have since learned makes it difficult for some customers to find information about this issue.
Summit For Democracy | The Flip Side
The left worries that the US is not dealing with its own domestic threats to democracy
but supports the general goal of the summit
Zack Beauchamp, Vox
When you read about near misses [of failure of a democracy], two factors prove decisive again and again:
when a society’s elite stands up to an authoritarian faction, using their power to beat it back, and
when the mass public organizes and demonstrates in favor of democracy…
In the United States, we are experiencing failures on both the elite and mass public level
Fred Kaplan, Slate
Pressed on this point, White House Press Secretary Jen Psaki said, ‘Inclusion on an invitation is not a stamp of approval on their approach to democracy—nor is exclusion a stamp … of disapproval.’
Well, then, what does inclusion or exclusion indicate?
Mainly it’s about international politics.
Russia and China weren’t invited because the summit’s premise—the contest between democracy and authoritarianism—is largely a contest against them.
Iraq was invited because the U.S. backed Iraq in a long war and because Israel can’t be the only Middle Eastern country to attend.
Pakistan was invited because India was, and it would offend the Pakistanis—still needed for support in counterterrorism operations—to leave them out
The right supports the general goal of the summit
but criticizes the invitation list
and notes the undemocratic measures that have been put in place since the pandemic began.