Cloud Hosting Blogs

New – VPC Traffic Mirroring – Capture & Inspect Network Traffic

Amazon Web Services Blog -

Running a complex network is not an easy job. In addition to simply keeping it up and running, you need to keep an ever-watchful eye out for unusual traffic patterns or content that could signify a network intrusion, a compromised instance, or some other anomaly. VPC Traffic Mirroring Today we are launching VPC Traffic Mirroring. This is a new feature that you can use with your existing Virtual Private Clouds (VPCs) to capture and inspect network traffic at scale. This will allow you to: Detect Network & Security Anomalies – You can extract traffic of interest from any workload in a VPC and route it to the detection tools of your choice. You can detect and respond to attacks more quickly than is possible with traditional log-based tools. Gain Operational Insights – You can use VPC Traffic Mirroring to get the network visibility and control that will let you make security decisions that are better informed. Implement Compliance & Security Controls – You can meet regulatory & compliance requirements that mandate monitoring, logging, and so forth. Troubleshoot Issues – You can mirror application traffic internally for testing and troubleshooting. You can analyze traffic patterns and proactively locate choke points that will impair the performance of your applications. You can think of VPC Traffic Mirroring as a “virtual fiber tap” that gives you direct access to the network packets flowing through your VPC. As you will soon see, you can choose to capture all traffic or you can use filters to capture the packets that are of particular interest to you, with an option to limit the number of bytes captured per packet. You can use VPC Traffic Mirroring in a multi-account AWS environment, capturing traffic from VPCs spread across many AWS accounts and then routing it to a central VPC for inspection. You can mirror traffic from any EC2 instance that is powered by the AWS Nitro system (A1, C5, C5d, M5, M5a, M5d, R5, R5a, R5d, T3, and z1d as I write this). Getting Started with VPC Traffic Mirroring Let’s review the key elements of VPC Traffic Mirroring and then set it up: Mirror Source – An AWS network resource that exists within a particular VPC, and that can be used as the source of traffic. VPC Traffic Mirroring supports the use of Elastic Network Interfaces (ENIs) as mirror sources. Mirror Target – An ENI or Network Load Balancer that serves as a destination for the mirrored traffic. The target can be in the same AWS account as the Mirror Source, or in a different account for implementation of the central-VPC model that I mentioned above. Mirror Filter – A specification of the inbound or outbound (with respect to the source) traffic that is to be captured (accepted) or skipped (rejected). The filter can specify a protocol, ranges for the source and destination ports, and CIDR blocks for the source and destination. Rules are numbered, and processed in order within the scope of a particular Mirror Session. Traffic Mirror Session – A connection between a mirror source and target that makes use of a filter. Sessions are numbered, evaluated in order, and the first match (accept or reject) is used to determine the fate of the packet. A given packet is sent to at most one target. You can set this up using the VPC Console, EC2 CLI, or the EC2 API, with CloudFormation support in the works. I’ll use the Console. I already have ENI that I will use as my mirror source and destination (in a real-world use case I would probably use an NLB destination): The MirrorTestENI_Source and MirrorTestENI_Destination ENIs are already attached to suitable EC2 instances. I open the VPC Console and scroll down to the Traffic Mirroring items, then click Mirror Targets: I click Create traffic mirror target: I enter a name and description, choose the Network Interface target type, and select my ENI from the menu. I add a Blog tag to my target, as is my practice, and click Create: My target is created and ready to use: Now I click Mirror Filters and Create traffic mirror filter. I create a simple filter that captures inbound traffic on three ports (22, 80, and 443), and click Create: Again, it is created and ready to use in seconds: Next, I click Mirror Sessions and Create traffic mirror session. I create a session that uses MirrorTestENI_Source, MainTarget, and MyFilter, allow AWS to choose the VXLAN network identifier, and indicate that I want the entire packet mirrored: And I am all set. Traffic from my mirror source that matches my filter is encapsulated as specified in RFC 7348 and delivered to my mirror target. I can then use tools like Suricata to capture, analyze, and visualize it. Things to Know Here are a couple of things to keep in mind: Sessions Per ENI – You can have up to three active sessions on each ENI. Cross-VPC – The source and target ENIs can be in distinct VPCs as long as they are peered to each other or connected through Transit Gateway. Scaling & HA – In most cases you should plan to mirror traffic to a Network Load Balancer and then run your capture & analysis tools on an Auto Scaled fleet of EC2 instances behind it. Bandwidth – The replicated traffic generated by each instance will count against the overall bandwidth available to the instance. If traffic congestion occurs, mirrored traffic will be dropped first. From our Partners During the beta test of VPC Traffic Mirroring, a collection of AWS partners were granted early access and provided us with tons of helpful feedback. Here are some of the blog posts that they wrote in order to share their experiences: Big Switch Networks – AWS Traffic Monitoring with Big Monitoring Fabric. Blue Hexagon – Unleashing Deep Learning-Powered Threat Protection for AWS. Corelight – Bring Network Security Monitoring to the Cloud with Corelight and Amazon VPC Traffic Mirroring. cPacket Networks – It’s Cloudy Today with a High Chance of Packets. ExtraHop – ExtraHop brings Network Detection & Response to the cloud-first enterprise with Amazon Web Services. Fidelis – Expanding Traffic Visibility Natively in AWS with Fidelis Network Sensors and Amazon VPC Traffic Mirroring. Flowmon – Flowmon Taking Advantage of Amazon VPC Traffic Mirroring. Gigamon – Gigamon GigaVUE Cloud Suite for Amazon Web Services and New Amazon VPC Traffic Mirroring. IronNet – IronDefense and IronDome Support for Amazon VPC Traffic Mirroring. JASK – Amazon VPC Traffic Mirroring. Netscout – AWS Traffic Mirroring Contributes to NETSCOUT’s Smart Data Intelligence. Nubeva – Decrypted Visibility With Amazon VPC Traffic Mirroring. Palo Alto Networks – See the Unseen in AWS Mirrored Traffic With the VM-Series. Riverbed – SteelCentral AppResponse Cloud to Support New Amazon VPC Traffic Mirroring. Vectra – Securing your AWS workloads with Vectra Cognito. Now Available VPC Traffic Mirroring is available now and you can start using it today in all commercial AWS Regions except Asia Pacific (Sydney), China (Beijing), and China (Ningxia). Support for those regions will be added soon. You pay an hourly fee (starting at $0.015 per hour) for each mirror source; see the VPC Pricing page for more info. — Jeff;  

AWS Security Hub Now Generally Available

Amazon Web Services Blog -

I’m a developer, or at least that’s what I tell myself while coming to terms with being a manager. I’m definitely not an infosec expert. I’ve been paged more than once in my career because something I wrote or configured caused a security concern. When systems enable frequent deploys and remove gatekeepers for experimentation, sometimes a non-compliant resource is going to sneak by. That’s why I love tools like AWS Security Hub, a service that enables automated compliance checks and aggregated insights from a variety of services. With guardrails like these in place to make sure things stay on track, I can experiment more confidently. And with a single place to view compliance findings from multiple systems, infosec feels better about letting me self-serve. With cloud computing, we have a shared responsibility model when it comes to compliance and security. AWS handles the security of the cloud: everything from the security of our data centers up to the virtualization layer and host operating system. Customers handle security in the cloud: the guest operating system, configuration of systems, and secure software development practices. Today, AWS Security Hub is out of preview and available for general use to help you understand the state of your security in the cloud. It works across AWS accounts and integrates with many AWS services and third-party products. You can also use the Security Hub API to create your own integrations. Getting Started When you enable AWS Security Hub, permissions are automatically created via IAM service-linked roles. Automated, continuous compliance checks begin right away. Compliance standards determine these compliance checks and rules. The first compliance standard available is the Center for Internet Security (CIS) AWS Foundations Benchmark. We’ll add more standards this year. The results of these compliance checks are called findings. Each finding tells you severity of the issue, which system reported it, which resources it affects, and a lot of other useful metadata. For example, you might see a finding that lets you know that multi-factor authentication should be enabled for a root account, or that there are credentials that haven’t been used for 90 days that should be revoked. Findings can be grouped into insights using aggregation statements and filters. Integrations In addition to the Compliance standards findings, AWS Security Hub also aggregates and normalizes data from a variety of services. It is a central resource for findings from AWS Guard Duty, Amazon Inspector, Amazon Macie, and from 30 AWS partner security solutions. AWS Security Hub also supports importing findings from custom or proprietary systems. Findings must be formatted as AWS Security Finding Format JSON objects. Here’s an example of an object I created that meets the minimum requirements for the format. To make it work for your account, switch out the AwsAccountId and the ProductArn. To get your ProductArn for custom findings, replace REGION and ACCOUNT_ID in the following string: arn:aws:securityhub:REGION:ACCOUNT_ID:product/ACCOUNT_ID/default. { "Findings": [{ "AwsAccountId": "12345678912", "CreatedAt": "2019-06-13T22:22:58Z", "Description": "This is a custom finding from the API", "GeneratorId": "api-test", "Id": "us-east-1/12345678912/98aebb2207407c87f51e89943f12b1ef", "ProductArn": "arn:aws:securityhub:us-east-1:12345678912:product/12345678912/default", "Resources": [{ "Type": "Other", "Id": "i-decafbad" }], "SchemaVersion": "2018-10-08", "Severity": { "Product": 2.5, "Normalized": 11 }, "Title": "Security Finding from Custom Software", "Types": [ "Software and Configuration Checks/Vulnerabilities/CVE" ], "UpdatedAt": "2019-06-13T22:22:58Z" }] } Then I wrote a quick node.js script that I named importFindings.js to read this JSON file and send it off to AWS Security Hub via the AWS JavaScript SDK. const fs = require('fs'); // For file system interactions const util = require('util'); // To wrap fs API with promises const AWS = require('aws-sdk'); // Load the AWS SDK AWS.config.update({region: 'us-east-1'}); // Create our Security Hub client const sh = new AWS.SecurityHub(); // Wrap readFile so it returns a promise and can be awaited const readFile = util.promisify(fs.readFile); async function getFindings(path) { try { // wait for the file to be read... let fileData = await readFile(path); // ...then parse it as JSON and return it return JSON.parse(fileData); } catch (error) { console.error(error); } } async function importFindings() { // load the findings from our file const findings = await getFindings('./findings.json'); try { // call the AWS Security Hub BatchImportFindings endpoint response = await sh.batchImportFindings(findings).promise(); console.log(response); } catch (error) { console.error(error); } } // Engage! importFindings(); A quick run of node importFindings.js results in { FailedCount: 0, SuccessCount: 1, FailedFindings: [] }. And now I can see my custom finding in the Security Hub console: Custom Actions AWS Security Hub can integrate with response and remediation workflows through the use of custom actions. With custom actions, a batch of selected findings is used to generate CloudWatch events. With CloudWatch Rules, these events can trigger other actions such as sending notifications via a chat system or paging tool, or sending events to a visualization service. First, we open Settings from the AWS Security Console, and select Custom Actions. Add a custom action and note the ARN. Then we create a CloudWatch Rule using the custom action we created as a resource in the event pattern, like this: { "source": [ "aws.securityhub" ], "detail-type": [ "Security Hub Findings - Custom Action" ], "resources": [ "arn:aws:securityhub:us-west-2:123456789012:action/custom/DoThing" ] } Our CloudWatch Rule can have many different kinds of targets, such as Amazon Simple Notification Service (SNS) Topics, Amazon Simple Queue Service (SQS) Queues, and AWS Lambda functions. Once our action and rule are in place, we can select findings, and then choose our action from the Actions dropdown list. This will send the selected findings to Amazon CloudWatch Events. Those events will match our rule, and the event targets will be invoked. Important Notes AWS Config must be enabled for Security Hub compliance checks to run. AWS Security Hub is available in 15 regions: US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), Canada (Central), South America (São Paulo), Europe (Ireland), Europe (London), Europe (Paris), Europe (Frankfurt), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Seoul), and Asia Pacific (Mumbai). AWS Security Hub does not transfer data outside of the regions where it was generated. Data is not consolidated across multiple regions. AWS Security Hub is already the type of service that I’ll enable on the majority of the AWS accounts I operate. As more compliance standards become available this year, I expect it will become a standard tool in many toolboxes. A 30-day free trial is available so you can try it out and get an estimate of what your costs would be. As always, we want to hear your feedback and understand how you’re using AWS Security Hub. Stay in touch, and happy building! — Brandon

AWS Control Tower – Set up & Govern a Multi-Account AWS Environment

Amazon Web Services Blog -

Earlier this month I met with an enterprise-scale AWS customer. They told me that they are planning to go all-in on AWS, and want to benefit from all that we have learned about setting up and running AWS at scale. In addition to setting up a Cloud Center of Excellence, they want to set up a secure environment for teams to provision development and production accounts in alignment with our recommendations and best practices. AWS Control Tower Today we are announcing general availability of AWS Control Tower. This service automates the process of setting up a new baseline multi-account AWS environment that is secure, well-architected, and ready to use. Control Tower incorporates the knowledge that AWS Professional Service has gained over the course of thousands of successful customer engagements, and also draws from the recommendations found in our whitepapers, documentation, the Well-Architected Framework, and training. The guidance offered by Control Tower is opinionated and prescriptive, and is designed to accelerate your cloud journey! AWS Control Tower builds on multiple AWS services including AWS Organizations, AWS Identity and Access Management (IAM) (including Service Control Policies), AWS Config, AWS CloudTrail, and AWS Service Catalog. You get a unified experience built around a collection of workflows, dashboards, and setup steps. AWS Control Tower automates a landing zone to set up a baseline environment that includes: A multi-account environment using AWS Organizations. Identity management using AWS Single Sign-On (SSO). Federated access to accounts using AWS SSO. Centralize logging from AWS CloudTrail, and AWS Config stored in Amazon S3. Cross-account security audits using AWS IAM and AWS SSO. Before diving in, let’s review a couple of key Control Tower terms: Landing Zone – The overall multi-account environment that Control Tower sets up for you, starting from a fresh AWS account. Guardrails – Automated implementations of policy controls, with a focus on security, compliance, and cost management. Guardrails can be preventive (blocking actions that are deemed as risky), or detective (raising an alert on non-conformant actions). Blueprints – Well-architected design patterns that are used to set up the Landing Zone. Environment – An AWS account and the resources within it, configured to run an application. Users make requests (via Service Catalog) for new environments and Control Tower uses automated workflows to provision them. Using Control Tower Starting from a brand new AWS account that is both Master Payer and Organization Master, I open the Control Tower Console and click Set up landing zone to get started: AWS Control Tower will create AWS accounts for log arching and for auditing, and requires email addresses that are not already associated with an AWS account. I enter two addresses, review the information within Service permissions, give Control Tower permission to administer AWS resources and services, and click Set up landing zone: The setup process runs for about an hour, and provides status updates along the way: Early in the process, Control Tower sends a handful of email requests to verify ownership of the account, invite the account to participate in AWS SSO, and to subscribe to some SNS topics. The requests contain links that I must click in order for the setup process to proceed. The second email also requests that I create an AWS SSO password for the account. After the setup is complete, AWS Control Tower displays a status report: The console offers some recommended actions: At this point, the mandatory guardrails have been applied and the optional guardrails can be enabled: I can see the Organizational Units (OUs) and accounts, and the compliance status of each one (with respect to the guardrails):   Using the Account Factory The navigation on the left lets me access all of the AWS resources created and managed by Control Tower. Now that my baseline environment is set up, I can click Account factory to provision AWS accounts for my teams, applications, and so forth. The Account factory displays my network configuration (I’ll show you how to edit it later), and gives me the option to Edit the account factory network configuration or to Provision new account: I can control the VPC configuration that is used for new accounts, including the regions where VPCs are created when an account is provisioned: The account factory is published to AWS Service Catalog automatically. I can provision managed accounts as needed, as can the developers in my organization. I click AWS Control Tower Account Factory to proceed: I review the details and click LAUNCH PRODUCT to provision a new account: Working with Guardrails As I mentioned earlier, Control Tower’s guardrails provide guidance that is either Mandatory or Strongly Recommended: Guardrails are implemented via an IAM Service Control Policy (SCP) or an AWS Config rule, and can be enabled on an OU-by-OU basis: Now Available AWS Control Tower is available now and you can start using it today in the US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland) Regions, with more to follow. There is no charge for the Control Tower service; you pay only for the AWS resources that it creates on your behalf. In addition to adding support for more AWS regions, we are working to allow you to set up a parallel landing zone next to an existing AWS account, and to give you the ability to build and use custom guardrails. — Jeff;  

New – UDP Load Balancing for Network Load Balancer

Amazon Web Services Blog -

The Network Load Balancer is designed to handle tens of millions of requests per second while maintaining high throughput at ultra low latency, with no effort on your part (read my post, New Network Load Balancer – Effortless Scaling to Millions of Requests per Second to learn more). In response to customer requests, we have added several new features since the late-2017 launch, including cross-zone load balancing, support for resource-based and tag-based permissions, support for use across an AWS managed VPN tunnel, the ability to create a Network Load Balancer using the AWS Elastic Beanstalk Console, support for Inter-Region VPC Peering, and TLS Termination. UDP Load Balancing Today we are adding support for another frequent customer request, the ability to load balance UDP traffic. You can now use Network Load Balancers to deploy connectionless services for online gaming, IoT, streaming, media transfer, and native UDP applications. If you are hosting DNS, SIP, SNMP, Syslog, RADIUS, and other UDP services in your own data center, you can now move the services to AWS. You can also deploy services to handle Authentication, Authorization, and Accounting, often known as AAA. You no longer need to maintain a fleet of proxy servers to ingest UDP traffic, and you can now use the same load balancer for both TCP and UDP traffic. You can simplify your architecture, reduce your costs, and increase your scalability. Creating a UDP Network Load Balancer I can create a Network Load Balancer with UDP support using the Console, CLI (create-load-balancer), API (CreateLoadBalancer), or a CloudFormation template (AWS::ElasticLoadBalancing::LoadBalancer), as usual. The console lets me choose the desired load balancer; I click the Create button underneath Network Load Balancer: I name my load balancer, choose UDP from the protocol menu, and select a port (514 is for Syslog): I already have suitable EC2 instances in us-east-1b and us-east-1c so I’ll use those AZs: Then I set up a target group for the UDP protocol on port 514: I choose my instances and click Add to registered: I review my settings on the next page, and my new UDP Load Balancer is ready to accept traffic within a minute or so (the state starts out as provisioning and transitions to active when it is ready): I’ll test this out by configuring my EC2 instances as centralized Syslogd servers. I simply edit the configuration file (/etc/rsyslog.conf) on the instances to make them listen on port 514, and restart the service: Then I launch another EC2 instance and configure it to use my NLB endpoint: And I can see log entries in my servers (ip-172-31-29-40 is my test instance): I did have to do make one small configuration change in order to get this to work! Using UDP to check on the health of a service does not really make sense, so I clicked override and specified a health check on port 80 instead: In a real-world scenario you would want to build a TCP-style health check into your service, of course. And, needless to say, I would run a custom implementation of Syslog that stores the log messages centrally and in a highly durable form. Things to Know Here are a couple of things to know about this important new NLB feature: Supported Targets – UDP on Network Load Balancers is supported for Instance target types (IP target types and PrivateLink are not currently supported). Health Checks – As I mentioned above, health checks must be done using TCP, HTTP, or HTTPS. Multiple Protocols – A single Network Load Balancer can handle both TCP and UDP traffic. You can add another listener to an existing load balancer to gain UDP support, as long as you use distinct ports. In situations such as DNS where you need support for both TCP and UDP on the same port, you can set up a multi-protocol target group and a multi-protocol listener (use TCP_UDP for the listener type and the TargetGroup). New CloudWatch Metrics – The existing CloudWatch metrics (ProcessedBytes, ActiveFlowCount, and NewFlowCount) now represent the aggregate traffic processed by the TCP, UDP, and TLS listeners on a given Network Load Balancer. Available Now This feature is available now and you can start using it today in all commercial AWS Regions. For pricing, see the Elastic Load Balancing Pricing page. — Jeff;  

Now Available: New C5 instance sizes and bare metal instances

Amazon Web Services Blog -

Amazon EC2 C5 instances are very popular for running compute-heavy workloads like batch processing, distributed analytics, high-performance computing, machine/deep learning inference, ad serving, highly scalable multiplayer gaming, and video encoding. Today, we are happy to expand the Amazon EC2 C5 family with: New larger virtualized instance sizes: 12xlarge and 24xlarge, A bare metal option. The new C5 instance sizes run on Intel’s Second Generation Xeon Scalable processors (code-named Cascade Lake) with sustained all-core turbo frequency of 3.6GHz and maximum single core turbo frequency of 3.9GHz. The new processors also enable a new feature called Intel Deep Learning Boost, a capability based on the AVX-512 instruction set. Thanks to the new Vector Neural Network Instructions (AVX-512 VNNI), deep learning frameworks will speed up typical machine learning operations like convolution, and automatically improve inference performance over a wide range of workloads. These instances are also based on the AWS Nitro System, with dedicated hardware accelerators for EBS processing (including crypto operations), the software-defined network inside of each Virtual Private Cloud (VPC), and ENA networking. New C5 instance sizes: 12xlarge and 24xlarge Previously, the largest C5 instance available was C5.18xlarge, with 72 logical processors and 144 GiB of memory. As you can see, the new 24xlarge size increases available resources by 33%, in order to scale up and reduce the time required to compute intensive tasks. Instance Name Logical Processors Memory EBS-Optimized Bandwidth Network Bandwidth c5.12xlarge 48 96 GiB 7 Gbps 12 Gbps c5.24xlarge 96 192 GiB 14 Gbps 25 Gbps Bare metal C5 Just like for existing bare metal instances (M5, M5d, R5, R5d, z1d, and so forth), your operating system runs directly on the underlying hardware with direct access to the processor. As described in a previous blog post, you can leverage bare metal instances for applications that: do not want to take the performance hit of nested virtualization, need access to physical resources and low-level hardware features, such as performance counters and Intel VT that are not always available or fully supported in virtualized environments, are intended to run directly on the hardware, or licensed and supported for use in non-virtualized environments. Bare metal instances can also take advantage of Elastic Load Balancing, Auto Scaling, Amazon CloudWatch, and other AWS services. Instance Name Logical Processors Memory EBS-Optimized Bandwidth Network Bandwidth c5.metal 96 192 GiB 14 Gbps 25 Gbps Now Available! You can start using these new instances today in the following regions: US East (N. Virginia), US West (Oregon), Europe (Ireland), Europe (London), Europe (Frankfurt), Europe (Stockholm), Europe (Paris), Asia Pacific (Singapore), Asia Pacific (Sydney), and AWS GovCloud (US-West). Please send us feedback and help us build the next generation of compute-optimized instances. — Julien;

Amazon S3 Update – SigV2 Deprecation Period Extended & Modified

Amazon Web Services Blog -

Every request that you make to the Amazon S3 API must be signed to ensure that it is authentic. In the early days of AWS we used a signing model that is known as Signature Version 2, or SigV2 for short. Back in 2012, we announced SigV4, a more flexible signing method, and made it the sole signing method for all regions launched after 2013. At that time, we recommended that you use it for all new S3 applications. Last year we announced that we would be ending support for SigV2 later this month. While many customers have updated their applications (often with nothing more than a simple SDK update), to use SigV4, we have also received many requests for us to extend support. New Date, New Plan In response to the feedback on our original plan, we are making an important change. Here’s the summary: Original Plan – Support for SigV2 ends on June 24, 2019. Revised Plan – Any new buckets created after June 24, 2020 will not support SigV2 signed requests, although existing buckets will continue to support SigV2 while we work with customers to move off this older request signing method. Even though you can continue to use SigV2 on existing buckets, and in the subset of AWS regions that support SigV2, I encourage you to migrate to SigV4, gaining some important security and efficiency benefits in the process. The newer signing method uses a separate, specialized signing key that is derived from the long-term AWS access key. The key is specific to the service, region, and date. This provides additional isolation between services and regions, and provides better protection against key reuse. Internally, our SigV4 implementation is able to securely cache the results of authentication checks; this reduces latency and adds to the overall resiliency of your application. To learn more, read Changes in Signature Version 4. Identifying Use of SigV2 S3 has been around since 2006 and some of the code that you or your predecessors wrote way back then might still be around, dutifully making requests that are signed with SigV2. You can use CloudTrail Data Events or S3 Server Access Logs to find the old-school requests and target the applications for updates: CloudTrail Data Events – Look for the SignatureVersion element within the additionalDataElement of each CloudTrail event entry (read Using AWS CloudTrail to Identify Amazon S3 Signature Version 2 Requests to learn more). S3 Server Access Logs – Look for the SignatureVersion element in the logs (read Using Amazon S3 Access Logs to Identify Signature Version 2 Requests to learn more). Updating to SigV4 “Do we need to change our code?” The Europe (Frankfurt), US East (Ohio), Canada (Central), Europe (London), Asia Pacific (Seoul), Asia Pacific (Mumbai), Europe (Paris), China (Ningxia), Europe (Stockholm), Asia Pacific (Osaka Local), AWS GovCloud (US-East), and Asia Pacific (Hong Kong) Regions were launched after 2013, and support SigV4 but not SigV2. If you have code that accesses S3 buckets in that region, it is already making exclusive use of SigV4. If you are using the latest version of the AWS SDKs, you are either ready or just about ready for the SigV4 requirement on new buckets beginning June 24, 2020. If you are using an older SDK, please check out the detailed version list at Moving from Signature Version 2 to Signature Version 4 for more information. There are a few situations where you will need to make some changes to your code. For example, if you are using pre-signed URLs with the AWS Java, JavaScript (node.js), or Python SDK, you need to set the correct region and signature version in the client configuration. Also, be aware that SigV4 pre-signed URLs are valid for a maximum of 7 days, while SigV2 pre-signed URLs can be created with a maximum expiry time that could be many weeks or years in the future (in almost all cases, using time-limited URLs is a much better practice). Using SigV4 will improve your security profile, but might also mandate a change in the way that you create, store, and use the pre-signed URLs. While using long-lived pre-signed URLs was easy and convenient for developers, using SigV4 with URLs that have a finite expiration is a much better security practice. If you are using Amazon EMR, you should upgrade your clusters to version 5.22.0 or later so that all requests to S3 are made using SigV4 (see Amazon EMR 5.x Release Versions for more info). If your S3 objects are fronted by Amazon CloudFront and you are signing your own requests, be sure to update your code to use SigV4. If you are using Origin Access Identities to restrict access to S3, be sure to include the x-amz-content-sha256 header and the proper regional S3 domain endpoint. We’re Here to Help The AWS team wants to help make your transition to SigV4 as smooth and painless as possible. If you run in to problems, I strongly encourage you to make use of AWS Support, as described in Getting Started with AWS Support. You can also Discuss this Post on Reddit! — Jeff;  

Amazon Personalize is Now Generally Available

Amazon Web Services Blog -

Today, we’re happy to announce that Amazon Personalize is available to all AWS customers. Announced in preview at AWS re:Invent 2018, Amazon Personalize is a fully-managed service that allows you to create private, customized personalization recommendations for your applications, with little to no machine learning experience required. Whether it is a timely video recommendation inside an application or a personalized notification email delivered just at the right moment, personalized experiences, based on your data, deliver more relevant experiences for customers often with much higher business returns. The task of developing an efficient recommender system is quite challenging: building, optimizing, and deploying real-time personalization requires specialized expertise in analytics, applied machine learning, software engineering, and systems operations. Few organizations have the knowledge, skills, and experience to overcome these challenges, and simple rule-based systems become brittle and costly to maintain as new products and promotions are introduced, or customer behavior changes. For over 20 years, Amazon.com has perfected machine learning models that provide personalized buying experiences from product discovery to checkout. With Amazon Personalize, we are bringing developers that same capability to build custom models without having to deal with the complexity of infrastructure and machine learning that typically accompanies these types of solutions. With Amazon Personalize, you provide the unique signals in your activity data (page views, signups, purchases, and so forth) along with optional customer demographic information (age, location, etc.). You then provide the inventory of the items you want to recommend, such as articles, products, videos, or music as an example. Then, entirely under the covers, Amazon Personalize will process and examine the data, identify what is meaningful, select the right algorithms, and train and optimize a personalization model that is customized for your data, and accessible via an API. All data analyzed by Amazon Personalize is kept private and secure and only used for your customized recommendations. The resulting models are yours and yours alone. With a single API call, you can make recommendations for your users and personalize the customer experience, driving more engagement, higher conversion, and increased performance on marketing campaigns. Domino’s Pizza, for instance, is using Amazon Personalize to deliver customized communications such as promotional deals through their digital properties. Sony Interactive Entertainment uses Personalize with Amazon SageMaker to automate and accelerate their machine learning development and drive more effective personalization at scale. Personalize is like having your own Amazon.com machine learning personalization team at your beck and call, 24 hours a day. Introducing Amazon Personalize Amazon Personalize can make recommendations based on your historical data stored in Amazon S3, or on streaming data sent in real-time by your applications, or on both. This gives customers a lot of flexibility to build recommendation solutions. For instance, you could build an initial recommender based on historical data, and retrain it periodically when you’ve accumulated enough live events. Alternatively, if you have no historical data to start from, you could ingest events for a while, and then build your recommender. Having covered historical data in my previous blog post, I will focus on ingesting live events this time. The high-level process looks like this: Create a dataset group in order to store events sent by your application. Create an interaction dataset and define its schema (no data is needed at this point). Create an event tracker in order to send events to Amazon Personalize. Start sending events to Amazon Personalize. Select a recommendation recipe, or let Amazon Personalize pick one for you thanks to AutoML. Create a solution, i.e. train the recipe on your dataset. Create a campaign and start recommending items. Creating a dataset group Let’s say we’d like to capture a click stream of movie recommendations. Using the the first time setup wizard, we create a dataset group to store these events. Here, let’s assume we don’t have any historical data to start with: all events are generated by the click stream, and are ingested using the event ingestion SDK. Creating a dataset group just requires a name. Then, we have to create the interaction dataset, which shows how users are interacting with items (liking, clicking, etc.). Of course, we need to define a schema describing the data: here, we’ll simply use the default schema provided by Amazon Personalize. Optionally, we could now define an import job, in order to add historical data to the data set: as mentioned above, we’ll skip this step as all data will come from the stream. Configuring the event tracker The next step is to create the event tracker that will allow us to send streaming events to the dataset group. After a minute or so, our tracker is ready. Please take note of the tracking id: we’ll need it to send events. Creating the dataset When Amazon Personalize creates an event tracker, it automatically creates a new dataset in the dataset group associated with the event tracker. This dataset has a well-defined schema storing the following information: user_id and session_id: these values are defined by your application. tracking_id: the event tracker id. timestamp, item_id, event_type, event_value: these values describe the event itself and must be passed by your application. Real-time events can be sent to this dataset in two different ways: Server-side, via the AWS SDK: please note ingestion can happen from any source, whether your code is hosted inside of AWS (e.g. in Amazon EC2 or AWS Lambda) or outside. With the AWS Amplify JavaScript library. Let’s look at both options. Sending real-time events with the AWS SDK This is a very easy process: we can simply use the PutEvents API to send either a single event, or a list of up to 10 events. Of course, we could use any of the AWS SDKs: since my favourite language is Python, this is how we can send events using the boto3 SDK. import boto3 personalize_events = boto3.client('personalize-events') personalize_events.put_events( trackingId = <TRACKING_ID>, userId = <USER_ID>, sessionId = <SESSION_ID>, eventList = [ { "eventId": "event1", "sentAt": 1549959198, "eventType": "rating", "properties": """{\"itemId\": \"123\", \"eventValue\": \"4\"}""" }, { "eventId": "event2", "sentAt": 1549959205, "eventType": "rating", "properties": """{\"itemId\": \"456\", \"eventValue\": \"2\"}""" } ] ) In our application, we rated movie 123 as a 4, and movie 456 as a 2. Using the appropriate tracking identifier, we sent two Events to our event tracker: eventId: an application-specific identifier. sentAt: a timestamp, matching the timestamp property defined in the schema. This value seconds since the Unix Epoch (1 January 1970 00:00:00.000 UTC), and is independent of any particular time zone. eventType: the event type, matching the event_type property defined in the schema, properties: the item id and event value, matching the item_id and event_value properties defined in the schema. Here’s a similar code snippet in Java. List<Event> eventList = new ArrayList<>(); eventList.add(new Event().withProperties(properties).withType(eventType)); PutEventsRequest request = new PutEventsRequest() .withTrackingId(<TRACKING_ID>) .withUserId(<USER_ID>) .withSessionId(<SESSION_ID>) .withEventList(eventList); client.putEvents(request) You get the idea! Sending real-time events with AWS Amplify AWS Amplify is a JavaScript library that makes it easy to create, configure, and implement scalable mobile and web apps powered by AWS. It’s integrated with the event tracking service in Amazon Personalize. A couple of setup steps are required before we can send events. For the sake of brevity, please refer to these detailed instructions in the Amazon Personalize documentation: Create an identity pool in Amazon Cognito, in order to authenticate users. Configure the Amazon Personalize plug-in with the pool id and tracker id. Once this is taken care of, we can send events to Amazon Personalize. We can still use any text string for event types, but please note that a couple of special types are available: Identify lets you send the userId for a particular user to Amazon Personalize. The userId then becomes an optional parameter in subsequent calls. MediaAutoTrack automatically calculates the play, pause and resume position for media events, and Amazon Personalize uses the position as event value. Here is how to send some sample events with AWS Amplify: Analytics.record({ eventType: "Identify", properties: { "userId": "<USER_ID>" } }, "AmazonPersonalize"); Analytics.record({ eventType: "<EVENT_TYPE>", properties: { "itemId": "<ITEM_ID>", "eventValue": "<EVENT_VALUE>" } }, "AmazonPersonalize"); Analytics.record({ eventType: "MediaAutoTrack", properties: { "itemId": "<ITEM_ID>", "domElementId": "MEDIA DOM ELEMENT ID" } }, "AmazonPersonalize"); As you can see, this is pretty simple as well. Creating a recommendation solution Now that we know how to ingest events, let’s define how our recommendation solution will be trained. We first need to select a recipe, which is much more than an algorithm: it also includes predefined feature transformations, initial parameters for the algorithm as well as automatic model tuning. Thus, recipes remove the need to have expertise in personalization. Amazon Personalize comes with several recipes suitable for different use cases. Still, if you’re new to machine learning, you may wonder which one of these recipes best fits your use case. No worry: as mentioned earlier, Amazon Personalize supports AutoML, a new technique that automatically searches for the most optimal recipe, so let’s enable it. While we’re at it, let’s also ask Amazon Personalize to automatically tune recipe parameters. All of this is very straightforward in the AWS console: as you’ll probably want to automate from now on, let’s use the AWS CLI instead. $ aws personalize create-solution \ --name jsimon-movieclick-solution \ --perform-auto-ml --perform-hpo \ --dataset-group-arn $DATASET_GROUP_ARN Now we’re ready to train the solution. No servers to worry about, training takes places on fully-managed infrastructure. $ aws personalize create-solution-version \   --solution-arn $SOLUTION_ARN Once training is complete, we can use the solution version to create a recommendation campaign. Deploying a recommendation campaign Still no servers to worry about! In fact, campaigns scale automatically according to incoming traffic: we simply need to define the minimum number of transactions per second (TPS) that we want to support. This number is used to size the initial fleet for hosting the model. It also impacts how much you will be charged for recommendations ($0.20 per TPS-hour). Here, I’m setting that parameter to 10, which means that I will initially be charged $2 per hour. If traffic exceeds 10 TPS, Personalize will scale up, increasing my bill according to the new TPS setting. Once traffic drops, Personalize will scale down, but it won’t go below my minimum TPS setting. $ aws personalize create-campaign \ --name jsimon-movieclick-campaign \ --min-provisioned-tps 10 \ --solution-version-arn $SOLUTION_VERSION_ARN Should you later need to update the campaign with a new solution version, you can simply use the UpdateCampaign API and pass the ARN of the new solution version. Once the campaign has been deployed, we can quickly test that it’s able to recommend new movies. Recommending new items in real-time I don’t think this could be simpler: just pass the id of the user and receive recommendations. $ aws personalize-rec get-recommendations \ --campaign-arn $CAMPAIGN_ARN \ --user-id 123 --query "itemList[*].itemId" ["1210", "260", "2571", "110", "296", "1193", ...] At this point, we’re ready to integrate our recommendation model in your application. For example, a web application would have to implement the following steps to display a list of recommended movies: Use the GetRecommendations API in our favorite language to invoke the campaign and receive movie recommendation for a given user, Read movie metadata from a backend (say, image URL, title, genre, release date, etc.), Generate HTML code to be rendered in the user’s browser. Amazon Personalize in action Actually, my colleague Jake Wells has built a web application recommending books. Using an open dataset containing over 19 million book reviews, Jake first used a notebook hosted on Amazon SageMaker to clean and prepare the data. Then, he trained a recommendation model with Amazon Personalize, and wrote a simple web application demonstrating the recommendation process. This is a really cool project, which would definitely be worthy of its own blog post! Available now! Whether you work with historical data or event streams, a few simple API calls are all it takes to train and deploy recommendation models. Zero machine learning experience is required, so please visit aws.amazon.com/personalize, give it a try and let us know what you think. Amazon Personalize is available in the following regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Singapore), and EU (Ireland) The service is also part of the AWS free tier. For the first two months after sign-up, you will be offered: 1. Data processing and storage: up to 20 GB per month 2. Training: up to 100 training hours per month 3. Prediction: up to 50 TPS-hours of real-time recommendations per month We’re looking forward to your feedback! — Julien;

Meet the Newest AWS Heroes! June 2019

Amazon Web Services Blog -

At the heart of the global AWS community are the builders and influencers whose passion for AWS leads them to actively share their technical know-how with others. The most prominent of these AWS community leaders are recognized as AWS Heroes. Heroes regularly share their extensive AWS knowledge online via blogs, social media, and open source contributions, and in person by speaking at industry conferences and running workshops or AWS-focused User Groups. Today we are thrilled to introduce the latest cohort of AWS Heroes: Anton Babenko – Oslo, Norway Community Hero Anton Babenko is a long time developer and CTO who runs a consulting company Betajob AS in Norway. He helps companies around the globe build solutions using AWS, and specializes in infrastructure as code, DevOps, and reusable infrastructure components. Anton spends a large amount of his time as an open-source contributor on various Terraform & AWS projects, including terraform-aws-modules and modules.tf. He enjoys solving real cloud architecture tasks, figuring out generic solutions by getting to the core, and making them available as open source to let the whole AWS community benefit. Anton also co-founded and co-organizes AWS, DevOps, and HashiCorp User Groups in Norway, DevOpsDays Oslo, and often speaks at various technical Meetups and conferences.           Bhuvaneswari Subramani – Bengaluru, India Community Hero Bhuvaneswari Subramani is Director Engineering Operations at Infor. She has almost two decades of IT experience, specializing in Cloud Computing, DevOps, and Performance Testing. She holds the AWS Certified Solution Architect Professional certification, is a co-organizer of the AWS User Group Bengaluru, and is instrumental in organizing Meetups & AWS Community Day Bengaluru. Bhuvaneswari is also an active speaker at AWS community events, industry conferences and delivers guest lectures on Cloud Computing for staff & students at engineering colleges affiliated to Anna University. She is a technophile & IT Blogger, who meticulously and picturesquely depicts the events that inspires & influences her. Her passion for technical writing is exemplified in the form of tech blog DevOps and CloudComputing for over a decade and of late, she constantly writes about AWS conferences and Meetups on the AWS User Group Bengaluru Blog.       Colin Percival – Vancouver, Canada Community Hero Colin Percival is the founder of Tarsnap, a secure online backup service which combines the flexibility and scriptability of the standard UNIX “tar” utility with strong encryption, deduplication, and the reliability of Amazon S3 storage. Having started work on Tarsnap in 2006, Colin is among the first generation of users of Amazon Web Services, and has written dozens of articles about his experiences with AWS on his blog. Colin has been a member of the FreeBSD project for 15 years and has served in that time as the project Security Officer and a member of the Core team; starting in 2008 he led the efforts to bring FreeBSD to the Amazon EC2 platform, and for the past 7 years he has been maintaining this support, keeping FreeBSD up to date with all of the latest changes and functionality in Amazon EC2.         Francesco Pochetti – Luxembourg Machine Learning Hero Francesco Pochetti first got in touch with Machine Learning back in 2013, taking Stanford’s ML MOOC by Andrew Ng. Now he leverages the wonders of AWS AI infrastructure, plays around with new services, builds ML solutions and lets the world know about his projects on his blog. This is where he regularly posts all his experiments in Machine Learning and Deep Learning. Most notably, within the AWS domain, Inferring movie genre from its poster in AWS SageMaker, Analyzing IMDb reviews sentiment with Amazon Comprehend or Running Neural Style Transfer with Lambda and GPU-powered EC2s.               Guy Ernest – Tel Aviv, Israel Machine Learning Hero Guy Ernest is busy in taking machine learning and AI to the masses to three audiences. The main audience he engages are software developers (SDE) and converting them to machine learning engineers (MLE), using the popular fastai library, PyTorch, and Amazon AI services. The next audience is business people in large enterprises that are learning the applicability of machine learning to their business and the way to conduct the AI transformation of their organization. Finally, Guy works with kids who are starting their AI/ML learning journey by enabling them with Alexa skills, computer vision, and robots in after school and summer camp activities.               Kesha Williams – Atlanta, USA Machine Learning Hero Kesha Williams has over 20 years of experience in software development. She successfully transformed her engineering skills from Java Developer to ML Practitioner by leaning hands on with AWS AI solutions like AWS DeepLens, Amazon Rekognition, and Amazon SageMaker. Kesha believes that as we all benefit from integrating technology into our everyday lives, we still struggle to make meaningful relationships with each other. To solve this, Kesha develops ML models on Amazon SageMaker using computer vision and natural language processing to help us better connect with people around us. Kesha is also very active in the community helping others find their path to machine learning. She authors courses for learning platforms such as Manning Publications, Packt, LinkedIn Learning, A Cloud Guru, and Cloud Academy.         Manoj Fernando – Sri Lanka Community Hero Manoj Fernando is a Technical Lead at 99X Technology in Sri Lanka and the CTO of Whatif AS in Norway. He is passionate about designing scalable and cost-effective cloud architectures on the AWS cloud platform. His team was one of the early adopters of AWS Lambda back in 2015, and he is one of the co-organizers of Serverless Sri Lanka Meetup. During his leisure time, he creates cloud training videos for the community on his YouTube channel. The training videos are focused on Modern Application Development, Cloud Certifications, and Cutting-edge cloud services. He is also a technical blogger, blogging on medium as well as on his website, and a public speaker, conducting cloud workshops for university students in Sri Lanka.             Margaret Valtierra – Chicago, USA Community Hero Margaret Valtierra is a Program Manager for the Cloud Service team at Morningstar. She is responsible for managing the AWS relationship and promoting cloud skills and best practices. She has organized and led the Chicago AWS user group since 2013. She is a member of the Global AWS Community Day Core Team, promoting user groups and organizing the annual Midwest AWS Community Day. Margaret is also a co-organizer for DevOpsDays Chicago and is an AWS Certified Solutions Architect Associate.               Marco Viganò – Milan, Italy Community Hero Marco Viganò is the CTO of Condé Nast Italy. He has more than 20 years of experience in IT, with a specific focus on media and publishing sector. He is a frequent speaker at AWS Summits and events, sharing design patterns for developing and operating highly scalable cloud solutions for the media and publishing industry. He is focused on Serverless and Machine Learning and one of his main topics is finding new technologies to improve systems. Also, he operates as Voice First evangelist inside the company using Alexa and AWS services.               Pavlos Mitsoulis – London, United Kingdom Machine Learning Hero Pavlos Mitsoulis has 7 years of Machine Learning and Software Engineering experience. Currently, he is a Staff Software Engineer (Machine Learning) at HomeAway (an Expedia Group brand), leading Machine Learning initiatives to support growth marketing. Additionally, he is the creator of Sagify, an open-source library that simplifies training, tuning, evaluating, and deploying ML models to SageMaker. Recently Pavlos authored a Packt video course, “Hands-On Machine Learning Using Amazon SageMaker“.               Vicky Seno – Los Angeles, USA Container Hero Vicky “Tanya” Seno is a Computer Science Professor at Santa Monica College. At SMC she teaches numerous AWS courses covering Computing Services, Containers, Kubernetes, ECS, Serverless, Networking and Security. Vicky has helped develop an AWS Cloud Computing College Degree and is part of a team that helps train and mentor faculty from nineteen local colleges in AWS, to help expand AWS course offerings in the Los Angeles area. She is also a co-organizer of AWS Cloud Day Conference at SMC that includes SA speakers, AWS workshops and a AWS CTF attended by over 130+ students at each event. In an effort to increase female representation in this field, Vicky has been involved in various speaking and training activities. Vicky hosts a YouTube Channel with over 34,000 followers and 100+ beginners tech tutorials. She has also spoken at AWS Summits on Containers, Kubernetes, and Amazon EKS.             You can learn about all the AWS Heroes from around the globe by checking out the Hero website.

Learn about AWS Services & Solutions – June AWS Online Tech Talks

Amazon Web Services Blog -

Join us this June to learn about AWS services and solutions. The AWS Online Tech Talks are live, online presentations that cover a broad range of topics at varying technical levels. These tech talks, led by AWS solutions architects and engineers, feature technical deep dives, live demonstrations, customer examples, and Q&A with AWS experts. Register Now! Note – All sessions are free and in Pacific Time. Tech talks this month: AR/VR June 20, 2019 | 1:00 PM – 2:00 PM PT – Create a Digital Asset with Amazon Sumerian and AWS IoT – Learn to create a “digital asset” of a physical machine using Amazon Sumerian and AWS IoT to create simulations, alerts, and diagnostic instructions. Compute June 25, 2019 | 9:00 AM – 10:00 AM PT – Running Enterprise CI/CD workloads with Amazon EC2 Spot Instances and CloudBees – Learn how to launch your CI/CD pipelines in the AWS cloud more quickly, and cost-effectively with CloudBees Core and Amazon EC2 Spot Instances. June 26, 2019 | 11:00 AM – 12:00 PM PT – Gain Control Over Your License Usage and Reduce Costs Using AWS License Manager – See how you can gain control over your license usage and reduce costs using AWS License Manager through effective and proactive license management. Containers June 18, 2019 | 9:00 AM – 10:00 AM PT – Serverless E-commerce Application Using AWS AppSync – Learn how to get started with AWS Amplify, create a GraphQL API using AppSync, and see how easy it is to collect and visualize analytics data using Amazon Pinpoint, Amazon Kinesis Firehose, Amazon Redshift, and Amazon QuickSight. June 20, 2019 | 11:00 AM – 12:00 PM PT – Container Security 101 and Beyond – Join a discussion on container-specific security considerations, from container image hygiene to running containers securely, handling sensitive data, permission management, as well as auditing. Data Lakes & Analytics June 17, 2019 | 11:00 AM – 12:00 PM PT – Build ETL Processes for Data Lakes with AWS Glue – Learn how you can build, automate, and manage ETL jobs for your data lake, using AWS Glue as a scalable, serverless platform for Apache Spark and Python shell jobs. June 19, 2019 | 9:00 AM – 10:00 AM PT – Ditching the Overhead: Moving Apache Kafka Workloads into Amazon MSK – Learn how to remove the complexity of managing Apache Kafka by moving your workloads to Amazon MSK with minimal downtime. June 20, 2019 | 9:00 AM – 10:00 AM PT – Unify Your Data Warehouse and Data Lake with AWS – Learn how to break data silos among analytical engines and between various groups in an organization by combining data warehouse and data lake architectures with AWS. Databases June 18, 2019 | 1:00 PM – 2:00 PM PT – Achieve Continuous Availability with Amazon Aurora Multi-Master – Learn how to achieve Continuous Availability with Amazon Aurora Multi-Master. DevOps June 19, 2019 | 1:00 PM – 2:00 PM PT – Root Cause and End-user Impact Analysis Using AWS X-Ray – Learn how to pinpoint application performance bottlenecks and assess the root cause of discovered issues using AWS X-Ray. End-User Computing June 24, 2019 | 9:00 AM – 10:00 AM PT – Replace Your On-Premises VDI with End-User Computing on AWS – Learn how you can improve the flexibility and scalability of your EUC solutions by moving to EUC services on AWS. Enterprise & Hybrid June 25, 2019 | 11:00 AM – 12:00 PM PT – How to Build Your Cloud Enablement Engine with the People You Already Have – Learn how to build the skills, organization, and operating model to accelerate cloud adoption. IoT June 24, 2019 | 11:00 AM – 12:00 PM PT – Securing Your Devices from the Edge to the Cloud – Learn how to use AWS IoT services to reduce the complexity of keeping data secure, restricting access to devices and cloud resources, securely connecting to the cloud, and auditing device usage. Machine Learning June 18, 2019 | 11:00 AM – 12:00 PM PT – Building Secure Machine Learning Environments Using Amazon SageMaker – Learn how to build secure machine learning environments using Amazon SageMaker. June 26, 2019 | 1:00 PM – 2:00 PM PT – Fraud Detection Using Machine Learning with Amazon SageMaker – Learn how to do fraud detection using machine learning with Amazon SageMaker. Migration June 27, 2019 | 9:00 AM – 10:00 AM PT – How AWS Migration Hub Helps You Plan, Track, and Complete Your Application Migrations – Learn how to use AWS Migration Hub and AWS Application Discovery Service to discover, plan and track your application migrations. Networking & Content Delivery June 17, 2019 | 1:00 PM – 2:00 PM PT – Customer Showcase: Exploiting Multi-Region Data Locality with Lambda@Edge – Learn how to run your applications in multi-regions and how serverless can help. Robotics June 26, 2019 | 9:00 AM – 10:00 AM PT – Developing Intelligent Robots with AWS RoboMaker – Learn how to develop, test and deploy intelligent robotic applications with AWS RoboMaker. Security, Identity, & Compliance June 17, 2019 | 9:00 AM – 10:00 AM PT – Continuous Compliance with AWS Security Hub – Learn how AWS Security Hub provides you with a comprehensive view of your security state within AWS and helps you with continuous compliance checks. June 19, 2019 | 11:00 AM – 12:00 PM PT – Customer Showcase: Learn How Amazon.com Uses Amazon GuardDuty to Protect its Infrastructure – Learn how one of AWS largest customers, Amazon.com, uses Amazon GuardDuty to protect their infrastructure. Serverless June 27, 2019 | 11:00 AM – 12:00 PM PT – Serverless Streams, Topics, Queues, & APIs! How to Pick the Right Serverless Application Pattern – Learn how to choose between streams, topics, queues, or APIs with AWS Lambda – pick right design pattern for your serverless application. Storage June 24, 2019 | 1:00 PM – 2:00 PM PT – Build Your Data Lake on Amazon S3 – Learn why Amazon S3 is the best destination for your data lake.

Amazon Managed Streaming for Apache Kafka (MSK) – Now Generally Available

Amazon Web Services Blog -

I am always amazed at how our customers are using streaming data. For example, Thomson Reuters, one of the world’s most trusted news organizations for businesses and professionals, built a solution to capture, analyze, and visualize analytics data to help product teams continuously improve the user experience. Supercell, the social game company providing games such as Hay Day, Clash of Clans, and Boom Beach, is delivering in-game data in real-time, handling 45 billion events per day. Since we launched Amazon Kinesis at re:Invent 2013, we have continually expanded the ways in in which customers work with streaming data on AWS. Some of the available tools are: Kinesis Data Streams, to capture, store, and process data streams with your own applications. Kinesis Data Firehose, to transform and collect data into destinations such as Amazon S3, Amazon Elasticsearch Service, and Amazon Redshift. Kinesis Data Analytics, to continuously analyze data using SQL or Java (via Apache Flink applications), for example to detect anomalies or for time series aggregation. Kinesis Video Streams, to simplify processing of media streams. At re:Invent 2018, we introduced in open preview Amazon Managed Streaming for Apache Kafka (MSK), a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. I am excited to announce that Amazon MSK is generally available today! How it works Apache Kafka (Kafka) is an open-source platform that enables customers to capture streaming data like click stream events, transactions, IoT events, application and machine logs, and have applications that perform real-time analytics, run continuous transformations, and distribute this data to data lakes and databases in real time. You can use Kafka as a streaming data store to decouple applications producing streaming data (producers) from those consuming streaming data (consumers). While Kafka is a popular enterprise data streaming and messaging framework, it can be difficult to setup, scale, and manage in production. Amazon MSK takes care of these managing tasks and makes it easy to set up, configure, and run Kafka, along with Apache ZooKeeper, in an environment following best practices for high availability and security. Your MSK clusters always run within an Amazon VPC managed by the MSK service. Your MSK resources are made available to your own VPC, subnet, and security group through elastic network interfaces (ENIs) which will appear in your account, as described in the following architectural diagram: Customers can create a cluster in minutes, use AWS Identity and Access Management (IAM) to control cluster actions, authorize clients using TLS private certificate authorities fully managed by AWS Certificate Manager (ACM), encrypt data in-transit using TLS, and encrypt data at rest using AWS Key Management Service (KMS) encryption keys. Amazon MSK continuously monitors server health and automatically replaces servers when they fail, automates server patching, and operates highly available ZooKeeper nodes as a part of the service at no additional cost. Key Kafka performance metrics are published in the console and in Amazon CloudWatch. Amazon MSK is fully compatible with Kafka versions 1.1.1 and 2.1.0, so that you can continue to run your applications, use Kafka’s admin tools, and and use Kafka compatible tools and frameworks without having to change your code. Based on our customer feedback during the open preview, Amazon MSK added may features such as: Encryption in-transit via TLS between clients and brokers, and between brokers Mutual TLS authentication using ACM private certificate authorities Support for Kafka version 2.1.0 99.9% availability SLA HIPAA eligible Cluster-wide storage scale up Integration with AWS CloudTrail for MSK API logging Cluster tagging and tag-based IAM policy application Defining custom, cluster-wide configurations for topics and brokers AWS CloudFormation support is coming in the next few weeks. Creating a cluster Let’s create a cluster using the AWS management console. I give the cluster a name, select the VPC I want to use the cluster from, and the Kafka version. I then choose the Availability Zones (AZs) and the corresponding subnets to use in the VPC. In the next step, I select how many Kafka brokers to deploy in each AZ. More brokers allow you to scale the throughtput of a cluster by allocating partitions to different brokers. I can add tags to search and filter my resources, apply IAM policies to the Amazon MSK API, and track my costs. For storage, I leave the default storage volume size per broker. I select to use encryption within the cluster and to allow both TLS and plaintext traffic between clients and brokers. For data at rest, I use the AWS-managed customer master key (CMK), but you can select a CMK in your account, using KMS, to have further control. You can use private TLS certificates to authenticate the identity of clients that connect to your cluster. This feature is using Private Certificate Authorities (CA) from ACM. For now, I leave this option unchecked. In the advanced setting, I leave the default values. For example, I could have chosen here a different instance type for my brokers. Some of these settings can be updated using the AWS CLI. I create the cluster and monitor the status from the cluster summary, including the Amazon Resource Name (ARN) that I can use when interacting via CLI or SDKs. When the status is active, the client information section provides specific details to connect to the cluster, such as: The bootstrap servers I can use with Kafka tools to connect to the cluster. The Zookeper connect list of hosts and ports. I can get similar information using the AWS CLI: aws kafka list-clusters to see the ARNs of your clusters in a specific region aws kafka get-bootstrap-brokers --cluster-arn <ClusterArn> to get the Kafka bootstrap servers aws kafka describe-cluster --cluster-arn <ClusterArn> to see more details on the cluster, including the Zookeeper connect string Quick demo of using Kafka To start using Kafka, I create two EC2 instances in the same VPC, one will be a producer and one a consumer. To set them up as client machines, I download and extract the Kafka tools from the Apache website or any mirror. Kafka requires Java 8 to run, so I install Amazon Corretto 8. On the producer instance, in the Kafka directory, I create a topic to send data from the producer to the consumer: bin/kafka-topics.sh --create --zookeeper <ZookeeperConnectString> \ --replication-factor 3 --partitions 1 --topic MyTopic Then I start a console-based producer: bin/kafka-console-producer.sh --broker-list <BootstrapBrokerString> \ --topic MyTopic On the consumer instance, in the Kafka directory, I start a console-based consumer: bin/kafka-console-consumer.sh --bootstrap-server <BootstrapBrokerString> \ --topic MyTopic --from-beginning Here’s a recording of a quick demo where I create the topic and then send messages from a producer (top terminal) to a consumer of that topic (bottom terminal): Pricing and availability Pricing is per Kafka broker-hour and per provisioned storage-hour. There is no cost for the Zookeeper nodes used by your clusters. AWS data transfer rates apply for data transfer in and out of MSK. You will not be charged for data transfer within the cluster in a region, including data transfer between brokers and data transfer between brokers and ZooKeeper nodes. You can migrate your existing Kafka cluster to MSK using tools like MirrorMaker (that comes with open source Kafka) to replicate data from your clusters into a MSK cluster. Upstream compatibility is a core tenet of Amazon MSK. Our code changes to the Kafka platform are released back to open source. Amazon MSK is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Singapore), Asia Pacific (Sydney), EU (Frankfurt), EU (Ireland), EU (Paris), and EU (London). I look forward to see how are you going to use Amazon MSK to simplify building and migrating streaming applications to the cloud!

Now Available – AWS IoT Things Graph

Amazon Web Services Blog -

We announced AWS IoT Things Graph last November and described it as a tool to let you build IoT applications visually. Today I am happy to let you know that the service is now available and ready for you to use! As you will see in a moment, you can represent your business logic in a flow composed of devices and services. Each web service and each type of device (sensor, camera, display, and so forth) is represented in Things Graph as a model. The models hide the implementation details that are peculiar to a particular brand or model of device, and allow you to build flows that can evolve along with your hardware. Each model has a set of actions (inputs), events (outputs), and states (attributes). Things Graph includes a set of predefined models, and also allows you to define your own. You can also use mappings as part of your flow to convert the output from one device into the form expected by other devices. After you build your flow, you can deploy it to the AWS Cloud or an AWS IoT Greengrass-enabled device for local execution. The flow, once deployed, orchestrates interactions between locally connected devices and web services. Using AWS IoT Things Graph Let’s take a quick walk through the AWS IoT Things Graph Console! The first step is to make sure that I have models which represent the devices and web services that I plan to use in my flow. I click Models in the console navigation to get started: The console outlines the three steps that I must follow to create a model, and also lists my existing models: The presence of aws/examples in the URN for each of the devices listed above indicates that they are predefined, and part of the public AWS IoT Things Graph namespace. I click on Camera to learn more about this model; I can see the Properties, Actions, and Events: The model is defined using GraphQL; I can view it, edit it, or upload a file that contains a model definition. Here’s the definition of the Camera: This model defines an abstract Camera device. The model, in turn, can reference definitions for one or more actual devices, as listed in the Devices section: Each of the devices is also defined using GraphQL. Of particular interest is the use of MQTT topics & messages to define actions: Earlier, I mentioned that models can also represent web services. When a flow that references a model of this type is deployed, activating an action on the model invokes a Greengrass Lambda function. Here’s how a web service is defined: Now I can create a flow. I click Flows in the navigation, and click Create flow: I give my flow a name and enter a description: I start with an empty canvas, and then drag nodes (Devices, Services, or Logic) to it: For this demo (which is fully explained in the AWS IoT Things Graph User Guide), I’ll use a MotionSensor, a Camera, and a Screen: I connect the devices to define the flow: Then I configure and customize it. There are lots of choices and settings, so I’ll show you a few highlights, and refer you to the User Guide for more info. I set up the MotionSensor so that a change of state initiates this flow: I also (not shown) configure the Camera to perform the Capture action, and the Screen to display it. I could also make use of the predefined Services: I can also add Logic to my flow: Like the models, my flow is ultimately defined in GraphQL (I can view and edit it directly if desired): At this point I have defined my flow, and I click Publish to make it available for deployment: The next steps are: Associate – This step assigns an actual AWS IoT Thing to a device model. I select a Thing, and then choose a device model, and repeat this step for each device model in my flow: Deploy – I create a Flow Configuration, target it at the Cloud or Greengrass, and use it to deploy my flow (read Creating Flow Configurations to learn more). Things to Know I’ve barely scratched the surface here; AWS IoT Things Graph provides you with a lot of power and flexibility and I’ll leave you to discover more on your own! Here are a couple of things to keep in mind: Pricing – Pricing is based on the number of steps executed (for cloud deployments) or deployments (for edge deployments), and is detailed on the AWS IoT Things Graph Pricing page. API Access – In addition to console access, you can use the AWS IoT Things Graph API to build your models and flows. Regions – AWS IoT Things Graph is available in the US East (N. Virginia), US West (Oregon), Europe (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo) Regions. — Jeff;    

New – Data API for Amazon Aurora Serverless

Amazon Web Services Blog -

If you have ever written code that accesses a relational database, you know the drill. You open a connection, use it to process one or more SQL queries or other statements, and then close the connection. You probably used a client library that was specific to your operating system, programming language, and your database. At some point you realized that creating connections took a lot of clock time and consumed memory on the database engine, and soon after found out that you could (or had to) deal with connection pooling and other tricks. Sound familiar? The connection-oriented model that I described above is adequate for traditional, long-running programs where the setup time can be amortized over hours or even days. It is not, however, a great fit for serverless functions that are frequently invoked and that run for time intervals that range from milliseconds to minutes. Because there is no long-running server, there’s no place to store a connection identifier for reuse. Aurora Serverless Data API In order to resolve this mismatch between serverless applications and relational databases, we are launching a Data API for the MySQL-compatible version of Amazon Aurora Serverless. This API frees you from the complexity and overhead that come along with traditional connection management, and gives you the power to quickly and easily execute SQL statements that access and modify your Amazon Aurora Serverless Database instances. The Data API is designed to meet the needs of both traditional and serverless apps. It takes care of managing and scaling long-term connections to the database and returns data in JSON form for easy parsing. All traffic runs over secure HTTPS connections. It includes the following functions: ExecuteStatement – Run a single SQL statement, optionally within a transaction. BatchExecuteStatement – Run a single SQL statement across an array of data, optionally within a transaction. BeginTransaction – Begin a transaction, and return a transaction identifier. Transactions are expected to be short (generally 2 to 5 minutes). CommitTransaction – End a transaction and commit the operations that took place within it. RollbackTransaction – End a transaction without committing the operations that took place within it. Each function must run to completion within 1 minute, and can return up to 1 megabyte of data. Using the Data API I can use the Data API from the Amazon RDS Console, the command line, or by writing code that calls the functions that I described above. I’ll show you all three in this post. The Data API is really easy to use! The first step is to enable it for the desired Amazon Aurora Serverless database. I open the Amazon RDS Console, find & select the cluster, and click Modify: Then I scroll down to the Network & Security section, click Data API, and Continue: On the next page I choose to apply the settings immediately, and click Modify cluster: Now I need to create a secret to store the credentials that are needed to access my database. I open the Secrets Manager Console and click Store a new secret. I leave Credentials for RDS selected, enter a valid database user name and password, optionally choose a non-default encryption key, and then select my serverless database. Then I click Next: I name my secret and tag it, and click Next to configure it: I use the default values on the next page, click Next again, and now I have a brand new secret: Now I need two ARNs, one for the database and one for the secret. I fetch both from the console, first for the database: And then for the secret: The pair of ARNs (database and secret) provides me with access to my database, and I will protect them accordingly! Using the Data API from the Amazon RDS Console I can use the Query Editor in the Amazon RDS Console to run queries that call the Data API. I open the console and click Query Editor, and create a connection to the database. I select the cluster, enter my credentials, and pre-select the table of interest. Then I click Connect to database to proceed: I enter a query and click Run, and view the results within the editor: Using the Data API from the Command Line I can exercise the Data API from the command line: $ aws rds-data execute-statement \ --secret-arn "arn:aws:secretsmanager:us-east-1:123456789012:secret:aurora-serverless-data-api-sl-admin-2Ir1oL" \ --resource-arn "arn:aws:rds:us-east-1:123456789012:cluster:aurora-sl-1" \ --database users \ --sql "show tables" \ --output json I can use jq to pick out the part of the result that is of interest to me: ... | jq .records [ { "values": [ { "stringValue": "users" } ] } ] I can query the table and get the results (the SQL statement is "select * from users where userid='jeffbarr'"): ... | jq .records [ { "values": [ { "stringValue": "jeffbarr" }, { "stringValue": "Jeff" }, { "stringValue": "Barr" } ] } If I specify --include-result-metadata, the query also returns data that describes the columns of the result (I’ll show only the first one in the interest of frugality): ... | jq .columnMetadata[0] { "type": 12, "name": "userid", "label": "userid", "nullable": 1, "isSigned": false, "arrayBaseColumnType": 0, "scale": 0, "schemaName": "", "tableName": "users", "isCaseSensitive": false, "isCurrency": false, "isAutoIncrement": false, "precision": 15, "typeName": "VARCHAR" } The Data API also allows me to wrap a series of statements in a transaction, and then either commit or rollback. Here’s how I do that (I’m omitting --secret-arn and --resource-arn for clarity): $ $ID=`aws rds-data begin-transaction --database users --output json | jq .transactionId` $ echo $ID "ATP6Gz88GYNHdwNKaCt/vGhhKxZs2QWjynHCzGSdRi9yiQRbnrvfwF/oa+iTQnSXdGUoNoC9MxLBwyp2XbO4jBEtczBZ1aVWERTym9v1WVO/ZQvyhWwrThLveCdeXCufy/nauKFJdl79aZ8aDD4pF4nOewB1aLbpsQ==" $ aws rds-data execute-statement --transaction-id $ID --database users --sql "..." $ ... $ aws rds-data execute-statement --transaction-id $ID --database users --sql "..." $ aws rds-data commit-transaction $ID If I decide not to commit, I invoke rollback-transaction instead. Using the Data API with Python and Boto Since this is an API, programmatic access is easy. Here’s some very simple Python / Boto code: import boto3 client = boto3.client('rds-data') response = client.execute_sql( secretArn = 'arn:aws:secretsmanager:us-east-1:123456789012:secret:aurora-serverless-data-api-sl-admin-2Ir1oL', database = 'users', resourceArn = 'arn:aws:rds:us-east-1:123456789012:cluster:aurora-sl-1', sql = 'select * from users' ) for user in response['records']: userid = user[0]['stringValue'] first_name = user[1]['stringValue'] last_name = user[2]['stringValue'] print(userid + ' ' + first_name + ' ' + last_name) And the output: $ python data_api.py jeffbarr Jeff Barr carmenbarr Carmen Barr Genuine, production-quality code would reference the table columns symbolically using the metadata that is returned as part of the response. By the way, my Amazon Aurora Serverless cluster was configured to scale capacity all the way down to zero when not active. Here’s what the scaling activity looked like while I was writing this post and running the queries: Now Available You can make use of the Data API today in the US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland) Regions. There is no charge for the API, but you will pay the usual price for data transfer out of AWS. — Jeff;

New – AWS IoT Events: Detect and Respond to Events at Scale

Amazon Web Services Blog -

As you may have been able to tell from many of the announcements that we have made over the last four or five years, we are working to build a wide-ranging set of Internet of Things (IoT) services and capabilities. Here’s a quick recap: October 2015 – AWS IoT Core – A fundamental set of Cloud Services for Connected Devices. Jun 2017 – AWS Greengrass – The ability to Run AWS Lambda Functions on Connected Devices. November 2017 – AWS IoT Device Management – Onboarding, Organization, Monitoring, and Remote Management of Connected Devices. November 2017 – AWS IoT Analytics – Advanced Data Analysis for IoT Devices. November 2017 – Amazon FreeRTOS – An IoT Operating System for Microcontrollers. April 2018 – Greengrass ML Inference – The power to do Machine Learning Inference at the Edge. August 2018 – AWS IoT Device Defender – A service that helps to Keep Your Connected Devices Safe. Last November we also announced our plans to launch four new IoT Services: AWS IoT SiteWise to collect, structure, and search data from industrial equipment at scale. AWS IoT Events to detect and respond to events at scale. AWS IoT Things Graph to build IoT applications visually. AWS IoT Greengrass Connectors to simplify and accelerate the process of connecting devices. You can use these services individually or together to build all sorts of powerful, connected applications! AWS IoT Events Now Available Today we are making AWS IoT Events available in production form in four AWS Regions. You can use this service to monitor and respond to events (patterns of data that identify changes in equipment or facilities) at scale. You can detect a misaligned robot arm, a motion sensor that triggers outside of business hours, an unsealed freezer door, or a motor that is running outside of tolerance, all with the goal of driving faster and better-informed decisions. As you will see in a moment, you can easily create detector models that represent your devices, their states, and the transitions (driven by sensors and events, both known as inputs) between the states. The models can trigger actions when critical events are detected, allowing you to build robust, highly automated systems. Actions can, for example, send a text message to a service technician or invoke an AWS Lambda function. You can access AWS IoT Events from the AWS IoT Event Console or by writing code that calls the AWS IoT Events API functions. I’ll use the Console, and I will start by creating a detector model. I click Create detector model to begin: I have three options; I’ll go with the demo by clicking Launch demo with inputs: This shortcut creates an input and a model, and also enables some “demo” functionality that sends data to the model. The model looks like this: Before examining the model, let’s take a look at the input. I click on Inputs in the left navigation to see them: I can see all of my inputs at a glance; I click on the newly created input to learn more: This input represents the battery voltage measured from a device that is connected to a particular powerwallId: Ok, let’s return to (and dissect) the detector model! I return to the navigation, click Detector models, find my model, and click it: There are three Send options at the top; each one sends data (an input) to the detector model. I click on Send data for Charging to get started. This generates a message that looks like this; I click Send data to do just that: Then I click Send data for Charged to indicate that the battery is fully charged. The console shows me the state of the detector: Each time an input is received, the detector processes it. Let’s take a closer look at the detector. It has three states (Charging, Charged, and Discharging): The detector starts out in the Charging state, and transitions to Charged when the Full_charge event is triggered. Here’s the definition of the event, including the trigger logic: The trigger logic is evaluated each time an input is received (your IoT app must call BatchPutMessage to inform AWS IoT Events). If the trigger logic evaluates to a true condition, the model transitions to the new (destination) state, and it can also initiate an event action. This transition has no actions; I can add one (or more) by clicking Add action. My choices are: Send MQTT Message – Send a message to an MQTT topic. Send SNS Message – Send a message to an SNS target, identifed by an ARN. Set Timer – Set, reset, or destroy a timer. Times can be expressed in seconds, minutes, hours, days, or months. Set Variable – Set, increment, or decrement a variable. Returning (once again) to the detector, I can modify the states as desired. For example, I could fine-tune the Discharging aspect of the detector by adding a LowBattery state: After I create my inputs and my detector, I Publish the model so that my IoT devices can use and benefit from it. I click Publish and fill in a few details: The Detector generation method has two options. I can Create a detector for each unique key value (if I have a bunch of devices), or I can Create a single detector (if I have one device). If I choose the first option, I need to choose the key that separates one device from another. Once my detector has been published, I can send data to it using AWS IoT Analytics, IoT Core, or from a Lambda function. Get Started Today We are launching AWS IoT Events in the US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland) Regions and you can start using it today! — Jeff;    

New – Opt-in to Default Encryption for New EBS Volumes

Amazon Web Services Blog -

My colleagues on the AWS team are always looking for ways to make it easier and simpler for you to protect your data from unauthorized access. This work is visible in many different ways, and includes the AWS Cloud Security page, the AWS Security Blog, a rich collection of AWS security white papers, an equally rich set of AWS security, identity, and compliance services, and a wide range of security features within individual services. As you might recall from reading this blog, many AWS services support encryption at rest & in transit, logging, IAM roles & policies, and so forth. Default Encryption Today I would like to tell you about a new feature that makes the use of encrypted Amazon EBS (Elastic Block Store) volumes even easier. This launch builds on some earlier EBS security launches including: EBS Encryption for Additional Data Protection. Encrypting EBS Snapshots Via Copying. Encrypted EBS Boot Volumes. Encryption with Custom Keys at Instance Launch Time. Sharing of Encrypted AMIs Across AWS Accounts. You can now specify that you want all newly created EBS volumes to be created in encrypted form, with the option to use the default key provided by AWS, or a key that you create. Because keys and EC2 settings are specific to individual AWS regions, you must opt-in on a region-by-region basis. This new feature will let you reach your protection and compliance goals by making it simpler and easier for you to ensure that newly created volumes are created in encrypted form. It will not affect existing unencrypted volumes. If you use IAM policies that require the use of encrypted volumes, you can use this feature to avoid launch failures that would occur if unencrypted volumes were inadvertently referenced when an instance is launched. Your security team can enable encryption by default without having to coordinate with your development team, and with no other code or operational changes. Encrypted EBS volumes deliver the specified instance throughput, volume performance, and latency, at no extra charge. I open the EC2 Console, make sure that I am in the region of interest, and click Settings to get started: Then I select Always encrypt new EBS volumes: I can click Change the default key and choose one of my keys as the default: Either way, I click Update to proceed. One thing to note here: This setting applies to a single AWS region; I will need to repeat the steps above for each region of interest, checking the option and choosing the key. Going forward, all EBS volumes that I create in this region will be encrypted, with no additional effort on my part. When I create a volume, I can use the key that I selected in the EC2 Settings, or I can select a different one: Any snapshots that I create are encrypted with the key that was used to encrypt the volume: If I use the volume to create a snapshot, I can use the original key or I can choose another one: Things to Know Here are some important things that you should know about this important new AWS feature: Older Instance Types – After you enable this feature, you will not be able to launch any more C1, M1, M2, or T1 instances or attach newly encrypted EBS volumes to existing instances of these types. We recommend that you migrate to newer instance types. AMI Sharing – As I noted above, we recently gave you the ability to share encrypted AMIs with other AWS accounts. However, you cannot share them publicly, and you should use a separate account to create community AMIs, Marketplace AMIs, and public snapshots. To learn more, read How to Share Encrypted AMIs Across Accounts to Launch Encrypted EC2 Instances. Other AWS Services – AWS services such as Amazon Relational Database Service (RDS) and Amazon WorkSpaces that use EBS for storage perform their own encryption and key management and are not affected by this launch. Services such as Amazon EMR that create volumes within your account will automatically respect the encryption setting, and will use encrypted volumes if the always-encrypt feature is enabled. API / CLI Access – You can also access this feature from the EC2 CLI and the API. No Charge – There is no charge to enable or use encryption. If you are using encrypted AMIs and create a separate one for each AWS account, you can now share the AMI with other accounts, leading to a reduction in storage utilization and charges. Per-Region – As noted above, you can opt-in to default encryption on a region-by-region basis. Available Now This feature is available now and you can start using it today in all public AWS regions and in GovCloud. It is not available in the AWS regions in China. — Jeff;  

AWS Ground Station – Ready to Ingest & Process Satellite Data

Amazon Web Services Blog -

Last fall I told you about AWS Ground Station and gave you a sneak preview of the steps that you would take to downlink data from a satellite. I am happy to report that the first two ground stations are now in operation, and that you can start using AWS Ground Station today. Using AWS Ground Station As I noted at the time, the first step is to Add satellites to your AWS account by sharing the satellite’s NORAD ID and other information with us: The on-boarding process generally takes a couple of days. For testing purposes, the Ground Station team added three satellites to my account: Terra (NORAD ID 25994) – This satellite was launched in 1989 and orbits at an altitude of 705 km. It carries five sensors that are designed to study the Earth’s surface. Aqua (NORAD ID 27424) – This satellite was launched in 2002 and also orbits at an altitude of 705 km. It carries six sensors that are designed to study surface water. NOAA-20 (NORAD ID 43013) – This satellite was launched in 2017 and orbits at an altitude of 825 km. It carries five sensors that observe both land and water. While the on-boarding process is under way, the next step is to choose the ground station that you will use to receive your data. This is dependent on the path your satellite takes as it orbits the Earth and the time at which you want to receive data. Our first two ground stations are located in Oregon and Ohio, with other locations under construction. Each ground station is associated with an adjacent AWS region and you need to set up your AWS infrastructure in that region ahead of time. I’m going to use the US East (Ohio) Region for this blog post. Following the directions in the AWS Ground Station User Guide, I use a CloudFormation template to set up my infrastructure within my VPC: The stack includes an EC2 instance, three Elastic Network Interfaces (ENIs), and the necessary IAM roles, EC2 security groups, and so forth: The EC2 instance hosts Kratos DataDefender (a lossless UDP transport mechanism). I can also use the instance to host the code that processes the incoming data stream. DataDefender makes the incoming data stream available on a Unix domain socket at port 55892. My code is responsible for reading the raw data, splitting it in to packets, and then processing each packet. You can also create one or more Mission Profiles. Each profile outlines the timing requirements for a contact, lists the resources needed for the contact, and defines how data flows during the contact. You can use the same Mission Profile for multiple satellites, and you can also use different profiles (as part of distinct contacts) for the same satellite. Scheduling a Contact With my satellite configured and my AWS infrastructure in place, I am ready to schedule a contact! I open the Ground Station Console, make sure that I am in the AWS Region that corresponds to the ground station that I want to use, and click Contacts. I review the list of upcoming contacts, select the desired one (If you are not accustomed to thinking in Zulu time, a World Clock / Converter is helpful), and click Reserve contact: Then I confirm my intent by clicking Reserve: The status of the connection goes to SCHEDULING and then to SCHEDULED, all within a minute or so: The next step is to wait for the satellite to come within range of the chosen ground station. During this time, I can connect to the EC2 instance in two ways: SSH – I can SSH to the instance’s IP address, verify that my code is in place and ready to run, and confirm that DataDefender is running: WEB – I can open up a web browser and see the DataDefender web interface: One thing to note: you may need to edit the security group attached to the instance in order to allow it to be accessed from outside of the VPC: 3-2-1 Contact! Ok, now I need to wait for Terra to come within range of the ground station that I selected. While not necessary, it can be fun (and educational) to use a real-time satellite tracker such as the one at n2yo.com: When my satellite comes in to range, DataDefender shows me that the data transfer is under way (at an impressive 781 Mbps), as indicated by the increased WAN Data Rate: As I noted earlier, the incoming data stream is available within the instance in real time on a Unix domain socket. After my code takes care of all immediate, low-level processing, it can route the data to Amazon Kinesis Data Streams for real-time processing, store it in Amazon S3 for safe-keeping or further analysis, and so forth. Customer Perspective – Spire While I was writing this blog post I spoke with Robert Sproles, a Program Manager with AWS customer Spire to learn about their adoption of Ground Station. Spire provides data & analytics from orbit, and runs the space program behind it. They design and build their own cubesats in-house, and currently have about 70 in orbit. Collectively, the satellites have more sensors than any of Spire’s competitors, and collect maritime, aviation, and weather data. Although Spire already operates a network of 30 ground stations, they were among the first to see the value of (and to start using) AWS Ground Station. In addition to being able to shift from a CapEx (capital expense) to OpEx (operating expense) model, Ground Station gives them the ability to collect fresh data more quickly, with the goal of making it available to their customers even more rapidly. Spire’s customers are wide-ranging and global, but can all benefit from rapid access to high-quality data. Their LEMUR (Low Earth Multi-Use Repeater) satellites go around the globe every 90 minutes, but this is a relatively long time when the data is related to aviation or weather. Robert told me that they can counter this by adding additional satellites in the same orbit or by making use of additional ground stations, all with the goal of reducing latency and delivering the freshest possible data. Spire applies machine learning to the raw data, with the goal of going from a “lump of data” to actionable insights. For example, they use ML to make predictions about the future positions of cargo ships, using a combination of weather and historical data. The predicted ship positions can be used to schedule dock slots and other scarce resources ahead of time. Now Available You can get started with AWS Ground Station today. We have two ground stations in operation, with ten more in the works and planned for later this year. — Jeff;  

New – Updated Pay-Per-Use Pricing Model for AWS Config Rules

Amazon Web Services Blog -

AWS Config rules give you the power to perform Dynamic Compliance Checking on your Cloud Resources. Building on the AWS Resource Configuration Tracking provided by AWS Config, you can use a combination of predefined and custom rules to continuously and dynamically check that all changes made to your AWS resources are compliant with the conditions specified in the rules, and to take action (either automatic or manual) to remediate non-compliant resources. You can currently select from 84 different predefined rules, with more in the works. These are managed rules that are refined and updated from time to time. Here are the rules that match my search for EC2: Custom rules are built upon AWS Lambda functions, and can be run periodically or triggered by a configuration change. Rules can optionally be configured to execute a remediation action when a noncompliant resource is discovered. There are many built-in actions, and the option to write your own action using AWS Systems Manager documents as well: New Pay-Per-Use Pricing Today I am happy to announce that we are switching to a new, pay-per-use pricing model for AWS Config rules. Effective August 1st, 2019 you will be charged based on the number of rule evaluations that you run each month. Here is the new pricing for AWS Public Regions: Rule Evaluations Per Month Price Per Evaluation 0-100,000 $0.0010 100,001-500,000 $0.0008 500,001 and above $0.0005 You will no longer pay for active config rules, which can grow costly when used across multiple accounts and regions. You will continue to pay for configuration items recorded, and any additional costs such as use of S3 storage, SNS messaging, and the invocation of Lambda functions. The pricing works in conjunction with AWS Consolidated Billing, and is designed to provide almost all AWS customers with a significant reduction in their Config Rules bill. The new model will let you expand globally and cost-effectively, and will probably encourage you to make even more use of AWS Config rules! — Jeff;  

How AWS helps our Customers to go Global – Report from Korea

Amazon Web Services Blog -

Amazon Web Services Korea LLC (AWS Korea) opened an office in Seoul, South Korea in 2012. This office has educated and supported many customers from startups to large enterprises. Owing to high customer demand, we launched our Asia Pacific (Seoul) Region with 2 Availability Zones and two edge locations in January 2016. This Region has given AWS customers in Korea low-latency access to our suite of AWS infrastructure services. Andy Jassy, CEO of Amazon Web Services announced to launch Seoul Region in AWS Cloud 2016. Following this launch, Amazon CloudFront announced two new Edge locations and one Edge cache: the third in May 2016, and the fourth in Feb 2018. CloudFront’s expansion across Korea further improves the availability and performance of content delivery to users in the region. Today I am happy to announce that AWS added a third Availability Zone (AZ) to the AWS Asia Pacific (Seoul) Region to support the high demand of our growing Korean customer base. This third AZ provides customers with additional flexibility to architect scalable, fault-tolerant, and highly available applications in AWS Asia Pacific (Seoul), and will support additional AWS services in Korea. This launch brings AWS’s global AZ total to 66 AZs within 21 geographic Regions around the world. AZs located in AWS Regions consist of one or more discrete data centers, each with redundant power, networking, and connectivity, and each housed in separate facilities. Now AWS serves tens of thousands of active customers in Korea, ranging from startups and enterprises to educational institutions. One of the examples that reflects this demand is AWS Summit Seoul 2019, a part of our commitment to investing in education. More than 16,000 builders attended, a greater than tenfold increase from the 1,500 attendees of our first Summit in 2015. AWS Summit 2018 – a photo of keynote by Dr. Werner Vogels, CTO of Amazon.com So, how have Korean customers migrated to the AWS Cloud and what has motivated them? They have learned that the AWS Cloud is the new normal in the IT industry and quick adoption to their business has allowed them to regain global competitiveness. Let us look at some examples of how our customers are utilizing the benefit of the broad and deep AWS Cloud platform in the global market by replicating their services in Korea. Do you know Korean Wave? The Korean Wave represents the increase in global popularity of South Korean culture such as Korean Pop and Drama. The top three broadcasting companies in Korea (KBS, MBC, and SBS) use AWS. They co-invested to found Content Alliance Platform (CAP) that launched POOQ, which offers real-time OTT broadcasting to 600,000+ subscribers for TV programs including popular K-Dramas and has been able to reduce the buffer times on its streaming services by 20 percents. CAP also used AWS’s video processing and delivery services to stream Korea’s largest sports event, the PyeongChang 2018 Olympic Winter Games. Lots of K-Pop fans from KCON Concert 2016 in France – Wikipedia SM Entertainment, a South Korean entertainment company to lead K-Pop influences with NCT 127, EXO, Super Junior, and Girls’ Generation. The company uses AWS to deliver its websites and mobile applications. By using AWS, the company was able to scale to support more than 3 million new users of EXO-L mobile app in three weeks. The company also developed its mobile karaoke app, Everysing, on AWS, saving more than 50 percent in development costs. The scalability, flexibility, and pay-as-you-go pricing of AWS encouraged them to develop more mobile apps. Global Enterprises on the Cloud Korean Enterprises rapidly adopted AWS cloud to offer scalable global scale services as well as focus on their own business needs. Samsung Electronics uses the breadth of AWS services to save infrastructure costs and achieve rapid deployments, which provides high availability to customers and allows them to scale their services globally to support Galaxy customers worldwide. For example, Samsung Electronics increased reliability and reduced costs by 40 percent within a year after migrating its 860TB Samsung Cloud database to AWS. Samsung chose Amazon DynamoDB for its stability, scalability, and low latency to maintain the database used by 300 million Galaxy smartphone users worldwide. LG Electronics has selected AWS to run its mission-critical services for more than 35 million LG Smart TVs across the globe to handle the dramatic instant traffic peaks that come with broadcasting live sports events such as the World Cup and Olympic Games. Also, it built a new home appliance IoT platform called ThinQ. LG Electronics uses a serverless architecture and secure provisioning on AWS to reduce the development costs for this platform by 80 percent through increased efficiency in managing its developer and maintenance resources. Recently Korean Air decided to move its entire infrastructure to AWS over the next three years – including its website, loyalty program, flight operations, and other mission-critical operations — and will shut down its data centers after this migration. “This will enable us to bring new services to market faster and more efficiently, so that customer satisfaction continues to increase.” said Kenny Chang, CIO of Korean Air. AWS Customers in Korea – From Startups to Enterprises in each industries AI/ML on Traditional Manufacturers AWS is helping Korean manufacturing companies realize the benefits of digitalization and regain global competitiveness by leveraging over collective experience gained from working with customers and partners around the world. Kia Motors produces three million vehicles a year to customers worldwide. It uses Amazon Rekognition and Amazon Polly to develop a car log-in feature using face analysis and voice services. Introduced in CES 2018, this system welcomes drivers and adjusts settings such as seating, mirrors and in-vehicle infotainment based on individual preferences to create a personalized driving experience. Coway, a Korean home appliance company uses AWS for IoCare, its IoT service for tens of thousands of air & water purifiers. It migrated IoCare from on-premises to AWS for speed and efficiency to handle increasing traffic as their business grew. Coway uses AWS managed services such as AWS IoT, Amazon Kinesis, Amazon DynamoDB, AWS Lambda, Amazon RDS, and Amazon ElastiCache, which also integrated Alexa Skills with AWS Lambda with their high-end air purifier Airmega for the global market. Play Amazing Games AWS has transformed the nature of Korean gaming companies, allowing them to autonomously launch and expand their businesses globally without help from local publishers. As a result, the top 15 gaming companies in Korea are currently using AWS, including Nexon, NC Soft, Krafton, Netmarble, and KaKao Games. Krafton is the developer of the hit video game Player Unknown’s Battle Grounds (PUBG), which was developed on AWS in less than 18 months. The game uses AWS Lambda, Amazon SQS, and AWS CodeDeploy for its core backend service, Amazon DynamoDB as its primary game database, and Amazon Redshift as its data analytics platform. PUBG broke records upon release, with more than 3 million concurrent players connected to the game. Nexon, a top Korean gaming company to produce top mobile games such as Heroes of Incredible Tales (HIT). They achieved cost savings of more than 30 percent for global infrastructure management and can now launch new games quicker by using AWS. Nexon uses Amazon DynamoDB for its game database and first started using AWS to respond to unpredictable spikes in user demand. Startups to go Global Lots of hot startups in Korea are using AWS to grow the local market, but here are great examples to go global although they are based on Korea. Azar is Hyperconnect’s video-based social discovery mobile app recorded 300 million downloads and now widely accessible in over 200 countries around the world with 20 billion cumulative matches in last year. Overcoming complex matching issues for reliable video chats between users, Hyperconnect utilizes various AWS services efficiently, which uses Amazon EC2, Amazon RDS, and Amazon SES to save cost managing global infra, and Amazon S3 and Amazon CloudFront to store and deliver service data to global users faster. They also use Amazon EMR to manage the vast amount of data generated by 40 million matches per day. SendBird provides chat APIs and messaging SDK in more than 10 thousand apps globally processing about 700 million messages per month. It uses AWS global regions to provide a top-class customer experience by keeping low latency under 100 ms everywhere in the world. Amazon ElastiCache is currently used to handle large volumes of chat data, and all the data are stored in the encrypted Amazon Aurora for integrity and reliability. Server log data are analyzed and processed using the Amazon Kinesis Data Firehose as well as Amazon Athena. Freedom to Local Financial Industry We also see Korean enterprises in the financial services industry leverage AWS to digitally transform their businesses by using data analytics, fintech, and digital banking initiatives. Financial services companies in Korea are leveraging AWS to deliver an enhanced customer experience, and examples of these customers include Shinhan Financial Group, KB Kookmin Bank, Kakao Pay, Mirae Asset, and Yuanta Securities. Shinhan Financial Group achieved a 50 percent cost reduction and a 20 percent response-time reduction after migrating its North American and Japanese online banking services to AWS. Shinhan’s new Digital Platform unit now uses Amazon ECS, Amazon CloudFront, and other services to reduce development time for new applications by 50 percent. Shinhan is currently pursuing an all-in migration to AWS including moving more than 150 workloads. Hyundai Card, a top Korean credit card company and a financial subsidiary of the Hyundai Kia Motor Group, built a dev/test platform called Playground on AWS to prototype new software and services by the development team. The customer uses Amazon EMR, AWS Glue, and Amazon Kinesis for cost and architecture optimization. It allowed quick testing of new projects without waiting for resource allocation from on-premises infrastructure, reducing the development period by 3-4 months Security and Compliance At AWS, the security, privacy, and protection of customer data always come first, which AWS provides local needs as well as global security and compliances. Our most recent example of this commitment is that AWS became the first global cloud service provider to achieve the Korea-Information Security Management System certification (K-ISMS) in December 2017. With this certification, enterprises and organizations across Korea are able to meet its compliance requirements more effectively and accelerate business transformation by using best-in-class technology delivered from the highly secure and reliable AWS Cloud. AWS also completed its first annual surveillance audit for the K-ISMS certification in 2018. In April 2019, AWS achieved the Multi-Tier Cloud Security Standard (MTCS) Level-3 certification for Seoul region. AWS is also the first cloud service provider in Korea to do so. With the MTCS, FSI customers in Korea can accelerate cloud adoption by no longer having to validate 109 controls, as required in the relevant regulations (Financial Security Institute’s Guideline on Use of Cloud Computing Services in Financial Industry and the Regulation on Supervision on Electronic Financial Transactions (RSEFT). AWS also published a workbook for Korean FSI customer, covering those and 32 additional controls from the RSEFT. What to support and enable Korean customers AWS Korea has made significant investments in education and training in Korea. Tens of thousands of people including IT professionals, developers, and students have been trained in AWS cloud skills over the last two years. AWS Korea also supports community-driven activities to enhance the developer ecosystem of cloud computing in Korea. To date, the AWS Korean User Group has tens of thousands of members, who hold hundreds of meetups across Korea annually. AWS Educate program is expected to accelerate Korean students’ capabilities in cloud computing skills, helping them acquire cloud expertise that is becoming increasingly relevant for their future employment. Tens of universities including Sogang University, Yonsei University, and Seoul National University have joined this program with thousands of students participating in AWS-related classes and non-profit e-learning programs such as Like a Lion, a non-profit organization that teaches coding to students. AWS is building a vibrant cloud ecosystem with hundreds of partners ― Systems Integrator (SI) partners include LG CNS, Samsung SDS, Youngwoo Digital, Saltware, NDS, and many others. Among them, Megazone, GS Neotek, and Bespin Global are AWS Premier Consulting Partners. Independent Software Vendor (ISV) partners include AhnLab, Hancom, SK Infosec, SendBird, and IGAWorks. They help our customers to enable AWS services in their workloads to migrate from on-premise or launch new services. The customer’s celebration whiteboard for 5th anniversary of AWS Summit Seoul Finally, I want to introduce lots of customer’s feedback in our whiteboard of AWS Summit 2019 although they were written in Korean. Here is one voice from them ― “It made me decide to become an AWS customer voluntary to climb on the shoulders of the giant to see the world.” We always will hear customer’s voices and build the broadest and deepest cloud platform for them to leverage ours and be successful in both Korea and global market. – Channy Yun; This article was translated into Korean(한국어) in AWS Korea Blog.

Amazon S3 Path Deprecation Plan – The Rest of the Story

Amazon Web Services Blog -

Last week we made a fairly quiet (too quiet, in fact) announcement of our plan to slowly and carefully deprecate the path-based access model that is used to specify the address of an object in an S3 bucket. I spent some time talking to the S3 team in order to get a better understanding of the situation in order to write this blog post. Here’s what I learned… We launched S3 in early 2006. Jeff Bezos’ original spec for S3 was very succinct – he wanted malloc (a key memory allocation function for C programs) for the Internet. From that starting point, S3 has grown to the point where it now stores many trillions of objects and processes millions of requests per second for them. Over the intervening 13 years, we have added many new storage options, features, and security controls to S3. Old vs. New S3 currently supports two different addressing models: path-style and virtual-hosted style. Let’s take a quick look at each one. The path-style model looks like either this (the global S3 endpoint): https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg https://s3.amazonaws.com/jeffbarr-public/classic_amazon_door_desk.png Or this (one of the regional S3 endpoints): https://s3-us-east-2.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg https://s3-us-east-2.amazonaws.com/jeffbarr-public/classic_amazon_door_desk.png In this example, jbarr-public and jeffbarr-public are bucket names; /images/ritchie_and_thompson_pdp11.jpeg and /jeffbarr-public/classic_amazon_door_desk.png are object keys. Even though the objects are owned by distinct AWS accounts and are in different S3 buckets (and possibly in distinct AWS regions), both of them are in the DNS subdomain s3.amazonaws.com. Hold that thought while we look at the equivalent virtual-hosted style references (although you might think of these as “new,” they have been around since at least 2010): https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg https://jeffbarr-public.s3.amazonaws.com/classic_amazon_door_desk.png These URLs reference the same objects, but the objects are now in distinct DNS subdomains (jbarr-public.s3.amazonaws.com and jeffbarr-public.s3.amazonaws.com, respectively). The difference is subtle, but very important. When you use a URL to reference an object, DNS resolution is used to map the subdomain name to an IP address. With the path-style model, the subdomain is always s3.amazonaws.com or one of the regional endpoints; with the virtual-hosted style, the subdomain is specific to the bucket. This additional degree of endpoint specificity is the key that opens the door to many important improvements to S3. Out with the Old In response to feedback on the original deprecation plan that we announced last week, we are making an important change. Here’s the executive summary: Original Plan – Support for the path-style model ends on September 30, 2020. Revised Plan – Support for the path-style model continues for buckets created on or before September 30, 2020. Buckets created after that date must be referenced using the virtual-hosted model. We are moving to virtual-hosted references for two reasons: First, anticipating a world with billions of buckets homed in many dozens of regions, routing all incoming requests directly to a small set of endpoints makes less and less sense over time. DNS resolution, scaling, security, and traffic management (including DDoS protection) are more challenging with this centralized model. The virtual-hosted model reduces the area of impact (which we call the “blast radius” internally) when problems arise; this helps us to increase availability and performance. Second, the team has a lot of powerful features in the works, many of which depend on the use of unique, virtual-hosted style subdomains. Moving to this model will allow you to benefit from these new features as soon as they are announced. For example, we are planning to deprecate some of the oldest security ciphers and versions (details to come later). The deprecation process is easier and smoother (for you and for us) if you are using virtual-hosted references. In With the New As just one example of what becomes possible when using virtual-hosted references, we are thinking about providing you with increased control over the security configuration (including ciphers and cipher versions) for each bucket. If you have ideas of your own, feel free to get in touch. Moving Ahead Here are some things to know about our plans: Identifying Path-Style References – You can use S3 Access Logs (look for the Host Header field) and AWS CloudTrail Data Events (look for the host element of the requestParameters entry) to identify the applications that are making path-style requests. Programmatic Access – If your application accesses S3 using one of the AWS SDKs, you don’t need to do anything, other than ensuring that your SDK is current. The SDKs already use virtual-hosted references to S3, except if the bucket name contains one or more “.” characters. Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020. Non-Routable Names – Some characters that are valid in the path component of a URL are not valid as part of a domain name. Also, paths are case-sensitive, but domain and subdomain names are not. We’ve been enforcing more stringent rules for new bucket names since last year. If you have data in a bucket with a non-routable name and you want to switch to virtual-host requests, you can use the new S3 Batch Operations feature to move the data. However, if this is not a viable option, please reach out to AWS Developer Support. Documentation – We are planning to update the S3 Documentation to encourage all developers to build applications that use virtual-host requests. The Virtual Hosting documentation is a good starting point. We’re Here to Help The S3 team has been working with some of our customers to help them to migrate, and they are ready to work with many more. Our goal is to make this deprecation smooth and uneventful, and we want to help minimize any costs you may incur! Please do not hesitate to reach out to us if you have questions, challenges, or concerns. — Jeff; PS – Stay tuned for more information on tools and other resources.

New – The Next Generation (I3en) of I/O-Optimized EC2 Instances

Amazon Web Services Blog -

Amazon’s Customer Obsession leadership principle says: Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers. Starting from the customer and working backwards means that we do not invent in a vacuum. Instead, we speak directly to our customers (both external and internal), ask detailed questions, and pay attention to what we learn. On the AWS side, we often hear about new use cases that help us to get a better understanding of what our customers are doing with AWS. For example, large-scale EC2 users provide us with another set of interesting data points, often expressed in terms of ratios between dollars, vCPUs, memory size, storage size, and networking throughput. We launched the I3 instances (Now Available – I3 Instances for Demanding, I/O Intensive Workloads) just about two years ago. Our customers use them to host distributed file systems, relational & NoSQL databases, in-memory caches, key-value stores, data warehouses, and MapReduce clusters. Because our customers are always (in Jeff Bezos’ words) “divinely discontent”, they want I/O-optimized instances with even more power & storage. To be specific, they have asked us for: A lower price per TB of storage Increased storage density to allow consolidation of workloads and scale-up processing A higher ratio of network bandwidth and instance storage to vCPUs The crucial element here is that our customers were able to express their needs in a detailed and specific manner. Simply asking for something to be better, faster, and cheaper does not help us to make well-informed decisions. New I3en Instances Today I am happy to announce the I3en instances. Designed to meet these needs and to do an even better job of addressing the use cases that I mentioned above, these instances are powered by AWS-custom Intel Xeon Scalable (Skylake) processors with 3.1 GHz sustained all-core turbo performance, up to 60 TB of fast NVMe storage, and up to 100 Gbps of network bandwidth. Here are the specs: Instance Name vCPUs Memory Local Storage (NVMe SSD) Random Read IOPS (4 K Block) Read Throughput (128 K Block) EBS-Optimized Bandwidth Network Bandwidth i3en.large 2 16 GiB 1 x 1.25 TB 42.5 K 325 MB/s Up to 3,500 Mbps Up to 25 Gbps i3en.xlarge 4 32 GiB 1 x 2.50 TB 85 K 650 MB/s Up to 3,500 Mbps Up to 25 Gbps i3en.2xlarge 8 64 GiB 2 x 2.50 TB 170 K 1.3 GB/s Up to 3,500 Mbps Up to 25 Gbps i3en.3xlarge 12 96 GiB 1 x 7.5 TB 250 K 2 GB/s Up to 3,500 Mbps Up to 25 Gbps i3en.6xlarge 24 192 GiB 2 x 7.5 TB 500 K 4 GB/s 3,500 Mbps 25 Gbps i3en.12xlarge 48 384 GiB 4 x 7.5 TB 1 M 8 GB/s 7,000 Mbps 50 Gbps i3en.24xlarge 96 768 GiB 8 x 7.5 TB 2 M 16 GB/s 14,000 Mbps 100 Gbps In comparison to the I3 instances, the I3en instances offer: A cost per GB of SSD instance storage that is up to 50% lower Storage density (GB per vCPU) that is roughly 2.6x greater Ratio of network bandwidth to vCPUs that is up to 2.7x greater You will need HVM AMIs with the NVMe 1.0e and ENA drivers. You can also make use of the new Elastic Fabric Adapter (EFA) if you are using the i3en.24xlarge (read my recent post to learn more). Now Available You can launch I3en instances today in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) Regions in On-Demand and Spot form. Reserved Instances, Dedicated Instances, and Dedicated Hosts are available. — Jeff;    

Learn about AWS Services & Solutions – May AWS Online Tech Talks

Amazon Web Services Blog -

Join us this May to learn about AWS services and solutions. The AWS Online Tech Talks are live, online presentations that cover a broad range of topics at varying technical levels. These tech talks, led by AWS solutions architects and engineers, feature technical deep dives, live demonstrations, customer examples, and Q&A with AWS experts. Register Now! Note – All sessions are free and in Pacific Time. Tech talks this month: Compute May 30, 2019 | 11:00 AM – 12:00 PM PT – Your First HPC Cluster on AWS – Get a step-by-step walk-through of how to set up your first HPC cluster on AWS. Containers May 20, 2019 | 11:00 AM – 12:00 PM PT – Managing Application Deployments with the AWS Service Operator – Learn how the AWS Service Operator helps you provision AWS resources and your applications using kubectl CLI. May 22, 2019 | 9:00 AM – 10:00 AM PT – Microservice Deployment Strategies with AWS App Mesh – Learn how AWS App Mesh makes the process of deploying microservices on AWS easier and more reliable while providing you with greater control and visibility to support common deployment patterns. Data Lakes & Analytics May 20, 2019 | 9:00 AM – 10:00 AM PT – EKK is the New ELK: Aggregating, Analyzing and Visualizing Logs – Learn how to aggregate, analyze, & visualize your logs with Amazon Elasticsearch Service, Amazon Kinesis Data Firehose, and Kibana May 22, 2019 | 11:00 AM – 12:00 PM PT – Building a Data Streaming Application Leveraging Apache Flink – Learn how to build and manage a real-time streaming application with AWS using Java and leveraging Apache Flink. Databases May 20, 2019 | 1:00 PM – 2:00 PM PT – Migrating to Amazon DocumentDB – Learn how to migrate MongoDB workloads to Amazon DocumentDB, a fully managed MongoDB compatible database service designed from the ground up to be fast, scalable, and highly available. May 29, 2019 | 1:00 PM – 2:00 PM PT – AWS Fireside Chat: Optimize Your Business Continuity Strategy with Aurora Global Database – Join us for a discussion with the General Manager of Amazon Aurora to learn how you can use Aurora Global Database to build scalable and reliable apps. DevOps May 21, 2019 | 9:00 AM – 10:00 AM PT – Infrastructure as Code Testing Strategies with AWS CloudFormation – Learn about CloudFormation testing best practices, including what tools to use and when, both while authoring code and while testing in a continuous integration and continuous delivery pipeline. End-User Computing May 23, 2019 | 1:00 PM – 2:00 PM PT – Managing Amazon Linux WorkSpaces at Scale using AWS OpsWorks for Chef Automate – Learn how to simplify your Amazon Linux Workspaces management at scale by integrating with AWS OpsWorks for Chef Automate. Enterprise & Hybrid May 28, 2019 | 11:00 AM – 12:00 PM PT – What’s New in AWS Landing Zone – Learn how the AWS Landing Zone can automate the setup of best practice baselines when setting up new AWS environments. IoT May 29, 2019 | 11:00 AM – 12:30 PM PT – Customer Showcase: Extending Machine Learning to Industrial IoT Applications at the Edge – Learn how AWS IoT customers are building smarter products and bringing them to market quickly with Amazon FreeRTOS and AWS IoT Greengrass. Machine Learning May 28, 2019 | 9:00 AM – 10:00 AM PT – Build a Scalable Architecture to Automatically Extract and Import Form Data – Learn how to build a scalable architecture to process thousands of forms or documents with Amazon Textract. May 30, 2019 | 1:00 PM – 2:00 PM PT – Build Efficient and Accurate Recommendation Engines with Amazon Personalize – Learn how to build efficient and accurate recommendation engines with high impact to the bottom line of your business. Mobile May 21, 2019 | 1:00 PM – 2:00 PM PT – Deploying and Consuming Serverless Functions with AWS Amplify – Learn how to build and deploy serverless applications using React and AWS Amplify. Networking & Content Delivery May 31, 2019 | 1:00 PM – 2:00 PM PT – Simplify and Scale How You Connect Your Premises to AWS with AWS Direct Connect on AWS Transit Gateway – Learn how to use multi account Direct Connect gateway to interface your on-premises network with your AWS network through a AWS Transit Gateway. Security, Identity, & Compliance May 30, 2019 | 9:00 AM – 10:00 AM PT – Getting Started with Cross-Account Encryption Using AWS KMS, Featuring Slack Enterprise Key Management – Learn how to manage third-party access to your data keys with AWS KMS. May 31, 2019 | 9:00 AM – 10:00 AM PT – Authentication for Your Applications: Getting Started with Amazon Cognito – Learn how to use Amazon Cognito to add sign up and sign in to your web or mobile application. Serverless May 22, 2019 | 1:00 PM – 2:00 PM PT – Building Event-Driven Serverless Apps with AWS Event Fork Pipelines – Learn how to use Event Fork Pipelines, a new suite of open-source nested applications in the serverless application repository, to easily build event driven apps. Storage May 28, 2019 | 1:00 PM – 2:00 PM PT – Briefing: AWS Hybrid Cloud Storage and Edge Computing – Learn about AWS hybrid cloud storage and edge computing capabilities. May 23, 2019 | 11:00 AM – 12:00 PM PT – Managing Tens to Billions of Objects at Scale with S3 Batch Operations – Learn about S3 Batch Operations and how to manage billions of objects with a single API request in a few clicks.

Pages

Recommended Content

Subscribe to Complete Hosting Guide aggregator - Cloud Hosting Official Blogs