Cloud Hosting Blogs

Amazon Prime Day 2019 – Powered by AWS

Amazon Web Services Blog -

What did you buy for Prime Day? I bought a 34″ Alienware Gaming Monitor and used it to replace a pair of 25″ monitors that had served me well for the past six years:   As I have done in years past, I would like to share a few of the many ways that AWS helped to make Prime Day a reality for our customers. You can read How AWS Powered Amazon’s Biggest Day Ever and Prime Day 2017 – Powered by AWS to learn more about how we evaluate the results of each Prime Day and use what we learn to drive improvements to our systems and processes. This year I would like to focus on three ways that AWS helped to support record-breaking amounts of traffic and sales on Prime Day: Amazon Prime Video Infrastructure, AWS Database Infrastructure, and Amazon Compute Infrastructure. Let’s take a closer look at each one… Amazon Prime Video Infrastructure Amazon Prime members were able to enjoy the second Prime Day Concert (presented by Amazon Music) on July 10, 2019. Headlined by 10-time Grammy winner Taylor Swift, this live-streamed event also included performances from Dua Lipa, SZA, and Becky G. Live-streaming an event of this magnitude and complexity to an audience in over 200 countries required a considerable amount of planning and infrastructure. Our colleagues at Amazon Prime Video used multiple AWS Media Services including AWS Elemental MediaPackage and AWS Elemental live encoders to encode and package the video stream. The streaming setup made use of two AWS Regions, with a redundant pair of processing pipelines in each region. The pipelines delivered 1080p video at 30 fps to multiple content distribution networks (including Amazon CloudFront), and worked smoothly. AWS Database Infrastructure A combination of NoSQL and relational databases were used to deliver high availability and consistent performance at extreme scale during Prime Day: Amazon DynamoDB supports multiple high-traffic sites and systems including Alexa, the Amazon.com sites, and all 442 Amazon fulfillment centers. Across the 48 hours of Prime Day, these sources made 7.11 trillion calls to the DynamoDB API, peaking at 45.4 million requests per second. Amazon Aurora also supports the network of Amazon fulfillment centers. On Prime Day, 1,900 database instances processed 148 billion transactions, stored 609 terabytes of data, and transferred 306 terabytes of data. Amazon Compute Infrastructure Prime Day 2019 also relied on a massive, diverse collection of EC2 instances. The internal scaling metric for these instances is known as a server equivalent; Prime Day started off with 372K server equivalents and scaled up to 426K at peak. Those EC2 instances made great use of a massive fleet of Elastic Block Store (EBS) volumes. The team added an additional 63 petabytes of storage ahead of Prime Day; the resulting fleet handled 2.1 trillion requests per day and transferred 185 petabytes of data per day. And That’s a A Wrap These are some impressive numbers, and show you the kind of scale that you can achieve with AWS. As you can see, scaling up for one-time (or periodic) events and then scaling back down afterward, is easy and straightforward, even at world scale! If you want to run your own world-scale event, I’d advise you to check out the blog posts that I linked above, and also be sure to read about AWS Infrastructure Event Management. My colleagues are ready (and eager) to help you to plan for your large-scale product or application launch, infrastructure migration, or marketing event. Here’s an overview of their process:   — Jeff;

New – Trigger a Kernel Panic to Diagnose Unresponsive EC2 Instances

Amazon Web Services Blog -

When I was working on systems deployed in on-premises data centers, it sometimes happened I had to debug an unresponsive server. It usually involved asking someone to physically press a non-maskable interrupt (NMI) button on the frozen server or to send a signal to a command controller over a serial interface (yes, serial, such as in RS-232).This command triggered the system to dump the state of the frozen kernel to a file for further analysis. Such a file is usually called a core dump or a crash dump. The crash dump includes an image of the memory of the crashed process, the system registers, program counter, and other information useful in determining the root cause of the freeze. Today, we are announcing a new Amazon Elastic Compute Cloud (EC2) API allowing you to remotely trigger the generation of a kernel panic on EC2 instances. The EC2:SendDiagnosticInterrupt API sends a diagnostic interrupt, similar to pressing a NMI button on a physical machine, to a running EC2 instance. It causes the instance’s hypervisor to send a non-maskable interrupt (NMI) to the operating system. The behaviour of your operating system when a NMI interrupt is received depends on its configuration. Typically, it involves entering into kernel panic. The kernel panic behaviour also depends on the operating system configuration, it might trigger the generation of the crash dump data file, obtain a backtrace, load a replacement kernel or restart the system. You can control who in your organisation is authorized to use that API through IAM Policies, I will give an example below. Cloud and System Engineers, or specialists in kernel diagnosis and debugging, find in the crash dump invaluable information to analyse the causes of a kernel freeze. Tools like WinDbg (on Windows) and crash (on Linux) can be used to inspect the dump. Using Diagnostic Interrupt Using this API is a three step process. First you need to configure the behavior of your OS when it receives the interrupt. By default, our Windows Server AMIs have memory dump already turned on. Automatic restart after the memory dump has been saved is also selected. The default location for the memory dump file is %SystemRoot% which is equivalent to C:\Windows. You can access these options by going to : Start > Control Panel > System > Advanced System Settings > Startup and Recovery On Amazon Linux 2, you need to install and configurekdump & kexec. This is a one-time setup. $ sudo yum install kexec-tools Then edit the file /etc/default/grub to allocate the amount of memory to be reserved for the crash kernel. In this example, we reserve 160M by adding crashkernel=160M. The amount of memory to allocate depends on your instance’s memory size. The general recommendation is to test kdump to see if the allocated memory is sufficient. The kernel doc has the full syntax of the crashkernel kernel parameter. GRUB_CMDLINE_LINUX_DEFAULT="crashkernel=160M console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0" And rebuild the grub configuration: $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg Finally edit /etc/sysctl.conf and add a line : kernel.unknown_nmi_panic=1. This tells the kernel to trigger a kernel panic upon receiving the interrupt. You are now ready to reboot your instance. Be sure to include these commands in your user data script or in your AMI to automatically configure this on all your instances. Once the instance is rebooted, verify that kdump is correctly started. $ systemctl status kdump.service ● kdump.service - Crash recovery kernel arming Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2019-07-05 15:09:04 UTC; 3h 13min ago Process: 2494 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS) Main PID: 2494 (code=exited, status=0/SUCCESS) CGroup: /system.slice/kdump.service Jul 05 15:09:02 ip-172-31-15-244.ec2.internal systemd[1]: Starting Crash recovery kernel arming... Jul 05 15:09:04 ip-172-31-15-244.ec2.internal kdumpctl[2494]: kexec: loaded kdump kernel Jul 05 15:09:04 ip-172-31-15-244.ec2.internal kdumpctl[2494]: Starting kdump: [OK] Jul 05 15:09:04 ip-172-31-15-244.ec2.internal systemd[1]: Started Crash recovery kernel arming. Our documentation contains the instructions for other operating systems. Once this one-time configuration is done, you’re ready for the second step, to trigger the API. You can do this from any machine where the AWS CLI or SDK is configured. For example : $ aws ec2 send-diagnostic-interrupt --region us-east-1 --instance-id <value> There is no return value from the CLI, this is expected. If you have a terminal session open on that instance, it disconnects. Your instance reboots. You reconnect to your instance, you find the crash dump in /var/crash. The third and last step is to analyse the content of the crash dump. On Linux systems, you need to install the crash utility and the debugging symbols for your version of the kernel. Note that the kernel version should be the same that was captured by kdump. To find out which kernel you are currently running, use the uname -r command. $ sudo yum install crash $ sudo debuginfo-install kernel $ sudo crash /usr/lib/debug/lib/modules/4.14.128-112.105.amzn2.x86_64/vmlinux /var/crash/127.0.0.1-2019-07-05-15\:08\:43/vmcore crash 7.2.6-1.amzn2.0.1 ... output suppressed for brevity ... KERNEL: /usr/lib/debug/lib/modules/4.14.128-112.105.amzn2.x86_64/vmlinux DUMPFILE: /var/crash/127.0.0.1-2019-07-05-15:08:43/vmcore [PARTIAL DUMP] CPUS: 2 DATE: Fri Jul 5 15:08:38 2019 UPTIME: 00:07:23 LOAD AVERAGE: 0.00, 0.00, 0.00 TASKS: 104 NODENAME: ip-172-31-15-244.ec2.internal RELEASE: 4.14.128-112.105.amzn2.x86_64 VERSION: #1 SMP Wed Jun 19 16:53:40 UTC 2019 MACHINE: x86_64 (2500 Mhz) MEMORY: 7.9 GB PANIC: "Kernel panic - not syncing: NMI: Not continuing" PID: 0 COMMAND: "swapper/0" TASK: ffffffff82013480 (1 of 2) [THREAD_INFO: ffffffff82013480] CPU: 0 STATE: TASK_RUNNING (PANIC) Collecting kernel crash dumps is often the only way to collect kernel debugging information, be sure to test this procedure frequently, in particular after updating your operating system or when you will create new AMIs. Control Who Is Authorized to Send Diagnostic Interrupt You can control who in your organisation is authorized to send the Diagnostic Interrupt, and to which instances, through IAM policies with resource-level permissions, like in the example below. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ec2:SendDiagnosticInterrupt", "Resource": "arn:aws:ec2:region:account-id:instance/instance-id" } ] } Pricing There are no additional charges for using this feature. However, as your instance continues to be in a ‘running’ state after it receives the diagnostic interrupt, instance billing will continue as usual. Availability You can send Diagnostic Interrupts to all EC2 instances powered by the AWS Nitro System, except A1 (Arm-based). This is C5, C5d, C5n, i3.metal, I3en, M5, M5a, M5ad, M5d, p3dn.24xlarge, R5, R5a, R5ad, R5d, T3, T3a, and Z1d as I write this. The Diagnostic Interrupt API is now available in all public AWS Regions and GovCloud (US), you can start to use it today. -- seb

Amplify Console – Hosting for Fullstack Serverless Web Apps

Amazon Web Services Blog -

AWS Amplify Console is a fullstack web app hosting service, with continuous deployment from your preferred source code repository. Amplify Console has been introduced in November 2018 at AWS re:Invent. Since then, the team has been listening to customer feedback and iterated quickly to release several new features, here is a short re:Cap. Instant Cache Invalidation Amplify Console allows to host single page web apps or static sites with serverless backends via a content delivery network, or CDN. A CDN is a network of distributed servers that cache files at edge locations across the world enabling low latency distribution of your web file assets. Previously, updating content on the CDN required manually invalidating the cache and waiting 15-20 minutes for changes to propagate globally. To make frequent updates, developers found workarounds such as setting lower time-to-live (TTLs) on asset headers which enables faster updates, but adversely impacts performance. Now, you no longer have to make a tradeoff between faster deployments and faster performance. On every commit code to your repository, the Amplify Console builds and deploys changes to the CDN that are viewable immediately in the browser. “Deploy To Amplify Console” Button When publishing your project source code on GitHub, you can make it easy for other developers to build and deploy your application by providing a “Deploy To Amplify Console” button in the Readme document. Clicking on that button will open Amplify Console and propose a three step process to deploy your code. You test this yourself with these example projects and have a look at the documentation. Adding a button to your own code repository is as easy as adding this line in your Readme document (be sure to replace the username and repository name in the GitHub URL): [![amplifybutton](https://oneclick.amplifyapp.com/button.svg)](https://console.aws.amazon.com/amplify/home#/deploy?repo=https://github.com/username/repository) Manual Deploy I think it is a good idea to version control everything, including simple web site where you are the only developer. But just in case you do not want to use a source code repository as source for your deployment, Amplify Console allows to deploy a zip file, a local folder on your laptop, an Amazon S3 bucket or any HTTPS URL, such as a shared repository on Dropbox. When creating a new Amplify Console project, select Deploy without Git Provider option.  Then choose your source file (your laptop, Amazon S3 or an HTTPS URI) AWS CloudFormation Integration Developers love automation. Deploying code or infrastructure is no different : you must ensure your infrastructure deployments are automated and repeatable. AWS CloudFormation allows you to automate the creation of infrastruture in the cloud based on a YAML or JSON description. Amplify Console added three new resource types to AWS CloudFormation: AWS::Amplify::App AWS::Amplify::Branch AWS::Amplify::Domain These allows you respectively to create a new Amplify Console app, to define the Git branch, and the DNS domain name to use. AWS CloudFormation connects to your source code repository to add a webhook to it. You need to include your Github Personal Access Token to allow this to happen, this blog post has all the details. Remember to not hardcode credentials (or OAuth tokens) into your Cloudformation templates, use parameters instead. Deploy Multiple Git Branches We believe your CI/CD tools must adapt to your team workflow, not the other way around. Amplify Console supports branch pattern deployments, allowing you to automatically deploy branches that match a specific pattern without any extra configuration. Pattern matching is based on regular expresssions. When you want to test a new feature, you typically create a new branch in Git. Amplify Console and the Amplify CLI are now detecting this and will provision a separate backend and hosting infrastructure for your serverless app. To enable branch detection, use the left menu, click on General > Edit and turn on Branch Autodetection: Custom HTTP Headers You can customize Amplify Console to send customized HTTP response headers. Response headers can be used for debugging, security, or informational purposes. To add your custom headers, you select App Settings > Build Settings and then edit the buildspec. For example, to enforce TLS transport and prevent XSS attacks, you can add the following headers: customHeaders: - pattern: '**/*' headers: - key: 'Strict-Transport-Security' value: 'max-age=31536000; includeSubDomains' - key: 'X-Frame-Options' value: 'X-Frame-Options: SAMEORIGIN' - key: 'X-XSS-Protection' value: 'X-XSS-Protection: 1; mode=block' - key: 'X-Content-Type-Options' value: 'X-Content-Type-Options: nosniff' - key: 'Content-Security-Policy' value: "default-src 'self'" The documentation has more details. Custom Containers for Build Last but not least, we made several changes to the build environment. Amplify Console uses AWS CodeBuild behind the scenes. The default build container image is now based on Amazon Linux 2 and has Serverless Application Model (SAM) CLI pre-installed. If, for whatever reasons you want to use your own container for the build, you can configure Amplify Console to do so. Select App Settings > Build Settings : And then edit the build image setting There are a few requirements on the container image: it has to have cURL, git, OpenSSH and, if you are building NodeJS projects, node and npm. As usual, the details are in the documentation. Each of these new features has been driven by your feedback, so please continue to tell us what is important for you by submittin, and expect to see more changes coming in the second part of the year and beyond. -- seb

Using callback URLs for approval emails with AWS Step Functions

Amazon Web Services Blog -

Guest post by Cloud Robotics Research Scientist at iRobot and AWS Serverless Hero, Ben Kehoe AWS Step Functions is a serverless workflow orchestration service that lets you coordinate processes using the declarative Amazon States Language. When you have a Step Functions task that takes more than fifteen minutes, you can’t use an AWS Lambda function—Step Functions provides the callback pattern for us in this situation. Approval emails are a common use case in this category. In this post, I show you how to create a Step Functions state machine that uses the sfn-callback-urls application for an email approval step. The app is available in the AWS Serverless Application Repository. The state machine sends an email containing approve/reject links, and later a confirmation email. You can easily expand this state machine for your use cases. Solution overview An approval email must include URLs that send the appropriate result back to Step Functions when the user clicks on them. The URL should be valid for an extended period of time, longer than presigned URLs—what if the user is on vacation this week? Ideally, this doesn’t involve storage of the token and the maintenance that requires. Luckily, there’s an AWS Serverless Application Repository app for that! The sfn-callback-urls app allows you to generate one-time-use callback URLs through a call to either an Amazon API Gateway or a Lambda function. Each URL has an associated name, whether it causes success or failure, and what output should be sent back to Step Functions. Sending an HTTP GET or POST to a URL sends its output to Step Functions. sfn-callback-urls is stateless, and it also supports POST callbacks with JSON bodies for use with webhooks. Deploying the app First, deploy the sfn-callback-urls serverless app and make note of the ARN for the Lambda function that it exposes. In the AWS Serverless Application Repository console, select Show apps that create custom IAM roles or resource policies, and search for sfn-callback-urls.  You can also access the application. Under application settings, select the box to acknowledge the creation of IAM resources. By default, this app creates a KMS key. You can disable this by setting the DisableEncryption parameter to true, but first read the Security section in the Readme to the left. Scroll down and choose Deploy. On the deployment confirmation page, choose CreateUrls, which opens the Lambda console for that function. Make note of the function ARN because you need it later. Create the application by doing the following: Create an SNS topic and subscribe your email to it. Create the Lambda function that handles URL creation and email sending, and add proper permissions. Create an IAM role for the state machine to invoke the Lambda function. Create the state machine. Start the execution and send yourself some emails! Create the SNS topic In the SNS console, choose Topics, Create Topic. Name the topic ApprovalEmailsTopic, and choose Create Topic. Make a note of the topic ARN, for example arn:aws:sns:us-east-2:012345678912:ApprovalEmailsTopic. Now, set up a subscription to receive emails. Choose Create subscription. For Protocol, choose Email, enter an email address, and choose Create subscription. Wait for an email to arrive in your inbox with a confirmation link. It confirms the subscription, allowing messages published to the topic to be emailed to you. Create the Lambda function Now create the Lambda function that handles the creation of callback URLs and sending of emails. For this short post, create a single Lambda function that completes two separate steps: Creating callback URLs Sending the approval email, and later sending the confirmation email There’s an if statement in the code to separate the two, which requires the state machine to tell the Lambda function which state is invoking it. The best practice here would be to use two separate Lambda functions. To create the Lambda function in the Lambda console, choose Create function, name it ApprovalEmailsFunction, and select the latest Python 3 runtime. Under Permissions, choose Create a new Role with basic permissions, Create. Add permissions by scrolling down to Configuration. Choose the link to see the role in the IAM console. Add IAM permissions In the IAM console, select the new role and choose Add inline policy. Add permissions for sns:Publish to the topic that you created and lambda:InvokeFunction to the sfn-callback-urls CreateUrls function ARN. Back in the Lambda console, use the following code in the function: import json, os, boto3 def lambda_handler(event, context): print('Event:', json.dumps(event)) # Switch between the two blocks of code to run # This is normally in separate functions if event['step'] == 'SendApprovalRequest': print('Calling sfn-callback-urls app') input = { # Step Functions gives us this callback token # sfn-callback-urls needs it to be able to complete the task "token": event['token'], "actions": [ # The approval action that transfers the name to the output { "name": "approve", "type": "success", "output": { # watch for re-use of this field below "name_in_output": event['name_in_input'] } }, # The rejection action that names the rejecter { "name": "reject", "type": "failure", "error": "rejected", "cause": event['name_in_input'] + " rejected it" } ] } response = boto3.client('lambda').invoke( FunctionName=os.environ['CREATE_URLS_FUNCTION'], Payload=json.dumps(input) ) urls = json.loads(response['Payload'].read())['urls'] print('Got urls:', urls) # Compose email email_subject = 'Step Functions example approval request' email_body = """Hello {name}, Click below (these could be better in HTML emails): Approve: {approve} Reject: {reject} """.format( name=event['name_in_input'], approve=urls['approve'], reject=urls['reject'] ) elif event['step'] == 'SendConfirmation': # Compose email email_subject = 'Step Functions example complete' if 'Error' in event['output']: email_body = """Hello, Your task was rejected: {cause} """.format( cause=event['output']['Cause'] ) else: email_body = """Hello {name}, Your task is complete. """.format( name=event['output']['name_in_output'] ) else: raise ValueError print('Sending email:', email_body) boto3.client('sns').publish( TopicArn=os.environ['TOPIC_ARN'], Subject=email_subject, Message=email_body ) print('done') return {} Now, set the following environment variables TOPIC_ARN and CREATE_URLS_FUNCTION to the ARNs of your topic and the sfn-callback-urls function noted earlier. After updating the code and environment variables, choose Save.   Create the state machine You first need a role for the state machine to assume that can invoke the new Lambda function. In the IAM console, create a role with Step Functions as its trusted entity. This requires the AWSLambdaRole policy, which gives it access to invoke your function. Name the role ApprovalEmailsStateMachineRole. Now you’re ready to create the state machine. In the Step Functions console, choose Create state machine, name it ApprovalEmails, and use the following code: { "Version": "1.0", "StartAt": "SendApprovalRequest", "States": { "SendApprovalRequest": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "Parameters": { "FunctionName": "ApprovalEmailsFunction", "Payload": { "step.$": "$$.State.Name", "name_in_input.$": "$.name", "token.$": "$$.Task.Token" } }, "ResultPath": "$.output", "Next": "SendConfirmation", "Catch": [ { "ErrorEquals": [ "rejected" ], "ResultPath": "$.output", "Next": "SendConfirmation" } ] }, "SendConfirmation": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "ApprovalEmailsFunction", "Payload": { "step.$": "$$.State.Name", "output.$": "$.output" } }, "End": true } } } This state machine has two states. It takes as input a JSON object with one field, “name”. Each state is a Lambda task. To shorten this post, I combined the functionality for both states in a single Lambda function. You pass the state name as the step field to allow the function to choose the block of code to run. Using the best practice of using separate functions for different responsibilities, this field would not be necessary. The first state, SendApprovalRequest, expects an input JSON object with a name field. It packages that name along with the step and the task token (required to complete the callback task), and invokes the Lambda function with it. Whatever output is received as part of the callback, the state machine stores it in the output JSON object under the output field. That output then becomes the input to the second state. The second state, SendConfirmation, takes that output field along with the step and invokes the function again. The second invocation does not use the callback pattern and doesn’t involve a task token. Start the execution To run the example, choose Start execution and set the input to be a JSON object that looks like the following: {   "name": "Ben" } You see the execution graph with the SendApprovalRequest state highlighted. This means it has started and is waiting for the task token to be returned. Check your inbox for an email with approve and reject links. Choose a link and get a confirmation page in the browser saying that your response has been accepted. In the State Machines console, you see that the execution has finished, and you also receive a confirmation email for approval or rejection. Conclusion In this post, I demonstrated how to use the sfn-callback-urls app from the AWS Serverless Application Repository to create URLs for approval emails. I also showed you how to build a system that can create and send those emails and process the results. This pattern can be used as part of a larger state machine to manage your own workflow. This example is also available as an AWS CloudFormation template in the sfn-callback-urls GitHub repository. Ben Kehoe

Preview Release of the new AWS Tools for PowerShell

Amazon Web Services Blog -

In 2012 we announced the first version of the AWS Tools for PowerShell module for Windows PowerShell, containing around 550 cmdlets supporting 20 or so services. In the years since, the growth of AWS has expanded the module to almost 6000 cmdlets spanning 160+ services, plus an additional (but identical) module for users of PowerShell 6 or higher that is capable of being run cross-platform. There are downsides to putting all those cmdlets into a single module (AWSPowerShell for PowerShell v2 through v5.1 on Windows, AWSPowerShell.NetCore for PowerShell v6 or higher on Windows, macOS, and Linux). First, the import time of the modules has grown significantly. On my 8th Gen Core i7 laptop the time to import either module has grown beyond 25 seconds. Second, the team discovered an issue with listing all of the cmdlets in the module manifests and subsequently had to revert to specifying ‘*’ for the CmdletsToExport manifest property. This prevents PowerShell from determining the cmdlets in the modules until they are explicitly imported, impacting tab completion of cmdlet names. In my shell profile I use the Set-AWSCredential and Set-DefaultAWSRegion cmdlets to set an initial scope for my shells. Thus I have to first explicitly import the module and then wait for the shell to become usable. This slow load time is obviously unsustainable, even more so when writing AWS Lambda functions in PowerShell when we particularly want a fast startup time. Announcing the Refactored AWS Tools for PowerShell Modules (Preview) Today the team released a new set of modules to the PowerShell Gallery to address this issue. These modules are in preview so that the team can gather your feedback (good or bad) which we hope you’ll give! In the preview release each AWS service now has its own PowerShell module, all depending on a common shared module named AWS.Tools.Common (this is the same modular approach that we take with the AWS SDK for .NET on NuGet). This has a number of implications: Instead of downloading and installing a single large module for all services, you can now install only the modules for services you actually need. The common module will be installed automatically when you install a service-specific module. You no longer need to explicitly import any of the preview modules before use, as the CmdletsToExport manifest property for each module is now properly specified. The versioning strategy for the new modules currently follows the AWSPowerShell and AWSPowerShell.NetCore modules. The strategy is detailed on the team’s GitHub repository notice for the preview and we welcome your feedback on it. Shell startup time is fast! On the same system I noted earlier the load time for my command shells is now between 1 and 2 seconds on average. The only change to my shell profile was to remove the explicit module import. The new modules follow the name pattern AWS.Tools.ServiceName. In some cases the more common contraction is used for the name. For example: AWS.Tools.EC2 AWS.Tools.S3 AWS.Tools.DirectoryService AWS.Tools.ElasticLoadBalancingV2 AWS.Tools.Polly AWS.Tools.Rekognition etc If you are writing PowerShell functions for AWS Lambda, be sure to update your script dependencies (using the #Requires statement) to use the new modules. You also need to add a #Requires statement for the common module. For example if I am writing a Lambda function in PowerShell that uses Amazon Simple Storage Service (S3) then I need to add the following two statements to my function’s script file: #Requires -Modules @{ModuleName='AWS.Tools.Common';ModuleVersion='3.3.563.0'} #Requires -Modules @{ModuleName='AWS.Tools.S3';ModuleVersion='3.3.563.0'} Mandatory Parameters The team has also addressed another long-standing and popular request from users – that of marking parameters as mandatory. Mandatory parameters are a great feature of PowerShell, helping guide users who are unfamiliar with APIs, and we’re very pleased to now be in a position to support them. The marking of mandatory parameters is dependent on data in the service models so if you discover any issues please let the team know in the link at the end of this post so that they can investigate and have the service models corrected if need be. Other Preview Changes The development team has also taken the opportunity to remove some old and obsolete cmdlets. If you need to use any of these removed cmdlets you will need to continue using the existing modules for the time being, but be sure to raise an issue on GitHub so that the team can consider supporting them in the new version: CloudHSM (HSM) is removed in favor of CloudHSMV2 (HSM2) ElasticLoadBalancing (ELB) is removed in favor of ElasticLoadBalancingV2 (ELB2) CloudWatchEvents (CWE) is removed in favor of EventBridge (EVB) KinesisAnalytics (KINA) is removed in favor of KinesisAnalyticsV2 (KINA2) What happens to the AWSPowerShell and AWSPowerShell.NetCore modules? Nothing! These modules will remain and will be updated in sync with the preview for the foreseeable future. Backwards compatibility is taken very seriously at AWS and we don’t want to deter use of these modules until we know the community is happy with the replacements. Note that you cannot mix the two different sets of modules. For example if you have the AWSPowerShell (or AWSPowerShell.NetCore) module loaded then an attempt to load modules from the preview will fail with an error. Get involved! The new preview modules are now available in the PowerShell Gallery and more details about the release can be found in Matteo’s notice on GitHub. The development team is eager to hear your feedback on the preview – do you like (or not like) the new modular format? Have you encountered any issues with the new support for marking mandatory parameters? Any other backwards compatibility issues you’ve found? Thoughts on the versioning strategy that should be adopted? Be sure to let them know on their GitHub issues repository! — Steve  

AWS Lake Formation – Now Generally Available

Amazon Web Services Blog -

As soon as companies started to have data in digital format, it was possible for them to build a data warehouse, collecting data from their operational systems, such as Customer relationship management (CRM) and Enterprise resource planning (ERP) systems, and use this information to support their business decisions. The reduction in costs of storage, together with an even greater reduction in complexity for managing large quantities of data, made possible by services such as Amazon S3, has allowed companies to retain more information, including raw data that is not structured, such as logs, images, video, and scanned documents. This is the idea of a data lake: to store all your data in one, centralized repository, at any scale. We are seeing this approach with customers like Netflix, Zillow, NASDAQ, Yelp, iRobot, FINRA, and Lyft. They can run their analytics on this larger dataset, from simple aggregations to complex machine learning algorithms, to better discover patterns in their data and understand their business. Last year at re:Invent we introduced in preview AWS Lake Formation, a service that makes it easy to ingest, clean, catalog, transform, and secure your data and make it available for analytics and machine learning. I am happy to share that Lake Formation is generally available today! With Lake Formation you have a central console to manage your data lake, for example to configure the jobs that move data from multiple sources, such as databases and logs, to your data lake. Having such a large and diversified amount of data makes configuring the right access permission also critical. You can secure access to metadata in the Glue Data Catalog and data stored in S3 using a single set of granular data access policies defined in Lake Formation. These policies allow you to define table and column-level data access. One thing I like the most of Lake Formation is that it works with your data already in S3! You can easily register your existing data with Lake Formation, and you don’t need to change existing processes loading your data to S3. Since data remains in your account, you have full control. You can also use Glue ML Transforms to easily deduplicate your data. Deduplication is important to reduce the amount of storage you need, but also to make analyzing your data more efficient because you don’t have neither the overhead nor the possible confusion of looking at the same data twice. This problem is trivial if duplicate records can be identified by a unique key, but becomes very challenging when you have to do a “fuzzy match”. A similar approach can be used for record linkage, that is when you are looking for similar items in different tables, for example to do a “fuzzy join” of two databases that do not share a unique key. In this way, implementing a data lake from scratch is much faster, and managing a data lake is much easier, making these technologies available to more customers. Creating a Data Lake Let’s build a data lake using the Lake Formation console. First I register the S3 buckets that are going to be part of my data lake. Then I create a database and grant permission to the IAM users and roles that I am going to use to manage my data lake. The database is registered in the Glue Data Catalog and holds the metadata required to analyze the raw data, such as the structure of the tables that are going to be automatically generated during data ingestion. Managing permissions is one of the most complex tasks for a data lake. Consider for example the huge amount of data that can be part of it, the sensitive, mission-critical nature of some of the data, and the different structured, semi-structured, and unstructured formats in which data can reside. Lake Formation makes it easier with a central location where you can give IAM users, roles, groups, and Active Directory users (via federation) access to databases, tables, optionally allowing or denying access to specific columns within a table. To simplify data ingestion, I can use blueprints that create the necessary workflows, crawlers and jobs on AWS Glue for common use cases. Workflows enable orchestration of your data loading workloads by building dependencies between Glue entities, such as triggers, crawlers and jobs, and allow you to track visually the status of the different nodes in the workflows on the console, making it easier to monitor progress and troubleshoot issues. Database blueprints help load data from operational databases. For example, if you have an e-commerce website, you can ingest all your orders in your data lake. You can load a full snapshot from an existing database, or incrementally load new data. In case of an incremental load, you can select a table and one or more of its columns as bookmark keys (for example, a timestamp in your orders) to determine previously imported data. Log file blueprints simplify ingesting logging formats used by Application Load Balancers, Elastic Load Balancers, and AWS CloudTrail. Let’s see how that works more in depth. Security is always a top priority, and I want to be able to have a forensic log of all management operations across my account, so I choose the CloudTrail blueprint. As source, I select a trail collecting my CloudTrail logs from all regions into an S3 bucket. In this way, I’ll be able to query account activity across all my AWS infrastructure. This works similarly for a larger organization having multiple AWS accounts: they just need, when configuring the trail in the CloudTrial console, to apply the trail to their whole organization. I then select the target database, and the S3 location for the data lake. As data format I use Parquet, a columnar storage format that will make querying the data faster and cheaper. The import frequency can be hourly to monthly, with the option to choose the day of the week and the time. For now, I want to run the workflow on demand. I can do that from the console or programmatically, for example using any AWS SDK or the AWS Command Line Interface (CLI). Finally, I give the workflow a name, the IAM role to use during execution, and a prefix for the tables that will be automatically created by this workflow. I start the workflow from the Lake Formation console and select to view the workflow graph. This opens the AWS Glue console, where I can visually see the steps of the workflow and monitor the progress of this run. When the workflow is completed a new table is available in my data lake database. The source data remain as logs in the S3 bucket output of CloudTrail, but now I have them consolidated, in Parquet format and partitioned by date, in my data lake S3 location. To optimize costs, I can set up an S3 lifecycle policy that automatically expires data in the source S3 bucket after a safe amount of time has passed. Securing Access to the Data Lake Lake Formation provides secure and granular access to data stores in the data lake, via a new grant/revoke permissions model that augments IAM policies. It is simple to set up these permissions, for example using the console: I simply select the IAM user or role I want to grant access to. Then I select the database and optionally the tables and the columns I want to provide access to. It is also possible to select which type of access to provide. For this demo, simple select permissions are sufficient. Accessing the Data Lake Now I can query the data using tools like Amazon Athena or Amazon Redshift. For example, I open the query editor in the Athena console. First, I want to use my new data lake to look into which source IP addresses are most common in my AWS Account activity: SELECT sourceipaddress, count(*) FROM my_trail_cloudtrail GROUP BY sourceipaddress ORDER BY 2 DESC; Looking at the result of the query, you can see which are the AWS API endpoints that I use the most. Then, I’d like to check which user identity types are used. That is an information stored in JSON format inside one of the columns. I can use some of the JSON functions available with Amazon Athena to get that information in my SQL statements: SELECT json_extract_scalar(useridentity, '$.type'), count(*) FROM "mylake"."my_trail_cloudtrail" GROUP BY json_extract_scalar(useridentity, '$.type') ORDER BY 2 DESC; Most of the times, AWS services are the ones creating activities in my trail. These queries are just an example, but give me quickly a deeper insight in what is happening in my AWS account. Think of what could be a similar impact for your business! Using database and logs blueprints, you can quickly create workflows to ingest data from multiple sources within your organization, set the right permission at column level of who can have access to any information collected, clean and prepare your data using machine learning transforms, and correlate and visualize the information using tools like Amazon Athena, Amazon Redshift, and Amazon QuickSight. Customizing Data Access with Column-Level Permissions In order to follow data privacy guidelines and compliance, the mission-critical data stored in a data lake requires to create custom views for different stakeholders inside the company. Let’s compare the visibility of two IAM users in my AWS account, one that has full permissions on a table, and one that has only select access to a subset of the columns of the same table. I already have a user with full access to the table containing my CloudTrail data, it’s called danilop. I create a new limitedview IAM user and I give it access to the Athena console. In the Lake Formation console, I only give this new user select permissions on three of the columns. To verify the different access to the data in the table, I log in with one user at a time and go to the Athena console. On the left I can explore which tables and columns the logged-in user can see in the Glue Data Catalog. Here’s a comparison for the two users, side-by-side: The limited user has access only to the three columns that I explicitly configured, and to the four columns used for partitioning the table, whose access is required to see any data. When I query the table in the Athena console with a select * SQL statement, logged in as the limitedview user, I only see data from those seven columns: Available Now There is no additional cost in using AWS Lake Formation, you pay for the use of the underlying services such as Amazon S3 and AWS Glue. One of the core benefits of Lake Formation are the security policies it is introducing. Previously you had to use separate policies to secure data and metadata access, and these policies only allowed table-level access. Now you can give access to each user, from a central location, only to the the columns they need to use. AWS Lake Formation is now available in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo). Redshift integration with Lake Formation requires Redshift cluster version 1.0.8610 or higher, your clusters should have been automatically updated by the time you read this. Support for Apache Spark with Amazon EMR will follow over the next few months. I only scratched the surface of what you can do with Lake Formation. Building and managing a data lake for your business is now much easier, let me know how you are using these new capabilities! — Danilo

New – Local Mocking and Testing with the Amplify CLI

Amazon Web Services Blog -

The open source Amplify Framework provides a set of libraries, user interface (UI) components, and a command line interface (CLI) that make it easier to add sophisticated cloud features to your web or mobile apps by provisioning backend resources using AWS CloudFormation. A comment I often get when talking with our customers, is that when you are adding new features or solving bugs, it is important to iterate as fast as possible, getting a quick feedback from your actions. How can we improve their development experience? Well, last week the Amplify team launched the new Predictions category, to let you quickly add machine learning capabilities to your web or mobile app. Today, they are doing it again. I am very happy to share that you can now use the Amplify CLI to mock some of the most common cloud services it provides, and test your application 100% locally! By mocking here I mean that instead of using the actual backend component, an API in the case of cloud services, a local, simplified emulation of that API is available instead. This emulation provides the basic functionality that you need for testing during development, but not the full behavior you’d get from the production service. With this new mocking capability you can test your changes quickly, without the need of provisioning or updating the cloud resources you are using at every step. In this way, you can set up unit and integration tests that can be executed rapidly, without affecting your cloud backend. Depending on the architecture of your app, you can set up automatic testing in your CI/CD pipeline without provisioning backend resources. This is really useful when editing AWS AppSync resolver mapping templates, written in Apache Velocity Template Language (VTL), which take your requests as input, and output a JSON document containing the instructions for the resolver. You can now have immediate feedback on your edits, and test if your resolvers work as expected without having to wait for a deployment for every update. For this first release, the Amplify CLI can mock locally: AppSync GraphQL APIs, including resolver mapping templates and storage backed by Amazon DynamoDB. AWS Lambda functions invoked directly or as resolvers of a GraphQL API. Amazon Simple Storage Service (S3) buckets used as storage for your application. Amazon Cognito User Pool authentication for GraphQL APIs, but you need first to get a JSON Web Token (JWT) from the actual service; after that, the JWT is honored locally. API Mocking Let’s do a quick overview of what you can do. For example, let’s create a sample app that helps people store and share the location of those nice places that allow you to refill your reusable water bottle and reduce plastic waste. To install the Amplify CLI, I need Node.js (version >= 8.11.x) and npm (version >= 5.x): npm install -g @aws-amplify/cli amplify configure Amplify supports lots of different frameworks, for this example I am using React and I start with a sample app (npx requires npm >= 5.2.x): npx create-react-app refillapp cd refillapp I use the Amplify CLI to inizialize the project and add an API. The Amplify CLI are interactive, asking you questions that drive the configuration of your backend. In this case, when asked, I select to add a GraphQL API. amplify init amplify add api During the creation of the API, I edit the GraphQL schema, and define a RefillLocation in this way: type RefillLocation @model { id: ID! name: String! description: String streetAddress: String! city: String! stateProvinceOrRegion: String zipCode: String! countryCode: String! } The fields that have an exclamation mark ! at the end are mandatory. The other fields are optional, and can be omitted when creating a new object. The @model you see in the first line is a directive using GraphQL Transform to define top level object types in your API that are backed by DynamoDB and generate for you all the necessary CRUDL (create, read, update, delete, and list) queries and mutations, and the subscriptions to be notified of such mutations. Now, I would normally need to run amplify push to configure and provision the backend resources required by the project (AppSync and DynamoDB in this case). But to get a quick feedback, I use the new local mocking capability running this command: amplify mock Alternatively, I can use the amplify mock api command to specifically mock just my GraphQL API. It would be the same at this stage, but it can be handy when using more than one mocking capability at a time. The output of the mock command gives you some information on what it does, and what you can do, including the AppSync Mock endpoint: GraphQL schema compiled successfully. Edit your schema at /MyCode/refillapp/amplify/backend/api/refillapp/schema.graphql or place .graphql files in a directory at /MyCode/refillapp/amplify/backend/api/refillapp/schema Creating table RefillLocationTable locally Running GraphQL codegen ✔ Generated GraphQL operations successfully and saved at src/graphql AppSync Mock endpoint is running at http://localhost:20002 I keep the mock command running in a terminal window to get feedback of possible errors in my code. For example, when I edit a VTL template, the Amplify CLI recognizes that immediately, and generates the updated code for the resolver. In case of a mistake, I get an error from the running mock command. The AppSync Mock endpoint gives you access to: the GraphQL transformations required by your API DynamoDB Local to manage your API data locally the Amplify GraphQL Explorer, based on the open source OneGraph graphiql-explorer plugin I can now run GraphQL queries, mutations, and subscriptions locally for my API, using a web interface. For example, to create a new RefillLocation I build the mutation visually, like this: To get the list of the RefillLocation objects in a city, I build the query using the same web interface, and run it against the local DynamoDB storage: When I am confident that my data model is correct, I start building the frontend code of my app, editing the App.js file of my React app, and add functionalities that I can immediately test, thanks to local mocking. To add the Amplify Framework to my app, including the React extensions, I use Yarn: yarn add aws-amplify yarn add aws-amplify-react Now, using the Amplify Framework library, I can write code like this to run a GraphQL operation: import API, { graphqlOperation } from '@aws-amplify/api'; import { createRefillLocation } from './graphql/mutations'; const refillLocation = { name: "My Favorite Place", streetAddress: "123 Here or There", zipCode: "12345" city: "Seattle", countryCode: "US" }; await API.graphql(graphqlOperation(createRefillLocation, { input: refillLocation })); Storage Mocking I now want to add a new feature to my app, to let users upload and share pictures of a RefillLocation. To do so, I add the Storage category to the configuration of my project and select “Content” to use S3: amplify add storage Using the Amplify Framework library, I can now, straight from the browser, put, get, or remove objects from S3 using the following syntax: import Storage from '@aws-amplify/storage'; Storage.put(name, file, { level: 'public' }) .then(result => console.log(result)) .catch(err => console.log(err)); Storage.get(file, { level: 'public' }) .then(result => { console.log(result); this.setState({ imageUrl: result }); fetch(result); }) .catch(err => alert(err)); All those interactions with S3 are marked as public, because I want my users to share their pictures with each other publicly, but the Amplify Framework supports different access levels, such as private, protected, and public. You can find more information on this in the File Access Levels section of the Amplify documentation. Since S3 storage is supported by this new mocking capability, I use again amplify mock to test my whole application locally, including the backend used by my GraphQL API (AppSync and DynamoDB) and my content storage (S3). If I want to test only part of my application locally, I can use amplify mock api or amplify mock storage to have only the GraphQL API, or the S3 storage, mocked locally. Availabe Now There are lots of other features that I didn’t have time to cover in this post, the best way to learn is to be curious and get hands on! You can start using Amplify by following the get-started tutorial. Being able to mock and test your application locally can help you build and refine your ideas faster, let us know what you think in the Amplify CLI GitHub repository. — Danilo

Amplify Framework Update – Quickly Add Machine Learning Capabilities to Your Web and Mobile Apps

Amazon Web Services Blog -

At AWS, we want to put machine learning in the hands of every developer. For example, we have pre-trained AI services for areas such as computer vision and language that you can use without any expertise in machine learning. Today we are making another step in that direction with the addition of a new Predictions category to the Amplify Framework. In this way, you can add and configure AI/ML uses cases for your web or mobile application with few lines of code! AWS Amplify consists of a development framework and developer services that make super easy to build mobile and web applications on AWS. The open-source Amplify Framework provides an opinionated set of libraries, user interface (UI) components, and a command line interface (CLI) to build a cloud backend and integrate it with your web or mobile apps. Amplify leverages a core set of AWS services organized into categories, including storage, authentication & authorization, APIs (GraphQL and REST), analytics, push notifications, chat bots, and AR/VR. Using the Amplify Framework CLI, you can interactively initialize your project with amplify init. Then, you can go through your storage (amplify add storage) and user authentication & authorization (amplify add auth) options. Now, you can also use amplify add predictions to configure your app to: Identify text, entities, and labels in images using Amazon Rekognition, or identify text in scanned documents to get the contents of fields in forms and information stored in tables using Amazon Textract. Convert text into a different language using Amazon Translate, text to speech using Amazon Polly, and speech to text using Amazon Transcribe. Interpret text to find the dominant language, the entities, the key phrases, the sentiment, or the syntax of unstructured text using Amazon Comprehend. You can select to have each of the above actions available only to authenticated users of your app, or also for guest, unauthenticated users. Based on your inputs, Amplify configures the necessary permissions using AWS Identity and Access Management (IAM) roles and Amazon Cognito. Let’s see how Predictions works for a web application. For example, to identify text in an image using Amazon Rekognition directly from the browser, you can use the following JavaScript syntax and pass a file object: Predictions.identify({ text: { source: file format: "PLAIN" # "PLAIN" uses Amazon Rekognition } }).then((result) => {...}) If the image is stored on Amazon S3, you can change the source to link to the S3 bucket selected when adding storage to this project. You can also change the format to analyze a scanned document using Amazon Textract. Here’s how to extract text from a form in a document stored on S3: Predictions.identify({ text: { source: { key: "my/image" } format: "FORM" # "FORM" or "TABLE" use Amazon Textract } }).then((result) => {...}) Here’s an example of how to interpret text using all the pre-trained capabilities of Amazon Comprehend: Predictions.interpret({ text: { source: { text: "text to interpret", }, type: "ALL" } }).then((result) => {...}) To convert text to speech using Amazon Polly, using the language and the voice selected when adding the prediction, and play it back in the browser, you can use the following code: Predictions.convert({ textToSpeech: { source: { text: "text to generate speech" } } }).then(result => { var audio = new Audio(); audio.src = result.speech.url; audio.play(); }) Available Now You can start building you next web or mobile app using Amplify today by following the get-started tutorial here and give us your feedback in the Amplify Framework Github repository. There are lots of other options and features available in the Predictions category of the Amplify Framework. Please see this walkthrough on the AWS Mobile Blog for an in-depth example of building a machine-learning powered app. It has never been easier to add machine learning functionalities to a web or mobile app, please let me know what you’re going to build next. — Danilo

AWS CloudFormation Update – Public Coverage Roadmap & CDK Goodies

Amazon Web Services Blog -

I launched AWS CloudFormation in early 2011 with a pair of posts: AWS CloudFormation – Create Your AWS Stack From a Recipe and AWS CloudFormation in the AWS Management Console. Since that launch, we have added support for many AWS resource types, launched many new features, and worked behind the scenes to ensure that CloudFormation is efficient, scalable, and highly available. Public Coverage Roadmap CloudFormation use is growing even faster than AWS itself, and the team has prioritized scalability over complete resource coverage. While our goal of providing 100% coverage remains, the reality is that it will take us some time to get there. In order to be more transparent about our priorities and to give you an opportunity to manage them, I am pleased to announce the much-anticipated CloudFormation Coverage Roadmap: Styled after the popular AWS Containers Roadmap, the CloudFormation Coverage Roadmap contains four columns: Shipped – Available for use in production form in all public AWS regions. Coming Soon – Generally a few months out. We’re working on It – Work in progress, but further out. Researching – We’re thinking about the right way to implement the coverage. Please feel free to create your own issues, and to give a thumbs-up to those that you need to have in order to make better use of CloudFormation: Before I close out, I would like to address one common comment – that AWS is part of a big company, and that we should simply throw more resources at it. While the team is growing, implementing robust, secure coverage is still resource-intensive. Please consider the following quote, courtesy of the must-read Mythical Man-Month: Good cooking takes time. If you are made to wait, it is to serve you better, and to please you. Cloud Development Kit Goodies The Cloud Development Kit (CDK) lets you model and provision your AWS resources using a programming language that you are already familiar with. You use a set of CDK Constructs (VPCs, subnets, and so forth) to define your application, and then use the CDK CLI to synthesize a CloudFormation template, deploy it to AWS, and create a stack. Here are some resources to help you to get started with the CDK: CDK Home Page – Start here! AWS Cloud Development Kit (CDK) – TypeScript and Python are Now Generally Available – Danilo’s recent blog post announcing general availability. CDK Developer Guide – Learn how to use the CDK. CDK Workshop – A detailed workshop with step-by-step directions, in both TypeScript and Python. CDK All The Things: A Whirlwind Tour – Cloud consultant Kevin Lin of Thence Consulting wrote this very detailed review of CDK. CDK Custom Resource Example – Learn how to add a custom resource to a CDK app. Stay Tuned The CloudFormation Coverage Roadmap is an important waypoint on a journey toward open source that started out with cfn-lint, with some more stops in the works. Stay tuned and I’ll tell you more just as soon as I can! — Jeff;

Amazon Polly Introduces Neural Text-To-Speech and Newscaster Style

Amazon Web Services Blog -

From Robbie the Robot to Jarvis, science fiction writers have long understood how important it was for an artificial being to sound as lifelike as possible. Speech is central to human interaction, and beyond words, it helps us express feelings and emotions: who can forget HAL 9000’s haunting final scene in 2001: A Space Odyssey? In the real world, things are more complicated of course. Decades before the term ‘artificial intelligence’ had even been coined, scientists were designing systems that tried to mimic the human voice. In 1937, almost 20 years before the seminal Dartmouth workshop, Homer Dudley invented the Voder, the first attempt to synthesize human speech with electronic components: this video has sound samples and extra information on this incredible device. We’ve come a long way since then! At AWS re:Invent 2016, we announced Polly, a managed service that turns text into lifelike speech, allowing customers to create applications that talk, and build entirely new categories of speech-enabled products. Zero machine learning expertise required: just call an API and get the job done! Since then, the team has regularly added new voices, for a current total of 29 languages and 59 voices. Today, we’re happy to announce two major new features for Polly: Neural Text-To-Speech, and a groundbreaking newscaster style. Introducing Neural Text-To-Speech (NTTS) Through a new machine learning approach, NTTS delivers significant improvements in speech quality. It increases naturalness and expressiveness, two key factors in synthesizing lifelike speech that is getting closer than ever from human voices. Here’s an example of the quality you can expect. As of today, NTTS is available for 11 voices, both in real-time and in batch mode: All 3 UK English voices: Amy, Emma and Brian. All 8 US English voices: Ivy, Joanna, Kendra, Kimberly, Salli, Joey, Justin and Matthew. Why not head out to the AWS console for a quick test? Introducing the newscaster style Speech quality is certainly important, but more can be done to make a synthetic voice sound even more realistic and engaging. What about style? For sure, human ears can tell the difference between a newscast, a sportscast, a university class and so on; indeed, most humans adopt the right style of speech for the right context, and this certainly helps in getting their message across. Thanks to NTTS, it’s possible to apply styles to synthesized speech, and you can now use a newscaster style with Polly. Here’s an example. From news to blog posts, this makes narration sound even more realistic, and customers like The Globe and Mail already use it today. Thanks to Polly and the newscaster style, their readers (or should we say listeners now?) can enjoy articles read in a high-quality voice that sounds like what they might expect to hear on the TV or radio. Adding Amazon Translate, they can also listen to articles that are automatically translated to a language they understand. As of today, the newscaster style is available for two US English voices (Joanna and Matthew), both in real-time and in batch mode. Again, you can head out to the AWS console for a quick test, and here’s the same clip as above with the newscaster style. Using Polly APIs with the NTTS voices and the newscaster style is extremely easy. Please let me show you how to get started with both. Using NTTS Voices and the Newscaster Style Let’s grab a bit of text for Polly to read: how about this paragraph from Amazon Simple Storage Service (S3)‘s announcement in 2006? “Earlier today we rolled out Amazon S3, our reliable, highly scalable, and low-latency data storage service. Using SOAP and REST interfaces, developers can easily store any number of blocks of data in S3. Each block can be up to 5 GB in length, and is associated with a user-defined key and additional key-value metadata pairs. Further, each block is protected by an ACL (Access Control List) allowing the developer to keep the data private, share it for reading, or share it for reading and writing, as desired. The system was designed to provide a data availability factor of 99.99%; all data is transparently stored in multiple locations”. I will use batch mode in order to save sound files in S3 and let you grab them: I explicitly changed permissions to make them public, but don’t worry, your own files are completely private by default. Let’s first try the standard Matthew voice. $ aws polly start-speech-synthesis-task --voice-id Matthew --text file://s3.txt --output-s3-bucket-name "jsimon-polly" --output-format mp3 --query "SynthesisTask.TaskId" "e3db409c-419d-4a31-a3a7-72c1e712fe23" $ wget https://jsimon-polly.s3.amazonaws.com/e3db409c-419d-4a31-a3a7-72c1e712fe23.mp3 -O matthew-standard.mp3 Tell us a bit about S3, Matthew. Now, let’s use the NTTS version of the same voice: all we have to do is set the ‘engine‘ parameter to ‘neural‘. $ aws polly start-speech-synthesis-task --voice-id Matthew --engine neural --text file://s3.txt --output-s3-bucket-name "jsimon-polly" --output-format mp3 --query "SynthesisTask.TaskId" "e3902335-c1e6-450b-b6e9-f913d6d52055" $ wget https://jsimon-polly.s3.amazonaws.com/e3902335-c1e6-450b-b6e9-f913d6d52055.mp3 -O matthew-neural.mp3 You should immediately notice the quality improvement that NTTS brings. Of course, Polly has correctly picked up technical abbreviations, numbers, etc. Now let’s spice things up and apply the newscaster style. This requires that we use the SSML markup language. All we need to do is to enclose the text like so: <speak> <amazon:domain name="news"> Earlier today we rolled out Amazon S3, our reliable, highly scalable, and low-latency data storage service. Using SOAP and REST interfaces, developers can easily store any number of blocks of data in S3. Each block can be up to 5 GB in length, and is associated with a user-defined key and additional key-value metadata pairs. Further, each block is protected by an ACL (Access Control List) allowing the developer to keep the data private, share it for reading, or share it for reading and writing, as desired. The system was designed to provide a data availability factor of 99.99%; all data is transparently stored in multiple locations. </amazon:domain> </speak> Let’s synthesize this text again, making sure to set text type to SSML. $ aws polly start-speech-synthesis-task --voice-id Matthew --engine neural --text file://s3.ssml --text-type ssml --output-s3-bucket-name "jsimon-polly" --output-format mp3 --query "SynthesisTask.TaskId" "25c18bda-b32b-4485-a45f-eb9b757a513b" $ wget https://jsimon-polly.s3.amazonaws.com/25c18bda-b32b-4485-a45f-eb9b757a513b.mp3 -O matthew-neural-newscaster.mp3 I’m sure you can immediately tell the difference! Doesn’t this sound like a news reporter reading our text? If you’re curious about the Joanna voice, here are the equivalent clips: standard, neural, and neural with newscaster style. Available Now! As you can see, it’s extremely easy to use these new features, and they are available today in US East (N. Virginia), US West (Oregon) and Europe (Ireland). The free tier offers 1 million characters for NTTS voices per month for the first 12 months, starting from your first request for speech (standard or NTTS). We’re looking forward to your feedback! Please post it to the AWS Forum for Polly, or send it to your usual AWS support contacts. — Julien;

Now Open – AWS Middle East (Bahrain) 

Amazon Web Services Blog -

The first AWS Region in the Middle East is now open and it is available for you to use today. The official name is Middle East (Bahrain) and the API name is me-south-1. With today’s launch, the AWS Cloud now spans 69 Availability Zones within 22 geographic regions around the world. The Middle East (Bahrain) Region consists of three Availability Zones (AZ’s). Having three Availability Zones enables Middle East organizations to meet business continuity and disaster recovery requirements and also build highly available, fault-tolerant, and scalable applications. Instances and Services Applications running in this 3-AZ region can use C5, C5d, D2, I3, M5, M5d, R5, R5d, and T3 instances, and can make use of a long list of AWS services including Amazon API Gateway, Application Auto Scaling, AWS Certificate Manager (ACM), AWS Artifact, AWS CloudFormation, Amazon CloudFront, AWS CloudTrail, Amazon CloudWatch,CloudWatch Events, Amazon CloudWatch Logs, AWS CodeDeploy, AWS Config, AWS Config Rules, AWS Database Migration Service, AWS Direct Connect, Amazon DynamoDB, EC2 Auto Scaling, EC2 Dedicated Hosts, Amazon Elastic Block Store (EBS), Amazon Elastic Compute Cloud (EC2), Elastic Container Registry, Amazon ECS, Application Load Balancers (Classic, Network, and Application), Amazon EMR, Amazon ElastiCache, Amazon Elasticsearch Service, Amazon Glacier, AWS Identity and Access Management (IAM), Amazon Kinesis Data Streams, AWS Key Management Service (KMS), AWS Lambda, AWS License Manager, AWS Marketplace, Amazon Neptune, AWS Organizations, AWS Personal Health Dashboard, AWS Resource Groups, Amazon Redshift, Amazon Relational Database Service (RDS), Amazon Aurora, Amazon Route 53 (including Private DNS for VPCs), AWS Shield, AWS Server Migration Service, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Simple Storage Service (S3), Amazon Simple Workflow Service (SWF), AWS Step Functions, AWS Support API, AWS Systems Manager, AWS Transit Gateway, AWS Trusted Advisor, Amazon Virtual Private Cloud, and VM Import/Export. Edge Locations and Direct Connect Amazon CloudFront edge locations are already operational in two cities in the Middle East (Dubai, and Fujairah, in the United Arab Emirates). These cities also host existing AWS Direct Connect locations (Equinix DX1 in Dubai, and Etisalat Smart Hub datacenter in Fujairah). In addition we are now opening a new AWS Direct Connect location, AWS Bahrain DC53, in Manama, Bahrain. This new location is the first to be operated by AWS and features 1 Gbps and 10 Gbps connections. See the AWS Direct Connect website for details on our locations, associated home regions, logical redundancy, and pricing. AWS Customers in the Middle East Our customers in the Middle East are already using AWS to do incredible things, for example: Emirates NBD is a leading banking Group in the Middle East that is utilizing AWS’s artificial intelligence and machine learning services, data analytics, and natural language processing technologies to better engage with customers and simplify banking. Al Tayer Group, one of the largest privately held family-run conglomerates in the Middle East and North Africa (MENA) with interests in automobile sales and services, luxury and lifestyle retail, and hospitality, relies on several AWS managed database services such as Amazon Aurora, and was able to cut costs by 40%, while increasing performance by four times, compared with proprietary database technologies. Bahrain’s National Bureau of Revenue (NBR), needed a solution that would enable a seamless VAT rollout, without delays and with the highest levels of security. By launching SAP S/4 Hana on AWS, the NBR was able to go to market in just under two months while lowering costs by 40%. The most well-known and fastest growing businesses in the Middle East are also choosing AWS to launch and grow their businesses. Careem, the leading technology platform for the region, started working with AWS in 2012 to help it scale fast and wide across 14 countries in under seven years. Today, Careem hosts some 33 million customers and one million drivers on its platform, and has expanded its services to offer on demand deliveries as well its original core business, ride hailing. And that Makes 22 AWS Cloud now has 22 regions with 69 Availability Zones, and we’re not close to being done. We are currently working on nine more Availability Zones and three more AWS Regions in Cape Town, Jakarta, and Milan. Investing in the Future To support the growth in cloud adoption across the region, AWS has made significant investments in education, training, and certification programs to help those interested in the latest cloud computing technologies, best practices, and architectures, to advance their technical skills and further support Middle East organizations in their digital transformation. AWS Activate – This global program provides startups with credits, training, and support so that they can build their businesses on AWS. AWS Educate – This global program teaches students about cloud computing. It provides AWS credits to educators and students, along with discounts on training, access to curated content, personalized learning pathways, and collaboration tools. AWS Academy – This global program is designed to bridge the gap between academia and industry by giving students the knowledge that they need to have in order to qualify for jobs that require cloud skills. The program is built around hands-on experience, and includes an AWS-authored curriculum, access to AWS certification, accreditation for educators. Training and Certification – This global program helps developers to build cloud skills using digital or classroom training and to validate those skills by earning an industry-recognized credential. It includes learning paths for Cloud Practitioners, Architects, Developers, and Operations. The Middle East (Bahrain) Region is now open and you can start creating your AWS resources in it today! — Martin

Amazon Transcribe Streaming Now Supports WebSockets

Amazon Web Services Blog -

I love services like Amazon Transcribe. They are the kind of just-futuristic-enough technology that excites my imagination the same way that magic does. It’s incredible that we have accurate, automatic speech recognition for a variety of languages and accents, in real-time. There are so many use-cases, and nearly all of them are intriguing. Until now, the Amazon Transcribe Streaming API available has been available using HTTP/2 streaming. Today, we’re adding WebSockets as another integration option for bringing real-time voice capabilities to the things you build. In this post, we are going to transcribe speech in real-time using only client-side JavaScript in a browser. But before we can build, we need a foundation. We’ll review just enough information about Amazon Transcribe, WebSockets, and the Amazon Transcribe Streaming API to broadly explain the demo. For more detailed information, check out the Amazon Transcribe docs. If you are itching to see things in action, you can head directly to the demo, but I recommend taking a quick read through this post first. What is Amazon Transcribe? Amazon Transcribe applies machine learning models to convert speech in audio to text transcriptions. One of the most powerful features of Amazon Transcribe is the ability to perform real-time transcription of audio. Until now, this functionality has been available via HTTP/2 streams. Today, we’re announcing the ability to connect to Amazon Transcribe using WebSockets as well. For real-time transcription, Amazon Transcribe currently supports British English (en-GB), US English (en-US), French (fr-FR), Canadian French (fr-CA), and US Spanish (es-US). What are WebSockets? WebSockets are a protocol built on top of TCP, like HTTP. While HTTP is great for short-lived requests, it hasn’t historically been good at handling situations that require persistent real-time communications. While an HTTP connection is normally closed at the end of the message, a WebSocket connection remains open. This means that messages can be sent bi-directionally with no bandwidth or latency added by handshaking and negotiating a connection. WebSocket connections are full-duplex, meaning that the server can client can both transmit data at the same time. They were also designed for cross-domain usage, so there’s no messing around with cross-origin resource sharing (CORS) as there is with HTTP. HTTP/2 streams solve a lot of the issues that HTTP had with real-time communications, and the first Amazon Transcribe Streaming API available uses HTTP/2. WebSocket support opens Amazon Transcribe Streaming up to a wider audience, and makes integrations easier for customers that might have existing WebSocket-based integrations or knowledge. How the Amazon Transcribe Streaming API Works Authorization The first thing we need to do is authorize an IAM user to use Amazon Transcribe Streaming WebSockets. In the AWS Management Console, attach the following policy to your user: { "Version": "2012-10-17", "Statement": [ { "Sid": "transcribestreaming", "Effect": "Allow", "Action": "transcribe:StartStreamTranscriptionWebSocket", "Resource": "*" } ] } Authentication Transcribe uses AWS Signature Version 4 to authenticate requests. For WebSocket connections, use a pre-signed URL, that contains all of the necessary information is passed as query parameters in the URL. This gives us an authenticated endpoint that we can use to establish our WebSocket. Required Parameters All of the required parameters are included in our pre-signed URL as part of the query string. These are: language-code: The language code. One of en-US, en-GB, fr-FR, fr-CA, es-US. sample-rate: The sample rate of the audio, in Hz. Max of 16000 for en-US and es-US, and 8000 for the other languages. media-encoding: Currently only pcm is valid. vocabulary-name: Amazon Transcribe allows you to define custom vocabularies for uncommon or unique words that you expect to see in your data. To use a custom vocabulary, reference it here. Audio Data Requirements There are a few things that we need to know before we start sending data. First, Transcribe expects audio to be encoded as PCM data. The sample rate of a digital audio file relates to the quality of the captured audio. It is the number of times per second (Hz) that the analog signal is checked in order to generate the digital signal. For high-quality data, a sample rate of 16,000 Hz or higher is recommended. For lower-quality audio, such as a phone conversation, use a sample rate of 8,000 Hz. Currently, US English (en-US) and US Spanish (es-US) support sample rates up to 48,000 Hz. Other languages support rates up to 16,000 Hz. In our demo, the file lib/audioUtils.js contains a downsampleBuffer() function for reducing the sample rate of the incoming audio bytes from the browser, and a pcmEncode() function that takes the raw audio bytes and converts them to PCM. Request Format Once we’ve got our audio encoding as PCM data with the right sample rate, we need to wrap it in an envelope before we send it across the WebSocket connection. Each messages consists of three headers, followed by the PCM-encoded audio bytes in the message body. The entire message is then encoded as a binary event stream message and sent. If you’ve used the HTTP/2 API before, there’s one difference that I think makes using WebSockets a bit more straightforward, which is that you don’t need to cryptographically sign each chunk of audio data you send. Response Format The messages we receive follow the same general format: they are binary-encoded event stream messages, with three headers and a body. But instead of audio bytes, the message body contains a Transcript object. Partial responses are returned until a natural stopping point in the audio is determined. For more details on how this response is formatted, check out the docs and have a look at the handleEventStreamMessage() function in main.js. Let’s See the Demo! Now that we’ve got some context, let’s try out a demo. I’ve deployed it using AWS Amplify Console – take a look, or push the button to deploy your own copy. Enter the Access ID and Secret Key for the IAM User you authorized earlier, hit the Start Transcription button, and start speaking into your microphone. The complete project is available on GitHub. The most important file is lib/main.js. This file defines all our required dependencies, wires up the buttons and form fields in index.html, accesses the microphone stream, and pushes the data to Transcribe over the WebSocket. The code has been thoroughly commented and will hopefully be easy to understand, but if you have questions, feel free to open issues on the GitHub repo and I’ll be happy to help. I’d like to extend a special thanks to Karan Grover, Software Development Engineer on the Transcribe team, for providing the code that formed that basis of this demo.

Building a Modern CI/CD Pipeline in the Serverless Era with GitOps

Amazon Web Services Blog -

Guest post by AWS Community Hero Shimon Tolts, CTO and co-founder at Datree.io. He specializes in developer tools and infrastructure, running a company that is 100% serverless. In recent years, there was a major transition in the way you build and ship software. This was mainly around microservices, splitting code into small components, using infrastructure as code, and using Git as the single source of truth that glues it all together. In this post, I discuss the transition and the different steps of modern software development to showcase the possible solutions for the serverless world. In addition, I list useful tools that were designed for this era. What is serverless? Before I dive into the wonderful world of serverless development and tooling, here’s what I mean by serverless. The AWS website talks about four main benefits: No server management. Flexible scaling. Pay for value. Automated high availability. To me, serverless is any infrastructure that you don’t have to manage and scale yourself. At my company Datree.io, we run 95% of our workload on AWS Fargate and 5% on AWS Lambda. We are a serverless company; we have zero Amazon EC2 instances in our AWS account. For more information, see the following: io case study Migrating to AWS ECS Fargate in production CON320: Operational Excellence w/ Containerized Workloads Using AWS Fargate (re:Invent 2018) What is GitOps? Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. According to Luis Faceira, a CI/CD consultant, GitOps is a way of working. You might look at it as an approach in which everything starts and ends with Git. Here are some key concepts: Git as the SINGLE source of truth of a system Git as the SINGLE place where we operate (create, change and destroy) ALL environments ALL changes are observable/verifiable. How you built software before the cloud Back in the waterfall pre-cloud era, you used to have separate teams for development, testing, security, operations, monitoring, and so on. Nowadays, in most organizations, there is a transition to full developer autonomy and developers owning the entire production path. The developer is the King – or Queen :) Those teams (Ops/Security/IT/etc) used to be gatekeepers to validate and control every developer change. Now they have become more of a satellite unit that drives policy and sets best practices and standards. They are no longer the production bottleneck, so they provide organization-wide platforms and enablement solutions. Everything is codified With the transition into full developer ownership of the entire pipeline, developers automated everything. We have more code than ever, and processes that used to be manual are now described in code. This is a good transition, in my opinion. Here are some of the benefits: Automation: By storing all things as code, everything can be automated, reused, and re-created in moments. Immutable: If anything goes wrong, create it again from the stored configuration. Versioning: Changes can be applied and reverted, and are tracked to a single user who made the change. GitOps: Git has become the single source of truth The second major transition is that now everything is in one place! Git is the place where all of the code is stored and where all operations are initiated. Whether it’s testing, building, packaging, or releasing, nowadays everything is triggered through pull requests. This is amplified by the codification of everything. Useful tools in the serverless era There are many useful tools in the market, here is a list of ones that were designed for serverless. Code Always store your code in a source control system. In recent years, more and more functions are codified, such as, BI, ops, security, and AI. For new developers, it is not always obvious that they should use source control for some functionality. GitHub AWS CodeCommit GitLab BitBucket Build and test The most common mistake I see is manually configuring build jobs in the GUI. This might be good for a small POC but it is not scalable. You should have your job codified and inside your Git repository. Here are some tools to help with building and testing: AWS CodeBuild CodeFresh GitHub Actions Jenkins-x CircleCI TravisCI Security and governance When working in a serverless way, you end up having many Git repos. The number of code packages can be overwhelming. The demand for unified code standards remains as it was but now it is much harder to enforce it on top of your R&D org. Here are some tools that might help you with the challenge: Snyk Datree PureSec Aqua Protego Bundle and release Building a serverless application is connecting microservices into one unit. For example, you might be using Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. Instead of configuring each one separately, you should use a bundler to hold the configuration in one place. That allows for easy versioning and replication of the app for several environments. Here are a couple of bundlers: Serverless Framework AWS Serverless Application Model (AWS SAM) Package When working with many different serverless components, you should create small packages of tools to be able to import across different Lambda functions. You can use a language-specific store like npm or RubyGems, or use a more holistic solution. Here are several package artifact stores that allow hosting for multiple programming languages: GitHub Package Registry Jfrog Artifactory Sonatype Nexus Monitor This part is especially tricky when working with serverless applications, as everything is split into small pieces. It’s important to use monitoring tools that support this mode of work. Here are some tools that can handle serverless: Rookout Amazon CloudWatch Epsagon Lumigo NewRelic DataDog Summary The serverless era brings many transitions along with it like a codification of the entire pipeline and Git being the single source of truth. This doesn’t mean that the same problems that we use to have like security, logging and more disappeared, you should continue addressing them and leveraging tools that enable you to focus on your business.

AWS DeepLens – Now Orderable in Seven Additional Countries

Amazon Web Services Blog -

The new (2019) edition of the AWS DeepLens can now be purchased in six countries (US, UK, Germany, France, Spain, Italy, and Canada), and preordered in Japan. The 2019 edition is easier to set up, and (thanks to Amazon SageMaker Neo) runs machine learning models up to twice as fast as the earlier edition. New Tutorials We are also launch a pair of new tutorials to help you to get started: aws-deeplens-coffee-leaderboard – This tutorial focuses on a demo that uses face detection to track the number of people that drink coffee. It watches a scene, and triggers a Lambda function when a face is detected. Amazon Rekognition is used to detect the presence of a coffee mug, and the face is added to a DynamoDB database that is maintained by (and private to) the demo. The demo also includes a leaderboard that tracks the number of coffees over time. Here’s the architecture: And here’s the leaderboard: To learn more, read Track the number of coffees consumed using AWS DeepLens. aws-deeplens-worker-safety-project – This tutorial focuses on a demo that identifies workers that are not wearing safety helmets. The DeepLens detects faces, and uploads the images to S3 for further processing. The results are analyze using AWS IoT and Amazon CloudWatch, and are displayed on a web dashboard. Here’s the architecture: To learn more, register for and then take the free 30-minute course: Worker Safety Project with AWS DeepLens. Detecting Cats, and Cats with Rats Finally, I would like to share a really cool video featuring my colleague Ben Hamm. After growing tired of cleaning up the remains of rats and other creatures that his cat Metric had killed, Ben decided to put his DeepLens to work. Using a hand-labeled training set, Ben created a model that could tell when Metric was carrying an unsavory item its his mouth, and then lock him out. Ben presented his project at Ignite Seattle and the video has been very popular. Take a look for yourself: Order Your DeepLens Today If you are in one of the countries that I listed above, you can order your DeepLens today and get started with Machine Learning in no time flat! Visit the DeepLens home page to learn more. — Jeff;

AWS Named as a Leader in Gartner’s Infrastructure as a Service (IaaS) Magic Quadrant for the 9th Consecutive Year

Amazon Web Services Blog -

My colleagues on the AWS service teams work to deliver what customers want today, and also do their best to anticipate what they will need tomorrow. This Customer Obsession, along with our commitment to Hire and Develop the Best (two of the fourteen Amazon Leadership Principles), helps us to figure out, and then to deliver on, our vision. It is always good to see that our hard work continues to delight customers, and to be recognized by Gartner and other leading analysts. For the ninth consecutive year, AWS has secured the top-right corner of the Leader’s quadrant in Gartner’s Magic Quadrant for Cloud Infrastructure as a Service (IaaS), earning highest placement for Ability to Execute and furthest for Completeness of Vision: The full report contains a lot of detail and is a great summary of the features and factors that our customers examine when choosing a cloud provider. — Jeff; Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Meet the Newest AWS News Bloggers!

Amazon Web Services Blog -

I wrote my first post for this blog way back in 2004! Over the course of the first decade, the amount of time that I devoted to the blog grew from a small fraction of my day to a full day. In the early days my email inbox was my primary source of information about upcoming launches, and also my primary tool for managing my work backlog. When that proved to be unscalable, Ana came onboard and immediately built a ticketing system and set up a process for teams to request blog posts. Today, a very capable team (Greg, Devin, and Robin) takes care of tickets, platforms, comments, metrics, and so forth so that I can focus on what I like to do best: using new services and writing about them! Over the years we have experimented with a couple of different strategies to scale the actual writing process. If you are a long-time reader you may have seen posts from Mike, Jinesh, Randall, Tara, Shaun, and a revolving slate of guest bloggers. News Bloggers I would like to introduce you to our current lineup of AWS News Bloggers. Like me, the bloggers have a technical background and are prepared to go hands-on with every new service and feature. Here’s our roster: Steve Roberts (@bellevuesteve) – Steve focuses on .NET tools and technologies. Julien Simon (@julsimon) – Julien likes to help developers and enterprises to bring their ideas to life. Brandon West (@bwest) – Brandon leads our developer relations team in the Americas, and has written a book on the topic. Martin Beeby (@thebeebs) – Martin focuses on .NET applications, and has worked as a C# and VB developer since 2001. Danilo Poccia (@danilop) – Danilo works with companies of any size to support innovation. He is the author of AWS Lambda in Action. Sébastien Stormacq (@sebesto) – Sébastien works with builders to unlock the value of the AWS cloud, using his secret blend of passion, enthusiasm, customer advocacy, curiosity, and creativity. We are already gearing up for re:Invent 2019, and can’t wait to bring you a rich set of blog posts. Stay tuned! — Jeff;

Learn about AWS Services & Solutions – July AWS Online Tech Talks

Amazon Web Services Blog -

Join us this July to learn about AWS services and solutions. The AWS Online Tech Talks are live, online presentations that cover a broad range of topics at varying technical levels. These tech talks, led by AWS solutions architects and engineers, feature technical deep dives, live demonstrations, customer examples, and Q&A with AWS experts. Register Now! Note – All sessions are free and in Pacific Time. Tech talks this month: Blockchain July 24, 2019 | 11:00 AM – 12:00 PM PT – Building System of Record Applications with Amazon QLDB – Dive deep into the features and functionality of our first-of-its-kind, purpose-built ledger database, Amazon QLDB. Containers July 31, 2019 | 11:00 AM – 12:00 PM PT – Machine Learning on Amazon EKS – Learn how to use KubeFlow and TensorFlow on Amazon EKS for your machine learning needs. Data Lakes & Analytics July 31, 2019 | 1:00 PM – 2:00 PM PT – How to Build Serverless Data Lake Analytics with Amazon Athena – Learn how to use Amazon Athena for serverless SQL analytics on your data lake, transform data with AWS Glue, and manage access with AWS Lake Formation. August 1, 2019 | 11:00 AM – 12:00 PM PT – Enhancing Your Apps with Embedded Analytics – Learn how to add powerful embedded analytics capabilities to your applications, portals and websites with Amazon QuickSight. Databases July 25, 2019 | 9:00 AM – 10:00 AM PT – MySQL Options on AWS: Self-Managed, Managed, Serverless – Understand different self-managed and managed MySQL deployment options on AWS, and watch a demonstration of creating a serverless MySQL-compatible database using Amazon Aurora. DevOps July 30, 2019 | 9:00 AM – 10:00 AM PT – Build a Serverless App in Under 20 Minutes with Machine Learning Functionality Using AWS Toolkit for Visual Studio Code – Get a live demo on how to create a new, ready-to-deploy serverless application. End-User Computing July 23, 2019 | 1:00 PM – 2:00 PM PT – A Security-First Approach to Delivering End User Computing Services – Learn how AWS improves security and reduces cost by moving data to the cloud while providing secure, fast access to desktop applications and data. IoT July 30, 2019 | 11:00 AM – 12:00 PM PT – Security Spotlight: Best Practices for Edge Security with Amazon FreeRTOS – Learn best practices for building a secure embedded IoT project with Amazon FreeRTOS. Machine Learning July 23, 2019 | 9:00 AM – 10:00 AM PT– Get Started with Machine Learning: Introducing AWS DeepLens, 2019 Edition – Learn the basics of machine learning through building computer vision apps with the new AWS DeepLens. August 1, 2019 | 9:00 AM – 10:00 AM PT – Implementing Machine Learning Solutions with Amazon SageMaker – Learn how machine learning with Amazon SageMaker can be used to solve industry problems. Mobile July 31, 2019 | 9:00 AM – 10:00 AM PT – Best Practices for Android Authentication on AWS with AWS Amplify – Learn the basics of Android authentication on AWS and leverage the built in AWS Amplify Authentication modules to provide user authentication in just a few lines of code. Networking & Content Delivery July 23, 2019 | 11:00 AM – 12:00 PM PT – Simplify Traffic Monitoring and Visibility with Amazon VPC Traffic Mirroring – Learn to easily mirror your VPC traffic to monitor and secure traffic in real-time with monitoring appliances of your choice. Productivity & Business Solutions July 30, 2019 | 1:00 PM – 2:00 PM PT – Get Started in Minutes with Amazon Connect in Your Contact Center – See how easy it is to get started with Amazon Connect, based on the same technology used by Amazon Customer Service to power millions of customer conversations. Robotics July 25, 2019 | 11:00 AM – 12:00 PM PT – Deploying Robotic Simulations Using Machine Learning with Nvidia JetBot and AWS RoboMaker – Learn how to deploy robotic simulations (and find dinosaurs along the way) using machine learning with Nvidia JetBot and AWS RoboMaker. Security, Identity & Compliance July 24, 2019 | 9:00 AM – 10:00 AM PT – Deep Dive on AWS Certificate Manager Private CA – Creating and Managing Root and Subordinate Certificate Authorities – Learn how to quickly and easily create a complete CA hierarchy, including root and subordinate CAs, with no need for external CAs. Serverless July 24, 2019 | 1:00 PM – 2:00 PM PT – Getting Started with AWS Lambda and Serverless Computing – Learn how to run code without provisioning or managing servers with AWS Lambda.

AWS New York Summit 2019 – Summary of Launches & Announcements

Amazon Web Services Blog -

The AWS New York Summit just wrapped up! Here’s a quick summary of what we launched and announced: Amazon EventBridge – This new service builds on the event-processing model that forms the basis for Amazon CloudWatch Events, and makes it easy for you to integrate your AWS applications with SaaS applications such as Zendesk, Datadog, SugarCRM, and Onelogin. Read my blog post, Amazon EventBridge – Event-Driven AWS Integration for your SaaS Applications, to learn more. Werner announces EventBridge – Photo by Serena Cloud Development Kit – CDK is now generally available, with support for TypeScript and Python. Read Danilo‘s post, AWS Cloud Development Kit (CDK) – TypeScript and Python are Now Generally Available, to learn more. Fluent Bit Plugins for AWS – Fluent Bit is a multi-platform, open source log processor and forwarder that is compatible with Docker and Kubernetes environments. You can now build a container image that includes new Fluent Bit plugins for Amazon CloudWatch and Amazon Kinesis Data Firehose. The plugins routes logs to CloudWatch, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service. Read Centralized Container Logging with Fluent Bit to learn more. Nicki, Randall, Robert, and Steve – Photo by Deepak AWS Toolkit for VS Code – This toolkit lets you develop and test locally (including step-through debugging) in a Lambda-like environment, and then deploy to the AWS Region of your choice. You can invoke Lambda functions locally or remotely, with full control of the function configuration, including the event payload and environment variables. To learn more, read Announcing AWS Toolkit for Visual Studio Code. Amazon CloudWatch Container Insights (preview) – You can now create CloudWatch Dashboards that monitor the performance and health of your Amazon ECS and AWS Fargate clusters, tasks, containers, and services. Read Using Container Insights to learn more. CloudWatch Anomaly Detection (preview) – This cool addition to CloudWatch uses machine learning to continuously analyze system and application metrics, determine a nominal baseline, and surface anomalies, all without user intervention. It adapts to trends, and helps to identity unexpected changes in performance or behavior. Read the CloudWatch Anomaly Detection documentation to learn more. Amazon SageMaker Managed Spot Training (coming soon) – You will soon be able to use Amazon EC2 Spot to lower the cost of training your machine learning models. This upcoming enhancement to SageMaker will lower your training costs by up to 70%, and can be used in conjunction with Automatic Model Training. — Jeff;  

AWS Cloud Development Kit (CDK) – TypeScript and Python are Now Generally Available

Amazon Web Services Blog -

Managing your Infrastructure as Code provides great benefits and is often a stepping stone for a successful application of DevOps practices. In this way, instead of relying on manually performed steps, both administrators and developers can automate provisioning of compute, storage, network, and application services required by their applications using configuration files. For example, defining your Infrastructure as Code makes it possible to: Keep infrastructure and application code in the same repository Make infrastructure changes repeatable and predictable across different environments, AWS accounts, and AWS regions Replicate production in a staging environment to enable continuous testing Replicate production in a performance test environment that you use just for the time required to run a stress test Release infrastructure changes using the same tools as code changes, so that deployments include infrastructure updates Apply software development best practices to infrastructure management, such as code reviews, or deploying small changes frequently Configuration files used to manage your infrastructure are traditionally implemented as YAML or JSON text files, but in this way you’re missing most of the advantages of modern programming languages. Specifically with YAML, it can be very difficult to detect a file truncated while transferring to another system, or a missing line when copying and pasting from one template to another. Wouldn’t it be better if you could use the expressive power of your favorite programming language to define your cloud infrastructure? For this reason, we introduced last year in developer preview the AWS Cloud Development Kit (CDK), an extensible open-source software development framework to model and provision your cloud infrastructure using familiar programming languages. I am super excited to share that the AWS CDK for TypeScript and Python is generally available today! With the AWS CDK you can design, compose, and share your own custom components that incorporate your unique requirements. For example, you can create a component setting up your own standard VPC, with its associated routing and security configurations. Or a standard CI/CD pipeline for your microservices using tools like AWS CodeBuild and CodePipeline. Personally I really like that by using the AWS CDK, you can build your application, including the infrastructure, in your IDE, using the same programming language and with the support of autocompletion and parameter suggestion that modern IDEs have built in, without having to do a mental switch between one tool, or technology, and another. The AWS CDK makes it really fun to quickly code up your AWS infrastructure, configure it, and tie it together with your application code! How the AWS CDK works Everything in the AWS CDK is a construct. You can think of constructs as cloud components that can represent architectures of any complexity: a single resource, such as an S3 bucket or an SNS topic, a static website, or even a complex, multi-stack application that spans multiple AWS accounts and regions. To foster reusability, constructs can include other constructs. You compose constructs together into stacks, that you can deploy into an AWS environment, and apps, a collection of one of more stacks. How to use the AWS CDK We continuously add new features based on the feedback of our customers. That means that when creating an AWS resource, you often have to specify many options and dependencies. For example, if you create a VPC you have to think about how many Availability Zones (AZs) to use and how to configure subnets to give private and public access to the resources that will be deployed in the VPC. To make it easy to define the state of AWS resources, the AWS Construct Library exposes the full richness of many AWS services with sensible defaults that you can customize as needed. In the case above, the VPC construct creates by default public and private subnets for all the AZs in the VPC, using 3 AZs if not specified. For creating and managing CDK apps, you can use the AWS CDK Command Line Interface (CLI), a command-line tool that requires Node.js and can be installed quickly with: npm install -g aws-cdk After that, you can use the CDK CLI with different commands: cdk init to initialize in the current directory a new CDK project in one of the supported programming languages cdk synth to print the CloudFormation template for this app cdk deploy to deploy the app in your AWS Account cdk diff to compare what is in the project files with what has been deployed Just run cdk to see more of the available commands and options. You can easily include the CDK CLI in your deployment automation workflow, for example using Jenkins or AWS CodeBuild. Let’s use the AWS CDK to build two sample projects, using different programming languages. An example in TypeScript For the first project I am using TypeScript to define the infrastructure: cdk init app --language=typescript Here’s a simplified view of what I want to build, not entering into the details of the public/private subnets in the VPC. There is an online frontend, writing messages in a queue, and an asynchronous backend, consuming messages from the queue: Inside the stack, the following TypeScript code defines the resources I need, and their relations: First I define the VPC and an Amazon ECS cluster in that VPC. By using the defaults provided by the AWS Construct Library, I don’t need to specify any parameter here. Then I use an ECS pattern that in a few lines of code sets up an Amazon SQS queue and an ECS service running on AWS Fargate to consume the messages in that queue. The ECS pattern library provides higher-level ECS constructs which follow common architectural patterns, such as load balanced services, queue processing, and scheduled tasks. A Lambda function has the name of the queue, created by the ECS pattern, passed as an environment variable and is granted permissions to send messages to the queue. The code of the Lambda function and the Docker image are passed as assets. Assets allow you to bundle files or directories from your project and use them with Lambda or ECS. Finally, an Amazon API Gateway endpoint provides an HTTPS REST interface to the function. const myVpc = new ec2.Vpc(this, "MyVPC"); const myCluster = new ecs.Cluster(this, "MyCluster", { vpc: myVpc }); const myQueueProcessingService = new ecs_patterns.QueueProcessingFargateService( this, "MyQueueProcessingService", { cluster: myCluster, memoryLimitMiB: 512, image: ecs.ContainerImage.fromAsset("my-queue-consumer") }); const myFunction = new lambda.Function( this, "MyFrontendFunction", { runtime: lambda.Runtime.NODEJS_10_X, timeout: Duration.seconds(3), handler: "index.handler", code: lambda.Code.asset("my-front-end"), environment: { QUEUE_NAME: myQueueProcessingService.sqsQueue.queueName } }); myQueueProcessingService.sqsQueue.grantSendMessages(myFunction); const myApi = new apigateway.LambdaRestApi( this, "MyFrontendApi", { handler: myFunction }); I find this code very readable and easier to maintain than the corresponding JSON or YAML. By the way, cdk synth in this case outputs more than 800 lines of plain CloudFormation YAML. An example in Python For the second project I am using Python: cdk init app --language=python I want to build a Lambda function that is executed every 10 minutes: When you initialize a CDK project in Python, a virtualenv is set up for you. You can activate the virtualenv and install your project requirements with: source .env/bin/activate pip install -r requirements.txt Note that Python autocompletion may not work with some editors, like Visual Studio Code, if you don’t start the editor from an active virtualenv. Inside the stack, here’s the Python code defining the Lambda function and the CloudWatch Event rule to invoke the function periodically as target: myFunction = aws_lambda.Function( self, "MyPeriodicFunction", code=aws_lambda.Code.asset("src"), handler="index.main", timeout=core.Duration.seconds(30), runtime=aws_lambda.Runtime.PYTHON_3_7, ) myRule = aws_events.Rule( self, "MyRule", schedule=aws_events.Schedule.rate(core.Duration.minutes(10)), ) myRule.add_target(aws_events_targets.LambdaFunction(myFunction)) Again, this is easy to understand even if you don’t know the details of AWS CDK. For example, durations include the time unit and you don’t have to wonder if they are expressed in seconds, milliseconds, or days. The output of cdk synth in this case is more than 90 lines of plain CloudFormation YAML. Available Now There is no charge for using the AWS CDK, you pay for the AWS resources that are deployed by the tool. To quickly get hands-on with the CDK, start with this awesome step-by-step online tutorial! More examples of CDK projects, using different programming languages, are available in this repository: https://github.com/aws-samples/aws-cdk-examples You can find more information on writing your own constructs here. The AWS CDK is open source and we welcome your contribution to make it an even better tool: https://github.com/awslabs/aws-cdk Check out our source code on GitHub, start building your infrastructure today using TypeScript or Python, or try different languages in developer preview, such as C# and Java, and give us your feedback!

Amazon EventBridge – Event-Driven AWS Integration for your SaaS Applications

Amazon Web Services Blog -

Many AWS customers also make great use of SaaS (Software as a Service) applications. For example, they use Zendesk to manage customer service & support tickets, PagerDuty to handle incident response, and SignalFX for real-time monitoring. While these applications are quite powerful on their own, they are even more so when integrated into a customer’s own systems, databases, and workflows. New Amazon EventBridge In order to support this increasingly common use case, we are launching Amazon EventBridge today. Building on the powerful event processing model that forms the basis for CloudWatch Events, EventBridge makes it easy for our customers to integrate their own AWS applications with SaaS applications. The SaaS applications can be hosted anywhere, and simply publish events to an event bus that is specific to each AWS customer. The asynchronous, event-based model is fast, clean, and easy to use. The publisher (SaaS application) and the consumer (code running on AWS) are completely decoupled, and are not dependent on a shared communication protocol, runtime environment, or programming language. You can use simple Lambda functions to handle events that come from a SaaS application, and you can also route events to a wide variety of other AWS targets. You can store incident or ticket data in Amazon Redshift, train a machine learning model on customer support queries, and much more. Everything that you already know (and hopefully love) about CloudWatch Events continues to apply, with one important change. In addition to the existing default event bus that accepts events from AWS services, calls to PutEvents, and from other authorized accounts, each partner application that you subscribe to will also create an event source that you can then associate with an event bus in your AWS account. You can select any of your event buses, create EventBridge Rules, and select Targets to invoke when an incoming event matches a rule. As part of today’s launch we are also opening up a partner program. The integration process is simple and straightforward, and generally requires less than one week of developer time. All About Amazon EventBridge Here are some terms that you need to know in order to understand how to use Amazon EventBridge: Partner – An organization that has integrated their SaaS application with EventBridge. Customer – An organization that uses AWS, and that has subscribed to a partner’s SaaS application. Partner Name – A unique name that identifies an Amazon EventBridge partner. Partner Event Bus – An Event Bus that is used to deliver events from a partner to AWS. EventBridge can be accessed from the AWS Management Console, AWS Command Line Interface (CLI), or via the AWS SDKs. There are distinct commands and APIs for partners and for customers. Here are some of the most important ones: Partners – CreatePartnerEventSource, ListPartnerEventSourceAccounts, ListPartnerEventSources, PutPartnerEvents. Customers – ListEventSources, ActivateEventSource, CreateEventBus, ListEventBuses, PutRule, PutTargets. Amazon EventBridge for Partners & Customers As I noted earlier, the integration process is simple and straightforward. You need to allow your customers to enter an AWS account number and to select an AWS region. With that information in hand, you call CreatePartnerEventSource in the desired region, inform the customer of the event source name and tell them that they can accept the invitation to connect, and wait for the status of the event source to change to ACTIVE. Then, each time an event of interest to the customer occurs, you call PutPartnerEvents and reference the event source. The process is just as simple on the customer side. You accept the invitation to connect by calling CreateEventBus to create an event bus associated with the event source. You add rules and targets to the event bus, and prepare your Lambda functions to process the events. Associating the event source with an event bus also activates the source and starts the flow of events. You can use DeActivateEventSource and ActivateEventSource to control the flow. Here’s the overall flow (diagram created using SequenceDiagram): Each partner has the freedom to choose the events that are relevant to their application, and to define the data elements that are included with each event. Using EventBridge Starting from the EventBridge Console, I click Partner event sources, find the partner of interest, and click it to learn more: Each partner page contains additional information about the integration. I read the info, and click Set up to proceed: The page provides me with a simple, three-step procedure to set up my event source: After the partner creates the event source, I return to Partner event sources and I can see that the Zendesk event source is Pending: I click the pending event source, review the details, and then click Associate with event bus: I have the option to allow other AWS accounts, my Organization, or another Organization to access events on the event bus that I am about to create. After I have confirmed that I trust the origin and have added any additional permissions, I click Associate: My new event bus is now available, and is listed as a Custom event bus: I click Rules, select the event bus, and see the rules (none so far) associated with it. Then I click Create rule to make my first rule: I enter a name and a description for my first rule: Then I define a pattern, choosing Zendesk as the Service name: Next, I select a Lambda function as my target: I can also choose from many other targets: After I create my rule, it will be activated in response to activities that occur within my Zendesk account. The initial set of events includes TicketCreated, CommentCreated, TagsChanged, AgentAssignmentChanged, GroupAssignmentChanged, FollowersChanged, EmailCCsChanged, CustomFieldChanged, and StatusChanged. Each event includes a rich set of properties; you’ll need to consult the documentation to learn more. Partner Event Sources We are launching with ten partner event sources, with more to come: Datadog Zendesk PagerDuty Whispir Saviynt Segment SignalFx SugarCRM OneLogin Symantec If you have a SaaS application and you are ready to integrate, read more about EventBridge Partner Integration. Now Available Amazon EventBridge is available now and you can start using it today in all public AWS regions in the aws partition. Support for the AWS regions in China, and for the Asia Pacific (Osaka) Local Region, is in the works. Pricing is based on the number of events published to the event buses in your account, billed at $1 for every million events. There is no charge for events published by AWS services. — Jeff; PS – As you can see from this post, we are paying even more attention to the overall AWS event model, and have a lot of interesting goodies on the drawing board. With this launch, CloudWatch Events has effectively earned a promotion to a top-level service, and I’ll have a lot more to say about that in the future!

Pages

Recommended Content

Subscribe to Complete Hosting Guide aggregator - Cloud Hosting Official Blogs