Cloud Hosting Blogs

New – AWS Elemental Link – Deliver Live Video to the Cloud for Events & Streams

Amazon Web Services Blog -

Video is central to so many online experiences. Regardless of the origin or creator, today’s viewers expect a high-resolution, broadcast-quality experience. In sophisticated environments, dedicated hardware and an associated A/V team can capture, encode, and stream or store video that meets these expectations. However, cost and operational complexity have prevented others from delivering a similar experience. Classrooms, local sporting events, enterprise events, and small performance spaces do not have the budget or the specialized expertise needed to install, configure, and run the hardware and software needed to reliably deliver video to the cloud for processing, storage, and on-demand delivery or live streaming. Introducing AWS Elemental Link Today I would like to tell you about AWS Elemental Link. This new device connects live video sources to AWS Elemental MediaLive. The device is small (about 32 cubic inches) and weighs less than a pound. It draws very little power, is absolutely silent, and is available for purchase today at $995. You can order these devices from the AWS Management Console and have them shipped to the intended point of use. They arrive preconfigured, and need only be connected to power, video, and the Internet. You can monitor and manage any number of Link devices from the console, without the need for specialized expertise at the point of use. When connected to a video source, the Link device sends all video, audio, and metadata streams that arrive on the built-in 3G-SDI or HDMI connectors to AWS Elemental MediaLive, with automatic, hands-free tuning that adapts to available bandwidth. Once your video is in the cloud, you can use the full lineup of AWS Elemental Media Services to process, store, distribute, and monetize it. Ordering an AWS Elemental Link To get started, I visit the AWS Elemental Link Console and click Start order: I indicate that I understands the Terms of Service, and click Continue to place order to proceed: I enter my order, starting with contact information and an optional order name: Then I enter individual order lines, and click Add new order line after each one. Each line represents one or more devices destined for one physical address. All of the devices in an order line are provisioned for the same AWS region: I can see my Order summary at the bottom. Once I have created all of the desired order lines I click Next to proceed: I choose a payment option, verify my billing address, and click Next: Then I review my order and click Submit to place it: After I pay my invoice, I wait for my devices to arrive. Connection & Setup When my device arrives, I connect it to my network and my camera, and plug in the power supply. I wait a minute or so while the device powers up and connects to the network, AWS, and to my camera. When it is all set, the front panel looks like this: Next, I open the AWS Elemental MediaLive Console and click Devices: Now that everything is connected, I can create a MediaLive input (Studio1), selecting Elemental Link as the source and choosing one of the listed input devices: And that’s the setup and connection process. From here I would create a channel that references the input and then set up an output group to stream, archive, broadcast, or package the video stream. We’re building a CloudFormation-powered solution that will take care of all of this for you; stay tuned for details. You can order your AWS Elemental Link today and start delivering video to the cloud in minutes! — Jeff;  

Join the FORMULA 1 DeepRacer ProAm Special Event

Amazon Web Services Blog -

The AWS DeepRacer League gives you the opportunity to race for prizes and glory, while also having fun & learning about reinforcement learning. You can use the AWS DeepRacer 3D racing simulator to build, train, and evaluate your models. You can review the results and improve your models in order to ensure that they are race-ready. Winning a FORMULA 1 (F1) race requires a technologically sophisticated car, a top-notch driver, an outstanding crew, and (believe it or not) a healthy dose of machine learning. For the past couple of seasons AWS has been working with the Formula 1 team to find ways to use machine learning to make cars that are faster and more fuel-efficient than ever before (read The Fastest Cars Deserve the Fastest Cloud and Formula 1 Works with AWS to Develop Next Generation Race Car to learn more). Special Event Each month the AWS DeepRacer League runs a new Virtual Race in the AWS DeepRacer console and this month is a special one: the Formula 1 DeepRacer ProAm Special Event. During the month of May you can compete for the opportunity to race against models built and tuned by Formula drivers and their crews. Here’s the lineup: Rob Smedley – Director of Data Systems for F1 and AWS Technical Ambassador. Daniel Ricciardo – F1 driver for Renault, with 7 Grand Prix wins and 29 podium appearances. Tatiana Calderon – Test driver for the Alfa Romeo F1 team and 2019 F2 driver. Each pro will be partnered with a member of the AWS Pit Crew tasked with teaching them new skills and taking them on a learning journey. Here’s the week-by-week plan for the pros: Week 1 – Learn the basics of reinforcement learning and submit models using a standard, single-camera vehicle configuration. Week 2 – Add stereo cameras to vehicles and learn how to configure reward functions to dodge debris on the track. Week 3 – Add LIDAR to vehicles and use the rest of the month to prepare for the head-to-head qualifier. At the end of the month the top AWS DeepRacer amateurs will face off against the professionals, in an exciting head to head elimination race, scheduled for the week of June 1. The teams will be documenting their learning journey and you’ll be able to follow along as they apply real-life racing strategies and data science to the world of autonomous racing. Bottom line: You have the opportunity to build & train a model, and then race it against one from Rob, Daniel, or Tatiana. How cool is that? Start Your Engines And now it is your turn. Read Get Started with AWS DeepRacer, build your model, join the Formula 1 DeepRacer ProAm Special Event, train it on the Circuit de Barcelona-Catalunya track, and don’t give up until you are at the top of the chart. Training and evaluation using the DeepRacer Console are available at no charge for the duration of the event (Terms and Conditions apply), making this a great opportunity for you to have fun while learning a useful new skill. Good luck, and see you at the finish line! — Jeff;  

New – Amazon EventBridge Schema Registry is Now Generally Available

Amazon Web Services Blog -

Amazon EventBridge is a serverless event bus that makes it easy to connect applications together. It can use data from AWS services, your own applications, and integrations with Software-as-a-Service (SaaS) partners. Last year at re:Invent, we introduced in preview EventBridge schema registry and discovery, a way to store the structure of the events (the schema) in a central location, and simplify using events in your code by generating the code to process them for Java, Python, and Typescript. Today, I am happy to announce that the EventBridge schema registry is generally available, and that we added support for resource policies. Resource policies allow to share a schema repository across different AWS accounts and organizations. In this way, developers on different teams can search for and use any schema that another team has added to the shared registry. Using EventBridge Schema Registry Resource Policies It’s common for companies to have different development teams working on different services. To make a more concrete example, let’s take two teams working on services that have to communicate with each other: The CreateAccount development team, working on a frontend API that receives requests from a web/mobile client to create a new customer account for the company. the FraudCheck development team, working on a backend service checking the data for newly created accounts to estimate the risk that those are fake. Each team is using their own AWS account to develop their application. Using EventBridge, we can implement the following architecture: The frontend CreateAccount applications is using the Amazon API Gateway to process the request using a AWS Lambda function written in Python. When a new account is created, the Lambda function publishes the ACCOUNT_CREATED event on a custom event bus. The backend FraudCheck Lambda function is built in Java, and is expecting to receive the ACCOUNT_CREATED event to call Amazon Fraud Detector (a fully managed service we introduced in preview at re:Invent) to estimate the risk of that being a fake account. If the risk is above a certain threshold, the Lambda function takes preemptive actions. For example, it can flag the account as fake on a database, or post a FAKE_ACCOUNT event on the event bus. How can the two teams coordinate their work so that they both know the syntax of the events, and use EventBridge to generate the code to process those events? First, a custom event bus is created with permissions to access within the company organization. Then, the CreateAccount team uses EventBridge schema discovery to automatically populate the schema for the ACCOUNT_CREATED event that their service is publishing. This event contains all the information of the account that has just been created. In an event-driven architecture, services can subscribe to specific types of events that they’re interested in. To receive ACCOUNT_CREATED events, a rule is created on the event bus to send those events to the FraudCheck function. Using resource policies, the CreateAccount team gives read-only access to the FraudCheck team AWS account to the discovered schemas. The Principal in this policy is the AWS account getting the permissions. The Resource is the schema registry that is being shared. { "Version": "2012-10-17", "Statement": [ { "Sid": "GiveSchemaAccess", "Effect": "Allow", "Action": [ "schemas:ListSchemas", "schemas:SearchSchemas", "schemas:DescribeSchema", "schemas:DescribeCodeBinding", "schemas:GetCodeBindingSource",   "schemas:PutCodeBinding" ], "Principal": { "AWS": "123412341234" }, "Resource": [ "arn:aws:schemas:us-east-1:432143214321:schema/discovered-schemas", "arn:aws:schemas:us-east-1:432143214321:schema/discovered-schemas*" ] } ] } Now, the FraudCheck team can search the content of the discovered schema for the ACCOUNT_CREATED event. Resource policies allow you to make a registry available across accounts and organizations, but they will not automatically show up in the console. To access the shared registry, the FraudCheck team needs to use the AWS Command Line Interface (CLI) and specify the full ARN of the registry: aws schemas search-schemas \ --registry-name arn:aws:schemas:us-east-1:432143214321:registry/discovered-schemas \ --keywords ACCOUNT_CREATED In this way, the FraudCheck team gets the exact name of the schema created by the CreateAccount team. { "Schemas": [ { "RegistryName": "discovered-schemas", "SchemaArn": "arn:aws:schemas:us-east-1:432143214321:schema/discovered-schemas/CreateAccount@ACCOUNT_CREATED", "SchemaName": “CreateAccount@ACCOUNT_CREATED", "SchemaVersions": [ { "CreatedDate": "2020-04-28T11:10:15+00:00", "SchemaVersion": 1 } ] } ] } With the schema name, the FraudCheck team can describe the content of the schema: aws schemas describe-schema \ --registry-name arn:aws:schemas:us-east-1:432143214321:registry/discovered-schemas \ --schema-name CreateAccount@ACCOUNT_CREATED The result describes the schema using the OpenAPI specification: { "Content": "{\"openapi\":\"3.0.0\",\"info\":{\"version\":\"1.0.0\",\"title\":\"CREATE_ACCOUNT\"},\"paths\":{},\"components\":{\"schemas\":{\"AWSEvent\":{\"type\":\"object\",\"required\":[\"detail-type\",\"resources\",\"detail\",\"id\",\"source\",\"time\",\"region\",\"version\",\"account\"],\"x-amazon-events-detail-type\":\"CREATE_ACCOUNT\",\"x-amazon-events-source\":\”CreateAccount\",\"properties\":{\"detail\":{\"$ref\":\"#/components/schemas/CREATE_ACCOUNT\"},\"account\":{\"type\":\"string\"},\"detail-type\":{\"type\":\"string\"},\"id\":{\"type\":\"string\"},\"region\":{\"type\":\"string\"},\"resources\":{\"type\":\"array\",\"items\":{\"type\":\"object\"}},\"source\":{\"type\":\"string\"},\"time\":{\"type\":\"string\",\"format\":\"date-time\"},\"version\":{\"type\":\"string\"}}},\"CREATE_ACCOUNT\":{\"type\":\"object\",\"required\":[\"firstName\",\"surname\",\"id\",\"email\"],\"properties\":{\"email\":{\"type\":\"string\"},\"firstName\":{\"type\":\"string\"},\"id\":{\"type\":\"string\"},\"surname\":{\"type\":\"string\"}}}}}}", "LastModified": "2020-04-28T11:10:15+00:00", "SchemaArn": "arn:aws:schemas:us-east-1:432143214321:schema/discovered-schemas/CreateAccount@CREATE_ACCOUNT", "SchemaName": “CreateAccount@ACCOUNT_CREATED", "SchemaVersion": "1", "Tags": {}, "Type": "OpenApi3", "VersionCreatedDate": "2020-04-28T11:10:15+00:00" } Using the AWS Command Line Interface (CLI), the FraudCheck team can create a code binding if it isn’t already created, using the put-code-binding command, and then download the code binding to process that event: aws schemas get-code-binding-source \ --registry-name arn:aws:schemas:us-east-1:432143214321:registry/discovered-schemas \ --schema-name CreateAccount@ACCOUNT_CREATED \ --language Java8 CreateAccount.zip Another option for the FraudCheck team is to copy and paste (after unescaping the JSON string) the Content of the discovered schema to create a new custom schema in their AWS account. Once the schema is copied to their own account, the FraudCheck team can use the AWS Toolkit IDE plugins to view the schema, download code bindings, and generate serverless applications directly from their IDEs. The EventBridge team is working to add the capability to the AWS Toolkit to use a schema registry in a different account, making this step simpler. Stay tuned! Often customers have a specific team, with a different AWS account, managing the event bus. For the sake of simplicity, in this post I assumed that the CreateAccount team was the one configuring the EventBridge event bus. With more accounts, you can simplify permissions using IAM to share resources with groups of AWS accounts in AWS Organizations. Available Now The EventBridge Schema Registry is available now in all commercial regions except Bahrain, Cape Town, Milan, Osaka, Beijing, and Ningxia. For more information on how to use resource policies for schema registries, please see the documentation. Using Schema Registry resource policies, it is much easier to coordinate the work of different teams sharing information in an event-driven architecture. Let me know what are you going to build with this! — Danilo

New – Amazon Simple Email Service (SES) for VPC Endpoints

Amazon Web Services Blog -

Although chat and messaging applications have been popular, the email has retained its place as a ubiquitous channel with the highest Return on Investment (ROI) because of its low barrier to entry, affordability and ability to target specific recipients. To ensure that organization’s marketing and transactional messages are received by the end customer in a timely manner and to drive deeper engagement with them, you need to partner with a mature and trusted email service provider that has built specialized expertise in delivering email at scale. Amazon Simple Email Services(SES) has been the trustworthy, flexible and affordable email service provider for developers and digital marketers since 2011. Amazon SES is a reliable, cost-effective service for businesses of all sizes that use email to keep in contact with their customers. Many businesses operate in industries that are highly secure and have strict security policies. So we have enhanced security and compliance features in Amazon SES, such as enabling you to configure DKIM using your own RSA key pair, and support HIPAA Eligibility and FIPS 140-2 Compliant Endpoints as well as regional expansions. Today, I am pleased to announce that customers can now connect directly from Virtual Private Cloud (VPC) to Amazon SES through a VPC Endpoint, powered by AWS PrivateLink, in a secure and scalable manner. You can now access Amazon SES through your VPC without requiring an Internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. When you use an interface VPC Endpoint, communication between your VPC and Amazon SES APIs stays within the Amazon network, adding increased security. With this launch, the traffic to Amazon SES does not transit over the Internet and never leaves the Amazon network to securely connect their VPC to Amazon SES without imposing availability risks or bandwidth constraints on their network traffic. You can centralize Amazon SES across your multi-account infrastructure and provide it as a service to your accounts without the need to utilizing an Internet gateway. Amazon SES for VPC Endpoints – Getting Started If you want to test sending emails from your EC2 instance in default VPC, Create a Security Group with following inbound rules and set the private IP of your instance in the EC2 console. To create the VPC Endpoint for Amazon SES, use the Creating an Interface Endpoint procedure in the VPC console and select com.amazonaws.region.email-smtp service name, and attach security group that you just create it. After your endpoint will be available, you can ssh to your EC2 instance and use openssl command to test connection or send email through just created endpoint. You can interact with the same way of SMTP interface from your operating system’s command line. $ openssl s_client -crlf -quiet -starttls smtp -connect email-smtp.ap-southeast-2.amazonaws.com:465 ... depth=2 C = US, O = Amazon, CN = Amazon Root CA 1 verify return:1 depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon verify return:1 depth=0 CN = email-smtp.ap-southeast-2.amazonaws.com verify return:1 ... 220 email-smtp.amazonaws.com ESMTP SimpleEmailService-d-ZIFLXXX HELO email-smtp.amazonaws.com ... 250 Ok Note that VPC Endpoints currently do not support cross-region requests—ensure that you create your endpoint in the same region in which you plan to issue your API calls to Amazon SES. Now Available! Amazon SES for VPC Endpoints is generally available and you can use it in all regions where Amazon SES is available. There is no additional charge for using Amazon SES for VPC Endpoints. Take a look at the product page and the documentation to learn more. Please send feedback to AWS forum for Amazon SES or through your usual AWS support contacts. — Channy;

Now Open – AWS Europe (Milan) Region

Amazon Web Services Blog -

Today, I am very happy to announce that, as we anticipated some time ago, a new AWS Region is available in Italy! The Europe (Milan) Region is our sixth Region in Europe, and is composed of 3 availability zones (AZs) that you can use to reliably spread your applications across multiple data centers, for example configuring the subnets of your Amazon Virtual Private Cloud to use different AZs for your Amazon Elastic Compute Cloud (EC2) instances. Each AZ is a fully isolated partition of our infrastructure that contains one or more data centers. AZs are located in separate and distinct geographic locations with enough distance to significantly reduce the risk of a single event impacting availability in the Region, but near enough for business continuity applications that require rapid failover and synchronous replication. This gives you the ability to operate production applications that are more highly available, more fault tolerant, and more scalable than would be possible from a single data center. Fully managed services like Amazon Simple Storage Service (S3), AWS Lambda, and Amazon DynamoDB, replicate data and applications across AZs automatically. The AWS Region in Milan offers low latency for customers seeking to serve end-users in Italy, and also has a latency advantage over other existing AWS regions when serving customers from other countries such as Austria, Greece, and Bulgaria. Results may differ based on the quality, capacity, and distance of the connection in the end user’s last-mile network. An in-country infrastructure is also critical for Italian customers with data residency requirements and regulations, such as those operating in government, healthcare, and financial services. AWS in Italy Currently AWS has five edge locations in Italy (three in Milan, one in Palermo, and one in Rome) and an AWS Direct Connect location in Milan which connects to the Europe (Frankfurt) Region and to the new Region in Milan. We opened the first AWS office in Italy at the beginning of 2014 in Milan, there is now also an office in Rome, engineering teams in Piedmont and Sardinia, and a broad network of partners. AWS continues to build in Italy teams of account managers, solutions architects, business developers, and professional services consultants to help customers of all sizes build or move their workloads in the cloud. In 2016, AWS acquired Italy-based NICE Software, a leading provider of software and services for high performance and technical computing. I joined AWS in 2012, and I was immediately blown away by what Italian customers were building on AWS to lower their costs, become more agile, and innovate faster. For example: GEDI Gruppo Editoriale is an Italian multimedia giant that publishes some of the largest circulation newspapers in Italy, including La Repubblica and La Stampa. In March 2018, during the Italian general elections, they experienced over 80 million page views and 18.8 million unique visits, and were able to provide their readers with continuous special election-day coverage with real-time data of election results. Satispay is disrupting the mobile payment landscape, allowing their users to securely send money or pay using a smartphone app that relies on International Bank Account Numbers (IBANs), and directly connects consumers and merchants via their bank accounts. They are all-in on AWS, and benefit from AWS’s compliance accreditations and certifications, many of which are required to operate in the financial services industry. Adopting DevOps and CI/CD best practices, they went from one deployment per week to 16 deployments per day, giving them the freedom and flexibility to develop new features, and innovate faster. Musixmatch is a Bologna-based startup that has quickly become the world’s largest lyrics platform , with more than 50 million users and over 14 million lyrics in 58 languages. Musixmatch is using AWS to be able to scale quickly, and most importantly to innovate constantly. In just three days, Musixmatch started using Amazon SageMaker to train models to analyze the language of songs and identify the mood and emotion of the lyrics. This allowed Musixmatch to build a platform where users can find new music based on emotions and mood of the lyrics of songs. Musixmatch also used SageMaker to train umBERTo, a state-of-the-art Italian language model. Avio Aero is an aerospace business that designs, constructs, and maintains systems and components for military and civil aviation. Among other things, they developed a serverless application for their finance team to manage expense approvals and purchase orders. They are excited by the new Region because they have many applications that contain particularly critical data that needs to be stored in Italy. Available Now The new Europe (Milan) Region is ready to support your business. You can look at the Region Table for service availability. With this launch, AWS now has 76 AZs within 24 geographic Regions around the world, with 3 new Regions coming in Indonesia, Japan, and Spain. To build a secure, high-performing, resilient, and efficient infrastructure for your applications, you can leverage the best practices we shared in the AWS Well Architected Framework, and review your architecture using the AWS Well-Architected Tool. For more information on our global infrastructure, and the custom hardware we use, check out this interactive map. — Danilo

New – Serverless Streaming ETL with AWS Glue

Amazon Web Services Blog -

When you have applications in production, you want to understand what is happening, and how the applications are being used. To analyze data, a first approach is a batch processing model: a set of data is collected over a period of time, then run through analytics tools. To be able to react quickly, you can use a streaming model, where data is processed as it arrives, a record at a time or in micro-batches of tens, hundreds, or thousands of records. Managing continuous ingestion pipelines and processing data on-the-fly is quite complex, because it’s an always-on system that needs to be managed, patched, scaled, and generally taken care of. Today, we are making this easier and more cost-effective to implement by extending AWS Glue jobs, based on Apache Spark, to run continuously and consume data from streaming platforms such as Amazon Kinesis Data Streams and Apache Kafka (including the fully-managed Amazon MSK). In this way, Glue can provision, manage, and scale the infrastructure needed to ingest data to data lakes on Amazon S3, data warehouses such as Amazon Redshift, or other data stores. For example, you can store streaming data in a DynamoDB table for quick lookups, or in Elasticsearch to look for specific patterns. This procedure is usually referred to as extract, transform, load (ETL). As you process streaming data in a Glue job, you have access to the full capabilities of Spark Structured Streaming to implement data transformations, such as aggregating, partitioning, and formatting as well as joining with other data sets to enrich or cleanse the data for easier analysis. For example, you can access an external system to identify fraud in real-time, or use machine learning algorithms to classify data, or detect anomalies and outliers. Processing Streaming Data with AWS Glue To try this new feature, I want to collect data from IoT sensors and store all data points in an S3 data lake. I am using a Raspberry Pi with a Sense HAT to collect temperature, humidity, barometric pressure, and its position in space in real-time (using the integrated gyroscope, accelerometer, and magnetometer). Here’s an architectural view of what I am building: First, I register the device with AWS IoT Core, and run the following Python code to send, once per second, a JSON message with sensor data to the streaming-data MQTT topic. I have a single device in this setup, with more devices, I would use a subtopic per device, for example streaming-data/{client_id}. import time import datetime import json from sense_hat import SenseHat from awscrt import io, mqtt, auth, http from awsiot import mqtt_connection_builder sense = SenseHat() topic = "streaming-data" client_id = "raspberrypi" # Callback when connection is accidentally lost. def on_connection_interrupted(connection, error, **kwargs): print("Connection interrupted. error: {}".format(error)) # Callback when an interrupted connection is re-established. def on_connection_resumed(connection, return_code, session_present, **kwargs): print("Connection resumed. return_code: {} session_present: {}".format( return_code, session_present)) if return_code == mqtt.ConnectReturnCode.ACCEPTED and not session_present: print("Session did not persist. Resubscribing to existing topics...") resubscribe_future, _ = connection.resubscribe_existing_topics() # Cannot synchronously wait for resubscribe result because we're on the connection's event-loop thread, # evaluate result with a callback instead. resubscribe_future.add_done_callback(on_resubscribe_complete) def on_resubscribe_complete(resubscribe_future): resubscribe_results = resubscribe_future.result() print("Resubscribe results: {}".format(resubscribe_results)) for topic, qos in resubscribe_results['topics']: if qos is None: sys.exit("Server rejected resubscribe to topic: {}".format(topic)) # Callback when the subscribed topic receives a message def on_message_received(topic, payload, **kwargs): print("Received message from topic '{}': {}".format(topic, payload)) def collect_and_send_data(): publish_count = 0 while(True): humidity = sense.get_humidity() print("Humidity: %s %%rH" % humidity) temp = sense.get_temperature() print("Temperature: %s C" % temp) pressure = sense.get_pressure() print("Pressure: %s Millibars" % pressure) orientation = sense.get_orientation_degrees() print("p: {pitch}, r: {roll}, y: {yaw}".format(**orientation)) timestamp = datetime.datetime.fromtimestamp( time.time()).strftime('%Y-%m-%d %H:%M:%S') message = { "client_id": client_id, "timestamp": timestamp, "humidity": humidity, "temperature": temp, "pressure": pressure, "pitch": orientation['pitch'], "roll": orientation['roll'], "yaw": orientation['yaw'], "count": publish_count } print("Publishing message to topic '{}': {}".format(topic, message)) mqtt_connection.publish( topic=topic, payload=json.dumps(message), qos=mqtt.QoS.AT_LEAST_ONCE) time.sleep(1) publish_count += 1 if __name__ == '__main__': # Spin up resources event_loop_group = io.EventLoopGroup(1) host_resolver = io.DefaultHostResolver(event_loop_group) client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver) mqtt_connection = mqtt_connection_builder.mtls_from_path( endpoint="a1b2c3d4e5f6g7-ats.iot.us-east-1.amazonaws.com", cert_filepath="rapberrypi.cert.pem", pri_key_filepath="rapberrypi.private.key", client_bootstrap=client_bootstrap, ca_filepath="root-CA.crt", on_connection_interrupted=on_connection_interrupted, on_connection_resumed=on_connection_resumed, client_id=client_id, clean_session=False, keep_alive_secs=6) connect_future = mqtt_connection.connect() # Future.result() waits until a result is available connect_future.result() print("Connected!") # Subscribe print("Subscribing to topic '{}'...".format(topic)) subscribe_future, packet_id = mqtt_connection.subscribe( topic=topic, qos=mqtt.QoS.AT_LEAST_ONCE, callback=on_message_received) subscribe_result = subscribe_future.result() print("Subscribed with {}".format(str(subscribe_result['qos']))) collect_and_send_data() This is an example of the JSON messages sent by the device: { "client_id": "raspberrypi", "timestamp": "2020-04-16 11:33:23", "humidity": 39.35261535644531, "temperature": 30.10732078552246, "pressure": 1020.447509765625, "pitch": 4.044007304723748, "roll": 7.533848064912158, "yaw": 77.01560798660883, "count": 104 } In the Kinesis console, I create the my-data-stream data stream (1 shard is more than enough for my workload). Back in the AWS IoT console, I create an IoT rule to send all data from the MQTT topic to this Kinesis data stream. Now that all sensor data is sent to Kinesis, I can leverage the new Glue integration to process data as it arrives. In the Glue console, I manually add a table in the Glue Data Catalog. I select Kinesis as the type of source, and enter my stream name and the endpoint of the Kinesis Data Streams service. Note that for Kafka streams, before creating the table, you need to create a Glue connection. I select JSON as data format, and define the schema for the streaming data. If I don’t specify a column here, it will be ignored when processing the stream. After that, I confirm the final recap step, and create the my_streaming_data table. We are working to add schema inference to streaming ETL jobs. With that, specifying the full schema up front won’t be necessary. Stay tuned. To process the streaming data, I create a Glue job. For the IAM role, I create a new one attaching the AWSGlueServiceRole and AmazonKinesisReadOnlyAccess managed policies. Depending on your use case and the set up of your AWS accounts, you may want to use a role providing more fine-grained access. For the data source, I select the table I just created, receiving data from the Kinesis stream. To get a script generated by Glue, I select the Change schema transform type. As target, I create a new table in the Glue Data Catalog, using an efficient format like Apache Parquet. The Parquet files generated by this job are going to be stored in an S3 bucket whose name starts with aws-glue- (including the final hyphen). By following the naming convention for resources specified in the AWSGlueServiceRole policy, this job has the required permissions to access those resources. I leave the default mapping that keeps in output all the columns in the source stream. In this way, I can ingest all the records using the proposed script, without having to write a single line of code. I quickly review the proposed script and save. Each record is processed as a DynamicFrame, and I can apply any of the Glue PySpark Transforms or any transforms supported by Spark Structured Streaming. By default with this configuration, only ApplyMapping is used. I start the job, and after a few minutes I see the Parquet files containing the output of the job appearing in the output S3 bucket. They are partitioned by ingest date (year, month, day, and hour). To populate the Glue Data Catalog with tables based on the content of the S3 bucket, I add and run a crawler. In the crawler configuration, I exclude the checkpoint folder used by Glue to keep track of the data that has been processed. After less than a minute, a new table has been added. In the Amazon Athena console, I refresh database and tables, and select to preview the output_my_data containing ingest data from this year. In this way, I see the first ten records in the table, and get a confirmation that my setup is working! Now, as data is being ingested, I can run more complex queries. For example, I can get the minimum and maximum temperature, collected from the device sensors, and the overall number of records stored in the Parquet files. Looking at the results, I see more than 8,000 records have been processed, with a maximum temperature of 31 degrees Celsius (about 88 degrees Fahrenheit). Actually, it was never really this hot. Temperature is measured by these sensors very close to the device, and is growing as the device is warming up with usage. I am using a single device in this set up, but the solution implemented here can easily scale up with the number of data sources. Available Now Support for streaming sources is available in all regions where Glue is offered, as described in the AWS Region table. For more information, please have a look at the documentation. Managing a serverless ETL pipeline with Glue makes it easier and more cost-effective to set up and manage streaming ingestion processes, reducing implementation efforts so you can focus on the business outcomes of analytics. You can set up a whole ingestion pipeline without writing code, as I did in this walkthrough, or customize the proposed script based on your needs. Let me know what are you going to use this new feature for! — Danilo

New – Amazon Keyspaces (for Apache Cassandra) is Now Generally Available

Amazon Web Services Blog -

We introduced Amazon Managed Apache Cassandra Service (MCS) in preview at re:Invent last year. In the few months that passed, the service introduced many new features, and it is generally available today with a new name: Amazon Keyspaces (for Apache Cassandra). Amazon Keyspaces is built on Apache Cassandra, and you can use it as a fully managed, serverless database. Your applications can read and write data from Amazon Keyspaces using your existing Cassandra Query Language (CQL) code, with little or no changes. For each table, you can select the best configuration depending on your use case: With on-demand, you pay based on the actual reads and writes you perform. This is the best option for unpredictable workloads. With provisioned capacity, you can reduce your costs for predictable workloads by configuring capacity settings up front. You can also further optimize costs by enable auto scaling, which updates your provisioned capacity settings automatically as your traffic changes throughout the day. Using Amazon Keyspaces One of the first “serious” applications I built as a kid, was an archive for my books. I’d like to rebuild it now as a serverless API, using: Amazon Keyspaces to store data. AWS Lambda for the business logic. Amazon API Gateway with the new HTTP API. With Amazon Keyspaces, your data is stored in keyspaces and tables. A keyspace gives you a way to group related tables together. In the blog post for the preview, I used the console to configure my data model. Now, I can also use AWS CloudFormation to manage my keyspaces and tables as code. For example I can create a bookstore keyspace and a books table with this CloudFormation template: AWSTemplateFormatVersion: '2010-09-09' Description: Amazon Keyspaces for Apache Cassandra example Resources: BookstoreKeyspace: Type: AWS::Cassandra::Keyspace Properties: KeyspaceName: bookstore BooksTable: Type: AWS::Cassandra::Table Properties: TableName: books KeyspaceName: !Ref BookstoreKeyspace PartitionKeyColumns: - ColumnName: isbn ColumnType: text RegularColumns: - ColumnName: title ColumnType: text - ColumnName: author ColumnType: text - ColumnName: pages ColumnType: int - ColumnName: year_of_publication ColumnType: int Outputs: BookstoreKeyspaceName: Description: "Keyspace name" Value: !Ref BookstoreKeyspace # Or !Select [0, !Split ["|", !Ref BooksTable]] BooksTableName: Description: "Table name" Value: !Select [1, !Split ["|", !Ref BooksTable]] If you don’t specify a name for a keyspace or a table in the template, CloudFormation generates a unique name for you. Note that in this way keyspaces and tables may contain uppercase characters that are outside of the usual Cassandra conventions, and you need to put those names between double quotes when using Cassandra Query Language (CQL). When the creation of the stack is complete, I see the new bookstore keyspace in the console: Selecting the books table, I have an overview of its configuration, including the partition key, the clustering columns, and all the columns, and the option to change the capacity mode for the table from on-demand to provisioned: For authentication and authorization, Amazon Keyspaces supports AWS Identity and Access Management (IAM) identity-based policies, that you can use with IAM users, groups, and roles. Here’s a list of actions, resources, and conditions that you can use in IAM policies with Amazon Keyspaces. You can now also manage access to resources based on tags. You can use IAM roles using AWS Signature Version 4 Process (SigV4) with this open source authentication plugin for the DataStax Java driver. In this way you can run your applications inside an Amazon Elastic Compute Cloud (EC2) instance, a container managed by Amazon ECS or Amazon Elastic Kubernetes Service, or a Lambda function, and leverage IAM roles for authentication and authorization to Amazon Keyspaces, without the need to manage credentials. Here’s a sample application that you can test on an EC2 instance with an associated IAM role giving access to Amazon Keyspaces. Going back to my books API, I create all the resources I need, including a keyspace and a table, with the following AWS Serverless Application Model (SAM) template. AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Description: Sample Books API using Cassandra as database Globals: Function: Timeout: 30 Resources: BookstoreKeyspace: Type: AWS::Cassandra::Keyspace BooksTable: Type: AWS::Cassandra::Table Properties: KeyspaceName: !Ref BookstoreKeyspace PartitionKeyColumns: - ColumnName: isbn ColumnType: text RegularColumns: - ColumnName: title ColumnType: text - ColumnName: author ColumnType: text - ColumnName: pages ColumnType: int - ColumnName: year_of_publication ColumnType: int BooksFunction: Type: AWS::Serverless::Function Properties: CodeUri: BooksFunction Handler: books.App::handleRequest Runtime: java11 MemorySize: 2048 Policies: - Statement: - Effect: Allow Action: - cassandra:Select Resource: - !Sub "arn:aws:cassandra:${AWS::Region}:${AWS::AccountId}:/keyspace/system*" - !Join - "" - - !Sub "arn:aws:cassandra:${AWS::Region}:${AWS::AccountId}:/keyspace/${BookstoreKeyspace}/table/" - !Select [1, !Split ["|", !Ref BooksTable]] # !Ref BooksTable returns "Keyspace|Table" - Effect: Allow Action: - cassandra:Modify Resource: - !Join - "" - - !Sub "arn:aws:cassandra:${AWS::Region}:${AWS::AccountId}:/keyspace/${BookstoreKeyspace}/table/" - !Select [1, !Split ["|", !Ref BooksTable]] # !Ref BooksTable returns "Keyspace|Table" Environment: Variables: KEYSPACE_TABLE: !Ref BooksTable # !Ref BooksTable returns "Keyspace|Table" Events: GetAllBooks: Type: HttpApi Properties: Method: GET Path: /books GetBookByIsbn: Type: HttpApi Properties: Method: GET Path: /books/{isbn} PostNewBook: Type: HttpApi Properties: Method: POST Path: /books Outputs: BookstoreKeyspaceName: Description: "Keyspace name" Value: !Ref BookstoreKeyspace # Or !Select [0, !Split ["|", !Ref BooksTable]] BooksTableName: Description: "Table name" Value: !Select [1, !Split ["|", !Ref BooksTable]] BooksApi: Description: "API Gateway HTTP API endpoint URL" Value: !Sub "https://${ServerlessHttpApi}.execute-api.${AWS::Region}.amazonaws.com/" BooksFunction: Description: "Books Lambda Function ARN" Value: !GetAtt BooksFunction.Arn BooksFunctionIamRole: Description: "Implicit IAM Role created for Books function" Value: !GetAtt BooksFunctionRole.Arn In this template I don’t specify the keyspace and table names, and CloudFormation is generating unique names automatically. The function IAM policy gives access to read (cassandra:Select) and write (cassandra:Write) only to the books table. I am using CloudFormation Fn::Select and Fn::Split intrinsic functions to get the table name. The driver also needs read access to the system* keyspaces. To use the authentication plugin for the DataStax Java driver that supports IAM roles, I write the Lambda function in Java, using the APIGatewayV2ProxyRequestEvent and APIGatewayV2ProxyResponseEvent classes to communicate with the HTTP API created by the API Gateway. package books; import java.net.InetSocketAddress; import java.security.NoSuchAlgorithmException; import java.util.Collections; import java.util.List; import java.util.HashMap; import java.util.Map; import java.util.StringJoiner; import javax.net.ssl.SSLContext; import org.json.simple.JSONObject; import org.json.simple.parser.JSONParser; import org.json.simple.parser.ParseException; import com.datastax.oss.driver.api.core.ConsistencyLevel; import com.datastax.oss.driver.api.core.CqlSession; import com.datastax.oss.driver.api.core.cql.*; import software.aws.mcs.auth.SigV4AuthProvider; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.RequestHandler; import com.amazonaws.services.lambda.runtime.LambdaLogger; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyRequestEvent; import com.amazonaws.services.lambda.runtime.events.APIGatewayV2ProxyResponseEvent; public class App implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> { JSONParser parser = new JSONParser(); String[] keyspace_table = System.getenv("KEYSPACE_TABLE").split("\\|"); String keyspace = keyspace_table[0]; String table = keyspace_table[1]; CqlSession session = getSession(); PreparedStatement selectBookByIsbn = session.prepare("select * from \"" + table + "\" where isbn = ?"); PreparedStatement selectAllBooks = session.prepare("select * from \"" + table + "\""); PreparedStatement insertBook = session.prepare("insert into \"" + table + "\" " + "(isbn, title, author, pages, year_of_publication)" + "values (?, ?, ?, ?, ?)"); public APIGatewayV2ProxyResponseEvent handleRequest(APIGatewayV2ProxyRequestEvent request, Context context) { LambdaLogger logger = context.getLogger(); String responseBody; int statusCode = 200; String routeKey = request.getRequestContext().getRouteKey(); logger.log("routeKey = '" + routeKey + "'"); if (routeKey.equals("GET /books")) { ResultSet rs = execute(selectAllBooks.bind()); StringJoiner jsonString = new StringJoiner(", ", "[ ", " ]"); for (Row row : rs) { String json = row2json(row); jsonString.add(json); } responseBody = jsonString.toString(); } else if (routeKey.equals("GET /books/{isbn}")) { String isbn = request.getPathParameters().get("isbn"); logger.log("isbn: '" + isbn + "'"); ResultSet rs = execute(selectBookByIsbn.bind(isbn)); if (rs.getAvailableWithoutFetching() == 1) { responseBody = row2json(rs.one()); } else { statusCode = 404; responseBody = "{\"message\": \"not found\"}"; } } else if (routeKey.equals("POST /books")) { String body = request.getBody(); logger.log("Body: '" + body + "'"); JSONObject requestJsonObject = null; if (body != null) { try { requestJsonObject = (JSONObject) parser.parse(body); } catch (ParseException e) { e.printStackTrace(); } if (requestJsonObject != null) { int i = 0; BoundStatement boundStatement = insertBook.bind() .setString(i++, (String) requestJsonObject.get("isbn")) .setString(i++, (String) requestJsonObject.get("title")) .setString(i++, (String) requestJsonObject.get("author")) .setInt(i++, ((Long) requestJsonObject.get("pages")).intValue()) .setInt(i++, ((Long) requestJsonObject.get("year_of_publication")).intValue()) .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM); ResultSet rs = execute(boundStatement); statusCode = 201; responseBody = body; } else { statusCode = 400; responseBody = "{\"message\": \"JSON parse error\"}"; } } else { statusCode = 400; responseBody = "{\"message\": \"body missing\"}"; } } else { statusCode = 405; responseBody = "{\"message\": \"not implemented\"}"; } Map<String, String> headers = new HashMap<>(); headers.put("Content-Type", "application/json"); APIGatewayV2ProxyResponseEvent response = new APIGatewayV2ProxyResponseEvent(); response.setStatusCode(statusCode); response.setHeaders(headers); response.setBody(responseBody); return response; } private String getStringColumn(Row row, String columnName) { return "\"" + columnName + "\": \"" + row.getString(columnName) + "\""; } private String getIntColumn(Row row, String columnName) { return "\"" + columnName + "\": " + row.getInt(columnName); } private String row2json(Row row) { StringJoiner jsonString = new StringJoiner(", ", "{ ", " }"); jsonString.add(getStringColumn(row, "isbn")); jsonString.add(getStringColumn(row, "title")); jsonString.add(getStringColumn(row, "author")); jsonString.add(getIntColumn(row, "pages")); jsonString.add(getIntColumn(row, "year_of_publication")); return jsonString.toString(); } private ResultSet execute(BoundStatement bs) { final int MAX_RETRIES = 3; ResultSet rs = null; int retries = 0; do { try { rs = session.execute(bs); } catch (Exception e) { e.printStackTrace(); session = getSession(); // New session } } while (rs == null && retries++ < MAX_RETRIES); return rs; } private CqlSession getSession() { System.setProperty("javax.net.ssl.trustStore", "./cassandra_truststore.jks"); System.setProperty("javax.net.ssl.trustStorePassword", "amazon"); String region = System.getenv("AWS_REGION"); String endpoint = "cassandra." + region + ".amazonaws.com"; System.out.println("region: " + region); System.out.println("endpoint: " + endpoint); System.out.println("keyspace: " + keyspace); System.out.println("table: " + table); SigV4AuthProvider provider = new SigV4AuthProvider(region); List<InetSocketAddress> contactPoints = Collections.singletonList(new InetSocketAddress(endpoint, 9142)); CqlSession session; try { session = CqlSession.builder().addContactPoints(contactPoints).withSslContext(SSLContext.getDefault()) .withLocalDatacenter(region).withAuthProvider(provider).withKeyspace("\"" + keyspace + "\"") .build(); } catch (NoSuchAlgorithmException e) { session = null; e.printStackTrace(); } return session; } } To connect to Amazon Keyspaces with TLS/SSL using the Java driver, I need to include a trustStore in the JVM arguments. When using the Cassandra Java Client Driver in a Lambda function, I can’t pass parameters to the JVM, so I pass the same options as system properties, and specify the SSL context when creating the CQL session with the  withSslContext(SSLContext.getDefault()) parameter. Note that I also have to configure the pom.xml file, used by Apache Maven, to include the trustStore file as a dependency. System.setProperty("javax.net.ssl.trustStore", "./cassandra_truststore.jks"); System.setProperty("javax.net.ssl.trustStorePassword", "amazon"); Now, I can use a tool like curl or Postman to test my books API. First, I take the endpoint of the API from the output of the CloudFormation stack. At the beginning there are no books stored in the books table, and if I do an HTTP GET on the resource, I get an empty JSON list. For readability, I am removing all HTTP headers from the output. $ curl -i https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/books HTTP/1.1 200 OK [] In the code, I am using a PreparedStatement to run a CQL statement to select all rows from the books table. The names of the keystore and of the table are passed to the Lambda function in an environment variable, as described in the SAM template above. Let’s use the API to add a book, by doing an HTTP POST on the resource. $ curl -i -d '{ "isbn": "978-0201896831", "title": "The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd Edition)", "author": "Donald E. Knuth", "pages": 672, "year_of_publication": 1997 }' -H "Content-Type: application/json" -X POST https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/books HTTP/1.1 201 Created { "isbn": "978-0201896831", "title": "The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd Edition)", "author": "Donald E. Knuth", "pages": 672, "year_of_publication": 1997 } I can check that the data has been inserted in the table using the CQL Editor in the console, where I select all the rows in the table. I repeat the previous HTTP GET to get the list of the books, and I see the one I just created. $ curl -i https://a1b2c3d4e5-api.eu-west-1.amazonaws.com/books HTTP/1.1 200 OK [ { "isbn": "978-0201896831", "title": "The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd Edition)", "author": "Donald E. Knuth", "pages": 672, "year_of_publication": 1997 } ] I can get a single book by ISBN, because the isbn column is the primary key of the table and I can use it in the where condition of a select statement. $ curl -i https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/books/978-0201896831 HTTP/1.1 200 OK { "isbn": "978-0201896831", "title": "The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd Edition)", "author": "Donald E. Knuth", "pages": 672, "year_of_publication": 1997 } If there is no book with that ISBN, I return a “not found” message: $ curl -i https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/books/1234567890 HTTP/1.1 404 Not Found {"message": "not found"} It works! We just built a fully serverless API using CQL to read and write data using temporary security credentials, managing the whole infrastructure, including the database table, as code. Available Now Amazon Keyspace (for Apache Cassandra) is ready for your applications, please see this table for regional availability. You can find more information on how to use Keyspaces in the documentation. In this post, I built a new application, but you can get lots of benefits by migrating your current tables to a fully managed environment. For migrating data, you can now use cqlsh as described in this post. Let me know what are you going to use it for! — Danilo

New – AWS Transfer for FTP and FTPS, in addition to existing SFTP

Amazon Web Services Blog -

AWS Transfer for SFTP was launched on November 2018 as a fully managed service that enables the transfer of files directly into and out of Amazon S3 using the Secure File Transfer Protocol (SFTP). Today, we are happy to announce the expansion of the service to add support for FTPS and FTP, which makes it easy to migrate and securely run File Transfer Protocol over SSL (FTPS) and FTP workloads in AWS, in addition to the existing AWS Transfer for SFTP service. Supporting SFTP-, FTPS-, and FTP-based transfers for Amazon S3, we are also announcing the “AWS Transfer Family,” which is the aggregated name of AWS Transfer for SFTP, FTPS, and FTP. Some software archiving and scientific research applications use FTP to distribute software artifacts or public datasets, and CRM, ERP, and supply chain applications use FTPS for transferring sensitive data. Many of their existing applications cannot switch from FTP or FTPS to SFTP because this requires changing existing applications and processes – especially those involving third-parties – and is often impractical or infeasible. Customers are looking for an easy and secure way to migrate their file transfers without disrupting their existing integrations and end users. For these reasons, we are launching AWS Transfer for FTPS and AWS Transfer for FTP. Basic difference between SFTP and FTPS/FTP Let’s talk a bit about the differences among SFTP and FTPS/FTP before we start a walk through. These are actually different protocols, but they work similar to “File Transfer.” Secure File Transfer Protocol (SFTP) – Defined by the Internet Engineering Task Force (IETF) as an extended version of SSH 2.0, allowing file transfer over SSH and for use with Transport Layer Security (TLS) and VPN applications. File Transfer Protocol (FTP) – Defined by RFC114 originally, and replaced by RFC765 and RFC959 for TCP/IP basis. File Transfer Protocol over SSL/TLS (FTPS) – Used to encrypt FTP communication by SSL/TLS. Until now, customers with multiple protocol needs were using the service for SFTP or were waiting for this launch. With this announcement, customers who use either of the three protocols can migrate and leverage AWS services for their end to end file transfer needs. Availability of these new protocols increases accessibility to your data, while the same options that were available for SFTP can be used for FTPS and FTP to secure access. Access control features available include using use of IAM Roles and policies, Logical directories for S3 and Security Groups. Walk through This walk through provides a step-by-step guide for creating a fully managed FTP Server. FTP servers are only accessible inside your VPC, including AWS Direct Connect or VPN. You can use FTPS if you need access via the internet. You will see a new AWS console page when you access the AWS Transfer Family console. Click Create server to begin. There are now three protocol choices – SFTP, FTPS, and FTP. For this example, let’s start the walk through by selecting FTP. Click the FTP check box, and uncheck the SFTP check box. We can assign both protocols at the same time, but we are creating a FTP server as the new feature for this step. Click Next We now need to assign an Identity provider. The identity provider is used for authentication when logging on to the FTP server. Only Custom which is provided by Amazon API Gateway is supported for FTPS and FTP. To be able to invoke the API, we need to create an Invocation URL, which is an API Gateway endpoint, and also an IAM role. Here are guidelines for how to create an Invocation URL using CloudFormation with a yaml template. For servers enabled for SFTP only, you can also choose Service Managed authentication to store and manage identities within the service. Click Next, and an Endpoint configuration dialog comes up. We can only choose VPC as a VPC hosted endpoint for FTP. If we need access from the internet, we need to choose FTPS instead of FTP for security reasons. Then, we choose an appropriate VPC and its subnet to host the endpoint. Click Next, and the next dialog comes up. The next step is optional. We can enable CloudWatch logging by assigning an IAM role. The CloudFormation template above created an IAM role that we can use, or we can use a different role. We skip the Server Host key section because this is for SFTP. Assign the appropriate tags and click Next. Then, click Create server. The FTP server is created. Click Server ID, and we see the detail of the FTP server. It is time to test the FTP server! From Action, let’s select Test. Type “myuser” as Username and “MySuperSecretPassword” as Password. Status code of HTTP 200 is returned if your FTP server is successfully integrated with your identity provider. Now that we know your identity provider is all integrated correctly, let’s test using a ftp client. We can now perform cd/ls/put/get/rm operations using a FTP client against an existing Amazon S3 bucket(s). We use Amazon EC2 for this walk through. Create an instance if we do not have it in the subnet specified above, and Install lftp client. sudo yum install lftp To connect to the server, we will need its endpoint URL of the FTP server. We need to access the VPC Endpoint console to obtain it. If you were using an internet facing SFTP and/or FTPS server, you could get this information directly from the AWS Transfer Family Console. If we access the Endpoint from another subnet or other VPC, please be sure that Security Groups allows TCP port 21 and port 8192-8200. Then, we can try to login to the FTP server by below command; lftp -d ftp://{VPC End Point of your FTP Server} -u 'myuser, MySuperSecretPassword' (Click to enlarge the image) Next Step Username and Password for test is specified in the source code inside the Lambda function created by CloudFormation as guided. The blog article “Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager” is a good way to start to learn more about managing an authentication data, and this CloudFormation template is used for creating API Gateway and Lambda functions with AWS Secrets Manager. Closing Remarks: Only Passive mode is supported. Our service does not make outbound connections. Only Explicit mode for FTPS is supported. Our service does not support implicit mode. Renaming file name is supported, but renaming directory (S3 BucketName) is not supported, and also append operations are not supported Available Today AWS Transfer for FTPS and FTP are available in all Regions where AWS Transfer for SFTP is currently available. Take a look at the product page and the documentation to learn more. You can also check this video for a demo. – Kame;  

New – Use CloudWatch Synthetics to Monitor Sites, API Endpoints, Web Workflows, and More

Amazon Web Services Blog -

Today’s applications encompass hundreds or thousands of moving parts including containers, microservices, legacy internal services, and third-party services. In addition to monitoring the health and performance of each part, you need to make sure that the parts come together to deliver an acceptable customer experience. CloudWatch Synthetics (announced at AWS re:Invent 2019) allows you to monitor your sites, API endpoints, web workflows, and more. You get an outside-in view with better visibility into performance and availability so that you can become aware of and then fix any issues quicker than ever before. You can increase customer satisfaction and be more confident that your application is meeting your performance goals. You can start using CloudWatch Synthetics in minutes. You simply create canaries that monitor individual web pages, multi-page web workflows such as wizards and checkouts, and API endpoints, with metrics stored in Amazon CloudWatch and other data (screen shots and HTML pages) stored in an S3 bucket. as you create your canaries, you can set CloudWatch alarms so that you are notified when thresholds based on performance, behavior, or site integrity are crossed. You can view screenshots, HAR (HTTP archive) files, and logs to learn more about the failure, with the goal of fixing it as quickly as possible. CloudWatch Synthetics in Action Canaries were once used to provide an early warning that deadly gases were present in a coal mine. The canaries provided by CloudWatch Synthetics provide a similar early warning, and are considerably more humane. I open the CloudWatch Console and click Canaries to get started: I can see the overall status of my canaries at a glance: I created several canaries last month in preparation for this blog post. I chose a couple of sites including the CNN home page, my personal blog, the Amazon Movers and Shakers page, and the Amazon Best Sellers page. I did not know which sites would return the most interesting results, and I certainly don’t mean to pick on any one of them. I do think that it is important to show you how this (and every) feature performs with real data, so here we go! I can turn my attention to the Canary runs section, and look at individual data points. Each data point is an aggregation of runs for a single canary: I can click on the amzn_movers_shakers canary to learn more: I can see that there was a single TimeoutError issue in the last 24 hours. I can see the screenshots that were captured as part of each run, along with the HAR files and logs. Each HAR file contains a detailed log of the HTTP requests that were made when the canary was run, along with the responses and the amount of time that it took for the request to complete: Each canary run is implemented using a Lambda function. I can access the function’s execution metrics in the Metrics tab: And I can see the canary script and other details in the Configuration tab: Hatching a Canary Now that you have seen a canary in action, let me show you how to create one. I return to the list of canaries and click Create canary. I can use one of four blueprints to create my canary, or I can upload or import an existing one: All of these methods ultimately result in a script that is run either once or periodically. The canaries that I showed above were all built from the Heartbeat monitoring blueprint, like this: I can also create canaries for API endpoints, using either GET or PUT methods, any desired HTTP headers, and some request data: Another blueprint lets me create a canary that checks a web page for broken links (I’ll use this post): Finally, the GUI workflow builder lets me create a sophisticated canary that can include simulated clicks, content verification via CSS selector or text, text entry, and navigation to other URLs: As you can see from these examples, the canary scripts are using the syn-1.0 runtime. This runtime supports Node.JS scripts that can use the Puppeteer and Chromium packages. Scripts can make use of a set of library functions and can (with the right IAM permissions) access other AWS services and resources. Here’s an example script that calls AWS Secrets Manager: var synthetics = require('Synthetics'); const log = require('SyntheticsLogger'); const AWS = require('aws-sdk'); const secretsManager = new AWS.SecretsManager(); const getSecrets = async (secretName) => { var params = { SecretId: secretName }; return await secretsManager.getSecretValue(params).promise(); } const secretsExample = async function () { // Fetch secrets var secrets = await getSecrets("secretname") // Use secrets log.info("SECRETS: " + JSON.stringify(secrets)); }; exports.handler = async () => { return await secretsExample(); }; Scripts signal success by running to completion, and errors by raising an exception. After I create my script, I establish a schedule and a pair of data retention periods. I also choose an S3 bucket that will store the artifacts created each time a canary is run: I can also control the IAM role, set CloudWatch Alarms, and configure access to endpoints that are in a VPC: Watch the demo video to see CloudWatch Synthetics in action: Things to Know Here are a couple of things to know about CloudWatch Synthetics: Observability – You can use CloudWatch Synthetics in conjunction with ServiceLens and AWS X-Ray to map issues back to the proper part of your application. To learn more about how to do this, read Debugging with Amazon CloudWatch Synthetics and AWS X-Ray and Using ServiceLens to Monitor the Health of Your Applications. Automation – You can create canaries using the Console, CLI, APIs, and from CloudFormation templates. Pricing – As part of the AWS Free Tier you get 100 canary runs per month at no charge. After that, you pay per run, with prices starting at $0.0012 per run, plus the usual charges for S3 storage and Lambda invocations. Limits – You can create up to 100 canaries per account in the US East (N. Virginia), Europe (Ireland), US West (Oregon), US East (Ohio), and Asia Pacific (Tokyo) Regions, and up to 20 per account in other regions where CloudWatch Synthetics are available. Available Now CloudWatch Synthetics are available now and you can start using them today! — Jeff;

New – Announcing Amazon AppFlow

Amazon Web Services Blog -

Software as a service (SaaS) applications are becoming increasingly important to our customers, and adoption is growing rapidly. While there are many benefits to this way of consuming software, one challenge is that data is now living in lots of different places. To get meaningful insights from this data, we need to have a way to analyze it, and that can be hard when our data is spread out across multiple data islands. Developers spend huge amounts of time writing custom integrations so they can pass data between SaaS applications and AWS services so that it can be analysed; these can be expensive and can often take months to complete. If data requirements change, then costly and complicated modifications have to be made to the integrations. Companies that don’t have the luxury of engineering resources might find themselves manually importing and exporting data from applications, which is time-consuming, risks data leakage, and has the potential to introduce human error. Today it is my pleasure to announce a new service called Amazon AppFlow that will solve this issue. Amazon AppFlow allows you to automate the data flows between AWS services and SaaS applications such as Salesforce, Zendesk, and ServiceNow. SaaS application administrators, business analysts, and BI specialists can quickly implement most of the integrations they need without waiting months for IT to finish integration projects. As well as allowing data to flow in from SaaS applications to AWS services, it’s also capable of sending data from AWS services to SaaS applications. Security is our top priority at AWS, and so all of the data is encrypted while in motion. Some of the SaaS applications have integrated with AWS PrivateLink; this adds an extra layer of security and privacy. When the data flows between the SaaS application and AWS with AWS PrivateLink, the traffic stays on the Amazon network rather than using the public internet. If the SaaS application supports it, Amazon AppFlow automatically takes care of this connection, making private data transfer easy for everyone and minimizing the threat from internet-based attacks and the risk of sensitive data leakage. You can schedule the data transfer to happen on a schedule, in response to a business event, or on demand, giving you speed and flexibility with your data sharing. To show you the power of this new service, I thought it would be interesting to show you how to set up a simple flow. I run a Slack workspace in the United Kingdom and Ireland for web community organizers. Since Slack is one of the supported SaaS applications in Amazon AppFlow, I thought it would be nice to try and import some of the conversation data into S3. Once it was in S3 I could then start to analyze it using Amazon Athena and then ultimately create a visualization using Amazon QuickSight. To get started, I go to the Amazon AppFlow console and click the Create Flow button. In the next step, I enter the Flow name and a Flow description. There are also some options for data encryption. By default, all data is encrypted in transit, and in the case of S3 it’s also encrypted at rest. You have the option to supply your own encryption key, but for this demo, I’m just going to use the key in my account that is used by default. On this step you are also given the option to enter Tags for the resource, I have been getting into the habit of tagging demo infrastructure in my account as Demo which makes it easier for me to know which resources I can delete. On the next step, I select the source of my data. I pick Slack and go through the wizard to establish a connection with my Slack workspace. I also get to choose what data I want to import from my Slack Workspace. I select the Conversations object in the general slack channel. This will import any messages that are posted to the general channel and then send it to the destination that I configure next. There are a few destinations that I can pick, but to keep things simple, I ask for the data to be sent to an S3 bucket. I also set the frequency that I want to fetch the data on this step. I want the data to be retrieved every hour, so I select Run the flow on schedule and make the necessary configurations. Slack can be triggered on demand or on schedule; some other sources can be triggered by specific events, such as converting a lead in Salesforce. The next step is to map the data fields, I am just going to go with the defaults, but you could customize this and combine fields or take only the specific fields required for analysis. Now the flow has been created, and I have activated it; it runs automatically every hour, adding new data to my S3 bucket. I won’t go into the specifics of Amazon Athena or Amazon QuickSight, but I used both of these AWS services to take my data stored in S3 and produce a word cloud of the most common words that are used in my Slack Channel. The cool thing about Athena is that you can run SQL queries directly over the encrypted data in S3 without needing any additional data warehouse. You can see the results in the image below. I could now easily share this as a dashboard with anyone in my organization. Amazon AppFlow is launching today with support for S3 and 13 SaaS applications as sources of data, and S3, Amazon Redshift, Salesforce, and Snowflake as destinations, and you will see us add hundreds more as the service develops. The service automatically scales up or down to meet the demands you place on it, it also allows you to transfer 100GB in a single flow which means you don’t need to break data down into batches. You can trust Amazon AppFlow with your most valuable data as we have architected to be highly available and resilient. Amazon AppFlow is available from today in US East (Northern Virginia), US East (Ohio), US West (Northern California), US West (Oregon), Canada (Central), Asia Pacific (Singapore), Asia Pacific (Toyko), Asia Pacific (Sydney), Asia Pacific (Seoul), Asia Pacific (Mumbai), Europe (Paris), Europe (Ireland), Europe (Frankfurt), Europe (London), and South America (São Paulo) with more regions to come. Happy Data Flowing — Martin

AWS Chatbot – ChatOps for Slack and Chime

Amazon Web Services Blog -

Last year, my colleague Ilya Bezdelev wrote Introducing AWS Chatbot: ChatOps for AWS to launch the public beta of AWS Chatbot. He also participated in the re:Invent 2019 Launchpad and did an in-depth AWS Chatbot demo: In his initial post, Ilya showed you how you can practice ChatOps within Amazon Chime or Slack, receiving AWS notifications and executing commands in an environment that is intrinsically collaborative. In a later post, Running AWS commands from Slack using AWS Chatbot, Ilya showed how to configure AWS Chatbot in a Slack channel, display CloudWatch alarms, describe AWS resources, invoke a Lambda function and retrieve the logs, and create an AWS Support case. My colleagues Erin Carlson and Matt Cowsert wrote about AWS Budgets Integration with Chatbot and walked through the process of setting up AWS Budget alerts and arranging for notifications from within AWS Chatbot. Finally, Anushri Anwekar showed how to Receive AWS Developer Tools Notifications over Slack using AWS Chatbot. As you can see from the posts that I referred to above, AWS Chatbot is a unique and powerful communication tool that has the potential to change the way that you monitor and maintain your cloud environments. Now Generally Available I am happy to announce that AWS Chatbot has graduated from beta to general availability, and that you can use it to practice ChatOps across multiple AWS regions. We are launching with support for Amazon CloudWatch, the AWS Code* services, AWS Health, AWS Budgets, Amazon GuardDuty, and AWS CloudFormation. You can connect it to your Amazon Chime chatrooms and your Slack channels in minutes. Simply open the AWS Chatbot Console, choose your Chat client, and click Configure client to get started: As part of the configuration process you will have the opportunity to choose an existing IAM role or to create a new one from one or more templates. The role provides AWS Chatbot with access to CloudWatch metrics, and the power to run commands, invoke Lambda functions, respond to notification actions, and generate support cases: AWS Chatbot listens on Amazon Simple Notification Service (SNS) topics to learn about events and alarm notifications in each region of interest: You can set up CloudWatch Alarms in any region where you select a topic and use them to send notifications to AWS Chatbot. Special Offer from Slack Our friends at Slack have put together a special offer to help you and your team connect and stay productive through new and shifting circumstances: If you upgrade from the Free Plan to a Standard or Plus Plan you will receive a 25% discount for the first 12 months from your upgrade date. Available Now You can start using AWS Chatbot today at no additional charge. You pay for the underlying services (CloudWatch, SNS, and so forth) as if you were using them without AWS Chatbot, and you also pay any charges associated with the use of your chat client. — Jeff;  

Now Open – AWS Africa (Cape Town) Region

Amazon Web Services Blog -

The AWS Region in Africa that Jeff promised you in 2018 is now open. The official name is Africa (Cape Town) and the API name is af-south-1. You can start using this new Region today to deploy workloads and store your data in South Africa. The addition of this new Region enables all organizations to bring lower latency services to their end-users across Africa, and allows more African organisations to benefit from the performance, security, flexibility, scalability, reliability, and ease of use of the AWS cloud. It enables organisations of all sizes to experiment and innovate faster. AWS Regions meet the highest levels of security, compliance, and data protection. With the new Region, local customers with data residency requirements, and those looking to comply with the Protection of Personal Information Act (POPIA), will be able to store their content in South Africa with the assurance that they retain complete ownership of their data and it will not move unless they choose to move it. Africa (Cape Town) is the 23rd AWS Region, and the first one in Africa. It is comprised of three Availability Zones, bringing the Global AWS Infrastructure to a total of 73 Availability Zones (AZ). Instances and Services Applications running on this 3-AZs Region can use C5d, D2, I3, M5, M5d, R5, R5d, and T3 instances, and can use a long list of AWS services including Amazon API Gateway, Amazon Aurora (both MySQL and PostgreSQL), Amazon CloudWatch, Amazon CloudWatch Logs, CloudWatch Events, Amazon DynamoDB, Amazon Elastic Block Store (EBS), Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Registry, Amazon ECS, Elastic Load Balancing (Classic, Network, and Application), Amazon EMR, Amazon ElastiCache, Amazon Elasticsearch Service, Amazon Glacier, Amazon Kinesis Data Streams, Amazon Relational Database Service (RDS), Amazon Redshift, Amazon Route 53, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Simple Storage Service (S3), Amazon Simple Workflow Service (SWF), AWS Auto Scaling, AWS Artifact, AWS Certificate Manager, AWS CloudFormation, AWS CloudTrail, AWS CodeDeploy, AWS Config, AWS Personal Health Dashboard, AWS Database Migration Service, AWS Direct Connect, AWS Elastic Beanstalk, AWS Identity and Access Management (IAM), AWS Key Management Service (KMS), AWS Marketplace, AWS Mobile SDK, AWS Shield (regional), AWS Site-to-Site VPN, AWS Step Functions, AWS Support, AWS Systems Manager, AWS Trusted Advisor, AWS X-Ray, AWS Import/Export A Growing Presence in Africa This new Region is a continuation of the AWS investment in Africa. In 2004, Amazon opened a Development Center in Cape Town that focuses on building pioneering networking technologies, next generation software for customer support, and the technology behind Amazon Elastic Compute Cloud (EC2). AWS has also added a number of teams including account managers, customer service reps, partner managers, solutions architects, developer advocates, and more, helping customers of all sizes as they move to the cloud. In 2015, we continued our expansion, opening an office in Johannesburg, and in 2017 we brought the Amazon Global Network to Africa through AWS Direct Connect. In 2018 we launched infrastructure on the African continent introducing Amazon CloudFront to South Africa, with two edge locations in Cape Town and Johannesburg, and recently in Nairobi, Kenya. We also support the growth of technology education with AWS Academy and AWS Educate, and continue to support the growth of new businesses through AWS Activate. The addition of the AWS Region in South Africa helps builders in organisations of all sizes, from startups to enterprises, as well as educational institutions, NGOs, and the public sector across Africa, to innovate and grow. The new Region is open to all: existing AWS customers, partners, and new African customers working with local partners across the region. If you are planning to deploy workloads on the Africa (Cape Town) Region, don’t hesitate to contact us. I am also taking advantage of this post to remind we have dozens of open positions in the region, for many different roles, such as Account Management, Solution Architects, Customer Support, Product Management, Software Development and more. Visit amazon.jobs and send us your resume. More to Come We are continuously expanding our global infrastructure to allow you to deploy workloads close to your end-users. We already announced two future AWS Regions in APAC: Indonesia and Japan, and two in Europe: Italy and Spain. Stay tuned for more posts like this one. -- seb (Photo via Good Free Photos)

Announcing TorchServe, An Open Source Model Server for PyTorch

Amazon Web Services Blog -

PyTorch is one of the most popular open source libraries for deep learning. Developers and researchers particularly enjoy the flexibility it gives them in building and training models. Yet, this is only half the story, and deploying and managing models in production is often the most difficult part of the machine learning process: building bespoke prediction APIs, scaling them, securing them, etc. One way to simplify the model deployment process is to use a model server, i.e. an off-the-shelf web application specially designed to serve machine learning predictions in production. Model servers make it easy to load one or several models, automatically creating a prediction API backed by a scalable web server. They’re also able to run preprocessing and postprocessing code on prediction requests. Last but not least, model servers also provide production-critical features like logging, monitoring, and security. Popular model servers include TensorFlow Serving and the Multi Model Server. Today, I’m extremely happy to announce TorchServe, a PyTorch model serving library that makes it easy to deploy trained PyTorch models at scale without having to write custom code. Introducing TorchServe TorchServe is a collaboration between AWS and Facebook, and it’s available as part of the PyTorch open source project. If you’re interested in how the project was initiated, you can read the initial RFC on Github. With TorchServe, PyTorch users can now bring their models to production quicker, without having to write custom code: on top of providing a low latency prediction API, TorchServe embeds default handlers for the most common applications such as object detection and text classification. In addition, TorchServe includes multi-model serving, model versioning for A/B testing, monitoring metrics, and RESTful endpoints for application integration. As you would expect, TorchServe supports any machine learning environment, including Amazon SageMaker, container services, and Amazon Elastic Compute Cloud (EC2). Several customers are already enjoying the benefits of TorchServe. Toyota Research Institute Advanced Development, Inc. (TRI-AD) is developing software for automated driving at Toyota Motor Corporation. Says Yusuke Yachide, Lead of ML Tools at TRI-AD: “we continuously optimize and improve our computer vision models, which are critical to TRI-AD’s mission of achieving safe mobility for all with autonomous driving. Our models are trained with PyTorch on AWS, but until now PyTorch lacked a model serving framework. As a result, we spent significant engineering effort in creating and maintaining software for deploying PyTorch models to our fleet of vehicles and cloud servers. With TorchServe, we now have a performant and lightweight model server that is officially supported and maintained by AWS and the PyTorch community”. Matroid is a maker of computer vision software that detects objects and events in video footage. Says Reza Zadeh, Founder and CEO at Matroid Inc.: “we develop a rapidly growing number of machine learning models using PyTorch on AWS and on-premise environments. The models are deployed using a custom model server that requires converting the models to a different format, which is time-consuming and burdensome. TorchServe allows us to simplify model deployment using a single servable file that also serves as the single source of truth, and is easy to share and manage”. Now, I’d like to show you how to install TorchServe, and load a pretrained model on Amazon Elastic Compute Cloud (EC2). You can try other environments by following the documentation. Installing TorchServe First, I fire up a CPU-based Amazon Elastic Compute Cloud (EC2) instance running the Deep Learning AMI (Ubuntu edition). This AMI comes preinstalled with several dependencies that I’ll need, which will speed up setup. Of course you could use any AMI instead. TorchServe is implemented in Java, and I need the latest OpenJDK to run it. sudo apt install openjdk-11-jdk Next, I create and activate a new Conda environment for TorchServe. This will keep my Python packages nice and tidy (virtualenv works too, of course). conda create -n torchserve source activate torchserve Next, I install dependencies for TorchServe. pip install sentencepiece       # not available as a Conda package conda install psutil pytorch torchvision torchtext -c pytorch If you’re using a GPU instance, you’ll need an extra package. conda install cudatoolkit=10.1 Now that dependencies are installed, I can clone the TorchServe repository, and install TorchServe. git clone https://github.com/pytorch/serve.git cd serve pip install . cd model-archiver pip install . Setup is complete, let’s deploy a model! Deploying a Model For the sake of this demo, I’ll simply download a pretrained model from the PyTorch model zoo. In real life, you would probably use your own model. wget https://download.pytorch.org/models/densenet161-8d451a50.pth Next, I need to package the model into a model archive. A model archive is a ZIP file storing all model artefacts, i.e. the model itself (densenet161-8d451a50.pth), a Python script to load the state dictionary (matching tensors to layers), and any extra file you may need. Here, I include a file named index_to_name.json, which maps class identifiers to class names. This will be used by the built-in image_classifier handler, which is in charge of the prediction logic. Other built-in handlers are available (object_detector, text_classifier, image_segmenter), and you can implement your own. torch-model-archiver --model-name densenet161 --version 1.0 \ --model-file examples/image_classifier/densenet_161/model.py \ --serialized-file densenet161-8d451a50.pth \ --extra-files examples/image_classifier/index_to_name.json \ --handler image_classifier Next, I create a directory to store model archives, and I move the one I just created there. mkdir model_store mv densenet161.mar model_store/ Now, I can start TorchServe, pointing it at the model store and at the model I want to load. Of course, I could load several models if needed. torchserve --start --model-store model_store --models densenet161=densenet161.mar Still on the same machine, I grab an image and easily send it to TorchServe for local serving using an HTTP POST request. Note the format of the URL, which includes the name of the model I want to use. curl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg curl -X POST http://127.0.0.1:8080/predictions/densenet161 -T kitten.jpg The result appears immediately. Note that class names are visible, thanks to the built-in handler. [ {"tiger_cat": 0.4693356156349182}, {"tabby": 0.46338796615600586}, {"Egyptian_cat": 0.06456131488084793}, {"lynx": 0.0012828155886381865}, {"plastic_bag": 0.00023323005007114261} ] I then stop TorchServe with the ‘stop‘ command. torchserve --stop As you can see, it’s easy to get started with TorchServe using the default configuration. Now let me show you how to set it up for remote serving. Configuring TorchServe for Remote Serving Let’s create a configuration file for TorchServe, named config.properties (the default name). This files defines which model to load, and sets up remote serving. Here, I’m binding the server to all public IP addresses, but you can restrict it to a specific address if you want to. As this is running on an EC2 instance, I need to make sure that ports 8080 and 8081 are open in the Security Group. model_store=model_store load_models=densenet161.mar inference_address=http://0.0.0.0:8080 management_address=http://0.0.0.0:8081 Now I can start TorchServe in the same directory, without having to pass any command line arguments. torchserve --start Moving back to my local machine, I can now invoke TorchServe remotely, and get the same result. curl -X POST http://ec2-54-85-61-250.compute-1.amazonaws.com:8080/predictions/densenet161 -T kitten.jpg You probably noticed that I used HTTP. I’m guessing a lot of you will require HTTPS in production, so let me show you how to set it up. Configuring TorchServe for HTTPS TorchServe can use either the Java keystore or a certificate. I’ll go with the latter. First, I create a certificate and a private key with openssl. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem Then, I update the configuration file to define the location of the certificate and key, and I bind TorchServe to its default secure ports (don’t forget to update the Security Group). model_store=model_store load_models=densenet161.mar inference_address=https://0.0.0.0:8443 management_address=https://0.0.0.0:8444 private_key_file=mykey.key certificate_file=mycert.pem I restart TorchServe, and I can now invoke it with HTTPS. As I use a self-signed certificate, I need to pass the ‘–insecure’ flag to curl. curl --insecure -X POST https://ec2-54-85-61-250.compute-1.amazonaws.com:8443/predictions/densenet161 -T kitten.jpg There’s a lot more to TorchServe configuration, and I encourage you to read its documentation! Getting Started TorchServe is available now at https://github.com/pytorch/serve. Give it a try, and please send us feedback on Github. - Julien      

Capacity-Optimized Spot Instance Allocation in Action at Mobileye and Skyscanner

Amazon Web Services Blog -

Amazon EC2 Spot Instances were launched way back in 2009. The instances are spare EC2 compute capacity that is available at savings of up to 90% when compared to the On-Demand prices. Spot Instances can be interrupted by EC2 (with a two minute heads-up), but are otherwise the same as On-Demand instances. You can use Amazon EC2 Auto Scaling to seamlessly scale Spot Instances, On-Demand instances, and instances that are part of a Savings Plan, all within a single EC2 Auto Scaling Group. Over the years we have added many powerful Spot features including a Simplified Pricing Model, the Capacity-Optimized scaling strategy (more on that in a sec), integration with the EC2 RunInstances API, and much more. EC2 Auto Scaling lets you use two different allocation strategies for Spot Instances: lowest-price – Allocates instances from the Spot Instance pools that have the lowest price at the time of fulfillment. Spot pricing changes slowly over time based on long-term trends in supply and demand, but capacity fluctuates in real time. As the lowest-price strategy does not account for pool capacity depth as it deploys Spot Instances, this allocation strategy is a good fit for fault-tolerant workloads with a low cost of interruption. capacity-optimized – Allocates instances from the Spot Instance pools with the optimal capacity for the number of instances that are launching, making use of real-time capacity data. This allocation strategy is appropriate for workloads that have a higher cost of interruption. It thrives on flexibility, empowered by the instance families, sizes, and generations that you choose. Today I want to show you how you can use the capacity-optimized allocation strategy and to share a pair of customer stories with you. Using Capacity-Optimized Allocation First, I switch to the the new Auto Scaling console by clicking Go to the new console: The new console includes a nice diagram to explain how Auto Scaling works. I click Create Auto Scaling group to proceed: I name my Auto Scaling group and choose a launch template as usual, then click Next: If you are not familiar with launch templates, read Recent EC2 Goodies – Launch Templates and Spread Placement, to learn all about them. Because my launch template does not specify an instance type, Combine purchase options and instance types is pre-selected and cannot be changed. I ensure that the Capacity-optimized allocation strategy is also selected, and set the desired balance of On-Demand and Spot Instances: Then I choose a primary instance type, and the console recommends others. I can choose Family and generation (m3, m4, m5 for example) flexibility or just size flexibility (large, xlarge, 12xlarge, and so forth) within the generation of the primary instance type. As I noted earlier, this strategy thrives on flexibility, so choosing as many relevant instances as possible is to my benefit. I can also specify a weight for each of the instance types that I decide to use (this is especially useful when I am making use of size flexibility): I also (not shown) select my VPC and the desired subnets within it, click Next, and proceed as usual. Flexibility with regard to subnets/Availability Zones is also to my benefit; for more information, read Auto Scaling Groups with Multiple Instance Types and Purchase Options. And with that, let’s take a look at how AWS customers Skyscanner and Mobileye are making use of this feature! Capacity-Optimized Allocation at Skyscanner Skyscanner is an online travel booking site. They run the front-end processing for their site on Spot Instances, making good use of up to 40,000 cores per day. Skyscanner’s online platform runs on Kubernetes clusters powered entirely by Spot Instances (watch this video to learn more). Capacity-optimized allocation has delivered many benefits including: Faster Time to Market – The ability to access more compute power at a lower cost has allowed them to reduce the time to launch a new service from 6-7 weeks using traditional infrastructure to just 50 minutes on the AWS Cloud. Cost Savings – Diversifying Spot Instances across Availability Zones and instance types has resulted in an overall savings of 70% per core. Reduced Interruptions – A test that Skyscanner ran in preparation for Black Friday showed that their old configuration (lowest-price) had between 200 and 300 Spot interruptions and the new one (capacity-optimized) had between 10 and 15. Capacity-Optimized Allocation at Mobileye Mobileye (an Intel company) develops vision-based technology for self-driving vehicles and advanced driver assistant systems. Spot Instances are used to run their analytics, machine learning, simulation, and AWS Batch workloads packaged in Docker containers. They typically use between 200K and 300K concurrent cores, with peak daily usage of around 500K, all on Spot. Here’s a instance count graph over the course of a day: After switching to capacity-optimized allocation and making some changes in accord with our Spot Instance best practices, they reduced the overall interruption rate by about 75%. These changes allowed them to save money on their compute costs while increasing application uptime and reducing their time-to-insight. To learn more about how Mobileye uses Spot Instances, watch their re:Invent presentation, Navigating the Winding Road Toward Driverless Mobility. — Jeff;  

AWS Snowball Edge Update – Faster Hardware, OpsHub GUI, IAM, and AWS Systems Manager

Amazon Web Services Blog -

Over the last couple of years I’ve told you about several members of the “Snow” family of edge computing and data transfer devices – The original Snowball, the more-powerful Snowball Edge, and the exabyte-scale Snowmobile. Today I would like to tell you about the latest updates to Snowball Edge. Here’s what I have for you today: Snowball Edge Update – New storage optimized devices that are 25% faster, with more memory, more vCPUs, and support for 100 Gigabit networking. AWS OpsHub for Snow Family – A new GUI-based tool to simplify the management of Snowball Edge devices. IAM for Snowball Edge – AWS Identity and Access Management (IAM) can now be used to manage access to services and resources on Snowball Edge devices. Snowball Edge Support for AWS Systems Manager – Support for task automation to simplify common maintenance and deployment tasks on instances and other resources on Snowball Edge devices. Let’s take a closer look at each one… Snowball Edge Storage Optimized Update We’ve refreshed the hardware, more than doubling the processing power and boosting data transfer speed by up to 25%, all at the same price as the older devices. The newest Snowballl Edge Storage Optimized devices feature 40 vCPUs and 80 GB of memory, up from 24 and 48, respectively. The processor now runs at 3.2 GHz, allowing you to launch more powerful EC2 instances that can handle your preprocessing and analytics workloads even better than before. In addition to the 80 TB of storage for data processing and data transfer workloads, there’s now 1 TB of SATA SSD storage that is accessible to the EC2 instances that you launch on the device. The improved data transfer speed that I mentioned earlier is made possible by a new 100 Gigabit QSFP+ network adapter. Here are the instances that are available on the new hardware (you will need to rebuild any existing AMIs in order to use them): Instance Name Memory vCPUs sbe-c.small 2 1 sbe-c.medium 4 1 sbe-c.large 8 2 sbe-c.xlarge 16 4 sbe-c.2xlarge 32 8 sbe-c.4xlarge 64 16 You can cluster up to twelve Storage Optimized devices together in order to create a single S3-compatible bucket that can store nearly 1 petabyte of data. You can also run Lambda functions on this and on other Snowball Edge devices. To learn more and to order a Snowball Edge (or an entire cluster), visit the AWS Snowball Console. AWS OpsHub for Snow Family This is a new graphical user interface that you can use to manage Snowball Edge devices. You can unlock devices and configure devices, use drag-and-drop operations to copy data, launch applications (EC2 AMIs), monitor device metrics, and automate routine operations. Once downloaded and installed on your Windows or Mac, you can use AWS OpsHub even if you don’t have a connection to the Internet. This makes it ideal for use in some of the mobile and disconnected modes that I mentioned earlier, and also makes it a great fit for high-security environments. AWS OpsHub is available at no charge wherever Snowball Edge is available. To learn more and to get started with AWS OpsHub, visit the Snowball Resources Page. IAM for Snowball Edge You can now use user-based IAM policies to control access to services and resources running on Snowball Edge devices. If you have multiple users with access to the same device, you can use IAM policies to ensure that each user has the appropriate permissions. If you have applications that make calls to IAM, S3, EC2, or STS (newly available on Snowball Edge) API functions on a device, you should make sure that you specify the “snow” region in your calls. This is optional now, but will become mandatory for devices ordered after November 2, 2020. IAM support is available for devices ordered on or after April 16, 2020. To learn more, read Using Local IAM. Snowball Edge Support for AWS Systems Manager AWS Systems Manager gives you the power to automate common maintenance and deployment tasks in order to make you and your teams more efficient. You can now write scripts in Python or PowerShell and execute them in AWS OpsHub. The scripts can include any of the operations supported on the device. For example, here’s a simple script that restarts an EC2 instance: To learn more, read about Automating Tasks. — Jeff;

AWS Data Transfer Out (DTO) 40% Price Reduction in South America (São Paulo) Region

Amazon Web Services Blog -

I have good news for AWS customers using our South America (São Paulo) Region. Effective April 1, 2020 we are reducing prices for Data Transfer Out to the Internet (DTO) from the South America (São Paulo) Region by 40%. Data Transfer in remains free. Here are the new prices for DTO from EC2, S3, and many other AWS services to the Internet: Monthly Usage Tier Previous AWS Rate ($/GB) Price Adjustment New AWS Rate ($/GB) Less than 10 TB 0.250 -40% 0.150 Less than 50 TB 0.230 -40% 0.138 Less than 150 TB 0.210 -40% 0.126 More than 150 TB 0.190 -40% 0.114 At AWS, we focus on driving down our costs over time. As we do this, we pass the savings along to our customers. This is our 81st price reduction since 2006. If you want to get started with AWS, the AWS Free Tier includes 15 GB/month of global data transfer out and lets you explore more than 60 AWS services. — Jeff;  

AWS Online Tech Talks for April 2020

Amazon Web Services Blog -

Join us for live, online presentations led by AWS solutions architects and engineers. AWS Online Tech Talks cover a range of topics and expertise levels, and feature technical deep dives, demonstrations, customer examples, and live Q&A with AWS experts. Note – All sessions are free and in Pacific Time. Can’t join us live? Access webinar recordings and slides on our On-Demand Portal. Tech talks this month are: April 20, 2020 | 9:00 AM – 10:00 AM PT – Save Costs Running Kubernetes Clusters with EC2 Spot Instances – ​Learn how you can lower costs and improve application resiliency by running Kubernetes workloads on Amazon EKS with Spot Instances.​ April 20, 2020 | 11:00 AM – 12:00 PM PT – Hadoop 3.0 and Docker on Amazon EMR 6.0 – A deep dive into what’s new in EMR 6.0 including Apache Hadoop 3.0, Docker containers & Apache Hive performance improvements​. ​April 20, 2020 | 1:00 PM – 2:00 PM PT – Infrastructure as Code on AWS – ​Join this tech talk to learn how to use AWS CloudFormation and AWS CDK to provision and manage infrastructure, deploy code, and automate your software-release processes. April 21, 2020 | 9:00 AM – 10:00 AM PT – How to Maximize Results with a Cloud Contact Center, Featuring Aberdeen Research – ​Learn how to maximize results with a cloud contact center, featuring Aberdeen Research and Amazon Connect​. April 21, 2020 | 11:00 AM – 12:00 PM PT – Connecting Microcontrollers to the Cloud for IoT Applications – ​Learn how you can connect microcontrollers to the cloud for IoT applications​. April 21, 2020 | 1:00 PM – 2:00 PM PT – Reducing Machine Learning Inference Cost for PyTorch Models – ​Join us for a tech talk to learn about deploying your PyTorch models for low latency at low cost.​ April 22, 2020 | 11:00 AM – 12:00 PM PT – Top 10 Security Items to Improve in Your AWS Account – Learn about the top 10 security items to improve in your AWS environment and how you can automate them.​ April 22, 2020 | 1:00 PM – 2:00 PM PT – Building Your First Application with AWS Lambda – ​Learn how to build your first serverless application with AWS Lambda, including basic design patterns and best practices.​ April 23, 2020 | 9:00 AM – 10:00 AM PT – Persistent Storage for Containers with Amazon EFS – ​Learn how to securely store your containers in the cloud with Amazon EFS​. April 23, 2020 | 11:00 AM – 12:00 PM PT – Build Event Driven Graph Applications with AWS Purpose-Built Databases – ​Learn how to build event driven graph applications using AWS purpose-built database services including Amazon Neptune, Amazon DynamoDB, and Amazon ElastiCache.​ April 23, 2020 | 1:00 PM – 2:00 PM PT – Migrate with AWS – ​Introduction to best practice driven process for migrations to AWS, developed by the experience in helping thousands of enterprises migrate. April 27, 2020 | 9:00 AM – 10:00 AM PT – Best Practices for Modernizing On-Premise Big Data Workloads Using Amazon EMR – ​Learn about best practices to migrate from on-premises big data (Apache Spark and Hadoop) to Amazon EMR.​ April 27, 2020 | 11:00 AM – 12:00 PM PT – Understanding Game Changes and Player Behavior with Graph Databases – ​Learn how to solve problems with highly connected data in game datasets with Amazon Neptune. ​​April 27, 2020 | 1:00 PM – 2:00 PM PT – Assess, Migrate, and Modernize from Legacy Databases to AWS: Oracle to Amazon Aurora PostgreSQL Migration – ​Punitive licensing and high cost of on-premises legacy databases could hold you back. Join this tech talk to learn how to assess, migrate, and modernize your Oracle workloads over to Amazon Aurora PostgreSQL, using Amazon Database Migration Service (DMS).​ April 28, 2020 | 9:00 AM – 10:00 AM PT – Implementing SAP in the Cloud with AWS Tools and Services – ​This tech talk will help architects and administrators to understand the automation capabilities available that can assist your SAP migration.​ April 28, 2020 | 11:00 AM – 12:00 PM PT – Choosing Events, Queues, Topics, and Streams in Your Serverless Application – ​Learn how to choose between common Lambda event sources like EventBridge, SNS, SQS, and Kinesis Data Streams.​ April 30, 2020 | 9:00 AM – 10:00 AM PT – Inside Amazon DocumentDB: The Makings of a Managed Non-relational Database – Join Rahul Pathak, GM of Emerging Databases and Blockchain at AWS, to learn about the inner workings of Amazon DocumentDB and how it provides better performance, scalability, and availability while reducing operational overhead for managing your own non-relational databases.

New – Accelerate SAP Deployments with AWS Launch Wizard

Amazon Web Services Blog -

Last year, we announced AWS Launch Wizard for SQL Server, which enables quick and easy deployment of high availability SQL solutions on AWS for enterprise workloads. Today, I am excited to announce AWS Launch Wizard for SAP, a new service that is speedy, easy, flexible, secure, and cost effective for customers. This new service helps Customers to deploy SAP applications on AWS by orchestrating provision of underlying AWS resources using AWS CloudFormation and AWS Systems Manager. Thousands of AWS customers have built and migrated their SAP workloads using AWS Quick Start and Amazon Elastic Compute Cloud (EC2), including x1, x1e, and high memory instances. In addition, they also take advantage of the AWS Partner Network (APN) for SAP to find solutions that work for them. SAP Customers want well architected intuitive wizard to deploy SAP system with best use of AWS resources. AWS Launch Wizard for SAP is designed for customers who want to deploy new SAP workloads on AWS or migrate existing on-premises SAP workloads to AWS with the following benefits: Deployment efficiency: AWS Launch Wizard for SAP recommends the Amazon EC2 instances that fit your SAP workload requirements and automates the launch of AWS services for SAP systems with recommended configurations and minimum manual input. Launch Wizard helps customers achieve faster time to value for provisioning and accelerates deployment of SAP applications by 2X. Being able to quickly launch SAP environments improves the customer’s agility to support new business initiatives. Prescriptive guidance: AWS Launch Wizard for SAP guides customers with correct sizing and provisioning of AWS services for SAP systems based on best practices from thousands of SAP on AWS deployments. Faster learning curve: AWS Launch Wizard for SAP offers an SAP-oriented user experience for customers. It provides guided deployment steps aligned with SAP deployment patterns and uses SAP terminology which creates a familiar experience for SAP users. AWS Launch Wizard for SAP – Getting Started To get started with an SAP deployment, in the Launch Wizard console, you can click the Create deployment button and select the SAP application. When you click on the Next button, you can provide Deployment Name and Infrastructure settings. Infrastructure settings can be saved based on how you want to classify your deployment using the infrastructure. They can be reused for SAP systems deployments that share the same infra configuration. Assign a key pair and select the VPC which to deploy the SAP instances. After you select the Availability Zones and private subnets, you can assign security groups to the EC2 instances that will run the SAP applications. After setting SAP System Admin IDs, you can set the topic of Amazon Simple Notification Service (SNS) to get alerts about SAP deployments. By clicking the Next button, we can go to the application settings. If you save infrastructure configurations, you can reuse to apply to future deployments. Configuring application settings, AWS Launch Wizard for SAP supports two types of SAP applications: NetWeaver stack on SAP HANA database deployments and HANA database deployments. You can provide SAPSID, HANASID, and the instance number used in the SAP HANA installation and then configure the AWS resources based on these inputs. It supports two EBS volume types for SAP HANA Data and Log: HANA data and log – General Purpose SSD (gp2) and Provisioned IOPS SSD (io1). Optionally, a customer can choose to provide HANA software hosted on an S3 bucket to deploy HANA configured for high availability using SLES/RHE. Next, you can configure the deployment model with SAP supported Operating Systems such as SUSE Linux and RedHat Enterprise within a single instance, distributed instances , and high availability patterns in multi AZs. When you define the infrastructure requirement, you can use the recommended guide to provide vCPU/Memory or manually choose the instances from the list of SAP supported EC2 instances for the SAP component (ASCS or ERS or APP or HANA DB) and then deploy the SAP components on it. You will be able to see the cost estimates for the AWS resources – Amazon EC2, EBS, and EFS volumes – that will be provisioned for a particular deployment. After reviewing all your configurations, you can simply deploy by clicking Deploy button. Depending on the chosen deployment it takes 1 to 3 hours. You will be able to see which SAP systems have been deployed, what infrastructure configuration was used for the deployment, what components of SAP were deployed, and a mapping of SAP components to EC2 instances. Now Available! AWS Launch Wizard for SAP is generally available and you can use it in US East (N. Virginia), US West (Oregon), Europe (Ireland), US West (N. California), US East (Ohio), Europe (Ireland), Europe (Paris), Europe (Frankfurt), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Mumbai), South America (São Paulo), Europe (London), Canada (Central), Europe (Stockholm). There is no additional charge for using AWS Launch Wizard, only for the resources it creates. Take a look at the product page and the documentation to learn more. Please send feedback to SAP Contact Us, AWS SAP partners or through your usual AWS support contacts. — Channy;

Amazon Elastic Container Service now supports Amazon EFS file systems

Amazon Web Services Blog -

It has only been five years since Jeff wrote on this blog about the launch of the Amazon Elastic Container Service. I remember reading that post and thinking how exotic and unusual containers sounded. Fast forward just five years, and containers are an everyday part of most developers lives, but whilst customers are increasingly adopting container orchestrators such as ECS, there are still some types of applications that have been hard to move into this containerized world. Customers building applications that require data persistence or shared storage have faced a challenge since containers are temporary in nature. As containers are scaled in and out dynamically, any local data is lost as containers are terminated. Today we are changing that for ECS by launching support for Amazon Elastic File System (EFS) file systems. Both containers running on ECS and AWS Fargate will be able to use Amazon Elastic File System (EFS). This new capability will help customers containerize applications that require shared storage such as content management systems, internal DevOps tools, and machine learning frameworks. A whole new set of workloads will now enjoy the benefits containers bring, enabling customers to speed up their deployment process, optimize their infrastructure utilization, and build more resilient systems. Amazon Elastic File System (EFS) provides a fully managed, highly available, scalable shared file system; it means that you can store your data separately from your compute. It is also a regional service, meaning that the service is storing data within and across 3 Availability Zones for high availability and durability. Until now, it was possible to get EFS working with ECS if you were running your containers on a cluster of EC2 instances. However, if you wanted to use AWS Fargate as your container data plane then, prior to this announcement, you couldn’t mount an EFS file system. Fargate does not allow you as a customer to gain access to the managed instances inside the Fargate fleet and so you are unable to make the modifications required to the instances to setup EFS. I’m sure many of our customers will be delighted that they now have a way of connecting EFS file systems easily to ECS, personally I’m ecstatic that we can use this new feature in combination with Fargate, it will be perfect for a little side project that I am currently building and finally give us a way of having persistent storage work in combination with serverless containers. There is a good reason both ECS and EFS include the word Elastic in their name as both of these services can scale up and down as your application requires. EFS scales on demand from zero to petabytes with no disruptions, growing and shrinking automatically as you add and remove files. With ECS there are options to use either Cluster Auto Scaling or Fargate to ensure that your capacity grows and shrinks to meet demand. For you, our customer, this means that you are only ever paying for the storage and compute that you are actually going to use. So, enough talking, let’s get to the fun bit and see how we can get a containerized application working with Amazon Elastic File System (EFS). A Simple Shared File System Example For this example, I have built some basic infrastructure, so I can show you the before and after effect of adding EFS to a Fargate cluster. Firstly I have created a VPC that spans two Availability Zones. Secondly, I have created an ECS cluster. On the ECS Cluster, I plan to run two containers using Fargate, and this means that I don’t have to set up any EC2 instances as my containers will run on the Fargate fleet that is managed by AWS. To deploy my application, I create a Task Definition that uses a Docker Image of an Open Source application called Cloud Commander, which is a simple drag and drop, file manager. In the ECS console, I create a service and use the Task Definition that I created to deploy my application. Once the service is deployed, and the containers are provisioned, I head over to the Application Load Balancer URL which was created as part of my service, and I can see that my application appears to be working. I can drag a file to upload it to the application. However there is a problem. If I refresh the page, occasionally, the file that I dragged to upload disappears. This happens because I have two containers running my application, and they are both using their local file systems. As I refresh my browser, the load balancer sends me to one of the two containers, and only one of the containers is storing the image on its local volume. What I need is a shared file system that both the containers can mount and to which they can both write files. Next, I create a new file system inside the EFS console. In the wizard, I choose the same VPC that I used when I created my ECS cluster and select all of the availability zones that the VPC spans and ask the service to create mount targets in each one. These mount targets will mean that containers that are in different Availability Zones will still be able to connect to the file system. I select the defaults for all the other options in the wizard. In Step 3, I click the button to Add an access Point. An access point is a way of me giving a particular application access to the file system and gives me incredibly granular control over what data my application is allowed to access. You can add multiple Access points to your EFS file system and provide different applications, different levels of access to the same file system. The application I am deploying will handle user uploads for my web site, so I will create an EFS Access Point that gives this application full access to the /uploads directory, but nothing else. To do this, I will create an access point with a new User ID (1000) and Group ID (1000), and a home directory of /uploads. The directory will be created with this user and group as the owner with full permissions, giving read permissions to all other users. Security is the number one priority at AWS and the team have worked hard to ensure ECS integrates with EFS to provide multiple layers of security for protecting EFS filesystems from unauthorized access, including IAM role-based access control, VPC security groups, and encryption of data in transit. After working through the wizard, my file system is created, and I’m given a File system ID and an Access point ID. I will need these IDs to configure the task definitions in Fargate. I go back to the Task Definition inside my ECS Cluster and create a new revision of the Task Definition. I scroll down to the Volumes section of the definition and click Add volume. I can then add my EFS File System details, I select the correct File system ID and also the correct Access point ID that I created earlier. I have opted to Enable Encryption in transit, but for this example I have not enabled EFS IAM authorization which would be helpful in a larger application with many clients requiring different levels of access for different portions of a filesystem. This feature can simplify management by using IAM authorization, if you want more details on this check out the blog we wrote when the feature launched earlier in the year. Now that I have updated my task definition, I can update my ECS service to use this new definition. It’s also essential here to make sure that I set the platform version to 1.4.0. The service deploys my two new containers and decommissions the old two. The new containers will now be using the shared EFS file system, and so my application now works as expected. If I upload files and then revisit the application, my files will still be there. If my containers are replaced or are scaled up or down, the file system will persist. Looking to the Future I am loving the innovation that has been coming out of the containers teams recently, and looking at their public roadmap; they have some really exciting plans for the future. If you have ideas or feature requests, make sure you add your voice to the many customers that are already guiding their roadmap. The new feature is available in all regions where ECS and EFS are available and comes at no additional cost. So, go check it out in the AWS console and let us know what you think. Happy Containerizing — Martin  

Simplified Time-Series Analysis with Amazon CloudWatch Contributor Insights

Amazon Web Services Blog -

Inspecting multiple log groups and log streams can make it more difficult and time consuming to analyze and diagnose the impact of an issue in real time. What customers are affected? How badly? Are some affected more than others, or are outliers? Perhaps you performed deployment of an update using a staged rollout strategy and now want to know if any customers have hit issues or if everything is behaving as expected for the target customers before continuing further. All of the data points to help answer these questions is potentially buried in a mass of logs which engineers query to get ad-hoc measurements, or build and maintain custom dashboards to help track. Amazon CloudWatch Contributor Insights, generally available today, is a new feature to help simplify analysis of Top-N contributors to time-series data in CloudWatch Logs that can help you more quickly understand who or what is impacting system and application performance, in real-time, at scale. This saves you time during an operational problem by helping you understand what is contributing to the operational issue and who or what is most affected. Amazon CloudWatch Contributor Insights can also help with ongoing analysis for system and business optimization by easily surfacing outliers, performance bottlenecks, top customers, or top utilized resources, all at a glance. In addition to logs, Amazon CloudWatch Contributor Insights can also be used with other products in the CloudWatch portfolio, including Metrics and Alarms. Amazon CloudWatch Contributor Insights can analyze structured logs in either JSON or Common Log Format (CLF). Log data can be sourced from Amazon Elastic Compute Cloud (EC2) instances, AWS CloudTrail, Amazon Route 53, Apache Access and Error Logs, Amazon Virtual Private Cloud (VPC) Flow Logs, AWS Lambda Logs, and Amazon API Gateway Logs. You also have the choice of using structured logs published directly to CloudWatch, or using the CloudWatch Agent. Amazon CloudWatch Contributor Insights will evaluate these log events in real-time and display reports that show the top contributors and number of unique contributors in a dataset. A contributor is an aggregate metric based on dimensions contained as log fields in CloudWatch Logs, for example account-id, or interface-id in Amazon Virtual Private Cloud Flow Logs, or any other custom set of dimensions. You can sort and filter contributor data based on your own custom criteria. Report data from Amazon CloudWatch Contributor Insights can be displayed on CloudWatch dashboards, graphed alongside CloudWatch metrics, and added to CloudWatch alarms. For example customers can graph values from two Amazon CloudWatch Contributor Insights reports into a single metric describing the percentage of customers impacted by faults, and configure alarms to alert when this percentage breaches pre-defined thresholds. Getting Started with Amazon CloudWatch Contributor Insights To use Amazon CloudWatch Contributor Insights I simply need to define one or more rules. A rule is simply a snippet of data that defines what contextual data to extract for metrics reported from CloudWatch Logs. To configure a rule to identify the top contributors for a specific metric I supply three items of data – the log group (or groups), the dimensions for which the top contributors are evaluated, and filters to narrow down those top contributors. To do this, I head to the Amazon CloudWatch console dashboard and select Contributor Insights from the left-hand navigation links. This takes me to the Amazon CloudWatch Contributor Insights home where I can click Create a rule to get started. To get started quickly, I can select from a library of sample rules for various services that send logs to CloudWatch Logs. You can see above that there are currently a variety of sample rules for Amazon API Gateway, Amazon Route 53 Query Logs, Amazon Virtual Private Cloud Flow Logs, and logs for container services. Alternatively, I can define my own rules, as I’ll do in the rest of this post. Let’s say I have a deployed application that is publishing structured log data in JSON format directly to CloudWatch Logs. This application has two API versions, one that has been used for some time and is considered stable, and a second that I have just started to roll out to my customers. I want to know as early as possible if anyone who has received the new version, targeting the new api, is receiving any faults and how many faults are being triggered. My stable api version is sending log data to one log group and my new version is using a different group, so I need to monitor multiple log groups (since I also want to know if anyone is experiencing any error, regardless of version). The JSON to define my rule, to report on 500 errors coming from any of my APIs, and to use account ID, HTTP method, and resource path as dimensions, is shown below. { "Schema": { "Name": "CloudWatchLogRule", "Version": 1 }, "AggregateOn": "Count", "Contribution": { "Filters": [ { "Match": "$.status", "EqualTo": 500 } ], "Keys": [ "$.accountId", "$.httpMethod", "$.resourcePath" ] }, "LogFormat": "JSON", "LogGroupNames": [ "MyApplicationLogsV*" ] } I can set up my rule using either the Wizard tab, or I can paste the JSON above into the Rule body field on the Syntax tab. Even though I have the JSON above, I’ll show using the Wizard tab in this post and you can see the completed fields below. When selecting log groups I can either select them from the drop down, if they already exist, or I can use wildcard syntax in the Select by prefix match option (MyApplicationLogsV* for example). Clicking Create saves the new rule and makes it immediately start processing and analyzing data (unless I elect to create it in disabled state of course). Note that Amazon CloudWatch Contributor Insights processes new log data created once the rule is active, it does not perform historical inspection, so I need to build rules for operational scenarios that I anticipate happening in future. With the rule in place I need to start generating some log data! To do that I’m going to use a script, written using the AWS Tools for PowerShell, to simulate my deployed application being invoked by a set of customers. Of those customers, a select few (let’s call them the unfortunate ones) will be directed to the new API version which will randomly fail on HTTP POST requests. Customers using the old API version will always succeed. The script, which runs for 5000 iterations, is shown below. The cmdlets being used to work with CloudWatch Logs are the ones with CWL in the name, for example Write-CWLLogEvent. # Set up some random customer ids, and select a third of them to be our unfortunates # who will experience random errors due to a bad api update being shipped! $allCustomerIds = @( 1..15 | % { Get-Random }) $faultingIds = $allCustomerIds | Get-Random -Count 5 # Setup some log groups $group1 = 'MyApplicationLogsV1' $group2 = 'MyApplicationLogsV2' $stream = "MyApplicationLogStream" # When writing to a log stream we need to specify a sequencing token $group1Sequence = $null $group2Sequence = $null $group1, $group2 | % { if (!(Get-CWLLogGroup -LogGroupName $_)) { New-CWLLogGroup -LogGroupName $_ New-CWLLogStream -LogGroupName $_ -LogStreamName $stream } else { # When the log group and stream exist, we need to seed the sequence token to # the next expected value $logstream = Get-CWLLogStream -LogGroupName $_ -LogStreamName $stream $token = $logstream.UploadSequenceToken if ($_ -eq $group1) { $group1Sequence = $token } else { $group2Sequence = $token } } } # generate some log data with random failures for the subset of customers 1..5000 | % { Write-Host "Log event iteration $_" # just so we know where we are progress-wise $customerId = Get-Random $allCustomerIds # first select whether the user called the v1 or the v2 api $useV2Api = ((Get-Random) % 2 -eq 1) if ($useV2Api) { $resourcePath = '/api/v2/some/resource/path/' $targetLogGroup = $group2 $nextToken = $group2Sequence } else { $resourcePath = '/api/v1/some/resource/path/' $targetLogGroup = $group1 $nextToken = $group1Sequence } # now select whether they failed or not. GET requests for all customers on # all api paths succeed. POST requests to the v2 api fail for a subset of # customers. $status = 200 $errorMessage = '' if ((Get-Random) % 2 -eq 0) { $httpMethod = "GET" } else { $httpMethod = "POST" if ($useV2Api -And $faultingIds.Contains($customerId)) { $status = 500 $errorMessage = 'Uh-oh, something went wrong...' } } # Write an event and gather the sequence token for the next event $nextToken = Write-CWLLogEvent -LogGroupName $targetLogGroup -LogStreamName $stream -SequenceToken $nextToken -LogEvent @{ TimeStamp = [DateTime]::UtcNow Message = (ConvertTo-Json -Compress -InputObject @{ requestId = [Guid]::NewGuid().ToString("D") httpMethod = $httpMethod resourcePath = $resourcePath status = $status protocol = "HTTP/1.1" accountId = $customerId errorMessage = $errorMessage }) } if ($targetLogGroup -eq $group1) { $group1Sequence = $nextToken } else { $group2Sequence = $nextToken } Start-Sleep -Seconds 0.25 } I start the script running, and with my rule enabled, I start to see failures show up in my graph. Below is a snapshot after several minutes of running the script. I can clearly see a subset of my simulated customers are having issues with HTTP POST requests to the new v2 API. From the Actions pull down in the Rules panel, I could now go on to create a single metric from this report, describing the percentage of customers impacted by faults, and then configure an alarm on this metric to alert when this percentage breaches pre-defined thresholds. For the sample scenario outlined here I would use the alarm to halt the rollout of the new API if it fired, preventing the impact spreading to additional customers, while investigation of what is behind the increased faults is performed. Details on how to set up metrics and alarms can be found in the user guide. Amazon CloudWatch Contributor Insights is available now to users in all commercial AWS Regions, including China and GovCloud. — Steve

Pages

Recommended Content

Subscribe to Complete Hosting Guide aggregator - Cloud Hosting Official Blogs