How to use aws Rust SDK ,their design choices and useability.

How to use aws Rust SDK ,their design choices and useability.

I’m sure once you know this , you won't find any difficulties using other AWS services.

Sanjeevi's photo
Sanjeevi
·

14 min read

In the previous post, we saw why Rust was the best choice for developing a back-end application. In this post, we are about to learn about the design of the AWS Rust SDK and how to use it.

There is not much value to backend development if we are not utilizing the cloud behind the scenes. Our chosen cloud platform is AWS. Servers are independent of cloud providers, but AWS supports more services, and the best part is it has high-quality documentation for every service they offer. Whenever I encounter an error, my first preference is to search for specific service documentation.

I chose AWS not just because of its popularity, although that's an added advantage. Compared to Google Cloud and Azure, AWS has the best-in-class support for client libraries in Rust APIs. One of the foremost reasons I chose AWS cloud is because of the design of its Rust SDKs. It turns out that AWS is the number one cloud provider in the world (I only found out nine months ago when Rust SDKs were not stable), and it has the best-in-class documentation I have ever seen.

Anyone who wants to use AWS services with the Rust programming language can benefit from this blog post, as it covers all the necessary information to use any service on AWS.

AWS has announced the general availability of the AWS SDK for Rust, enabling interaction with AWS services through the software development kit. This SDK is similar to Python's boto3 and other client libraries in various programming languages.

How are AWS services structured?

Each AWS service or resource is a Rust crate. The significant advantage of this approach is that we don't need to pay for compile-time for services we don't use, as Rust code typically takes longer to compile. You don't have to include the entirety of large AWS services, which could otherwise take hours of your time to compile. Below is an example of how to import AWS Relational Database Service and Elastic Compute Cloud into your Rust project.

aws-sdk-rds = "1.10.0"
aws-sdk-ec2 = "1.12.0"

Regardless of the AWS services you choose to employ, it is essential to specify these crates in your cargo.toml file. These crates manage credentials, as all requests to AWS must be authenticated before performing any actions on AWS resources. Importantly, we need the tokio crate to run the code, as all AWS services are asynchronous.

aws-config = "1.1.1"
aws-credential-types = "1.1.1"
tokio = {version = "*",features = ["full"]}

The above crates provide types such as SdkConfig and a load_from_env function to build a SdkConfig type for use with the Client for all AWS services.

First Step

The entry point for any service is the Client type struct, which accepts credentials in the form of the SdkConfig type constructed using the aws_config crate. It then returns a builder to perform an action on AWS. Before utilizing any of the client methods, we must build the SdkConfig type.

There are multiple ways to construct an SdkConfig type.

Method 1:

This method has been deprecated and should not be used for new projects. For existing projects, use it behind the feature flag to avoid breaking changes to your code.

let sdk_config: SdkConfig = aws_config::load_from_env().await;

Method 2:

This method is preferable and has been newly added in the stable Rust SDK release. Beta SDKs used the above method

let sdk_config = load_defaults(BehaviorVersion::latest()).await;

let credentials = sdk_config
    .credentials_provider()
    .unwrap()
    .provide_credentials()
    .await;

match credentials {
    Ok(credentials) => {
        println!("Access Key ID: {}", credentials.access_key_id());
        println!("Secret Access Key ID: {}", credentials.secret_access_key());
        println!("Session token if any: {}", credentials.session_token().unwrap_or_default());
    }
    Err(error) => println!("{}", error.to_string()),
}

println!("Region: {:?}", sdk_config.region());

You can observe that the SdkConfig type contains all the information needed to make a request, such as access key, secret access key, region, and session token if temporary credentials are used. The load_defaults function loads the credential information from your system, and the path is different for Windows and Unix-based systems. It will be managed by the AWS CLI when you first log in. The two functions above are asynchronous, which means you must use await on the future to execute the code or to return a Sdkconfig type. This is because the future returns an opaque type, not a concrete type.

There is no need to retrieve the credential data from sdkconfig, as shown above. It's only for demonstration purposes.

Method 3

The third way to configure an sdkconfig type is to use a builder type on it. We can use this configuration to provide the credentials explicitly in the code instead of fetching them from disk, as the above two methods do. It's not the recommended way of providing credentials according to AWS best practices

    let build_credentials_provider = Credentials::new(
        "your_access_key_id",
        "your_secret_key_id",
        Some("your_session_token".into()),
        Some(SystemTime::UNIX_EPOCH), //expiration time
        "admin", //provider name
    );

    let sdk_config_builder = SdkConfig::builder()
        .region(Some(Region::new("ap-south-1")))
        .credentials_provider(SharedCredentialsProvider::new(build_credentials_provider))
        .build();

Once we have the skdconfig type, we are ready to build a client for a specific service. All services on AWS return the same name Client, so be sure to alias the client type appropriately for disambiguate the service you are using. For example,

use aws_sdk_rds::Client as RdsClient; // For relational database client
use aws_sdk_ses::Client as SesClient; // For Simple Email Service client
use aws_sdk_s3::Client as S3Client;// For Simple Storage Service client

Let's use Amazon Simple Email Service (SES) for sending templated emails. 

    let destination_emails= Destination::builder()
           .to_addresses(recipient_email).build();

    let template = Template::builder()
        .template_name("your_template_name")
        .template_data("A JSON containing key-value pairs for the chosen template above")
        .build();

    let email_content = EmailContent::builder().template(template).build();
    ses_client
        .send_email()
        .from_email_address("Your_verified_from_email_address")
        .destination(destination_emails)
        .content(email_content)
        .send()
        .await
        .expect("Error while sending emails");

I have created a cli application on top of SES service to easily send email and other options. Use this link if you are interested.

The patterns of performing action on AWS services.

1) Client - In this case, the Simple Email Service (SES) client is returned after successful verification of credentials. In older versions, if no credentials are found, this will result in a panic.

2) After obtaining the SES client, various SES actions can be called on the SesV2Client, such as sending emails, creating contacts, and more. Note that the credentials must have the SES service permission to perform these actions.

3) The builder pattern is heavily used throughout the AWS crates and also in the Rust ecosystem. It's common to have a builder static method on a type, and finally calling the build method on the builder will return the actual type we care about. The builder type has a name like TemplateBuilder, and the type we want would be Template. This is the best part of the API design.

4) Once the necessary request parameters are specified using the setter methods, calling the send method will dispatch the request to AWS and return whether the response is successful or not.

5) However, calling send won't execute anything unless we await on them. The send method returns a Future. In Rust, Futures are lazy unless consumed, so make sure to await on them.

6) After calling await on send will return the Result type with either Ok(OutputType) or Err(SDK Error). Without pattern matching or handling, we can't know whether the operation was successful or failed. The operation is executed even if we are not handling the error but the compiler issues a warning if you didn't handle the error. This is different from getting warning about not using await in that it's means no execution was happened.

The send method returns an error as a Result enum instead of an Option type because numerous issues may arise, and as users, we need to know the reason behind the failure. Some possible errors are:

  • Invalid credentials
  • Even with valid credentials, there may be a lack of permission
  • Cross-region access to resources, for example, attempting to access an S3 bucket in us-west-1 (North Virginia) from ap-south-1 (Mumbai).

These steps involved in above action not specific to SES. The procedure is the same for all services: build a specific service client, call an action on the client, send, await, and handle the error.

Fallible Operations

Following Rust best practices, all fallible operations in AWS services either return Result or Option, this way developers don't have to deal with new types and methods. Additionally, errors implement the Error trait, allowing us to call to_string on errors returned by the specific service which is useful for printing out the errors.

Modules

Each AWS service at least has these modules

  • Types: This module contains all the data structures for request inputs and outputs. For example, in the above send email code, we have used the EmailContent and Template from the types module. If you already have experience reading the API documentation of a particular service, these types are exactly equivalent to data types of that service.

  • Primitives: The most commonly used primitives are Blob, DateTime, and DateTimeFormat. These are also types but are used with other types to simplify. For example, to format the time from seconds to a date-like format, you can do the following:

let seconds = DateTime::from_secs(1209034013);
let format_seconds_to_http_date = seconds.fmt(DateTimeFormat::HttpDate).unwrap();
let format_seconds_to_date_time = seconds.fmt(DateTimeFormat::DateTime).unwrap();
println!("{}", format_seconds_to_http_date);
println!("{}", format_seconds_to_date_time);

If you need to know the crate version for some reason, you can obtain it using the PKG_VERSION constant in the meta module, as shown below:

println!("{}", aws_sdk_sesv2::meta::PKG_VERSION);

The other common modules are errors and configuration.

Method Names and Their Return Types

Naming convention: If you've read the Code Complete book, you'll see this in action in every corner of AWS doc.rs. Every function and struct has a meaningful name and purpose.

  • Methods starting with get_ return a reference to the data, such as emails or objects in buckets. These methods are used to retrieve information from the response.

  • Methods starting with set_ return &mut Self, allowing the setting of parameters for requests without consuming unless send is called finally. These set_ methods are used to provide multiple values to the request.

  • Plain verb function names serve the same purpose as set_ but for single inputs.

Example of the above method names in action in the AWS Rekognition service:

let collection_of_attributes = vec![
    Attribute::Gender,
    Attribute::EyesOpen,
    Attribute::Smile,
    Attribute::Sunglasses,
];
rekognition_client
    .detect_faces()
    .attributes(Attribute::Gender)
    .set_attributes(Some(collection_of_attributes))
    .send()
    .await
    .unwrap();

Both attributes and set_attributes are used to set values. The difference is that when using set_attributes, you can provide more values. You can't use the attributes method to set multiple values by calling it multiple times because only the last value is considered. Note that the above code is not able to detect faces without other requirements, and it's just demonstrating how the methods are named and used.

Enums are used extensively instead of error-prone string types for representing various data types. This approach:

  1. Is more type-safe and doesn't require runtime checking.
  2. Provides type suggestions in the IDE.
  3. Consumes less memory than a string type.

Most enums provides a way to build the variants from string slices, look for the from_str method in the appropriate crate documentation.

In AWS crates , Enums have an UnknownVariant for forward compatibility. If something is added in the future, existing code won't break because it defaults to an invariant value to use as new feature.

Methods can accept more types because of trait requirement specified in the function signature. For example, you can pass String or &str or any type that implements Display if the function parameter is impl Into<String>.

Field Access

Public field access is essential to pay attention to because most methods take ownership of data. This means we can't do much more with borrowed data. It's relieving that the fields of types are public otherwise we would have been fight with borrow checker. Why is this useful? Let's take an example:

let get_object_output = s3_client
    .get_object()
    .bucket("bucket_name")
    .key("Key object name")
    .send()
    .await
    .expect("Error while getting object from s3");
let get_data = get_object_output.body().collect();
// let bytes = get_object_output.body.collect();

If you compile the above code, it will result in an error similar to the one below. The solution is to access the body field directly instead of calling the method body, which returns a reference to the body (Don't be confused by the name 'body'). Then, we can call the collect method on it without the borrow checker complaining. The collect method is used to write the bytes to disk.

Asynchronous Operations

All service actions, including the credential loading function, are asynchronous. In Rust, running asynchronous code requires a third party like Tokio or Rust's official async-std to execute. This distinguishes Rust from most programming languages where runtimes are built into the language.

While various asynchronous runtimes are available, only Tokio is compatible with AWS services. Attempting to use the async-std runtime will result in an error, and some services may depend on a specific version of the Tokio crate.

The Price We Pay for the Builder Pattern

Builder patterns, known for their fluency and readability, come with a trade-off in efficiency. It is easy to miss the parameters of an action , leading to errors at runtime. On the other hand, state machines effectively encode the domain but are less user-friendly. There is always a trade-off in any choice.

To avoid runtime errors, read the appropriate API documentation for a specific service on the AWS documentation page. The AWS documentation specifies all the parameters a service accepts, highlighting which ones are required and which are not. You should not forget the required parameters, but you can leave others where they have default values that can be overridden if desired.

The Blob and Bytestream Types

In any AWS services within the Rust ecosystem, we primarily deal with String, Bytestream, and Blob data types for uploading data from our local system using HTTPS.

When uploading policy documents in IAM services or JSON key-value pairs in SES template operations, they are represented as the String type. Since String is a such a common type, we will focus on the other two types.

The following code uploads an image to an S3 bucket using the Bytestream type:

let byte_stream = ByteStream::from_path(image_path).await.unwrap();
s3_client
    .put_object()
    .bucket("Your_bucket_name")
    .key("use your image name as your key name")
    .body(byte_stream)
    .send()
    .await
    .expect("Error while uploading object to S3 bucket");

This code uploads an image to the detect_faces action on the Rekognition service using the Blob type to encode the image data:

let mut read_content = std::fs::File::open(image_path).expect("Error while opening the image path");
let mut read_into_vector = Vec::new();
read_content.read_to_end(&mut read_into_vector).unwrap();

let blob_type = Blob::new(read_into_vector);

let build_image = Image::builder().bytes(blob_type).build();

rekognition_client
    .detect_faces()
    .image(build_image)
    .send()
    .await
    .expect("Error while detecting face");

In the above code, we can also use an S3 URL, a go-to option for many use cases, because there are two ways to build an Image type, either from blob or specifying S3 image url.

Interesting Aspects of the Rust Type System

In this final topic, we will explore intriguing properties of the sharde_provide_fn in the credentials type crate. This is not just about AWS services but also how Rust can be employed to enhance the security of our APIs. Essentially, by encoding essential principles using the Rust type system, we can ensure that programmers never misuse the API.

Here is the type signature of provide_credentials_fn from here:

pub fn provide_credentials_fn<'c, T, F>(f: T) -> ProvideCredentialsFn<'c, T>
where
    T: Fn() -> F + Send + Sync + 'c,
    F: Future<Output = Result<Credentials, CredentialsError>> + Send + 'static,

Don't be intimidated by the symbols. Let's break it down to understand it more clearly.

  1. Because of the static lifetime, we can't pass the reference.
  2. Due to the Fn closure, we can't move data inside the closure and can't mutate the data. (Notes: In Rust, a closure is equivalent to a C++ or Python lambda. A closure is the only type that can capture variables from the outer scope. Functions, on the other hand, cannot capture anything other than the things passed as parameters.)

Now, let's decompose these points to gain a clearer understanding:

  1. The return type of the closure should be a Future. This forces us to use an asynchronous closure or function. Specifically, a Future that resolves into a Result.

  2. Due to the Fn() closure, we can't capture the types of T and &mut T. This means we can neither move the data inside the closure nor mutate the data that depends on the outside, avoiding potential side effects and confusing errors. The closure only accepts the type &T. Even doing so will result in an error because of the constraint on the return type. The signature F + Send + Sync + 'c indicates that the closure can't use captured elements inside it, regardless of their types. The only way to satisfy the compiler is to create any data inside the closure itself. Due to the static lifetime, we cannot use references that depend on anything outside the closure.

  3. The Send + Sync constraint disallows the use of non-thread-safe types. It also restricts types that are Send but not Sync, and vice versa.

I have a solution involving the use of static slices, but if you have a better solution, please share it by commenting

These restrictions around the API design have made it challenging to abstract credentials while creating the IAM client CLI application.

Now, I invite you to try each one of these options and see for yourself where the compiler does not allow certain actions:

  1. Pass an owned type of T like String::new() or Vec::new().
  2. Pass a reference type of &T like "ref" or &[12,34,34].
  3. Pass a non-sendable type like Rc::new(Cell::new()).

If you don't have time to do that yourself, don't worry. I've created separate functions for each situation here along with all the code examples mentioned above.

Share your thoughts on this API design.

If you want to learn more about how to use Rust AWS SDKs, head over to this project where various AWS services have been built. Explore the AWS APIs Github source code to learn more about SDK usage.

Don't forget that each API action or service may incur a cost depending on the service you use.

For the upcoming posts of the month, we will focus on learning Rust's language features instead of diving straight into developing a backend application in Rust. This is because Rust has some unique features that need to be addressed before anything else makes sense.

The cover image was edited using AWS Slides by me. I am not affiliated with AWS or anything. If the image violates anything, let me know.

AWSRust