AWS Tips 06/2024

Working with the AWS C++ SDK can be a great professional challenge. The SDK can be configured in many different ways and offers a variety of APIs for both synchronous and asynchronous processing. Especially interesting is the S3 CRT client.

Why Use AWS C++ SDK?

The SDK can be configured in many different ways, lining up with your specific needs. Here’s why you might love it:

Configure in numerous ways: Tailor the SDK to fit the needs of your projects.
Variety of APIs: Take advantage of APIs for both synchronous and asynchronous processing.
High-performance clients: Looking to work with high-performance S3 access? Definitely use the S3 CRT client, an optimal choice for any fast system, including those running on Windows, that need access to data stored in S3.

Configuring the S3 CRT Client

After you’ve installed and set up AWS SDK and the S3 CRT client, it’s time to configure the client. Configuration of the SDK in a typical C++ project is handled through CMake with a typical CMakeLists.txt configuration looking like this:

1
find_package(AWSSDK REQUIRED COMPONENTS s3 s3-crt)

Then, you’ll need to choose the right spot in your project to instantiate the S3 CRT client and configure it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
using namespace Aws::S3Crt;

// client configuration
ClientConfiguration config;
config.region = Aws::Region::US_EAST_1;
config.throughputTargetGbps = 1.0f;
config.partSize = 20 * 1024 * 1024;

// instantiate the configured client
auto client = std::make_unique<S3CrtClient>(config);

This chunk of code does the following:

It sets the region to us-east-1
It targets a throughput of 1 Gbps.
It sets the part size for uploads to 20 MB.
Finally, it creates a new S3CrtClient object with the provided configuration.

Downloading the S3 Objects

With the client configured, you can now use it to download S3 objects into files or memory.

First, you need to create a GET request and configure it with your desired save location.

1
2
3
4
5
6
7
8
Model::GetObjectRequest request;
request.SetBucket("<bucket_name>");
request.SetKey("<object_key>");

// you may want to write to a output stream or into memory
request.SetResponseStreamFactory([]() {
    return Aws::New<Aws::FStream>("<memory_tag>", "<file_path>", std::ios_base::out | std::ios_base::binary);
});

Here, you fill <bucket_name> with the name of your S3 bucket and <object_key> with the name of the object you want to download. <memory_tag> and <file_path> are placeholders for the memory allocator and the output stream or file path you want to write the S3 object to.

The request can then be passed to the client, which will execute it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
client->GetObjectAsync(request, [](
    const S3CrtClient* client,
    const Model::GetObjectRequest& request,
    Aws::Utils::Outcome<Model::GetObjectResult, S3CrtError> outcome,
    const std::shared_ptr<const Aws::client::AsyncCallerContext>& context) {
    
    // evaluate the outcome of the request and handle it
    if(outcome.IsSuccess) {
        std::out << "File download success" << std::endl;
    } else {
        std::out << "File download error" << std::endl;
    }
});

This snippet executes your GET request asynchronously, allowing your application to continue other work until the request completes.

Overcoming Streaming Issues

In multi-threaded applications, which are often the domain of the AWS C++ SDK, it’s important to be aware of potential issues and workarounds. One of the issues you might encounter happens when streaming the response body to a file.

In this situation, the client does not close the stream before leaving the result handler. This delay can cause a crash if you pass control to another thread that tries to quickly open the same stream. The following line of code helps to circumvent the problem.

1
outcome.GetResult().ReplaceBody(nullptr);

It forces the result object to replace its internal body object instance. The destructor of the body object closes the output stream. Once this stream is closed, another thread can open it for reading safely.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
client->GetObjectAsync(request, [](
    const S3CrtClient* client,
    const Model::GetObjectRequest& request,
    Aws::Utils::Outcome<Model::GetObjectResult, S3CrtError> outcome,
    const std::shared_ptr<const Aws::Client::AsyncCallerContext>& context) {
       
    // evaluate the outcome of the request and handle it
    if(outcome.IsSuccess) {
        std::out << "File download success" << std::endl;
    } else {
        std::out << "File download error" << std::endl;
    }
    
    // force close the output stream
    outcome.GetResult().ReplaceBody(nullptr);
    
    // pass a message to another thread
});

In this updated version of the handler lambda function, we added a line to forcibly close the output stream after the request outcome is evaluated.

This was a tricky issue to address, and it ultimately underscores the importance of having some SDKs available as open source code - it enables us to pinpoint and resolve tricky issues like this.