In my previous article, I introduced the AWS Distro for OpenTelemetry (ADOT) Lambda layer and showed how to use it to instrument a simple AWS Lambda function. Tyler Benson provided helpful feedback to me on the CNCF Slack, pointing out that the OpenTelemetry community provides its own Lambda layers. I wasn’t aware of this, so I decided to investigate further. Here, I’ll provide an overview of the OpenTelemetry community’s Lambda layers and compare them to the ADOT Lambda layer.

Comparing the OpenTelemetry community’s Lambda layers and the ADOT Lambda layer #

ADOT Lambda project (and layer) actually builds on top of the OpenTelemetry community’s Lambda project. Specifically, the ADOT project uses the opentelemetry-lambda project as a Git submodule. So, even if you are using the ADOT Lambda layer, you are still benefiting from the OpenTelemetry community’s Lambda project doing the heavy lifting.

This means that, regardless which set of layers you choose, your Lambda will have access to the same functional components:

1. an OpenTelemetry Collector, and
2. an OpenTelemetry instrumentation library.

A key difference between the two projects is that the OpenTelemetry community’s Lambda project provides the Collector and instrumentation as separate layers. The ADOT Lambda layer incorporates both the Collector and the instrumentation library into a single layer.

Also, according to the ADOT project, the ADOT Lambda layers “are preconfigured for use with AWS services.” So, if your goal is to send telemetry to AWS services such as AWS X-Ray, you may not need to provide any custom configuration for the collector.

Choosing between the Community and ADOT Lambda layers #

Here are some factors you might consider when choosing between the OpenTelemetry community Lambda layers and the ADOT Lambda layer:

• Convenience: The ADOT Lambda layer is a single layer, while the OpenTelemetry community’s Lambda project provides two layers, and is supposed to come pre-configured to work with AWS services (e.g. X-Ray).
• Independence: The OpenTelemetry community’s Lambda project allows you to use different versions of the Collector and instrumentation layers, while the ADOT Lambda layer is a single version.
• Currency: The OpenTelemetry community’s Lambda project may have more recent versions of the Collector and instrumentation layers than the ADOT Lambda layer.
• Support: The ADOT Lambda layer is maintained by AWS, and therefore has guaranteed response times for customers with AWS Enterpries Support.
• Documentation: This one is more subjective, but, at the current moment, I found the ADOT Lambda documentation to be more consolidated than the OpenTelemetry community’s Lambda project documentation. However, this may change over time.

Adding the OpenTelemetry Lambda layers from the OpenTelemetry community #

I took the OpenTelemetry community’s Lambda layers for a spin, myself, using the Hello World SAM project from the previous article. I added a community_layers branch to the companion Git repository to GitHub, which you can use to follow along with the steps below.

Similar to the ADOT Lambda project’s layer, we first need to find the appropriate ARNs for the community’s Lambda layers. At present, these can be found on the OpenTelemetry Lambda project’s releases page.

Adding the OpenTelemetry Lambda collector layer #

The first layer we need to locate is the Collector layer. Go to the community’s Lambda project releases page, and find the latest release of “layer-collector”. For example, at the time of writing, this is “layer-collector/0.4.0”. You should find a copy-able ARN pattern in the section labeled “ARN” in the release notes. See an example below:

At the time of writing, the ARN format is arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-<architecture>-<version>, where <region>, <architecture>, and <version> are placeholders. Similar to the previous article using the ADOT layer, we will use SAM template substitution to take care of the correct value for <region>. If you’re unsure of the correct value for <architecture>, please refer to the previous article’s instructions describing this choice for the ADOT layer.

At the time of writing, here’s what adding the Collector layer to the SAM template looks like:

      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-collector-amd64-0_4_0:1"

Adding the OpenTelemetry Lambda instrumentation layer #

Next, we need to find the ARN for the instrumentation layer. Go back to the community’s Lambda project releases page, and find the latest release of “layer-python”. For example, at the time of writing, this is “layer-python/0.4.0”. Again, you should find a copy-able ARN pattern in the section labeled “ARN” in the release notes, as shown below:

At the time of writing, the ARN format for the instrumentation layer is arn:aws:lambda:<region>:184161586896:layer:opentelemetry-python-<version>, where <region> and <version> are placeholders. Note that the Python instrumentation layer is architecture-agnostic, so we don’t need to worry about the <architecture> placeholder as we did for the Collector layer.

Here’s what adding the instrumentation layer to the SAM template looks like:

      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-collector-amd64-0_4_0:1"
        - !Sub "arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-python-0_4_0:1"

Taking a broader look at the SAM template, here’s what we now have:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - x86_64
      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-collector-amd64-0_4_0:1"
        - !Sub "arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-python-0_4_0:1"
      Environment:
        Variables:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
          OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/otel-collector-config.yaml
          OTEL_SERVICE_NAME: sam-hello-world
          OTEL_PROPAGATORS: tracecontext
          HONEYCOMB_API_KEY: "{{resolve:secretsmanager:honeycomb-test:SecretString:APIKey}}"
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

As we can see, the end result is very similar to the SAM template we used in the previous article, just with the two community layers swapped in for the ADOT Lambda layer.

At this point, we can build and deploy the SAM template as we did in the ADOT article and invoke the Lambda function:

sam build && sam deploy
sam remote invoke HelloWorldFunction --stack-name sam-hello-world

You should see the same telemetry data being sent to Honeycomb.

Conclusion #

In this article we looked at the OpenTelemetry community’s Lambda layers. We covered how the OpenTelemetry Lambda project provides the foundation for the ADOT Lambda project, and compared the two projects, including some criteria you might use to decide which to use. Finally, we showed the process of adding the OpenTelemetry community’s Lambda layers to a SAM template, similar to the process we used for adding the ADOT Lambda layer.

Special thanks again to Tyler Benson for his helpful feedback and for pointing me to the OpenTelemetry community’s Lambda project.

References and further reading #

• The OpenTelemetry Functions as a Service docs and OpenTelemetry Lambda GitHub project page.
• The AWS Distro for OpenTelemetry (ADOT) technical docs and ADOT Lambda docs, and ADOT Lambda GitHub project page.