OpenTelemetry

[florianbepunkt]

@floydspace Do you have any experience in using Lambda, OpenTelemetry and Effect?

We are considering adopting Coralogix as a SIEM platform. It provided an Otel lambda layer, that captures the lambda logs. Since effect also does support Otel, I wondered how would you instrument the code, when you use Effect as well?

[floydspace]

hi @florianbepunkt
this is how I do that

attach layer to the lambda like this:

  lambda:
    handler: src/index.handler
    layers:
      - arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-30-2:1
    environment:
      AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler

and then provide following layer in your function:

import { Resource, Tracer } from "@effect/opentelemetry";
import { OtelTracerProvider } from "@effect/opentelemetry/Tracer";
import { trace } from "@opentelemetry/api";
import { Layer } from "effect";

export const layer = (config: { readonly serviceName: string }) =>
  Tracer.layer.pipe(
    Layer.provide(Resource.layer(config)),
    Layer.provideMerge(
      Layer.sync(OtelTracerProvider, () => trace.getTracerProvider())
    )
  );

the idea that you do not have to configure NodeSdk with span exporter within effect layer, it is configured by lambda layer, you only need to call trace.getTracerProvider which will take it from global closure

that will print traces to xray by default

HOWEVER

the traces would not be linked under the root segment, as you can see they are printed as separate segments

this is what I still could not figure out how to solve.

if you compare it with powertools-tracer which is implemented in this repo, it print traces perfectly

it is why I prefer using the powertools-tracer for now

---

So I think it is a good starting point for you to explorer it further.

[floydspace]

UPDATE:

I actually was able to link them together with this trick:

import { Resource, Tracer } from "@effect/opentelemetry";
import { trace } from "@opentelemetry/api";
import { Layer } from "effect";
import { ParentSpan } from "effect/Tracer";

export const layer = (config: { readonly serviceName: string }) =>
  Tracer.layer.pipe(
    Layer.provide(Resource.layer(config)),
    Layer.provideMerge(
      Layer.sync(Tracer.OtelTracerProvider, () => trace.getTracerProvider())
    ),
    Layer.merge(
      Layer.sync(ParentSpan, () =>
        Tracer.makeExternalSpan(trace.getActiveSpan()!.spanContext())
      )
    )
  );

[florianbepunkt]

@floydspace Thank you. I will try to replicate your setup later today. Thank you!

[florianbepunkt]

@floydspace Is there a specific reason why you are using the Tracer directly instead of using the NodeSDK (https://effect.website/docs/observability/tracing/#printing-a-span-to-the-console)? Just curious how the pieces fit togther.

[floydspace]

@florianbepunkt
As I mentioned

you do not have to configure NodeSdk with span exporter within effect layer, it is configured by lambda layer, you only need to call trace.getTracerProvider which will take it from global closure

if you want to configure in effect way, than you do not need to use otel lambda layer
I suggest to look at how otel layer implemented and copy span exporter configuration to the effect NodeSdk layer

[berenddeboer]

Thanks @floydspace ! Using this I was able to come up with this lambda:

import { Effect, Layer } from "effect"
import { Resource, Tracer } from "@effect/opentelemetry"
import { trace } from "@opentelemetry/api"

// Hook into ADOT layer's pre-configured OpenTelemetry provider
// No need to configure our own exporter - ADOT layer handles that
const TracerLive = Tracer.layer.pipe(
  Layer.provide(Resource.layer({ serviceName: "effect-lambda-service" })),
  Layer.provideMerge(
    Layer.sync(Tracer.OtelTracerProvider, () => trace.getTracerProvider())
  )
)

// Function to simulate a task with possible subtasks
const task = (
  name: string,
  delay: number,
  children: ReadonlyArray<Effect.Effect<void>> = []
) =>
  Effect.gen(function* () {
    yield* Effect.log(name)
    yield* Effect.sleep(`${delay} millis`)
    for (const child of children) {
      yield* child
    }
    yield* Effect.sleep(`${delay} millis`)
  }).pipe(Effect.withSpan(name))

const poll = task("/poll", 1)

// Create a program with tasks and subtasks
const program = task("client", 2, [
  task("/api", 3, [
    task("/authN", 4, [task("/authZ", 5)]),
    task("/payment Gateway", 6, [
      task("DB", 7),
      task("Ext. Merchant", 8)
    ]),
    task("/dispatch", 9, [
      task("/dispatch/search", 10),
      Effect.all([poll, poll, poll], { concurrency: "inherit" }),
      task("/pollDriver/{id}", 11)
    ])
  ])
])

// Create an async Lambda handler using Effect.runPromise
const handler = async (event: unknown, context: unknown) => {
  return Effect.runPromise(
    program.pipe(
      Effect.provide(TracerLive),
      Effect.catchAllCause(Effect.logError)
    )
  )
}

// Note: Using module.exports instead of export for ADOT compatibility with esbuild
// The ADOT layer needs to hot-patch the handler at runtime
module.exports.handler = handler

CDK stack:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as nodejs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as path from 'path';

export class OpenTelemetryAwsStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create the Lambda function with OpenTelemetry and Effect
    const effectLambda = new nodejs.NodejsFunction(this, 'EffectLambda', {
      entry: path.join(__dirname, 'lambda.ts'),
      handler: 'handler',
      runtime: lambda.Runtime.NODEJS_22_X,
      architecture: lambda.Architecture.ARM_64,
      timeout: cdk.Duration.seconds(10),
      memorySize: 256,
      tracing: lambda.Tracing.ACTIVE, // Enable X-Ray tracing
      bundling: {
        minify: true,
        sourceMap: true,
        target: 'es2022',
        externalModules: [
          'aws-sdk', // Exclude aws-sdk as it's available in Lambda runtime
        ],
      },
      environment: {
        NODE_OPTIONS: '--enable-source-maps',
        // ADOT configuration
        AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
        OTEL_SERVICE_NAME: 'effect-lambda-service',
        OTEL_NODE_ENABLED_INSTRUMENTATIONS: 'aws-sdk,aws-lambda,http',
      },
    });

    // Add AWS ADOT layer for ARM64
    const adotLayer = lambda.LayerVersion.fromLayerVersionArn(
      this,
      'AdotLayer',
      'arn:aws:lambda:ap-southeast-2:901920570463:layer:aws-otel-nodejs-arm64-ver-1-30-2:1'
    );
    effectLambda.addLayers(adotLayer);

    // Add X-Ray write permissions
    effectLambda.role?.addManagedPolicy(
      iam.ManagedPolicy.fromAwsManagedPolicyName('AWSXRayDaemonWriteAccess')
    );
  }
}

And this is my trace: