Skip to main content

Create an Email Experiment

Email experiments can be used to gain insight into the efficacy of new emails or content changes.

Steps to start an email experiment​

Create and use Launchdarkly toggle​

In order to differentiate between emails during development (and analysis), we use Launchdarkly experiment toggles.

After creating your toggle, access the toggle's experiment variant for a given user1. If we are

@_variation_toggle ||= Toggles.new(user: user).enabled?('experiment-example-toggle-name')

Most legacy notification emails exist in the monolith as Haml files, sent with Rails ActionMailer using a call to mailgun_mail. We can attach our toggle variable as a parameter, with camel-case key as our LD toggle name:

  mailgun_mail(
    to: sample_email@test.com,
    subject: 'example subject',
    mailgun_variables: {
      experiment_example_toggle_name: @_variation_toggle
    }
  )

Activity-based email notifications​

If the email being experiment on is spawned by an Activity, we add the toggle variable to the notification's config:

  def config(notification)
    @notification ||= notification
      {
        email_experiment_running: variation_toggle_enabled,
        tag: 'example_mailgun_tag',
        subject: 'example subject',
        mailgun_variables: {
          experiment_example_toggle_name: @_variation_toggle
        },
      }
  end

  def variation_toggle_enabled
    @_variation_toggle == "on"
  end

Since email_experiment_running is added to the notification's config, we can access this variable within the notification's corresponding haml file (using a ruby block) to easily switch between treatment and control.

  @experiment_running = @metadata[:email_experiment_running] == "on"
A note on config variables​

  • Note that .enabled? returns the toggle variant, not a boolean value. For A/B experiments, we can also pass email_experiment_running as @_variation_toggle == "on".

  • For multivariant experiment, email_experiment_running could be renamed to email_experiment_running_a or any other text variant within reason; don't override special values1 used by the mailgun_mail ('to', 'tag', etc.) since they won't get added to @metadata.

Email Analysis​

BigQuery​

Using BigQuery, we can track downstream sessions in the app.

Example CTR and Sessions Query​

Use the following query to search for emails sent through mailgun, for a specific tag:

  with emails as (select
      distinct lcm.message_id,
      lcm.user_id,
      IF(JSON_EXTRACT_SCALAR(lcm.transactional_data, "$.experiment_event_email_refresh") = "true", "treatment", "control") as variant,
      lcm.tag,
      logical_or(opened) as opened,
      logical_or(clicked) as clicked,
      count(distinct sla.session_id) as count_sessions,
  from handshake_derived.lifecycle_communication_messages as lcm
  join student.student_dim as st
  on lcm.user_id = st.user_id
  left join handshake_derived.session_lifecycle_attribution sla
      on lcm.message_id = sla.message_id
      and sla.last_touch
      and sla.session_start_timestamp >= TIMESTAMP({{start_date}})
  where
      lcm.sent_at >= TIMESTAMP({{start_date}})
    and lcm.channel = 'email'
    and lcm.tag = 'attendee_welcome_email'
    and lcm.product_bucket = 'Events'
    and lcm.delivered
  group by 1,2,3,4
  )

  SELECT
    tag,
    variant,
    count(message_id) AS emails_sent,
    count(distinct user_id) as unique_users,
    countif(opened) as total_opens,
    countif(clicked) as total_clicks,
    SUM(count_sessions) as total_sessions,
    SAFE_DIVIDE(count(distinct case when clicked THEN message_id END), count(distinct message_id)) as ctr
  FROM
    emails
  group by 1, 2
  order by 1, 2
Additional Notes​

  • You do not need product_bucket for most emails, but Events and Career Fairs share the same mailgun tag, so it is necessary to query the correct email in this case.

  • TIMESTAMP({{start_date}}) uses a prior-set variable start_date, but you can use built-in functions like CURRENT_TIMESTAMP, TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 7 DAY (for 1 week) and more2.

  • To query for emails sent via Iterable campaign, replace tag with iterable_campaign_name or iterable_campaign_id.

Hex​

Hex is a useful app3 for building analytics notebooks.

Example Hex dashboards for Email Experiments:​

Footnotes​

  1. In activity handlers, we can acess the notification's user by modifying the config to return @notification, using @notification.user. ↩ ↩2

  2. https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions ↩

  3. Seats are limited. Requests go through the Data team, usually via slack. ↩