How to Calculate Weighted Mean and Weighted Standard Deviation with Python

The other day, I had to calculate the weighted mean, standard deviation, and variance of a dataset using Python and I struggled to find reliable guidance that explained how the calculation worked. Here, I’ll explain how I performed the calculations using the math and statistics libraries.

Why use weighed values?

Many of us assign different weights to values when making comparisons. Suppose you’re trying to choose between a set of products in an online store and are utilizing review scores to help you decide. In this context, a product with a high overall rating based on just a few reviews might be less impressive than a product with a lower overall score based on many more reviews. This is because we might value products with a longer track record over products without one, and the number of reviews represents that track record.

Overall product ratings on Amazon provide an instructive example. The summary includes both the overall rating itself and the number of ratings the overall rating is based on. (I say “based on” because the rating is not calculated by simple average of all ratings.) The number of ratings represents the weight of the overall rating: the more ratings the overall rating is based on, the more that overall rating should count in our calculation. Notice that we can make use of the number of ratings without actually knowing each individual rating, which Amazon doesn’t provide. So if we wanted to calculate the average rating of a set of products on Amazon, we could weigh each overall rating based on the number of ratings.

Calculating Weighted Mean in Python

To calculate the weighted mean using Python, I first made two lists: one of values and one of weights. I then used those two lists to generate a list of tuples of the value-weight pairs. In practice, you could populate the lists from a database or a data frame, but here I’ve just typed them into the script manually for demonstration.

vals = [4.6, 4.5, 4.0, 3.8, 4.2]
weights = [174, 230, 169, 275, 224]

vals_n_weights = [(vals[i], weights[i]) for i in range(0, len(weights))]

So far so good. Now for the math. A standard formula for weighted mean is as follows (from Wikipedia):

The numerator is the sum of the weights (w) times their values (x) and the denominator is the sum of the weights (w). The following Python function takes values and weights from the list of tuples, vals_n_weights, to generate a list, weighted_vals, which it then adds up to get the weighted mean:

def get_wmean(vals, weights):
    weighted_vals = []
    for tup in vals_n_weights:
        weighted_vals.append(round(tup[0]*tup[1]/sum(weights), 3))    
    answer = sum(weighted_vals)
    return answer

So far so good.

Calculating Weighted Standard Deviation in Python

Calculating weighted standard deviation is a bit trickier, and there are actually multiple formulas you can choose from to calculate it depending on what kind of “weight” you’re using. If the weights reflect importance, then you use one formula. If they reflect uncertainty, then you use a different one. (This paper provides a helpful summary of the two different cases.) I used the weigth-as-importance formula, which looks like this:

The source of this formula is a DATAPLOT reference manual from 1996. (Seems reputable to me!) Inside the square root symbol, the numerator is sum of the square of the weights (w) times the difference between the value (x) and the weighted mean (xˉ). (We square this value to keep the numbers positive.) The denominator is the number of weights (N’) minus 1 times the sum of the weights divided by the number of weights. The reason for the (N’-1)/N has to do with Bessel’s Correction, and is nicely explained in this paper here. Basically, the idea is that we use N’-1 to account for the fact that the values in a sample tend to be closer to the sample average than to the population average. I can’t really explain it more than that based no my current knowledge.

In Python, we can define a function that does this:

def get_wstd(vals, weights):
    numerator = []
    for i in range(0, len(weights)):
        numerator.append(weights[i]*(vals[i]-get_wmean(vals, weights))**2)
    var = sum(numerator)/(((len(weights)-1)*sum(weights))/len(weights))
    wstdev = math.sqrt(var)
    return wstdev

The function takes the items in vals and weights and turns them into a list of values that are summed to generate the numerator of the weighted standard deviation equation, which we then divide by the ((length of weights minus 1) times the sum of the weights) divided by the length of the weights. This gives us the weighed variance. The weighted standard deviation will be the square root of the weighted variance.

And that’s it! The whole script can be found below, and is also shared on GitHub.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Created on Fri Dec 16 14:47:08 2022

This program takes a list of values an a list of weights and recturns the
unweighted mean, weighted mean, standard deviation, and weighted standard
deviation using a standard formula that includes Bessel's Correction. The
formula can be found here:
    
    https://www.itl.nist.gov/div898/software/dataplot/refman2/ch2/weightsd.pdf


@author: FranksCodes

"""
import statistics
import math

vals = [4.6, 4.5, 4.0, 3.8, 4.2]
weights = [174, 230, 169, 275, 224]

vals_n_weights = [(vals[i], weights[i]) for i in range(0, len(weights))]

def get_wmean(vals, weights):
    weighted_vals = []
    for tup in vals_n_weights:
        weighted_vals.append(round(tup[0]*tup[1]/sum(weights), 3))    
    answer = sum(weighted_vals)
    return answer


def get_wstd(vals, weights):
    numerator = []
    for i in range(0, len(weights)):
        numerator.append(weights[i]*(vals[i]-get_wmean(vals, weights))**2)
    var = sum(numerator)/(((len(weights)-1)*sum(weights))/len(weights))
    wstdev = math.sqrt(var)
    return wstdev
    
print("Values:", vals)
print("Weights:", weights)
print("Unweighted mean:", sum(vals)/len(vals))     
print("Weighted mean:", get_wmean(vals, weights))
print("Standard deviation:", statistics.stdev(vals))
print("Weighted standard deviation", get_wstd(vals, weights))

References:

Why use weighed values?

Calculating Weighted Mean in Python

Calculating Weighted Standard Deviation in Python

Share this:

Like this:

Related

Leave a Reply Cancel reply