A/B Testing Notes

How It Works

TubeBuddy's A/B Testing works by having you create a variation of a video's metadata (this could include the Thumbnail, Title, Tags or Description). We then alternate your video's metadata every 24 hours at Midnight PST (to line up with YouTube Analytics statistics). The test completes either based on a certain number of days that you picked or based on statistical significance being achieved.

In an ideal world, we'd be able to have each impression throughout the day alternate between the Original and Variation which would be a 'true' A/B test. Since YouTube analytics are only provided in 24 hours blocks, our current system is basically the best that can be done taking into account YouTube's limitations.

Important Notes

• In a perfect world, when you run a A/B Test in TubeBuddy, the same video would have the highest CTR, the most views and the highest avg view duration. Unfortunately, that doesn't happen every time so it might take some user analysis to choose which version of the video you want to go with after the test is complete.
  
  
• One important thing to note is that CTR statistics not only have many many more data points (because the average video gets 40 to 80 impressions for each 'view' the video receives), but it also eliminates biases such as day of the week or how close to the publish date the test is. When you look at views, watch time, and all other stats displayed on screen in the lower section, those are 24-hour blocks of time and can be affected by the day of the week. 
  
  For example, maybe your particular audience is more active on Mondays than on Tuesday in general. When looking at CTR and the top line of stats, we're getting impressions and clicks (a ratio) and this ratio is not affected by the day of the week or anything else which is why we declare the winner of the video based on CTR.
  
  
• There are cases where the winning version has a higher CTR but a lower view count. Generally speaking, you should always go for a higher CTR. 
  
  For example, if you happen to get 100 views one day and 1,000 views the next day, the winner might still be the first day because your video was shown to 500 people on the first-day screen and 20% of them clicked whereas the second day it was shown to 100,000 people and only 1% clicked. The title/thumbnail on day 1 obviously was more appealing and clickable so in the future, you'd want to use that version.
  
  
• Be careful when running tests on brand-new videos. With all A/B Tests (either in TubeBuddy or not), it's important to try to compare apples to apples. When your video is first launched, it shows up in subscription feeds and more channel browse features but then that fades away after a couple of days where you then tend to get more views from search & related traffic sources. 
  
  When your video is in the subscription feed, most people clicking are your existing audience. So the CTR shows you how likely it is for your existing audience to click your video. After a few days, it's gone from subscription feeds for the most part so you're getting clicks from new viewers. These new viewers don't know your brand or who you are so you'd end up with a different CTR. So - keep this all in mind. 
  
  If you run a test immediately after a video is published, this is likely to show you what your existing audience prefers in a thumbnail and if you run a test 4+ days after your video is launched, it will show you how likely it is that your potential new audience clicks. Perhaps the 'danger' zone is if you start a test ~2 days after the video is launched where you get a mix between new and existing users and that could potentially skew the results in some cases.
  
  
• Keep in mind that when you update your thumbnail on YouTube, because of caching on their system, your old thumbnail can still show up for ~15 minutes in various areas based on our testing. In our A/B tests, the thumbnail is switched automatically at midnight Pacific Time (to line up with YouTube analytics) which for most people is a very slow time of day which works in our favor.
  
  
• Careful running Title/Tags/Description tests on high-performing videos. Generally speaking, if a video is doing very well, you probably want to leave it alone. When you change the Title/Description/Tags, it causes the video to get re-indexed in YouTube's system and has the potential to be detrimental to its performance. Updating a Thumbnail does not get it re-index so you don't have to worry there.  
  
  
• External embeds don't count towards a thumbnail's CTR (YouTube addresses this - basically that when embedded on another site, they cannot track impressions). So don't worry when you see your old thumbnail still showing on a Facebook post or embedded on your company's website for example.
  
  
• There are also cases where the winner version has a higher CTR but a lower average view duration. This example is a bit trickier and not always as clear-cut. Generally speaking, you always want a higher CTR but if you have a higher CTR and much lower avg view duration, then it could mean that your thumbnail doesn't accurately represent your video content. 
  
  People are clicking to watch based on your thumbnail but then the video itself isn't what they were looking for. In these cases, it's important to look at the 'Watch Time per Impression" box in the top right-hand corner. This will show you based on every time your video thumbnail is displayed on screen (whether or not the user clicks), the expected total watch time. So if it's displayed 100 times, clicked twice and each time the user watches 3 minutes of the video. Your watch time per impression is (3 * 2 * 60 seconds) / 100 impressions = 3.6 seconds. 
  
  The YouTube algorithm is always trying to maximize total watch time per impression so we recommend going with the highest value in those panel in these cases. That said, there might not be statistical significance in these cases and the test could be inconclusive. Try making a more meaningful change to thumbnail and running a new test or, use your best judgment as to whether or not the thumbnail change you made actually could be 'tricking' people or not accurately representing your content.

How CTR Increases are Calculated