Dealing with Overfitting When Crowdsourcing Financial Alpha

Dealing with Overfitting When Crowdsourcing Financial Alpha

Presented By Delaney Granizo-Mackenzie
Delaney  Granizo-Mackenzie
Delaney Granizo-Mackenzie
Director of Academia at Quantopian

Delaney’s background is in computer science, statistics, math, and computational genetics. As Director of Academia at Quantopian he oversees the firm’s worldwide educational and academic initiatives. While working with professors at schools including Princeton, MIT, Stanford, and Harvard, Delaney developed the free online Quantopian Lecture Series. The lecture series draws from academic rigor and industry realism and has since spawned a global workshop program teaching quantitative finance. Delaney currently focuses on maintaining the quality of the lectures as they grow, while also growing the audience of people who have learned from the lectures.

Presentation Description

Level: Intermediate 

Type: Talk 

Length: 1:27:33

 Tools used: Pyfolio


The financial services sector has traditionally been a very secretive environment. The barriers to entry are incredibly high, and people sign many NDAs and non-competes to join. Quantopian is attempting to change that by allowing anybody to research and design trading strategies. The flip side is that identifying which strategies are good is a very difficult problem. We'll walk through some research we've done and show some insights about whether traditional metrics of financial performance are useful when considering strategies.

Learning Outcomes:

  • You will know Quantopian’s services and mission
  • You will know about case studies on statistical learning and finance trading
  • You will learn about the role of ML at Quantopian and fin-tech
  • You will learn about the problem of overfitting in machine learning models

Talk Timeline

0:00 - 8:41: Introduction and Overview

  • Quantopian possesses a lot of open-source code.
  • Perils of human decision-making
  • Humans are subject to sub-conscious biases.

8:41 - 22:40: Case-study 

  • Judges making decisions in relation to the time they’ve eaten lunch
  • Quantopian provides a forum for users and free online lectures

22:40 - 41:19: Quantopian Strategy

  • Disclaimer about recommendations and investing
  • Example of a strategy submitted to Quantopian
  • Mention of paper “Pseudo - Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance”
  • Decisions are forecasts and forecasts are decisions
  • Simulation setup
  • Simulation results

41:19 - 56:45: What about applications to real life?

  • Goal is to offer a trading platform that anybody around the world can use
  • Data source: 7152 algorithms
  • Data cleaning
  • IS vs OOS performance
  • Extremely poor results because historical results cannot make forecasts
  • Looking at strategies with low and high volatility

56:45 - 1:05:23: Machine Learning Approaches

  • “In finance, machine learning tends to be a recipe for losing all your money.”
  • ML determined most important feature was tail_ratio
  • Hypothetical portfolios on hold-out set
  • “What happened since then”
  • Next Steps which includes more strategies, more OOS data and better features

1:05:23 - 1:10:18: Demo of Pyfolio

1:10:18 - 1:19:56Specific examples of Overfitting

1:19:56 - 1:27:33: Questions

Presentation Curriculum

Dealing with Overfitting when Crowdsourcing Financial Alpha
Hide Content