Machine Intelligence

Jeremy Nixon's compiled knowledge on AI, machine learning, and the path to general intelligence. Synthesized from 60+ raw entries spanning 2014-2020.

The Research Landscape

Jeremy tracked the major AI research labs intensively, breaking down their research agendas, strengths, and philosophical orientations:

• DeepMind: Neuro-inspired path to general intelligence. Emphasis on reinforcement learning, hierarchical representations, and drawing from neuroscience. See deepmind-research-overview, deepminds-path-to-neuro-inspired-general-intelligence, mind-of-demis-hassabis
• Google Brain: Broader portfolio spanning language models, generative models, and infrastructure. See google-brain-research-overview, google-brain-research-overview-technical
• OpenAI: Focus on scaling, safety, and alignment. See openai-research-overview, open-ai-research-breakdown
• Facebook AI Research (FAIR): Self-supervised learning, computer vision. See facebook-ai-research-overview

Key differentiating question: What separates labs that produce transformative research from those that don't? See properties-of-research-orgs, 17-11-03-differentiating-factors-between-brain-and-msr, 18-12-29-types-of-transcendence-applied-to-research-labs

Breakthroughs Toward Machine Intelligence

Jeremy maintained an ordered list of the key breakthroughs needed for machine intelligence:

1. Representation learning -- Learning useful abstractions from data. See abstract-representation-learning, 17-10-26-representation-learning-research-ideas, 18-08-27-new-concrete-representation-learning-ideas, 18-01-25-valuable-properties-of-representations, 18-08-16-implementations-of-concepts-in-representation-learning
2. Meta-learning -- Learning to learn. See 18-03-05-metalearning-research-ideas, metalearning-the-structure-of-information
3. Hierarchical/causal models -- Building structured world models. See generative-causal-hierarchical-model-based-reinforcement-learning, 17-08-11-hierarchical-structure
4. Multi-agent systems -- Emergent intelligence from interacting agents. See multi-agent, multi-agent-conversation-notes
5. Recursive self-improvement -- Systems that improve their own capabilities. See recursive-self-improvement-task-search-ai-gas-powerplay-beneficial-agi, 19-06-20-forms-of-recursive-self-improvement

Full timeline: 17-08-09-breakthroughs-leading-to-machine-intelligence, 17-10-21-ordered-list-of-breakthroughs-for-machine-intelligence, 19-05-02-recent-breakthroughs-in-deep-learning

Contrarian Views on ML

Jeremy systematically collected contrarian positions on machine learning:

• Deep learning's limitations and what it cannot do. See criticisms-of-machine-learning-deep-learning, 17-08-07-deep-problems-with-machine-learning
• Arguments against embodiment, intuitive physics, and neuro-inspiration as necessary paths. See against-embodiment-intuitive-physics-amp-neuroinspiration
• Contrarian truths worth testing empirically. See 18-01-24-ml-contrarian-truths-worth-testing, 18-12-28-my-mi-contrarian-truths, 17-08-10-machine-intelligence-contrarian
• Paradigms in machine intelligence that deserve questioning. See 19-03-17-paradigms-in-machine-intelligence-worth-questioning
• Contrarian observations at the algorithm level: 17-08-27-contrarian-truths-about-machine-learning-linear-regression

Technical Foundations

Interesting facts and deep knowledge across ML methods:

• Linear Regression: 17-08-27-interesting-facts-in-machine-learning-linear-regression
• Logistic Regression: 17-08-28-interesting-facts-in-machine-learning-logistic-regression
• Decision Trees: 17-08-29-interesting-facts-in-machine-learning-decision-trees
• Neural Networks: 17-09-02-interesting-facts-about-neural-networks, 17-10-14-interesting-facts-in-machine-learning-neural-networks
• General ML facts: interesting-facts-in-machine-learning, 18-11-13-powerful-concepts-in-machine-learning
• Optimization: 19-11-08-optimization-algorithms, 18-10-22-gradient-variance
• Bias-variance tradeoff: bias-variance-vs-variance-bias-variance-vs-sensitivity-specificity

See also: comprehensive-technical-machine-learning-topics, notes-deep-learning-textbook, notes-elements-of-statistical-learning, applied-predictive-modeling, deep-learning-frameworks

AI Safety and Alignment

Jeremy thought seriously about the safety implications of different paths to intelligence:

• Relative safety of different architectures: relative-safety-of-forms-of-machine-intelligence, relative-safety-of-paths-to-general-intelligence, 19-05-27-relative-safety-of-forms-of-general-intelligence
• Alignment and control approaches: 17-07-27-alignment-amp-control-solutions, 17-09-27-interesting-approaches-to-ai-safety
• New paradigms in safety research: 19-04-07-new-paradigms-in-safety, 19-06-05-research-ideas-in-robustness-alignment-security-long-term-safety

Research Process

How to actually do research well:

• research-experiments-processes-and-systematization, miming-great-scientists
• ideas-worth-implementing-as-research-org, properties-of-research-orgs
• publishing-decomposition-recombination, open-research
• 20-02-08-innovations-in-research-organization, 20-07-10-depth-in-research
• 19-05-01-research-failure-stories, 18-08-15-most-valuable-research-events, 18-08-31-thoughts-on-valuable-research-events

ML Startup Ideas

• 17-02-05-machine-learning-startup-ideas
• 17-08-31-machine-learning-projects
• 15-08-05-data-science-projects-i-would-love
• 20-01-22-productive-and-practical-research-ideas
• 19-10-07-novel-feedback-loop-based-machine-learning-algorithms