The Erlangen Programme of ML

Geometric Deep Learning is an attempt for geometric unification of a broad class of ML problems from the perspectives of symmetry and invariance. These principles not only underlie the breakthrough performance of convolutional neural networks and the recent success of graph neural networks but also provide a principled way to construct new types of problem-specific inductive biases.

This blog post was co-authored with Joan Bruna, Taco Cohen, and Petar Veličković and is based on the new “proto-book” M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021), Petar’s talk at Cambridge and Michael’s keynote talk at ICLR 2021.

In October 1872, the philosophy faculty of a small university in the Bavarian city of Erlangen appointed a new young professor. As customary, he was requested to deliver an inaugural research programme, which he published under the somewhat long and boring title Vergleichende Betrachtungen über neuere geometrische Forschungen (“A…

Thoughts and Theory, Rethinking GNNs

Graph neural networks (GNNs) are intimately related to differential equations governing information diffusion on graphs. Thinking of GNNs as partial differential equations (PDEs) leads to a new broad class of GNNs that are able to address in a principled way some of the prominent issues of current Graph ML models such as depth, oversmoothing, bottlenecks, and graph rewiring.

This blog post was co-authored with Ben Chamberlain and James Rowbottom, and is based on our paper B. Chamberlain, J. Rowbottom et al., GRAND: Graph Neural Diffusion (2021) ICML.

First page of “Scala graduum Caloris”, a 1701 paper by Sir Isaac Newton published anonymously in the Philosophical Transactions of the Royal Society. Shown is a temperature scale with 0 corresponding to the temperature of “winter air when water starts freezing” (aqua incipit gelu rigescere) and 12 representing the temperature measured upon “contact with the human body” (contactum corporis humani). The highest temperature of 210 is that of “kitchen fire urged by bellows”.

In March 1701, the Philosophical Transactions of the Royal Society published an anonymous note in Latin titled “A Scale of the Degrees of Heat” [1]. Though no name was indicated, it was no secret that Isaac Newton was the author (he would become “Sir Isaac” four years later). In a series of experiments, Newton observed that

the temperature a hot body loses in a given time is proportional to the temperature…

Year 2020 in Review & Predictions for 2021

The end of the year is a good time to recap and make predictions. 2020 has turned Graph ML into a celebrity of machine learning. For this post, I sought the opinion of prominent researchers in the field of graph ML and its applications trying to summarise the highlights of the past year and predict what is in store for 2021.

Image: Shutterstock

Beyond message passing

Will Hamilton, Assistant Professor at McGill University and CIFAR Chair at Mila, author of GraphSAGE.

“2020 saw the field of Graph ML come to terms with the fundamental limitations of the message-passing paradigm.

These limitations include the so-called “bottleneck” issue [1], problems with over-smoothing [2], and theoretical limits in terms of representational capacity [3,4]. Looking forward, I expect that in 2021 we will be searching for the next big paradigm for Graph ML. …

The best of Graph Deep Learning in 2020

Among many papers on Geometric and Graph ML, applications in biochemistry, drug design, and structural biology shone in 2020. Perhaps for the first time, we are finally starting to see the real impact of these machine learning techniques in fundamental science. In this post, I highlight three papers that excited me the most in the past year (disclaimer: I am a co-author of one of them).

Geometric ML methods were featured on the covers of the February 2020 issues of two major biology magazines, Cell and Nature Methods. Image credits: Amanda Cicero, Luca Vallescura, Darryl “Moose” Norris, and Chris Sinclair (left) and Laura Persat and Erin Dewalt (right).

J. M. Stokes et al., A deep learning approach to antibiotic discovery (2020) Cell 180(4):688–702.

What? A graph neural network-based deep learning pipeline for the discovery of new antibiotic drugs.

How? A graph neural network was trained to predict the growth inhibition of the bacterium Escherichia coli on a dataset of >2000 molecules (including approved antibiotic drugs as well as natural compounds from the plant and animal kingdoms) with known antibacterial activity. The prediction is based on the molecular graph only and does not rely on any side information such as the drug mechanism of action.

The trained model was…

Making Sense of Big Data, Recommender Systems 2020 challenge

This year, Twitter sponsored the RecSys 2020 Challenge, providing a large dataset of user engagements. In this post, we describe the challenge and the insights we had from the winning teams.

This blog post was co-authored with Luca Belli, Apoorv Sharma, Yuanpu Xie, Ying Xiao, Dan Shiebler, Max Hansmire, and Wenzhe Shi from Twitter Cortex.

Recommender systems are an important part of modern social networks and e-commerce platforms. They aim to maximise user satisfaction as well as other key business objectives. At the same time, there is a lack of large-scale public social network datasets for the scientific community to use when building and benchmarking new models to tailor content to user interests. In the past year, we have worked to address exactly that problem.

Twitter partnered with the RecSys conference

Let food be thy medicine

The food we eat contains thousands of bioactive molecules, some of which are similar to anti-cancer drugs. Modern machine learning techniques can discover such components and help design nutrition that will let us live longer and healthier.

Illustration: Bianca Dagheti.

This post was co-authored with Kirill Veselkov and Gabriella Sbordone and is based on the TEDx Lugano 2019 talk and the paper published in Nature journal Scientific Reports.

We now live longer than ever. Yet, we are not necessarily living healthier anymore: with a rapidly aging population, people are experiencing a continuous growth of chronic diseases such as cancer, metabolic, neurological, and heart disorders. This drives healthcare costs through the roof and puts a significant strain on the public health systems [1].

A large part of the problem resides in poor dietary choices. Unhealthy diets kill more than cigarettes and…

Deep learning with latent graphs

Graph neural networks exploit relational inductive biases for data that come in the form of a graph. However, in many cases, we do not have the graph readily available. Can graph deep learning still be applied in this case? In this post, I draw parallels between recent works on latent graph learning and older techniques of manifold learning.

The past few years have witnessed a surge of interest in developing ML methods for graph-structured data. Such data naturally arises in many applications such as social sciences (e.g. the Follow graph of users on Twitter or Facebook), chemistry (where molecules can be modelled as graphs of atoms connected by bonds), or biology (where interactions between different biomolecules are often modelled as a graph referred to as the interactome). …

Deep learning on giant graphs

One of the challenges that have so far precluded the wide adoption of graph neural networks in industrial applications is the difficulty to scale them to large graphs such as the Twitter follow graph. The interdependence between nodes makes the decomposition of the loss function into individual nodes’ contributions challenging. In this post, we describe a simple graph neural network architecture developed at Twitter that can work on very large graphs.

This post was co-authored with Fabrizo Frasca and Emanuele Rossi.

Graph Neural Networks (GNNs) are a class of ML models that have emerged in recent years for learning on graph-structured data. GNNs have been successfully applied to model systems of relation and interactions in a variety of different domains, including social science, computer vision and graphics, particle physics, chemistry, and medicine. Until recently, most of the research in the field has focused on developing new GNN models and testing them on small graphs (with Cora, a citation network containing only about 5K nodes, still being widely used [1]); relatively little…

Deep learning on dynamic graphs

Many real-world problems involving networks of transactions of various nature and social interactions and engagements are dynamic and can be modelled as graphs where nodes and edges appear over time. In this post, we describe Temporal Graph Network, a generic framework developed at Twitter for deep learning on dynamic graphs.

This post was co-authored with Emanuele Rossi.

A dynamic network of Twitter users interacting with tweets and following each other. All the edges have a timestamp. Given such a dynamic graph, we want to predict future interactions, e.g., which tweet a user will like or whom they will follow.

Graph neural networks (GNNs) research has surged to become one of the hottest topics in machine learning this year. GNNs have seen a series of recent successes in problems from the fields of biology, chemistry, social science, physics, and many others. So far, GNN models have been primarily developed for static graphs that do not change over time. However, many interesting real-world graphs are dynamic and evolving in time, with prominent examples including social networks, financial transactions, and recommender systems. …

Basics of deep learning

Have you ever wondered what is so special about convolution? In this post, I derive the convolution from first principles and show that it naturally emerges from translational symmetry.

La connoissance de certains principes supplée facilement à la connoissance de certains faits. (Claude Adrien Helvétius)

During my undergraduate studies, which I did in Electrical Engineering at the Technion in Israel, I was always appalled that such an important concept as convolution [1] just landed out of nowhere. This seemingly arbitrary definition disturbed the otherwise beautiful picture of the signal processing world like a grain of sand in one’s eye. How nice would it be to have the convolution emerge from first principles rather than have it postulated! …

Michael Bronstein

Professor @imperialcollege, Head of Graph ML Research @Twitter, ML Lead @ProjectCETI. Researcher, teacher, entrepreneur

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store