What I think constitutes good research, and why

 Now that I have actively started thinking about bringing people together for research, I think I should start by defining what I believe constitutes good research. While my thoughts are primarily geared towards research in Computer Science, I think much of what I believe applies to research in general.

I want to start with something that is somewhat unrelated to research: chess (unfortunately, if you do not know how to play chess, you'll have to take this bit of what I say for granted). Around a year ago, I was absolutely shocked by a game that I witnessed between Stockfish and Alliestein over at the Top Chess Engines Competition (TCEC). You can find an (excellent) analysis of it here. Right out of the opening, Alliestein 'walked into' what was (to a human) a moderately deep queen trap, which involved a weird  Re1-e5-b5-b3 rook maneuver, which for most humans would have been out of grasp. But this wasn't the amazing part. The amazing part was that Allie saw through this, and deduced that in giving up the queen, it could get two important passers on the queenside. It went on to 'fall into' this trap, and played on to make a draw, often coming dangerously close to demolishing Stockfish. The variations involving this are mind bogglingly deep, with some over 25-30 moves deep, well outside the reach of even prime Carlsen. This was a clear indicator to me that these chess engines understood chess at a much deeper level than we humans do. In short, we humans, even the best of us, are too dumb to comprehend the detailed intricacies of chess.

The reason I brought this up is that I believe a similar truth holds for Computer Science. I think there is a deep grey sea of knowledge out there that is outside our reach. And we try to comprehend the knowledge the same way as we try to understand chess: we try to understand spoonfuls of it at a time, by turning the little morsels of insight we generate into guiding principles. In chess, this takes the form of open files, king safety and so on. In computer science it takes the form of generally accepted motifs, such as exploiting parallelism as a primary means of achieving higher throughput. I should remark here that these principles are necessarily general. There can and will exist anecdotes for which they break down completely. This however does not alter the fact that the only way we understand how systems work is via abstracting away the gory details and the complexity in simple statements that filter out the underlying guiding principles.

This brings me to what I believe what good research is. 

Good research is the type of research that generates new insight.

The quality of research, then, depends little on how much gain in throughput you observed, or on how many workloads your proposal beat existing systems. All of these are important in today's world, but to me they (when considered alone) fit more into the implementation domain. A good example of this, in my opinion, is tradeoffs. In CS (and indeed everywhere else), we face tradeoff curves. One example of this is the tradeoff between memory capacity and latency (there is actually a third dimension here, namely cost), which is the key distinguisher between, say, caches, DRAM and storage. One approach that has worked particularly well to tackle tradeoffs is heterogeneity: when we can two achieve different points in the tradeoff space, we incorporate both into the system. This of course, has a cost. The cost is that of complexity. Incorporating both caches and DRAM into a system necessitates making the system aware of this heterogeneity, and this mandates that the system implement intelligent decision making policies to map data to each type of memory.  This tells us that in fact, the tradeoff is not two dimensional, but three (or maybe more), with the third dimension being the degree to which the system is aware of the internals of the various types of memory. We can push this all the way, generating what could be considered a 'new' insight:

Often, by breaking the barriers of abstractions between different components of a system, we can achieve better performance.

Indeed, this insight has been applied to some of the modern systems, for instance in PAR-BS, wherein the DRAM is made aware of the extent of parallelism achievable by multiple threads, enabling better scheduling of requests to it. Another example of excellent insight, is the famous end-to-end argument by Saltzer, coined way back in 1984:

“Using performance to justify placing functions in a low-level subsystem must be done carefully. Sometimes, by examining the problem thoroughly, the same or better performance can be achieved at the high level.” 

This was in fact the key insight used by the slides used in the presentation of eRPC, which won the best paper award at NSDI 2019. The paper, of course, combines this with a slew of clever insights, such as optimizing for the common case instead of the worst case, to achieve in software performance similar to specialized hardware implementations of RPCs in datacenter networks today.

This long winded argument finally brings me to what I want to conclude with. Computer Science is this delicate interplay of various tradeoffs, many of which seem fundamental but are not. It is an infinitely intriguing field to explore, but only when done with the right mindset. Unfortunately, since today most courses cover well tested material, CS is often reduced to a question answer session, reminiscent of our JEE coaching. 

By all means, asking questions is the best way to learn, but I'm here to suggest that the right questions be asked.