“If one does not know to which port one is sailing, no wind is favorable.” - Seneca

Below are a set of questions that you can productively ask at any stage of a research (or development) project. Revisit these questions often to guarantee that you are not fooling yourself regarding the direction you are pursuing and how much you have accomplished so far. The Heilmeier catechism is similar in spirit and I’ll leave it below for reference.

The Heilmeier catechism

  • What are you trying to do? Articulate your objectives using absolutely no jargon.
  • How is it done today, and what are the limits of current practice?
  • What is new in your approach and why do you think it will be successful?
  • Who cares? If you are successful, what difference will it make?
  • What are the risks?
  • How much will it cost?
  • How long will it take?
  • What are the mid-term and final “exams” to check for success?

Self-assessment questions

  • What is the problem?
  • Why are you working on this problem?
  • What is the main hypothesis of the project?
  • What is the next step?
  • Do you believe this problem to be important?
  • Do you think that there are other more important problems that you are better prepared to solve?
  • Would you care if you solved this problem?
  • Would anyone care if you solved this problem?
  • Why would people use your approach instead of something else?
  • Why wouldn’t people use your approach (e.g., too complex, works only in specific cases, too slow)?
  • How will you convince people that you have solved the problem?
  • Why hasn’t the problem been solved yet?
  • Why hasn’t the problem been approached this way yet?
  • What would be resounding success for this project?
  • What would be a resounding failure for this project?
  • What would be the smallest success still worth publishing?
  • Is the problem sufficiently important to offset any lack of expertise you may have?
  • What are possible theoretical results for it?
  • What are possible experimental results for it?
  • What parts of the project are potentially worth simplifying without losing the main message?
  • What are the minimal results (experimental or theoretical) that would convince you?
  • Has your problem been solved before in some form?
  • What is the closest work and ideas that have already been published on this problem?
  • How does your approach compare to other approaches that have been considered?
  • What is the main benefit of your approach compared to competing approaches?
  • Why do you believe you are going to improve on existing approaches?
  • What is a good baseline?
  • What is a good starting point to develop your methods such that the initial results will be competitive with the existing state-of-the-art?
  • Is getting competitive results with the state-of-art important?
  • What are the main limitations of your approach?
  • What is the most important improvement from your approach with respect to existing work?
  • What is the main takeaway from your approach?
  • What is some possible follow up work?
  • What is the minimal viable instantiation of your approach?
  • Why do you expect your approach to be successful?
  • What is likely to fail?
  • What parts of your approach are work intensive?
  • Can any of the work intensive parts be simplified in the initial stages without jeopardizing any insights that might result from the solution of the problem?
  • What is the simplest non-trivial baseline that you can implement in a few hours?
  • Can you tackle a more specific problem?
  • Can you tackle a more general problem?
  • Is your problem general enough?
  • Is your problem specific enough?
  • What is the simplest theoretical result that you can prove about your method?
  • How far do I need to simplify my method until I can prove a theoretical result about it?
  • What tools are likely to be useful to prove a theoretical result about my method?
  • What is the most similar theoretical analysis that should be useful to prove results about your method?
  • Is your claim too general?
  • Is your claim too specific?
  • What is the simplest test that would allow you to distinguish the results of your approach from competing approaches (e.g., examples where one fails and the other succeeds or the other way around)?
  • Can you construct an example where your approach does poorly?
  • Can you construct an example where your approach does well?
  • How much progress have you made since yesterday?
  • How much progress have you made since last week?
  • How have you failed so far?
  • How have you succeeded so far?
  • What are the main challenges that you’ve faced so far?
  • What challenges do you anticipate given the challenges that you’ve faced so far?
  • What is the one sentence-length description of your ideas?
  • What is the three sentence-length description of your ideas?
  • What is the one paragraph-length description of your ideas?
  • What is the abstract-length description of your ideas?
  • If you were reviewing this paper, what concerns might you raise?
  • How is that different from X (where X is one of the closest existing works)?
  • Why should I use that instead of X (where X is one of the closest existing works)?
  • Why would anyone believe your ideas to be vague?
  • Why would anyone criticize your ideas as lacking enough experimental validation?
  • Who are the stakeholders of the project?
  • What results would make each of them happy?
  • Are there compromises that must be made for the sake of time, complexity, or scope?
  • How is reporting going to vary between the different stakeholders?