Received feedback from Leila on WWW submission that requests a few good changes around language / descriptions of the work and desire to have a broader discussion section within the paper. All good feedback and will work to incorporate into a camera-ready or re-submission.

Weekly update:

• Rejected from WWW so meeting in two weeks to discuss next steps. Possibility of KDD sooner but similar venue and I suspect that's not enough time to make changes we want to to increase chances.

Weekly update:

• Meeting next week but I think a few possible strategies with this to consider:
  • Shift from state-of-art paper to qualitative exploration through a model: that is, make it less about achieving a high-quality model and more about establishing the research space, determining what the major challenges are around data etc., explore a few different approaches, greatly expand discussion section with recommendations for how to proceed.
  • Extend human evaluations: add more human evaluations to the work. this might be about showing that while the model isn't great, for those half of edit summaries with nothing, it at least produces something viable.
  • Pivot towards policy prediction aspect: move away from text-changing edits to be about improving revert summaries in particular. Could cast as classification challenge or keep text-generation focus. This would start to mix with other ideas we've discussed so from a collaboration standpoint, might be difficult (different people involved) but feels like the most promising direction to me in general for this area of research.
  • Status-quo: minimal changes and look for new venue. I don't love this as I think this could be a really strong paper with a bit more adjustments.

Weekly update:

• Met with team and a few takeaways:
  • Going to consider submitting to COLM: https://colmweb.org/ (March 29th full submission)
  • Discussed whether to train a larger model (LLaMa?) to show that we could achieve GPT-4-level performance and dispel notions that e.g., we're just bad at fine-tuning models. But low priority as I think we're still interested in what a much smaller model can do.
  • Focus on thicker evaluations -- error analysis, more human eval, etc.
  • Focus on expanded discussion / establishing of research area -- i.e. emphasizing no existing baseline approaches and GPT performance is incidenttal.
  • Not going to pivot towards policy prediction -- core of this paper is still good and interesting and that work can continue in parallel
  • If time, we're consider more ways of representing the diffs for the models to see how that changes things

Weekly update:

• No progress

Weekly update: no progress but will check in with team next week

Weekly update:

• Marija moved paper onto new template for COLM so going to do a pass on that next week

Weekly update:

• Made Leila's requested fixes including adding a future work section that touched on different dataset filtering processes and ways of representing an edit to language models. We're a bit over 9 pages (the limit) but working on bringing it within requirements.

Weekly update:

• Paper solidifying -- just style / readability aspects at this point. Should be good to submit to COLM this month. We'll upload the submitted paper to arXiv as well.

Paper submitted to COLM and we'll hear May 24. I'll link to arXiv paper when posted.

Closing this task out. We can re-open or create a new one in case substantial new work is required as a result of COLM etc. I'll still update with an arXiv link when available.

Paper: https://arxiv.org/abs/2404.03428