synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

This paper proposes Synatra, a system for generating large amounts of training data for digital agents. The goal is to overcome the problem of expensive human annotation by using indirect knowledge like online tutorials and random web pages as input. Synatra leverages LLMs to transform this indirect knowledge into direct demonstrations in the form of action sequences, which are then used to fine-tune an LLM for web navigation tasks. The paper presents empirical results showing that agents trained with Synatra outperform other models of comparable size, even surpassing GPT-3.5 on certain benchmarks. However, the authors also acknowledge limitations, such as the potential for overfitting to specific formats and the need to address computational costs.