how is the "Avg" column in the ALFWorld results table computed?

[socialism-redstar]

Thanks for the great work and for releasing the code. I have a question about how the Avg column in the ALFWorld results table (Table 1) is computed from the three per-task-type success rates (Pick&Place / Transform / Examine). I tried to reconstruct the Avg using ALFWorld game counts per category (Seen: PP=59, Transform=68, Examine=13, total=140; Unseen: PP=41, Transform=75, Examine=18, total=134). For several rows this reconstructs the reported Avg exactly, but for a few it doesn't seem to combine — by any weighting — into the reported value. For example, RESKILL Seen: the reported Avg is 90.0 (4B) / 90.2 (8B), but a game-count-weighted micro-average of the corresponding Pick&Place / Transform / Examine numbers comes out to ~89.0 for both (the Unseen rows do reconcile). Could you clarify how Avg is defined — is it a micro-average over all games (weighted by per-category counts), a macro-average over the three categories (unweighted), or something else — and are the per-category numbers and the Avg computed over the same set of runs/seeds? Or could you provide the complete, specific numerical values ​​for the different tasks in Table 1? I'd like to make sure I'm reproducing your metric correctly. Thanks in advance!

[ZLHe0]

Thanks for your question!

To your question, in fact, Avg is a micro-average over all games in the split. We calculated it by number of games solved / total games (140 seen, 134 unseen), computed directly from the per-episode success flags and then averaged over the 3 seeds; it is not recomputed from the rounded per-category cells.

The three columns group ALFWorld's six task types as: Pick&Place = pick_and_place + pick_two_obj_and_place (micro over the union), Examine = look_at_obj_in_light (single type), and Transform = macro-average of the three pick_{heat,cool,clean}_then_place_in_recep rates (an unweighted mean of the subtype rates, not a game-count-weighted rate over transform games). Because the transform subtypes have unequal game counts, the Transform cell isn't the micro-rate over transform games, so weighting the displayed columns by total category counts won't recover Avg. Thanks again for flagging it! We will clarify it in the following updates.