Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54651

This PR fleshes out the benchmarks to everything I could come up with. (166 individual cases when all is said and done.) If there's anything you feel warrants a spot in CI that I've missed, by all means let me know.

Test Plan: Imported from OSS

Reviewed By: pbelevich

Differential Revision: D27537824

Pulled By: robieta

fbshipit-source-id: 3819e8fec2131c6b5f29f5099cd41e79131bed90