The U.S. Open is making a big bet on the potential of mixed doubles, changing the format in ways that will draw curiosity and controversy.