1. Preheat oven to 450F. On a large rimmed baking sheet, toss potatoes with garlic, rosemary, and 1 Tblsp oil; season generously with salt and pepper. Arrange potatoes in a single layer, cut side down. Bake, tossing potatoes once, until beginning to brown, about 20 minutes.
2. Meanwhile, rub fillets with remaining teaspoon oil; season all sides with salt and pepper.
3. remove baking sheep from oven. Add tomatoes and olives to potatoes; stir to combine. Push vegetable mixture to one side; place fillets flat on baking sheet, next to vegetables. Return to oven, and roast until fish is cooked through and potatoes are brown and tender, 15 to 20 minutes. Transfer fish and vegetables to serving platter. Serve immediately, with lemon wedges if desired.