Cut the fish into 4 equal-size portions. Combine the olive oil, lemon juice, salt, pepper, garlic and rosemary in a bowl. Brush the mixture onto the fish.
To grill, arrange the fish on a grill rack or use a grill basket sprayed with olive oil cooking spray. Grill over medium-hot coals until the fish flakes easily (allow 4-6 minutes per 1/2" of thickness). If the fish is more than 1" thick, gently turn it halfway through grilling.
To broil, spray the rack of a broiler pan with olive oil cooking spray and arrange the fish on it. Broil 4" from the heat for 4-6 minutes per 1/2" of thickness. If fish is more than 1" thick, gently turn it halfway through broiling.
To serve, top the fish with capers, if using, and garnish with rosemary sprigs, if desired.