Cut the orange into quarters, then cut into thin slices and place in a large bowl.
Zest and juice the lemon and add the oranges.
Stir in the olives, thyme, rosemary, red pepper flakes and olive oil.
Cover and refrigerate to marinate for at least 4 hours.
Preheat oven to 400°. Place olive mixture in an 8 to 9-inch ovenproof skillet.
Bake olives for about 20 minutes.