In a small bowl stir together the sugar, nuts, and brandy. Cover and chill for at least 24 hours or up to 1 week.
Preheat oven to 500 degrees F.
Place the brie on an ovenproof platter or pie plate. Bake for 4 or 5 minutes or until the brie is slightly softened. Spread the sugar mixture in an even layer on top of the warm brie and bake for 2 to 3 minutes longer, or until the sugar melts. Brush the fruit wedges with lemon juice and arrange them around 1 side of the brie. Place crackers around the other side.