Activate yeast: Mix warm milk, yeast, and a pinch of sugar. Let it foam for 10 minutes.
Make dough: In a bowl, mix flour, sugar, eggs, and salt. Add yeast mixture. Knead until smooth.
Add butter: Knead in butter gradually until fully absorbed.
First rise: Cover the dough and let it rise until doubled (about 2 hours).
Shape and second rise: Punch down, shape into a round disc, place on parchment-lined tray. Let rise 45 minutes.
Bake: Brush with egg yolk, sprinkle with pearl sugar, and bake at 350°F (175°C) for 25–30 minutes. Cool completely.
Make cream: Heat milk with half the sugar. In a bowl, whisk yolks, cornstarch, and remaining sugar. Slowly pour hot milk into yolk mixture, whisking. Return to pan and cook until thick. Stir in butter, vanilla, and orange blossom water. Chill.
Assemble: Slice brioche in half. Fold whipped cream into pastry cream. Spread cream on bottom half, then place the top back on. Dust with powdered sugar if desired.