Heat oven to 400°F. Pierce potatoes; roast on a lined sheet 50 to 60 minutes until fully soft. Split and scoop flesh; reduce oven to 350°F.
Melt 5 tablespoons butter in a small light-colored saucepan over medium. Cook 3 to 5 minutes, swirling, until solids turn golden brown. Immediately pour into a heatproof bowl.
Mash flesh with brown butter, half-and-half, sugar, vanilla, salt, and nutmeg. Cool until warm, not hot; beat in eggs.
Grease a 2-quart dish with teaspoon butter; spread filling. Bake 25 to 30 minutes until center reaches 160°F.
Scatter marshmallows evenly; bake 5 minutes more until puffed and golden, watching closely. Rest 10 minutes.