Line a 9×9 inch baking pan with parchment paper and grease well.
Combine sweetened condensed milk, brown sugar, milk, and unsalted butter in a heavy-based saucepan and stir over medium heat until mixture is bubbling comfortably. Note: make sure you keep the fudge moving at all times especially around the edges so the sugar won’t burn.
Stir until the fudge has turned a golden brown colour, around 15 to 20 minutes. Test the fudge by picking up a small teaspoon of the fudge and drop it in a small amount of cold water. If the fudge forms a firm ball, then it’s cooked enough. If you are using a candy thermometer, this is at the 118 to 120C stage.
Remove fudge from the heat and add ½ teaspoon coarse sea salt. Beat with a wooden spoon for about 10 minutes or until the fudge is thick and lightly grainy.
Pour the fudge into your prepared baking pan and smooth into the corners.
Sprinkle on the remaining coarse sea salt and press in gently with your fingers.
Cool the fudge in the fridge for at least two hours before cutting into bite sized pieces.