The protection of allergic consumers is crucial to the food industry. Therefore, accurate methods for the detection of food allergens are required. Targeted detection of selected molecules by MS combines high selectivity with accurate quantification. A confirmatory method based on LC/selected reaction monitoring (SRM)-MS/MS was established and validated for the quantification of milk traces in food. Tryptic peptides of the major milk proteins β-lactoglobulin, β-casein, αS2-casein, and κ-casein were selected as quantitative markers. Precise quantification was achieved using internal standard peptides containing isotopically labeled amino acids. For each peptide, qualifier and quantifier fragments were selected according to Commission Decision 2002/657/EC. A simple sample preparation method was established without immunoaffinity or SPE enrichment steps for food matrixes containing different amounts of protein, such as baby food, breakfast cereals, infant formula, and cereals. Intermediate reproducibility, repeatability, accuracy, and measurement uncertainty were determined for each matrix. LOD values of 0.2–0.5 mg/kg, e.g., for β-lactoglobulin, were comparable to those obtained with ELISA kits. An LOQ of approximately 5 mg/kg, expressed as mass fraction skim milk powder, was validated in protein-rich infant cereals. The obtained validation data show that the described LC/SRM-MS/MS approach can serve as a confirmatory method for the determination of milk traces in selected food matrixes.