A Chip-Scale Packaging (CSP) LED device and a method of manufacturing the same are disclosed. The CSP LED device includes a flip-chip LED semiconductor die and a packaging structure, wherein the packaging structure comprises a soft buffer layer, a photoluminescent structure and an encapsulant structure. The soft buffer layer includes a top portion formed on top of the flip-chip LED semiconductor die, and an edge portion formed covering an edge surface of the flip-chip LED semiconductor die, wherein the top portion has a convex surface, and the edge portion has an extension surface smoothly adjoining the convex surface. The photoluminescent structure is formed on the soft buffer layer covering the convex surface and the extension surface of the soft buffer layer. The encapsulant structure, which has a hardness not lower than that of the buffer layer, is formed on the photoluminescent structure. Therefore, the CSP LED device has improved reliability by improving adhesion strength between the flip-chip LED semiconductor die and the packaging structure, and improved optical performance such as more consistent correlated color temperature (CCT), more uniform spatial color, and higher optical efficacy.