Purpose: Enhancing the quality of radiotherapy for skin cancer requires systematic evaluation of radiation-induced skin toxicity, particularly acute toxicity, which affects ~90% of patients receiving ionizing radiation. This study assessed the reliability of a newly developed visual grading tool, the Clinician Erythema Assessment-Radiation Therapy (CEA-RT), designed to help radiation clinicians evaluate early radiation dermatitis. Methods and Materials: The CEA-RT tool utilizes a 5-point scale. Its reliability was evaluated by analyzing intra-rater and inter-rater agreement among clinicians, as well as its accuracy against a reference standard. The assessment was conducted using digital images. Results: The study's results showed high intra-rater reliability, with 98% of scores falling within +/-1 grade and consistent grading for regions of interest (ROI). However, inter-rater agreement was only 60%, indicating variability among different clinicians. Accuracy against a reference standard was moderate, with 25-30% of radiation therapists achieving high concordance and over 50% showing moderate agreement. Conclusions: The findings support the CEA-RT scale as a reliable tool for individual use. However, study limitations included variable time gaps between sessions, recall bias, fatigue, and reliance on digital images. The results emphasize the need for additional training, in vivo validation, and inclusion of diverse skin types to improve overall accuracy and applicability.