This study assessed the International Critical Thinking Essay Test (ICTET-A) for inter-rater reliability, internal reliability, and criterion validity. A self-selecting sample of participants (N = 100) completed the ICTET-A and a comparison test online. We found the ICTET-A items to have moderate to good levels of inter-rater reliability, and overall excellent inter-rater consistency for total test scores. The test had good internal reliability. There was a strong correlation between scores on the ICTET-A and the comparison test. Factor analysis showed that scores were best explained with one factor, suggesting the test measures a single construct. The ICTET-A can therefore be considered a valid measure of critical thinking. Additionally, we propose a short form of the test.