The widespread adoption of encryption has greatly enhanced user privacy but also complicated
network security monitoring. Traditional intrusion detection systems (IDS) that rely on payload
inspection have become ineffective as most communications are now secured using Transport Layer
Security (TLS). This work proposes a privacy-preserving framework for detecting malware in
TLS-encrypted traffic by combining JA4 and JA4S fingerprinting techniques with supervised machine
learning algorithms on a large-scale dataset of 72,816 TLS flows spanning 19 malware families.
The proposed approach extracts unique TLS handshake fingerprints and leverages classifiers such as
Random Forest, XGBoost, and Logistic Regression to classify encrypted traffic. By analyzing
metadata rather than payloads, the system maintains data confidentiality and regulatory
compliance. A rigorous fingerprint-level splitting protocol ensures zero overlap between training
and test fingerprint combinations, with all 20,815 test flows corresponding to entirely unseen
fingerprints. To validate that ML classifiers contribute genuine generalization beyond fingerprint
memorization, we introduce a non-ML majority-label baseline, which achieves only 26.09% accuracy
when forced to generalize to unseen fingerprints. XGBoost outperforms this baseline by +72.71%,
achieving 98.80% accuracy and 97.69% F1-score, while Random Forest achieves 96.79% accuracy. Prior
statistical analysis demonstrated 87% fingerprint uniqueness for distinguishing malware families.
We show these distinctive fingerprints enable highly accurate, machine-learning classification and
provide an effective, privacy-preserving method for detecting malware in encrypted traffic.