Dorsal Finger Fold Recognition by Convolutional Neural Networks for the Detection and Monitoring of Joint Swelling in Patients with Rheumatoid Arthritis

Digital biomarkers such as wearables are of increasing interest in monitoring rheumatic diseases, but they usually lack disease specificity. In this study, we apply convolutional neural networks (CNN) to real-world hand photographs in order to automatically detect, extract, and analyse dorsal finger fold lines as a correlate of proximal interphalangeal (PIP) joint swelling in patients with rheumatoid arthritis (RA). Hand photographs of RA patients were taken by a smartphone camera in a standardized manner. Overall, 190 PIP joints were categorized as either swollen or not swollen based on clinical judgement and ultrasound. Images were automatically preprocessed by cropping PIP joints and extracting dorsal finger folds. Subsequently, metrical analysis of dorsal finger folds was performed, and a CNN was trained to classify the dorsal finger lines into swollen versus non-swollen joints. Representative horizontal finger folds were also quantified in a subset of patients before and after resolution of PIP swelling and in patients with disease flares. In swollen joints, the number of automatically extracted deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49, p < 0.01). The joint diameter/deep skinfold length ratio was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1, SD 0.6, p < 0.01). The CNN model successfully differentiated swollen from non-swollen joints based on finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75%. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification. After significant response to disease-modifying antirheumatic drug ± corticosteroid therapy, longitudinal metrical analysis of eight representative deep finger folds showed a decrease in the mean diameter/finger fold length (finger fold index, FFI) from 3.03 (SD 0.68) to 2.08 (SD 0.57). Conversely, the FFI increased in patients with disease flares. In conclusion, automated preprocessing and the application of CNN algorithms in combination with longitudinal metrical analysis of dorsal finger fold patterns extracted from real-world hand photos might serve as a digital biomarker in RA.

automatically extracted deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49, p < 0.01). The joint diameter/deep skinfold length ratio was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1, SD 0.6, p < 0.01). The CNN model successfully differentiated swollen from non-swollen joints based on finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75%. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification. After significant response to diseasemodifying antirheumatic drug ± corticosteroid therapy, longitudinal metrical analysis of eight representative deep finger folds showed a decrease in the mean diameter/finger fold length (finger fold index, FFI) from 3.03 (SD 0.68) to 2.08 (SD 0.57). Conversely, the FFI increased in patients with disease flares. In conclusion, automated preprocessing and the application of CNN algorithms in combination with longitudinal metrical analysis of dorsal finger fold patterns extracted from real-world hand photos might serve as a digital biomarker in RA.

Introduction
Biomarkers are defined as objectively measurable values to diagnose, monitor, or predict pathologic processes [1]. Ideally, biomarkers are easy to observe, cheap, and diseasespecific. To monitor disease activity in rheumatoid arthritis (RA), currently available biomarkers are limited to serum C-reactive protein and blood sedimentation rate [2]. Digital biomarkers are considered non-invasive and should be able to provide a continuous picture of a disease status. In rheumatology, digital biomarkers most commonly include information obtained from wearables or smartphones, such as data from gyroscopes, accelerometers, tab speed, or geolocation [3]. Thermography (where temperature is recorded by a camera), in combination with artificial neural networks, has recently been used to differentiate between RA patients and healthy individuals, but so far has not been shown to monitor disease activity sufficiently [4]. The core problem of biomarkers for RA is the insufficient use of realworld data and the low specificity of patient-reported outcomes, respectively. Thus, there is a need for more specific and easier to collect data to measure inflammation.
Joint swelling due to underlying synovitis and effusion is a hallmark of arthritis and is only quantifiable clinically by ultrasound or magnetic resonance imaging. Clinical grading of joint swelling in RA does not improve the performance of disease activity scores and is not recommended in practice [5,6]. We postulated that hand joint swelling leads to an early and reversible change of dorsal finger fold patterns. Automated assessment of finger folds in RA and other forms of arthritis on hand photographs over time may be used as a supplement for existing biomarkers and clinical assessment, respectively. If captured by patients themselves, such a tool might empower patients and improve self-management strategies [7,8]. In this proof-of-concept study, we investigated dorsal finger fold patterns of proximal interphalangeal (PIP) joints in patients diagnosed with RA according to the EULAR/ACR criteria [9].

Methods
Hand photographs of patients with RA were taken during consultations by smartphone cameras in a standardized manner (single hand on a white DIN A4 paper, no direct light, at a distance of 25 cm) after obtaining informed consent. 190 PIP joints were categorized as either swollen or not swollen based on clinical judgement and ultrasound. For preprocessing, PIP joints were cropped, and dorsal finger folds from PIP joints were identified and extracted from the hand photographs by a subsequent pipeline algorithm consisting of open-source hand keypoint detection, contrast limited adaptive histogram equalization, and Canny edge detection [10] algorithms ( Fig. 1a, b). A convolutional neural network (CNN) was trained to classify the resultant (cropped and preprocessed images) into swollen versus non-swollen joints on an open access platform www.giotto.ai (Fig. 1d), after dividing the images into a training (80%, 152 images) and validation set (20%, 38 images). The network architecture we used is a resnet34 to which we postpended a fully connected layer with 1,000 input nodes and 2 output nodes. We trained the model using stochastic gradient descent with a constant learning rate of 0.001 for 40 epochs. We used a cross-entropy loss to optimize the parameters of the model.
Accordingly, the CNN model successfully differentiated swollen from non-swollen joints based on extracted finger fold patterns with a validation accuracy of 0.84, a sensitivity of 88%, and a specificity of 75% (Fig. 1d). The main reasons for misinterpreted images were hairy skin or oblique finger positioning. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification (Fig. 1e).
Next, we metrically compared 8 representative deep finger folds from different swollen PIP joints of four RA patients with significant clinical and ultrasound improvement after disease modifying antirheumatic drug ± corticosteroid therapy in patients achieving a good DAS28 response (−1.2, DAS28 ≤ 3.2) (Fig. 2). The mean FFl decreased from 3.03 (SD 0.68) before therapy to 2.08 (SD 0.57) after therapy. Curving reappeared in some skin folds after therapy. Conversely, the finger fold index increased in patients with disease flares and increased joint swelling.

Discussion
Here we provide a proof-of-concept to use automated dorsal finger fold recognition via standard smartphone cameras as a potential real-world digital biomarker for joint swelling in RA. Both the CNN model and specific metrical analyses were able to differentiate swollen from non-swollen joints. this is the first application of CNNs to real-world images in the setting of RA or osteoarthritis, whilst in previous studies, CNNs were applied to radiographs and ultrasound images [11,12]. Such a patient-controlled biomarker could complement patient reported outcomes and wearables and thus empower patients in monitoring their disease. How such a digital biomarker will finally be deployed in rheumatic patient care needs to be clarified. A shifting point of care towards interoperable, patient-led digital platforms could be one possible scenario. Potentially, flare detection by finger fold patterns in combination with patient-reported outcomes will further increase the accuracy of flare detection (and prediction) and thus lead to earlier clinical visits. On the other side, fewer C-reactive protein measurements or ultrasound exams during the patient journey might be required and costs could be saved. Future algorithms could potentially combine CNN classification tasks and longitudinal observations of finger line metrics in order to monitor arthritis and predict disease flares. In return, treat-to-target strategies or drug holidays could be monitored more closely both by the patient and the rheumatologist and necessary decisions might be taken ear-lier. The pilot study here has a focus on PIP joints in RA but also could be used for distal interphalangeal joints in hand osteoarthritis. Our model also captures metacarpal phalangeal joint swelling via longitudinal reliefs of extensor tendons (data not shown).
Clearly, to validate this digital biomarker, it needs to be correlated with disease activity scores such as DAS28-CRP in a larger, prospective study. In the literature, clinical exam for the detection of synovitis has a sensitivity of 43% and a specificity of 89% [13]. In this study, the sensitivity for the detection of joint swelling by finger fold patterns is substantially higher at 88%, with a slightly lower specificity of 75%. Given the limited sensitivity of clinical detection and quantification of joint swelling, automated finger fold analysis could be an adjunct to clinical examination and improve inter-clinician assessment, respectively.
We observed a high variability of finger fold patterns due to age, and there is a possible intraindividual circadian variation. Variation in background lighting, the influence of skin pigmentation, hydration status, and the use of lotions and cremes on extracted hand folds need to be clarified in further studies. In any case, education via the user interface (app) itself, the rheumatologist or health care professionals will be necessary to ensure a sufficient quality of captured images. Whether the user interface will be used as a standalone application, remote patient moni-toring tool, or in combination with a digital therapeutic, needs to be clarified. In any case, an optimal user experience and interaction with the treating rheumatologist as well as interoperability, e.g., with electronical records, will be required for a successful integration into standard care. Finger fold patterns seem more useful for monitoring intraindividual swelling, rather than for new diagnosis of arthritis. To this end, the CNN model needs more training to provide a more sophisticated classification of finger swelling. Swelling might also be monitored in other forms of arthritis or dactylitis, notably in psoriasis arthritis, where skin or nail lesions could be integrated into the algorithm [14].
Taken together, this pilot study demonstrates the application of computer vision and image recognition as a potential digital biomarker for RA. Even more importantly than the CNN models themselves, the quality of the obtained data will ultimately determine the utility of the biomarker.

Statement of Ethics
Written informed consent was obtained from all participants and the study was approved by the Commission Cantonale d'Ethique de la Recherche sur l'Etre Humain, Canton de Vaud, Lausanne, Switzerland (CER-VD ID-2020-00033). The study was conducted ethically in accordance with the World Medical Association Declaration of Helsinki.

Conflict of Interest Statement
Thomas Hügle is patentholder of DETECTRA. Thomas Hügle and Marc Blanchard are scientific board members of Atreon S.A.

Funding Sources
This study was supported by a grant from the Fondation pour l'Innovation Technologique (FIT) Lausanne.

Author Contributions
Thomas

Data Availability Statement
All data generated or analysed during this study are included in this article. Further enquiries can be directed to the corresponding author.