ICTACT Journal on Soft Computing

C. Sunil Kumar ¹, R. J. Rama Sree ²

Affiliations
1 Research and Development Center, Bharathiar University, IN
2 Rashtriya Sanskrit Vidyapeetha, IN

Abstract

In this paper, we studied the performances of models built using various SVM implementations during the multiclass classification task of automated evaluation of descriptive answers. The performances were evaluated on five datasets each with 900 samples and with each of the datasets treated using symmetric uncertainty feature selection filter. We quantitatively analyzed the best SVM implementation technique from amongst the 17 different SVM implementation combinations derived by using various SVM classifier libraries, SVM types and Kernel methods. Accuracy, F Score, Kappa and Area under ROC curve are used as model evaluation metrics in order to evaluate the models and rank them according to their performances. Based on the results, we derived the conclusion that SMO classifier when used with Polynomial kernel is the overall best performing classifier applicable for auto evaluation of descriptive answers.

Keywords

Descriptive Answers, Auto Evaluation, SVM, LibLINEAR, LibSVM, SMO, Kernels.

Full Text

     

C. Sunil Kumar ¹, R. J. Rama Sree ²

Affiliations
1 Research and Development Center, Bharathiar University, IN
2 Rashtriya Sanskrit Vidyapeetha, IN

Abstract

In this paper, we study the performance of various models for automated evaluation of descriptive answers by using rank based feature selection filters for dimensionality reduction. We quantitatively analyze the best from amongst five rank based feature selection techniques, namely Chi Squared filter, Information Gain filter, Gain Ratio filter, Relief filter and Symmetrical Uncertainty filter. We use Sequential Minimal Optimization with Polynomial kernel to build models and we evaluate these models in terms of various parameters such as Accuracy, Time to build the models, Kappa, Mean Absolute Error and Root Mean Squared Error. For all except the Relief filter, the accuracies obtained are at least 4% better than the accuracies obtained with the same models without any filters applied. We found that the accuracies recorded are same for Chi Squared filter, Information Gain filter, Gain Ratio filter and Symmetrical Uncertainty filter. Therefore accuracy alone is not the sole determinant in selecting the best filter. The time taken to build the models, Kappa, Mean Absolute Error and Root Mean Squared Error play a major role in determining the effectiveness of these filters. The overall rank aggregation metric of Symmetrical Uncertainty filter is 45 and this is better by 1 rank than the rank aggregation metric of Information gain filter. Symmetric Uncertainty filter's rank aggregation metric is better by 3, 6, 112 ranks respectively when compared to the rank aggregation metrics of Chi Squared filter, Gain Ratio filter and Relief filters. Through these quantitative measurements, we conclude that Symmetrical Uncertainty attribute evaluation is the overall best performing rank based feature selection algorithm applicable for auto evaluation of descriptive answers.

Keywords

Descriptive Answers, Text Classification, Rank Based Filters, Feature Selection, Dimensionality Reduction.

Full Text

     

C. Sunil Kumar ¹, R. J. Rama Sree ²

Affiliations
1 Research and Development Center, Bharathiar University, IN
2 Rashtriya Sanskrit Vidyapeetha, IN

Abstract

Automation of descriptive answers evaluation is the need of the hour because of the huge increase in the number of students enrolling each year in educational institutions and the limited staff available to spare their time for evaluations. In this paper, we use a machine learning workbench called LightSIDE to accomplish auto evaluation and scoring of descriptive answers. We attempted to identify the best supervised machine learning algorithm given a limited training set sample size scenario. We evaluated performances of Bayes, SVM, Logistic Regression, Random forests, Decision stump and Decision trees algorithms. We confirmed SVM as best performing algorithm based on quantitative measurements across accuracy, kappa, training speed and prediction accuracy with supplied test set.

Keywords

Descriptive Answers, Automated Evaluation, LightSIDE, Machine Learning Algorithms.

Full Text