[Modeling] SVM Support Vector Machine

1. 기본 구현

# -*- coding: utf-8 -*-

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

df = pd.read_csv('heart_disease_missing_processed.csv')

X = df.drop('Heart Disease Status', axis=1)
y = df['Heart Disease Status']

x_train, x_test, y_train, y_test, = train_test_split(X, y, test_size=0.2, random_state=42)

svm_model = SVC(kernel='rbf', class_weight='balanced', random_state=42)
svm_model.fit(x_train, y_train)

predictions = svm_model.predict(x_test)

print("\n=== SVM 최종 결과 ===")
train_predictions = svm_model.predict(x_train)
print(f"Train Accuracy: {accuracy_score(y_train, train_predictions):.4f}")
print(f"Test Accuracy: {accuracy_score(y_test, predictions):.4f}\n")

print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))

print("\nClassification_report:")
print(classification_report(y_test, predictions))

=== SVM 최종 결과 ===
Train Accuracy: 0.6707
Test Accuracy: 0.5315

Confusion Matrix:
[[909 704]
 [233 154]]

Classification_report:
              precision    recall  f1-score   support

           0       0.80      0.56      0.66      1613
           1       0.18      0.40      0.25       387

    accuracy                           0.53      2000
   macro avg       0.49      0.48      0.45      2000
weighted avg       0.68      0.53      0.58      2000

 

2. C와 gamma 튜닝

# -*- coding: utf-8 -*-

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

df = pd.read_csv('heart_disease_missing_processed.csv')

X = df.drop('Heart Disease Status', axis=1)
y = df['Heart Disease Status']

x_train, x_test, y_train, y_test, = train_test_split(X, y, test_size=0.2, random_state=42)

param_grid = {
    'C' : [0.1, 1, 10],
    'gamma'  :['scale', 'auto', 0.1, 1]
}

grid_search = GridSearchCV(SVC(kernel='rbf', class_weight='balanced', random_state=42),
                        param_grid,
                        cv=5,
                        scoring = 'f1',
                        n_jobs=-1,
                        verbose=2)
grid_search.fit(x_train,y_train)

print("가장 성능이 좋은 설정값: ", grid_search.best_params_)

best_svm_model = grid_search.best_estimator_

predictions = best_svm_model.predict(x_test)

print("\n=== 튜닝 완료된 SVM 최종 결과 ===")
train_predictions = best_svm_model.predict(x_train)
print(f"Train Accuracy: {accuracy_score(y_train, train_predictions):.4f}")
print(f"Test Accuracy: {accuracy_score(y_test, predictions):.4f}\n")

print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))

print("\nClassification_report:")
print(classification_report(y_test, predictions))

Fitting 5 folds for each of 12 candidates, totalling 60 fits
[CV] END ..................................C=0.1, gamma=auto; total time=   6.7s
[CV] END ..................................C=0.1, gamma=auto; total time=   6.9s
[CV] END .................................C=0.1, gamma=scale; total time=   6.9s
[CV] END .....................................C=0.1, gamma=1; total time=   6.2s
[CV] END .................................C=0.1, gamma=scale; total time=   6.9s
[CV] END ...................................C=0.1, gamma=0.1; total time=   6.7s
[CV] END ...................................C=0.1, gamma=0.1; total time=   6.7s
[CV] END ...................................C=0.1, gamma=0.1; total time=   6.8s
[CV] END ...................................C=0.1, gamma=0.1; total time=   8.3s
[CV] END ..................................C=0.1, gamma=auto; total time=   9.5s
[CV] END ..................................C=0.1, gamma=auto; total time=  10.0s
[CV] END ...................................C=0.1, gamma=0.1; total time=  10.7s
[CV] END .................................C=0.1, gamma=scale; total time=  11.8s
[CV] END ..................................C=0.1, gamma=auto; total time=  11.6s
[CV] END .................................C=0.1, gamma=scale; total time=  11.7s
[CV] END .................................C=0.1, gamma=scale; total time=  11.8s
[CV] END .....................................C=0.1, gamma=1; total time=   5.6s
[CV] END .....................................C=0.1, gamma=1; total time=   5.6s
[CV] END ...................................C=1, gamma=scale; total time=   5.5s
[CV] END ...................................C=1, gamma=scale; total time=   5.7s
[CV] END ...................................C=1, gamma=scale; total time=   6.7s
[CV] END ...................................C=1, gamma=scale; total time=   6.7s
[CV] END ...................................C=1, gamma=scale; total time=   5.8s
[CV] END ....................................C=1, gamma=auto; total time=   5.7s
[CV] END .....................................C=0.1, gamma=1; total time=   8.3s
[CV] END ....................................C=1, gamma=auto; total time=   6.0s
[CV] END .....................................C=0.1, gamma=1; total time=  10.2s
[CV] END ....................................C=1, gamma=auto; total time=   6.2s
[CV] END ....................................C=1, gamma=auto; total time=   7.2s
[CV] END .....................................C=1, gamma=0.1; total time=   5.7s
[CV] END .......................................C=1, gamma=1; total time=   6.0s
[CV] END .....................................C=1, gamma=0.1; total time=   6.6s
[CV] END .......................................C=1, gamma=1; total time=   6.1s
[CV] END .....................................C=1, gamma=0.1; total time=   8.3s
[CV] END ....................................C=1, gamma=auto; total time=   8.9s
[CV] END .....................................C=1, gamma=0.1; total time=   9.4s
[CV] END .....................................C=1, gamma=0.1; total time=   9.2s
[CV] END .......................................C=1, gamma=1; total time=   6.8s
[CV] END .......................................C=1, gamma=1; total time=   8.7s
[CV] END .......................................C=1, gamma=1; total time=   9.0s
[CV] END ..................................C=10, gamma=scale; total time=   7.6s
[CV] END ..................................C=10, gamma=scale; total time=   8.1s
[CV] END ..................................C=10, gamma=scale; total time=   8.7s
[CV] END ..................................C=10, gamma=scale; total time=   8.8s
[CV] END ..................................C=10, gamma=scale; total time=  11.9s
[CV] END ...................................C=10, gamma=auto; total time=   9.1s
[CV] END ...................................C=10, gamma=auto; total time=   8.4s
[CV] END ...................................C=10, gamma=auto; total time=   8.3s
[CV] END ...................................C=10, gamma=auto; total time=   9.4s
[CV] END ...................................C=10, gamma=auto; total time=   9.0s
[CV] END ....................................C=10, gamma=0.1; total time=   8.8s
[CV] END ....................................C=10, gamma=0.1; total time=   9.8s
[CV] END ....................................C=10, gamma=0.1; total time=   9.0s
[CV] END ....................................C=10, gamma=0.1; total time=   9.3s
[CV] END ....................................C=10, gamma=0.1; total time=  10.0s
[CV] END ......................................C=10, gamma=1; total time=   9.3s
[CV] END ......................................C=10, gamma=1; total time=   9.1s
[CV] END ......................................C=10, gamma=1; total time=   8.2s
[CV] END ......................................C=10, gamma=1; total time=   7.6s
[CV] END ......................................C=10, gamma=1; total time=   7.9s
가장 성능이 좋은 설정값:  {'C': 0.1, 'gamma': 1}

=== 튜닝 완료된 SVM 최종 결과 ===
Train Accuracy: 0.7984
Test Accuracy: 0.8065

Confusion Matrix:
[[1613    0]
 [ 387    0]]

Classification_report:
C:\Users\limye\anaconda3\envs\damvenv\Lib\site-packages\sklearn\metrics\_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\limye\anaconda3\envs\damvenv\Lib\site-packages\sklearn\metrics\_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
C:\Users\limye\anaconda3\envs\damvenv\Lib\site-packages\sklearn\metrics\_classification.py:1531: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", len(result))
              precision    recall  f1-score   support

           0       0.81      1.00      0.89      1613
           1       0.00      0.00      0.00       387

    accuracy                           0.81      2000
   macro avg       0.40      0.50      0.45      2000
weighted avg       0.65      0.81      0.72      2000

 

1. C: 0.1, gamma: 1

=> C는 margin을 넓게 잡으라는 뜻, 데이터가 너무 복잡하게 얽혀 있고 0이 압도적으로 많다 보니 모델이 이리저리 선을 그어 보다가 학습을 포기해 버리는 것. 그나마다 단순한 형태를 꼽은 결괏값이다.

 

3. SMOTE + SVM

# -*- coding: utf-8 -*-

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from imblearn.over_sampling import SMOTE

df = pd.read_csv('heart_disease_missing_processed.csv')

X = df.drop('Heart Disease Status', axis=1)
y = df['Heart Disease Status']

x_train, x_test, y_train, y_test, = train_test_split(X, y, test_size=0.2, random_state=42)

smote = SMOTE(random_state=42)
x_train_smote, y_train_smote = smote.fit_resample(x_train, y_train)

svm_model = SVC(kernel='rbf', C=1.0, gamma='scale', random_state=42)

svm_model.fit(x_train_smote, y_train_smote)

predictions = svm_model.predict(x_test)

print("\n=== SVM 최종 결과 ===")
train_predictions = svm_model.predict(x_train)
print(f"Train Accuracy: {accuracy_score(y_train, train_predictions):.4f}")
print(f"Test Accuracy: {accuracy_score(y_test, predictions):.4f}\n")

print("Confusion Matrix:")
print(confusion_matrix(y_test, predictions))

print("\nClassification_report:")
print(classification_report(y_test, predictions))

=== SVM 최종 결과 ===
Train Accuracy: 0.7248
Test Accuracy: 0.6345

Confusion Matrix:
[[1163  450]
 [ 281  106]]

Classification_report:
              precision    recall  f1-score   support

           0       0.81      0.72      0.76      1613
           1       0.19      0.27      0.22       387

    accuracy                           0.63      2000
   macro avg       0.50      0.50      0.49      2000
weighted avg       0.69      0.63      0.66      2000

=> 무조건 0으로 맞히는 환경은 벗어났지만 f1-score가 0.22~0.33를 벗어나지 못함.

 

Linear regression: 최대 accuracy 0.58 최대 f1-score 0.32

Random Forest: 최대 accuracy 0.75 최대 f1-score 0.33

SVM: 최대 accuracy 0.63 최대 f1-score 0.25