인프런 커뮤니티 질문&답변

Tree를 만들고 최초 한번 가중치 업데이트를 할 때,  그 값이 평균을 향하도록 학습률을 조정하는 것이 아닐까요?

누군가 테스트 한 내역이 하기와 같이 검색 되는데...

불균형한 데이터의 경우 처음 학습률이 평균이 되도록 하면 성능이 저하될 것 같은 상상을 해봅니다.

Tree=0
num_leaves=35
num_cat=2
split_feature=0 0 2 10 5 18 10 0 0 8 3 10 10 15 8 10 0 0 9 0 7 15 13 18 13 12 9 5 15 6 2 5 13 2
split_gain=1157.9 1200.12 589.218 266.5 214.587 170.697 135.443 132.99 96.2071 93.2053 80.8721 76.3493 69.4331 75.0835 67.8062 67.2649 109.225 59.3376 77.9133 54.1652 53.4583 55.6504 51.9849 51.9826 51.5137 51.0333 50.5939 49.1426 44.9309 45.1413 43.4918 43.1427 42.7473 40.7209
threshold=65.500000000000014 34.500000000000007 71.975000000000009 97.500000000000014 0.59500000000000008 211.50000000000003 226.50000000000003 78.500000000000014 84.500000000000014 7.5000000000000009 0 37.500000000000007 154.50000000000003 2.5000000000000004 12.500000000000002 80.500000000000014 62.500000000000007 69.500000000000014 5.5000000000000009 28.500000000000004 1 1.0000000180025095e-35 121.50000000000001 332.50000000000006 58.500000000000007 79.750000000000014 5.5000000000000009 1.4750000000000003 1.5000000000000002 42.500000000000007 54.07500000000001 0.015000000000000001 316.50000000000006 33.275000000000006
decision_type=8 10 2 8 8 10 8 10 10 2 1 8 8 10 2 8 10 8 2 10 1 10 2 10 2 2 2 8 8 10 2 8 2 2
left_child=1 6 4 7 9 27 19 20 14 26 12 -7 32 -14 17 24 -17 30 -19 -1 21 -2 -9 -15 -11 -12 -3 -6 33 -30 -5 -27 -4 -22
right_child=3 2 10 8 5 11 -8 22 -10 15 25 -13 13 23 -16 16 -18 18 -20 -21 28 -23 -24 -25 -26 31 -28 -29 29 -31 -32 -33 -34 -35
leaf_value=-1.3017993273605202 -1.2365519237007825 -1.1704605392075984 -1.1875759669484003 -1.1833302065593161 -1.2018216103277415 -1.1515881785938882 -1.3528821519057714 -1.356715522267989 -1.3843001904610113 -1.179889945384103 -1.2460417229323717 -1.2127216638057154 -1.2915602336119141 -1.2794855052648977 -1.3204334787470304 -1.1878180669769636 -1.2826554601297457 -1.3427217190527609 -1.2340237132810807 -1.256917319068303 -1.2606362037072381 -1.1663853564890327 -1.2018216103277415 -1.2091975110260338 -1.1275935361267115 -1.3576317831911671 -1.1444901123873226 -1.3155240700056141 -1.1988277640088607 -1.330077984621862 -1.2868820666124829 -1.273957548880817 -1.3168291197401223 -1.3904339241090904
leaf_count=547 209 1058 306 69 36 146 398 163 242 323 387 723 259 144 948 1303 140 221 96 606 61 134 20 372 141 126 711 226 93 42 74 197 33 60
internal_value=0 2.42746 2.68817 0.844371 2.95421 2.25245 0.693821 1.71992 0.300202 3.12342 1.72862 2.56473 2.02163 1.67897 0.493512 2.85941 2.66362 1.11488 0.544005 1.01683 2.07541 2.49045 -0.0250355 2.04779 3.35231 1.17775 3.36588 0.695489 1.35225 2.01773 2.04843 0.518342 2.64117 0.312707
internal_count=10614 8182 6631 2432 4807 1131 1551 782 1650 3676 1824 869 1114 775 1408 1907 1443 460 317 1153 599 343 183 516 464 710 1769 262 256 135 143 323 339 121
cat_boundaries=0 1 2
cat_threshold=2 1
shrinkage=1


Tree=1
num_leaves=35
num_cat=3
split_feature=0 0 10 4 5 2 0 0 0 0 4 2 11 0 5 14 7 4 6 18 11 15 7 17 0 17 15 9 2 13 11 6 4 4
split_gain=944.038 875.382 452.34 233.985 130.248 144.572 134.765 100.2 93.9869 81.3694 78.141 72.6584 58.5265 55.4876 55.7227 50.5636 50.4966 50.4294 50.2957 49.6577 45.8201 45.1958 45.0761 44.4216 47.5296 49.2615 50.8011 43.982 43.0995 42.1877 41.8501 41.0538 38.909 38.4231
threshold=69.500000000000014 32.500000000000007 154.50000000000003 58.500000000000007 7.3000000000000016 84.575000000000003 59.500000000000007 39.500000000000007 59.500000000000007 74.500000000000014 47.500000000000007 67.625000000000014 52.810000000000009 38.500000000000007 0.27500000000000008 0 1 70.500000000000014 35.500000000000007 421.50000000000006 108.325 2.5000000000000004 2 157.50000000000003 61.500000000000007 120.50000000000001 4.5000000000000009 5.5000000000000009 84.575000000000003 147.50000000000003 23.645000000000003 21.500000000000004 64.500000000000014 19.500000000000004
decision_type=10 8 8 10 8 2 8 10 10 10 10 2 8 10 10 1 1 10 10 8 10 10 1 2 10 2 10 2 2 2 8 10 8 10
left_child=1 10 7 -2 5 6 13 11 17 -5 31 33 15 14 19 -12 -10 27 -17 -4 32 -18 -11 24 25 -7 -27 -9 -19 -13 -31 -1 -6 -3
right_child=3 2 4 9 20 23 -8 8 16 22 12 29 -14 -15 -16 18 21 28 -20 -21 -22 -23 -24 -25 -26 26 -28 -29 -30 30 -32 -33 -34 -35
leaf_value=0.012077169455165057 -0.012629502288529668 0.00088220072728762203 0.018316423707753417 0.1217214158294474 0.10243713650000966 0.1203973861382328 0.061390377308026912 0.11954857648814066 0.12445927546884972 0.073425250407056761 0.032063979892072055 0.012472418648369059 -0.0086052959353287491 0.12009389383524052 -0.039432796901502254 0.12141774058225957 0.056751678626532298 0.15236717371212946 -0.026655596546584601 0.13184705313597286 -0.022674242795517965 0.11491648373676236 -0.019285171523004619 0.089865786193843972 0.09526334184119882 -0.035623451383552239 0.045665901194719798 0.14399439149167967 0.03301971722719 0.16231425172936223 0.0080844211574173496 -0.051890631137136746 0.022188730680620272 0.11483059170855676
leaf_count=289 1183 35 66 144 74 29 620 1126 498 113 351 144 227 1127 56 117 262 1008 35 69 157 177 122 141 72 237 139 708 32 25 32 251 188 760
internal_value=0 2.0935 2.26102 0.320569 1.66904 1.80237 2.01148 2.53403 2.63968 1.54707 0.394027 2.03888 0.804718 2.28163 1.30681 1.15121 2.1629 2.76455 1.96681 1.8184 0.563155 1.69056 0.682039 0.925351 0.589693 0.260019 -0.00170212 2.60076 3.01066 0.944138 2.12613 -0.295615 1.03737 2.23593
internal_count=10614 9052 7782 1562 2975 2556 1938 4807 3811 379 1270 996 730 1318 191 503 937 2874 152 135 419 439 235 618 477 405 376 1834 1040 201 57 540 262 795
cat_boundaries=0 1 2 3
cat_threshold=5 1 1
shrinkage=0.05

안녕하십니까,

boost_from_average에 대한 파라미터 설명 자료를 찾기 어려워서 저도 정확히는 알지 못합니다.

boosting 계열 알고리즘은 최초로 만들어진 Tree에서 오류를 보정하면서 지속적으로 Tree를 update합니다. 이때 최초 Tree를 만들때 오류를 update하기 위해 어떤 score값을 만드는데 이때 타겟값의 특정 평균값을 사용할것인지 아닌지를 나타내는 파라미터로 알고 있습니다.

boost_from_average 파라미터의 기능상 imbalanced dataset의 이진 분류에서 그 하나의 파라미터로 인하여 성능을 그렇게 크게 저하시키는 것은 어렵게 때문에  파라미터의 특성 때문에 발생하는 현상이 아니라 일종의 버그로 간주해도 좋을 것 같습니다.

감사합니다.