Have you solved all the problems?
If so, try your hand at creating derived variables from date data in Task Type 2 (Question 3)!If you haven't reached that point yet, I recommend organizing what you've studied so far step-by-step rather than rushing to learn new material.
Date derived variables (day of the week/month/weekend) — apply identically to both train and test! train['date'] = pd.to_datetime(train['date']) train['dayofweek'] = train['date'].dt.dayofweek # Day of week (Mon=0~Sun=6) train['month'] = train['date'].dt.month # Month (1~12) train['is_weekend'] = (train['date'].dt.dayofweek >= 5).astype(int) test['date'] = pd.to_datetime(test['date']) test['dayofweek'] = test['date'].dt.dayofweek test['month'] = test['date'].dt.month test['is_weekend'] = (test['date'].dt.dayofweek >= 5).astype(int)Keep it up!
The part you should pay close attention to in this data is
This is the training for the entire dataset (Comment No. 5). I have included the full training code after the comparison. Let's compare the results!
Adding or changing parameters (class_weight, hyperparameters, etc.) does not necessarily make things better. You must check them yourself using the validation set.
You must decide whether to adopt them by comparing the results.
If it is difficult to judge the parameters during the exam, go with the default values (the state where they are not set). Stability seems to be important!
Training on the entire dataset unconditionally is also good!
I have updated the Type 2 tasks on Coding Pang (https://code.sideonai.com/).
I have added lightgbm to all the codes in the Big Data Analysis Practical Exam lecture section. (
I am providing the baseline for Coding Pang Task Type 2. The EDA section has been excluded.
Question 1
import pandas as pd # 1) Load data train = pd.read_csv('data/car_train.csv') test = pd.read_csv('data/car_test.csv') # 2) One-hot encoding for categorical variables target = train.pop('target') print(train.shape, test.shape) train = pd.get_dummies(train) test = pd.get_dummies(test) print(train.shape, test.shape) # 3) Split for validation (Performance comparison before submission) from sklearn.model_selection import train_test_split X_tr, X_val, y_tr, y_val = train_test_split(train, target, test_size=0.2, random_state=0, stratify=target) # 4) Train three models from sklearn.ensemble import RandomForestClassifier rf = RandomForestClassifier(random_state=0) rf.fit(X_tr, y_tr) pred = rf.predict(X_val) from sklearn.metrics import f1_score print(f1_score(y_val, pred, average='macro')) from lightgbm import LGBMClassifier lgb = LGBMClassifier(random_state=0, verbose=-1) lgb.fit(X_tr, y_tr) pred = lgb.predict(X_val) print(f1_score(y_val, pred, average='macro')) from xgboost import XGBClassifier xgb = XGBClassifier(random_state=0, verbosity=0) xgb.fit(X_tr, y_tr) pred = xgb.predict(X_val) print(f1_score(y_val, pred, average='macro')) # 5) Re-train the selected model on the full train set and predict on test (Optional) # lgb.fit(train, target) # pred = lgb.predict(test) # 6) Save submission file (Only 1 pred column, remove index) result = pd.DataFrame({'pred': pred}) result.to_csv('result.csv', index=False) # Check submission file print("\n ===== Submission File (Sample) =====") print(pd.read_csv("result.csv").head()) print("\n ===== Submission File (Check Size) =====") print(pd.read_csv("result.csv").shape)Question 2
import pandas as pd # 1) Load data train = pd.read_csv('data/bike_train.csv') test = pd.read_csv('data/bike_test.csv') # 2) One-hot encoding for categorical variables target = train.pop('count') print(train.shape, test.shape) train = pd.get_dummies(train) test = pd.get_dummies(test) print(train.shape, test.shape) # 3) Split for validation (Performance comparison before submission) from sklearn.model_selection import train_test_split X_tr, X_val, y_tr, y_val = train_test_split(train, target, test_size=0.2, random_state=0) # 4) Train three models from sklearn.ensemble import RandomForestRegressor rf = RandomForestRegressor(random_state=0) rf.fit(X_tr, y_tr) pred = rf.predict(X_val) from sklearn.metrics import root_mean_squared_error print(root_mean_squared_error(y_val, pred)) from lightgbm import LGBMRegressor lgb = LGBMRegressor(random_state=0, verbose=-1) lgb.fit(X_tr, y_tr) pred = lgb.predict(X_val) print(root_mean_squared_error(y_val, pred)) from xgboost import XGBRegressor xgb = XGBRegressor(random_state=0, verbosity=0) xgb.fit(X_tr, y_tr) pred = xgb.predict(X_val) print(root_mean_squared_error(y_val, pred)) # 5) Re-train the selected model on the full train set and predict on test (Optional) # lgb.fit(train, target) # pred = lgb.predict(test) # 6) Save submission file (Only 1 'pred' column, remove index) result = pd.DataFrame({'pred': pred}) result.to_csv('result.csv', index=False) # Check submission file print("\n ===== Submission File (Sample) =====") print(pd.read_csv("result.csv").head()) print("\n ===== Submission File (Check Size) =====") print(pd.read_csv("result.csv").shape)I've updated the mock exams for Task Type 1 and Task Type 3! 🎉
I created the Coding Pang website so you can solve problems in an environment as similar to the actual exam as possible, and I have uploaded new questions there.
You can solve problems by writing code yourself, just like in the actual exam, so please come and practice 😊 I hope you pass the exam!
👉 Coding Pang Link: code.sideonai.com
It's still in beta, so please let me know if you find any bugs!
The exam is now just one week away.
I am sharing the recording of the Big Data Analysis Certification prep live session held one week prior (6.13).
It's an hour long, so try skimming through it at a faster playback speed!
Good luck to everyone :)
(This will be deleted after the 12th exam based on the total lecture time)
The announcement for the 12th exam has been posted.
It is the same as the existing exam content, and the changes are only at the level of improving phrasing and notation methods.
Similar remarks have been made before, but in Task Type 2
"To obtain excellent evaluation metrics, the optimal model must be explored" is included, so
You need to prepare at least two models (Random Forest, LightGBM).
https://www.dataq.or.kr/www/board/view.do?boardKind=notice&bbsKey=eyJiYnNhdHRyU2VxIjoxLCJiYnNTZXEiOjU3OTAxM30=
I'm rooting for you for the remaining period!!
