Linear Regression Regularization bias

Author: kotlyar-shapirovCreated Apr 26, 2023Updated Dec 17, 2023

Minor suggestion: Using all the weights (including bias) in regularization might end up in constraining aformentioned bias for non-normilized training data. e.g:

  class l1_regularization():
      """ Regularization for Lasso Regression """    
      def __call__(self, w):
          return self.alpha * np.linalg.norm(w) # this will constrain the bias too

It's extremely easy to fix, since you add bias as the zero'th column in data:

    def fit(self, X, y):
        # Insert constant ones for bias weights
        X = np.insert(X, 0, 1, axis=1)           
        self.training_errors = []
        self.initialize_weights(n_features=X.shape[1])

The new regularization should exclude zero'th weight from norms (and it's less than one line fix :)

  class l1_regularization():
      """ Regularization for Lasso Regression """
      def __init__(self, alpha):
          self.alpha = alpha
      
      def __call__(self, w):
          return self.alpha * np.linalg.norm(w[1:]) # here
  
      def grad(self, w):
          return self.alpha * np.sign(w[1:]) # and here

Same for the l2 and l1_l2

Source: eriklindernoren/ML-From-Scratch

View original on GitHub View discussion on GitHub