Linear regression is the simplest form of regression method where we supposedly have a linear relationship between our independent and dependent variables. Hence, it is one of the simplest methods of developing a machine learning model to predict a value of an unknown variable with their linear relationship to the independent variable\/s. <\/p>\n\n\n\n
Our focus is to understand the least squares regression<\/strong> and how to draw a least squares regression line<\/strong>. So we have to proceed with linear regression in mind. As a rule of thumb, a line is drawn when there is a linear relationship. You will understand what this means as you progress through this article.<\/p>\n\n\n\n Linear regression is used in machine learning to model linear relationships between a dependent variable and a single or multiple independent variables. Hence we have simple linear regression<\/strong> and multiple linear regression<\/strong> where multiple independent variables are considered.<\/p>\n\n\n\n The method is a widely used technique in regression analysis, hence in machine learning regression models as well. The least-squares regression technique for linear regression is a mathematical method of finding the best fit line that represents the relationship between the independent and corresponding dependent variable. When we mention the best fit<\/strong>, we are referring to minimize the errors (differences between real and anticipated values) as much as possible.<\/p>\n\n\n\n As I just explained the widely used application of least squares regression is the “linear<\/strong>” or “ordinary<\/strong>” method which is used for linear regression analysis. Its goal is to create a straight line that intends to minimize the total of the squares of the errors that are generated by the equations we use to generate the line. To calculate and minimize the errors, things like squared residuals (which we calculate by the differences in our predicted values and the real values) are considered based on our model.<\/p>\n\n\n\n Great! Now we have already started to embrace the core of this blog topic. The line of best fit<\/strong> drawn between two sets of variables by making the total of the squares of the individual errors<\/strong> as small as possible can be simply called the Least Squares Regression Line<\/strong>. This mathematical method to reduce the error is also known as the ordinary least squares method<\/strong>.<\/p>\n\n\n\n Are all these complicated mathematical terms starting to bothering you much? Let’s try to understand the concept by using a simple example. Shall we? \ud83d\ude42<\/p>\n\n\n\n Imagine that we’ve got a data set of a bunch of student grades. So, we’ve got many variables in that data set, including Grade 3<\/strong> data (G3<\/strong>) which are the students’ final grades<\/strong>, and Grade 2<\/strong> data (G2<\/strong>) which are their second-period grades<\/strong>. If we have to model the relationship between G2 <\/strong>and G3 <\/strong>in order to predict G3<\/strong> by using new G2 <\/strong>data, what should be the preferred method of doing so?<\/p>\n\n\n\n Let’s just plot the G2 vs G3 data on a scatter plot to see if they have a linear relationship or not.<\/p>\n\n\n\n As we can see most of the plot is describing a linear relationship between these variables. When we increase the G2 <\/strong>values, we can clearly see that G3 <\/strong>values also increase linearly. Therefore we can conclude that one of the best ways to create a final grade prediction model for the given scenario is by developing a linear regression model<\/strong>.<\/p>\n\n\n\n Before going into python machine learning to develop such a prediction model and get our hands dirty, let’s look at the basic math behind Least Squares Regression Line in depth.<\/p>\n\n\n\n Let’s assume that we want to draw a line to best fit these points. We could draw a line just judging by our eyes like this…<\/p>\n\n\n\n Unfortunately, that line can’t be so accurate. Because we drew it just by judging the best fit using our eyes and minds. We need a user-independent way of creating our line, something based on solid mathematics. We do not want a personal judgment that can be extremely dependable to the eye of the beholder.<\/p>\n\n\n\n To find such a solution we don’t have to look so far. We can mathematically define our best fit line by using the following very famous high school line equation and a little bit more… Hmm maths!<\/p>\n\n\n\n Let’s adopt this line equation to our problem. The equation consists of several variables, where y <\/strong>is final grades (dependant variable), x <\/strong>is second-period grades (independent variable), m <\/strong>is the slope of the line and finally, c <\/strong>is the y-intercept (where the line cuts y-axis).<\/p>\n\n\n\n Let’s assume that we have N <\/strong>points (x,y interceptions<\/strong>) in our plot and we want to find the best fit line for that plot;<\/p>\n\n\n\n *Note that the data set I will be using to build the model in python has 375 rows\/records, meaning that we will have 375 data points for each x and y value. It’s not practical to use such a big dataset just to explain the concept. Therefore I will pick just only 5 records. This will make our plot not so smooth though<\/em>.*<\/p>\n\n\n\n 2. calculate xy <\/strong>and x2<\/sup> <\/strong>for each of those points.<\/p>\n\n\n\n 3. Get values for \u03a3x<\/strong>, \u03a3y<\/strong>, \u03a3x2<\/sup><\/strong> and \u03a3xy<\/strong> (sums of x values, sums of y values, sums of x2<\/sup> values and sums of xy values)<\/p>\n\n\n\n N <\/strong>equal to 5 as we have 5 data points here.<\/p>\n\n\n\n = 5 * 578.5 – 48 * 51.5 \/ 5 * 546 – 48 * 48<\/p>\n\n\n\n = 2,892.5 – 2,472 \/ 2730 – 2304<\/p>\n\n\n\n = 420.5 \/ 426<\/p>\n\n\n\n m = 0.987<\/mark><\/strong><\/p>\n\n\n\n = 51.5 \u2212 0.987 * 48<\/em> \/ 5<\/p>\n\n\n\n = 4.124 \/ 5<\/p>\n\n\n\n c = 0.825<\/mark><\/strong><\/p>\n\n\n\n Now that we have missing m and c values, we can calculate the points for the best fit line<\/strong> using the y=mx+c<\/strong> equation and draw the line.<\/p>\n\n\n\n Let’s plugin values of m <\/strong>and c <\/strong>to our equation<\/p>\n\n\n\n y = 0.987x + 0.825<\/strong> <\/mark><\/strong><\/p>\n\n\n\n We can call this y <\/strong>value as y hat<\/strong> (\u0177<\/strong>). The meaning is the predicted y <\/strong>value or predicted dependant variable.<\/p>\n\n\n\n \u0177 = 0.987x + 0.825<\/strong><\/strong> <\/mark><\/strong><\/p>\n\n\n\n Let’s plot the points and the Least Squares Regression Line in a scatterplot now. Red points represent x,y <\/strong>points which are our G2 <\/strong>and G3 <\/strong>values respectively. Blue points represent x,\u0177<\/strong> points which are G2 <\/strong>and predicted G3 <\/strong>values respectively.<\/p>\n\n\n\n There we have it! Finally, we have the Least Squares Regression Line<\/strong> calculated and drawn on our scatterplot.<\/p>\n\n\n\n We call it so because as the process goes for creating the best fit line we keep the total of the squares of all errors as minimum as possible. In other words, we squared all the individual errors and added them all up making the total error as small as possible. <\/p>\n\n\n\n Now, do you see why we need a mathematical method or generating the best fit line for our data than just drawing a line by eye? \ud83d\ude42<\/p>\n\n\n\n Before getting into the nitty-gritty of developing a linear regression machine learning model in python<\/strong> and calculating the least squares regression line<\/strong>, let’s try to predict a y (G3)<\/strong> value for a new x (G2)<\/strong> value using our best fit line formula?<\/p>\n\n\n\n Let’s assume that a student called Michael scored 7.5 in his G2. What would be his final grade in the near future that we can expect according to our model?<\/p>\n\n\n\n Let’s plug in the data to our \u0177 = 0.987x + 0.825<\/strong><\/strong><\/strong> <\/strong>equation.<\/p>\n\n\n\n x<\/strong> is 7.5<\/strong>, so<\/p>\n\n\n\n \u0177 = 0.987x + 0.825<\/strong><\/strong><\/strong> <\/strong> <\/p>\n\n\n\n = 0.987 * 7.5 + 0.825 <\/p>\n\n\n\n \u0177 = 8.227<\/strong><\/p>\n\n\n\n This tells us that we can expect Michael’s final grade to be 8.2<\/strong>. (if we rounded that number to 1st decimal place) <\/p>\n\n\n\n Hmm, now it’s time for us to move into the interesting coding stuff. Developing machine learning models in python is very exciting, especially with the machine learning support packages and libraries.<\/p>\n\n\n\n The best thing about machine learning with python is that it has so many mathematical libraries and packages that we can use to simplify our code. For example, we can tell Numpy<\/strong>, Pandas<\/strong>, and sci-kit learn<\/strong> to do the heavy lifting, complicated mathematical coding, and data preparation by simply calling out their modules and functions into our code. By doing so, we both simply our code and reduce development time.<\/p>\n\n\n\n *I use Pycharm for my python coding. It’s a personal preference. You can use your favorite python IDE you are used to and comfortable with. It doesn’t matter, You can even use Jupyter notebooks in your browser.<\/em>*<\/p>\n\n\n\n Let’s import NumPy, pandas, sci-kit learn. Then we need matplotlib as well for drawing graphs and charts. (We need it here for the scatterplot and for plotting the least squares regression line). One of the best ways to install these useful machine learning libraries and packages at once is by installing anaconda distribution on your computer. <\/p>\n\n\n\n Do you already have anaconda on your computer?? Or aren’t you sure which version it is? Then read this article.<\/em> How to Check Anaconda Version in Windows?<\/a><\/strong><\/p>\n\n\n\n I almost forgot to mention that we need to download our dataset called Student Performance Data Set<\/a><\/strong> from the UCI Machine Learning Repository.<\/p>\n\n\n\n After downloading the student zip folder extract the files. Now move the student-mat.csv file to your python project folder. This makes our job’s a little easier, otherwise, we would have to define the path to the CSV file within our code.<\/p>\n\n\n\n Now let’s read the CSV file using pandas and save the data into a variable called data.<\/p>\n\n\n\n Strangely, this data set doesn’t include comma-separated values. But it has semicolons to separate the values. So we have to tell our model that we have used “;<\/strong>” to separate the values. That’s the purpose of the code sep=”;”<\/strong>.<\/p>\n\n\n\n Now let’s set that variable to define only the G2 and G3 values.<\/p>\n\n\n\n Let’s populate G2 and G3 values on a scatterplot to see if there’s a linear connection between them. If so, developing a linear model and plotting a least squares regression line for these variables makes sense.<\/p>\n\n\n\n Now that we know that the relationship is linear we can proceed with the rest of the code.<\/p>\n\n\n\n Let’s put that G3 into a different variable called “predictable”. <\/p>\n\n\n\n predictable = “G3”<\/p>\n\n\n\n Let’s define x and y variables for the model. Python doesn’t have inbuilt arrays. So, we have to use NumPy to create arrays and store x and y values.<\/p>\n\n\n\n Let’s create four more different arrays out of our data (from X<\/strong> and Y<\/strong>) to train and test the model. by running this code we get 90% of data randomly selected for the train sets and 10% for the test sets.<\/p>\n\n\n\n We can use the following code to create our linear regression model<\/strong>, using the LinearRegression class<\/strong> provided by the linear_model<\/strong> module in sci-kit-learn<\/strong>. I’m choosing my model name as nnl_leastsqureregression<\/strong>.<\/p>\n\n\n\n Now we can train the model using the test potion of our dataset, shown below<\/p>\n\n\n\n Now, at this point, our linear regression model has found the Least Squares Regression Line<\/strong>, which is the best fit line for our training data. By default, it has used the least-squares method<\/strong> we just learned above, to fit the line to our training data. The method has minimized the total of the squares of the individual errors of the data points.<\/p>\n\n\n\n Therefore, now we can get the slope<\/strong> and the y-intercept<\/strong> of the line by running the following lines of codes.<\/p>\n\n\n\n However, In case if you are wondering, yes coefficiency is the same as the slope of the regression line.<\/p>\n\n\n\n The slope of the line is 1.09879931<\/strong><\/p>\n\n\n\n The y-intercept is -1.3262551084047196<\/strong><\/p>\n\n\n\n Well, I know that these numbers are too long. But it’s just because we didn’t tell the program to round them up to our preferred decimal point.<\/p>\n\n\n\n Now, you may wonder why these values are different from the example that we used to understand the concept at the beginning. Let me tell you why. This is a real data set of 395 student records. And that example only had 5 records… Simple!<\/p>\n\n\n\n One more thing about the coefficiency and intercept values. It’s normal for each one of you to have slightly different values for them. We all get a different data split each time we run the code. Simply because of that random data split done at the train and test data split<\/strong> using scikit-learn’s train_test_split<\/strong> function. \ud83d\ude42<\/p>\n\n\n\n Let’s actually plot the Least Squares Regression Line on scatterplot now, shall we? We can plot the line on top of our training data.<\/p>\n\n\n\n The least squares method is by far the most popular and widely used mathematical method for drawing the best fit line in linear regression models<\/strong>. The resulting line is called as Least Squares Regression Line<\/strong>. The concept is mathematically a bit complexed over huge datasets. But we can rely on scikit learn<\/strong> to do the calculation for us. <\/p>\n\n\n\n Learn more about scikit learn by heading over to their official site. scikit learn official site<\/a><\/strong><\/p>\n\n\n\n Want to become an ML and Data Science Expert? And get hired by reputed companies? Enroll with Eduraka Today!<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Linear Regression in Machine Learning<\/h2>\n\n\n\n
What is Least Squares Regression & Line of Best Fit?<\/h2>\n\n\n\n
So what is Least Squares Regression Line<\/strong>? <\/h2>\n\n\n\n
Least Squares Regression Line Mathematical Example<\/h2>\n\n\n\n
![\"Independent](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/scatterplot-X-vs-Y-2.png?resize=609%2C452&ssl=1\")
Drawing the best fit line by the eye<\/h3>\n\n\n\n
![\"a](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/Untitled-design-1.jpg?resize=609%2C452&ssl=1\")
High school equation for the line<\/h3>\n\n\n\n
![\"y](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/y-mx-c.png?resize=181%2C28&ssl=1\")
<\/figure>\n\n\n\nLet’s calculate the least squares regression line one step at a time!<\/h3>\n\n\n\n
Calculating the slope (m) of the best fit line using the least squares method<\/h3>\n\n\n\n
![\"x](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/1-4.png?resize=166%2C166&ssl=1\")
<\/figure>\n\n\n\n![\"\u03a3x,](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/2-2.png?resize=372%2C220&ssl=1\")
<\/figure>\n<\/div>\n<\/div>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n\n\n![\"\"](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/calculate-the-slope-of-the-line-1.png?resize=198%2C44&ssl=1\")
<\/figure>\n\n\n\nCalculating the y-intercept (c) for the best fit line<\/h3>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/Formula-for-calculating-the-y-intercept.png?resize=135%2C35&ssl=1\")
<\/figure>\n\n\n\nFinal line equation for our model<\/h3>\n\n\n\n
Calculating the values of the points of our Least Squares Regression Line<\/h3>\n\n\n\n
![\"Calculating](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/Y-hat-and-y-2.png?resize=471%2C171&ssl=1\")
<\/figure>\n\n\n\n![\"](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/line-of-best-fit.jpg?resize=648%2C457&ssl=1\")
Why do we call it a least squares regression line?<\/h3>\n\n\n\n
![\"error](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/error.jpg?resize=652%2C459&ssl=1\")
![](\"https:\/\/i0.wp.com\/d1jnx9ba8s6j9r.cloudfront.net\/imgver.1551437392\/img\/co_img_781_1552626556.png?w=1200&ssl=1\")
<\/a>\n\n\n\n
Predicting a y<\/strong> value for a new x<\/strong> value using our model<\/h3>\n\n\n\n
Implementing a Linear Regression Model in Python & drawing the Least Squares Regression Line<\/h2>\n\n\n\n
Importing required libraries and packages into python<\/h3>\n\n\n\n
import numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as pt\nimport sklearn\nfrom sklearn import linear_model<\/code><\/pre>\n\n\n\nDownloading and extracting the CSV dataset into our project folder<\/h3>\n\n\n\n
![\"saving](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/project-folder-2.png?resize=346%2C167&ssl=1\")
<\/figure>\n\n\n\nReading the CSV using Pandas<\/h3>\n\n\n\n
data = pd.read_csv (\"student-mat.csv\", sep=\";\")<\/code><\/pre>\n\n\n\ndata = data [[\"G2\", \"G3\"]]<\/code><\/pre>\n\n\n\nDetermining if we have a linear relationship inbetwen our dependant and independant variables<\/h3>\n\n\n\n
pt.scatter(data.G2, data.G3, color='blue')\npt.xlabel(\"G2-\")\npt.ylabel(\"G3-\")\npt.show()<\/code><\/pre>\n\n\n\n![](\"https:\/\/i0.wp.com\/cdn2.hubspot.net\/hubfs\/493391\/Affiliate_April_2019\/Python-20.png?w=1200&ssl=1\")
<\/a>\n\n\n\n
![\"x](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/scatterplot-X-vs-Y-1-1.png?resize=609%2C452&ssl=1\")
Puting our x and y data in Numpy Arrays<\/h3>\n\n\n\n
X<\/strong> = np.array(data.drop([predictlable], 1))\nY<\/strong> = np.array(data[predictlable])<\/code><\/pre>\n\n\n\nSetting our trainning and testing data sets using sckit-learn train_test_split function<\/h3>\n\n\n\n
x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(X, Y, test_size=0.1)<\/code><\/pre>\n\n\n\nDeclaring our linear regresion model <\/h3>\n\n\n\n
nnl_leastsqureregression = linear_model.LinearRegression()<\/code><\/pre>\n\n\n\nTraining our model using the training data<\/h3>\n\n\n\n
nnl_leastsqureregression.fit(x_train, y_train)<\/code><\/pre>\n\n\n\nGeting the slope and intercept of our least squares regression line <\/h3>\n\n\n\n
print(\"slope :\", nnl_leastsqureregression.coef_)\n\nprint(\"intercept :\", nnl_leastsqureregression.intercept_)<\/code><\/pre>\n\n\n\n![\"calculating](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/slope-and-co-for-the-line-2.png?resize=1024%2C126&ssl=1\")
<\/figure>\n\n\n\nPlotting the least squares regression line on top of our training x and y data<\/h3>\n\n\n\n
pt.scatter(x_train, y_train, color='blue')\npt.plot(x_train, nnl_leastsqureregression.coef_*x_train + nnl_leastsqureregression.intercept_, '-r')\npt.xlabel(\"G2-\")\npt.ylabel(\"G3-\")\npt.show()<\/code><\/pre>\n\n\n\n![\"plotting](\"https:\/\/i0.wp.com\/neuralnetlab.com\/wp-content\/uploads\/2021\/08\/ploted-line-2.png?resize=573%2C444&ssl=1\")
Conclusion<\/h2>\n\n\n\n