text:
Use the offer code 20offearlybird to get 20% off.

# Logistic Regression Tutorial for Machine Learning

Last Updated on August 12, 2019

Logistic regression is one of the most popular machine learning algorithms for binary classification. This is because it is a simple algorithm that performs very well on a wide range of problems.

In this post you are going to discover the logistic regression algorithm for binary classification, step-by-step. After reading this post you will know:

• How to calculate the logistic function.
• How to learn the coefficients for a logistic regression model using stochastic gradient descent.
• How to make predictions using a logistic regression model.

This post was written for developers and does not assume a background in statistics or probability. Open a spreadsheet and follow along. If you have any questions about Logistic Regression ask in the comments and I will do my best to answer.

Kick-start your project with my new book Master Machine Learning Algorithms, including step-by-step tutorials and the Excel Spreadsheet files for all examples.

Let’s get started.

Update Nov/2016: Fixed a small typo in the update equation for b0.

Logistic Regression Tutorial for Machine Learning
Photo by Brian Gratwicke, some rights reserved.

## Tutorial Dataset

In this tutorial we will use a contrived dataset.

This dataset has two input variables (X1 and X2) and one output variable (Y). In input variables are real-valued random numbers drawn from a Gaussian distribution. The output variable has two values, making the problem a binary classification problem.

The raw data is listed below.

Below is a plot of the dataset. You can see that it is completely contrived and that we can easily draw a line to separate the classes.

This is exactly what we are going to do with the logistic regression model.

Logistic Regression Tutorial Dataset

## Logistic Function

Before we dive into logistic regression, let’s take a look at the logistic function, the heart of the logistic regression technique.

The logistic function is defined as:

transformed = 1 / (1 + e^-x)

Where e is the numerical constant Euler’s number and x is a input we plug into the function.

Let’s plug in a series of numbers from -5 to +5 and see how the logistic function transforms them:

You can see that all of the inputs have been transformed into the range [0, 1] and that the smallest negative numbers resulted in values close to zero and the larger positive numbers resulted in values close to one. You can also see that 0 transformed to 0.5 or the midpoint of the new range.

From this we can see that as long as our mean value is zero, we can plug in positive and negative values into the function and always get out a consistent transform into the new range.

Logistic Function

## Get your FREE Algorithms Mind Map

Sample of the handy machine learning algorithms mind map.

I've created a handy mind map of 60+ algorithms organized by type.

## Logistic Regression Model

The logistic regression model takes real-valued inputs and makes a prediction as to the probability of the input belonging to the default class (class 0).

If the probability is > 0.5 we can take the output as a prediction for the default class (class 0), otherwise the prediction is for the other class (class 1).

For this dataset, the logistic regression has three coefficients just like linear regression, for example:

output = b0 + b1*x1 + b2*x2

The job of the learning algorithm will be to discover the best values for the coefficients (b0, b1 and b2) based on the training data.

Unlike linear regression, the output is transformed into a probability using the logistic function:

p(class=0) = 1 / (1 + e^(-output))

p(class=0) = 1 / (1 + EXP(-output))

## Logistic Regression by Stochastic Gradient Descent

We can estimate the values of the coefficients using stochastic gradient descent.

This is a simple procedure that can be used by many algorithms in machine learning. It works by using the model to calculate a prediction for each instance in the training set and calculating the error for each prediction.

We can apply stochastic gradient descent to the problem of finding the coefficients for the logistic regression model as follows:

Given each training instance:

1. Calculate a prediction using the current values of the coefficients.
2. Calculate new coefficient values based on the error in the prediction.

The process is repeated until the model is accurate enough (e.g. error drops to some desirable level) or for a fixed number iterations. You continue to update the model for training instances and correcting errors until the model is accurate enough orc cannot be made any more accurate. It is often a good idea to randomize the order of the training instances shown to the model to mix up the corrections made.

By updating the model for each training pattern we call this online learning. It is also possible to collect up all of the changes to the model over all training instances and make one large update. This variation is called batch learning and might make a nice extension to this tutorial if you’re feeling adventurous.

### Calculate Prediction

Let’s start off by assigning 0.0 to each coefficient and calculating the probability of the first training instance that belongs to class 0.

B0 = 0.0

B1 = 0.0

B2 = 0.0

The first training instance is: x1=2.7810836, x2=2.550537003, Y=0

Using the above equation we can plug in all of these numbers and calculate a prediction:

prediction = 1 / (1 + e^(-(b0 + b1*x1 + b2*x2)))

prediction = 1 / (1 + e^(-(0.0 + 0.0*2.7810836 + 0.0*2.550537003)))

prediction = 0.5

### Calculate New Coefficients

We can calculate the new coefficient values using a simple update equation.

b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x

Where b is the coefficient we are updating and prediction is the output of making a prediction using the model.

Alpha is parameter that you must specify at the beginning of the training run. This is the learning rate and controls how much the coefficients (and therefore the model) changes or learns each time it is updated. Larger learning rates are used in online learning (when we update the model for each training instance). Good values might be in the range 0.1 to 0.3. Let’s use a value of 0.3.

You will notice that the last term in the equation is x, this is the input value for the coefficient. You will notice that the B0 does not have an input. This coefficient is often called the bias or the intercept and we can assume it always has an input value of 1.0. This assumption can help when implementing the algorithm using vectors or arrays.

Let’s update the coefficients using the prediction (0.5) and coefficient values (0.0) from the previous section.

b0 = b0 + 0.3 * (0 – 0.5) * 0.5 * (1 – 0.5) * 1.0

b1 = b1 + 0.3 * (0 – 0.5) * 0.5 * (1 – 0.5) * 2.7810836

b2 = b2 + 0.3 * (0 – 0.5) * 0.5 * (1 – 0.5) * 2.550537003

or

b0 = -0.0375

b1 = -0.104290635

b2 = -0.09564513761

### Repeat the Process

We can repeat this process and update the model for each training instance in the dataset.

A single iteration through the training dataset is called an epoch. It is common to repeat the stochastic gradient descent procedure for a fixed number of epochs.

At the end of epoch you can calculate error values for the model. Because this is a classification problem, it would be nice to get an idea of how accurate the model is at each iteration.

The graph below show a plot of accuracy of the model over 10 epochs.

Logistic Regression with Gradient Descent Accuracy versus Iteration

You can see that the model very quickly achieves 100% accuracy on the training dataset.

The coefficients calculated after 10 epochs of stochastic gradient descent are:

b0 = -0.4066054641

b1 = 0.8525733164

b2 = -1.104746259

### Make Predictions

Now that we have trained the model, we can use it to make predictions.

We can make predictions on the training dataset, but this could just as easily be new data.

Using the coefficients above learned after 10 epochs, we can calculate output values for each training instance:

These are the probabilities of each instance belonging to class=0. We can convert these into crisp class values using:

prediction = IF (output < 0.5) Then 0 Else 1

With this simple procedure we can convert all of the outputs to class values:

Finally, we can calculate the accuracy for the model on the training dataset:

accuracy = (correct predictions / num predictions made) * 100

accuracy = (10 /10) * 100

accuracy = 100%

## Summary

In this post you discovered how you can implement logistic regression from scratch, step-by-step. You learned:

• How to calculate the logistic function.
• How to learn the coefficients for a logistic regression model using stochastic gradient descent.
• How to make predictions using a logistic regression model.

## Discover How Machine Learning Algorithms Work!

#### See How Algorithms Work in Minutes

...with just arithmetic and simple examples

Discover how in my new Ebook:
Master Machine Learning Algorithms

It covers explanations and examples of 10 top algorithms, like:
Linear Regression, k-Nearest Neighbors, Support Vector Machines and much more...

### 148 Responses to Logistic Regression Tutorial for Machine Learning

1. Sean April 16, 2016 at 4:58 am #

Hi Jason,

Great blog! Nice job. Just wondering how to determine the form of the update equation? In this case, it’s in the form of b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x. How do I know it should be like this?

Thanks.

• Chris Edlinger May 8, 2016 at 8:30 pm #

Actually I have the same question. Plus I tried to reproduce your example but I can’t get the same (b0,b1,b2) after 10 iterations. I have instead (-0.02,0.312,-0.095)
thank you

• Van Le May 28, 2016 at 5:29 pm #

I got the wrong value too, but i found my error.I don’t know how to descible so i leave my code, it’s coded in C++.It maybe helpful for you. Here is it:

int k = 0;
for (int j = 0; j < 100; j++) { // I forgot this loop, i added it then i got the right value.
k = 0;
while (k < 10) {
for (int i = 0; i < 3; i++){
b[i] = b[i] + alpha * (y[k] – prediction) * prediction * (1 – prediction)* x[k][i];
cout << b[i]<<"\n";
}
k++;
output = 0;
for (int i = 0; i < 3; i++)
output += b[i] * x[k][i];

prediction = 1 / (1 + exp(-output));
}

}

• jack December 9, 2017 at 5:11 am #

please send me the whole source code

• Dylan November 6, 2016 at 6:13 am #

Hi I managed to reproduce the same result after 10 epochs. There is a typo in the code provided here. The updating equation for:

b0 = 0.0 + 0.3 * (0 – 0.5) * 0.5 * (1 – 0.5) * 1.0

should be changed as:

b0 = b0 + 0.3 * (0 – 0.5) * 0.5 * (1 – 0.5) * 1.0

Hope this helps 🙂

• Jason Brownlee November 7, 2016 at 7:10 am #

Thanks Dylan, I have updated the example.

• Jeetendra Dhall February 22, 2021 at 8:57 pm #

This is a great article indeed.

For the mathematical background, I referred:
“[2010] GENERATIVE AND DISCRIMINATIVE CLASSIFIERS- NAIVE BAYES AND LOGISTIC REGRESSION – Tom Mitchell”.

The weights my program finally learnt are:
[-0.48985449 0.99888324 -1.4328438 ]

My notebook can be found at:
https://j-dhall.github.io/ml/2021/02/22/logistic-regression-for-binary-classification-of-demo-data.html

• Atlas August 18, 2016 at 6:59 am #

I have the same question

• Zhanlue Zhao January 14, 2018 at 12:27 am #

It seems that the weight update is not right..

Old: b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x
New : b = b + alpha * (y – prediction) * x

The derivative is:

• Jason Brownlee January 14, 2018 at 6:38 am #

I took the equation from AI a modern approach by norvig, not a random cheat sheet.

• Zhanlue Zhao January 15, 2018 at 11:31 pm #

Thanks a lot for providing the reference. The difference comes from their different cost functions. The formula in the book –AI a modern approach– uses the quadratic cost function (y-h(x))^2. The formula in the cheat sheet uses the cross entropy as the cost function.

If a user expects the logistic regression to compute the class category value 0 and 1, the quadratic cost function makes better sense.
If a user expects the logistic regression to compute the class category probability, the cross entropy function makes better sense.

This is very interesting. Thanks.

• Zhanlue Zhao January 15, 2018 at 11:43 pm #

By the way. Professor Geoffrey Hinton has the following comments when introducing the cross entropy for softmax output function in his coursera course– Neural Networks for Machine Learning — lecture 4:

• The squared error measure has some drawbacks:
– If the desired output is 1 and the actual output is 0.00000001
there is almost no gradient for a logistic unit to fix up the error.
– If we are trying to assign probabilities to mutually exclusive class
labels, we know that the outputs should sum to 1, but we are
depriving the network of this knowledge

• Jason Brownlee January 16, 2018 at 7:34 am #

Nicely put!

• Antonio Sanchez April 24, 2020 at 10:08 pm #

The loss function in that book seems to be at odds to other books such as “Elements of Statistical Learning”, the wikipedia, and what my common sense tells me. I find it surprising, given how popular it is. I will not say it is wrong because it works as a cost function, but it is clearly going to produce a different result than what is called “logistic regression” elsewhere, and I think there should be some mention to it in the article.

• Jason Brownlee April 25, 2020 at 6:48 am #

There are many ways to find coefficients for a logistic regression model.

2. EDLINGER May 9, 2016 at 12:44 am #

Hi,

I can’t reproduce the value of your 10th iteration of vector (b0,b1,b2)
can you explain again the iteration process.
thank you

3. Gyanendro Loitongbam August 10, 2016 at 2:48 pm #

Hi all,
In this blog it is written that the coefficients calculated after 10
epochs of stochastic gradient descent are:

b0 = -0.4066054641

b1 = 0.8525733164

b2 = -1.104746259

Through my understanding I have a matrix of inputs of size (3×10,
x1,x2,y1) and weight matrix (3×10, b0,b1,b2) after 10 epochs.

My doubts is that this learned coefficients will only be for that
particular input vector (say first input) or for all the rest of the
input?

1) If it is for single input vector then, while testing the model that were
build using above described method, how to pass any unseen input? Will every
input pass through whole neurons?

2) If not then how to compute the coefficient?

4. Tom September 30, 2016 at 12:51 pm #

This article was very well-written and helped me immensely in understanding logistic regression. Thank you so much.

5. Karthik October 6, 2016 at 2:53 pm #

Jason, elucidated it very well! Clean and neat, I have come across many Blogs on ML. But yours always keep me hooked to your Knowledge Materials.

• Jason Brownlee October 7, 2016 at 7:50 am #

Thanks Karthik.

• Satish April 14, 2020 at 5:46 pm #

I strongly agree

6. Dylan November 14, 2016 at 11:12 pm #

Hi Jason, I have some questions for you.

[1] How can we produce the line separating these two classes? I have tried using ‘B0’ as the y-intercept, and ‘B1’ as the gradient. I am not sure if this is the case as ‘B2’ is left out of the equation.

Example:
Assuming the separator line will be from (x0,y0) to (x1,y1)

**Using the values of B0 and B1 after 10 epochs**

when x0 = minX1 = 1.38807
y0 = 1.38807(0.85257334) + (-0.40660548) = 0.77682614

when x1 = maxX1 = 8.675419
y1 = 8.675419(0.8527334) + (-0.40660548) = 6.9898252

This line (1.38807, 0.77682614) to (8.675419,6.9898252) perfectly separates the two classes, however upon increasing the number of epochs, the gradient changes and therefore produces false positives.

• Jason Brownlee November 15, 2016 at 7:56 am #

Hi Dylan, I agree with your intuition that the regression line best separates the two classes. Drawing it should draw the class boundary for the classifier I would expect.

It is possible that additional training is overfitting or that online gradient descent is resulting in noisy changes to the line.

• Shivani July 9, 2019 at 11:19 pm #

Really helpful, very nice explanation ,thank you so much.

7. Harish November 16, 2016 at 4:27 pm #

what’s the main use of logistic regression Algorithm which was explained by you ….?

• Jason Brownlee November 17, 2016 at 9:50 am #

Great question Harish,

Logistic regression is for binary classification problems, ideally where we can separate the 2 classes by a line or hyperplane.

If logistic regression works well on your problem, use it. If not, you may need to move on to more advanced methods, even non-linear methods.

• Peter November 5, 2017 at 6:14 am #

How would the approach vary if you were using poisson or log binomial regression to model binary outcomes?

8. Harish November 16, 2016 at 4:49 pm #

1)whats the use of calculating new coefficients i am unable to understand where these coefficients used??
2)How do you draw the accurate graph and values of MAKE PREDICTION??

• Jason Brownlee November 17, 2016 at 9:52 am #

Hi Harish, coefficients are used to make predictions on new data.

We use the prediction equation with the coefficients and the input values in order to make predict the class of a new piece of data.

If we make predictions on test data, then calculate the error of those predictions we can make a graph of accuracy each time the coefficients are updated. That is how the graph is created.

9. Jack Paterson February 17, 2017 at 3:25 am #

Hi Jason – Thanks for the article. It has been helpful but can you please explain the iteration process for each Epoch?

I don’t get the same values for the 10th Epoch. It isn’t clear if you are moving to the corresponding x1, x2 and Y values for each training instance in the Prediction and the calculation of the coefficients. For example the 2nd training instance I am assuming:

prediction = 1 / (1 + e^(-(-0.0375 + -1.043*2.7810836 + -0.0956*2.550537003)))

=0.3974

b0 = b0 + 0.3 * (0 – 0.0375) * 0.0375 * (1 – 0.0375) * 1.0

b1 = b1 + 0.3 * (0 – 0.0375) * 0.0375 * (1 – 0.0375) * 1.4655

b2 = b2 + 0.3 * (0 – 0.0375) * 0.5 * (1 – 0.0375) * 2.3621

I repeat this step another 8 times updating the y,X1 and X2 values but they dont match yours. I’d be grateful if you could point out there’s something I’m missing or post your results for each training instance.

• Jack Paterson February 17, 2017 at 3:29 am #

Sorry the prediction line should read:

prediction = 1 / (1 + e^(-(-0.0375 + -1.043*1.4655+ -0.0956*2.3621)))

• Alexandre CHIROUZE March 9, 2017 at 3:05 am #

You are right, but you missing the understanding of one epoch.

The first time, i did like you did : Updating 10 times b0, b1 and b2 and i did not have the result expected.

You have to update 10 * 10 times your b0, b1 and b2 :

One epoch is a complet turn of your data training.

Do what you did in your post, but repeat it 10 times and you’ll have the same result as the tutorial.

I did a C Program, I post the code here :

#include
#include
#include

float x1[10] = {2.7810836,
1.465489372,
3.396561688,
1.38807019,
3.06407232,
7.627531214,
5.332441248,
6.922596716,
8.675418651,
7.673756466
};
float x2[10] = {2.550537003,
2.362125076,
4.400293529,
1.850220317,
3.005305973,
2.759262235,
2.088626775,
1.77106367,
-0.2420686549,
3.508563011
};
int y_1[10] = {0,0,0,0,0,1,1,1,1,1};

float b0 = 0.00f;
float b1 = 0.00f;
float b2 = 0.00f;

float b[3];

float prediction = 0;
float output = 0;

int main()
{
int epoch = 0;
float alpha = 0.3;

while (epoch < 10)
{
int i = 0;

while (i < 10)
{
// CALCULATE PREDICTION
output = b0 + (b1 * x1[i]) + (b2 * x2[i]);
prediction = 1/(1 + exp(-output));
printf("Prediciton = %lf\n", prediction);

// AFFINEMENT DES COEFFICIENTS
b0 = b0 + alpha * (y_1[i] – prediction) * prediction * (1 – prediction) * 1.00f;
b1 = b1 + alpha * (y_1[i] – prediction) * prediction * (1 – prediction) * x1[i];
b2 = b2 + alpha * (y_1[i] – prediction) * prediction * (1 – prediction) * x2[i];
printf("New : b0 = %lf | b1 = %lf | b2 = %lf\n\n", b0, b1, b2);
i++;
}
epoch++;
}

int i = 0;
while (i < 10)
{
output = b0 + (b1 * x1[i]) + (b2 * x2[i]);
prediction = 1/(1 + exp(-output));
printf("Prediciton = %lf\n", prediction);
i++;
}

return 0;
}

• Max August 25, 2019 at 9:45 pm #

Dear Jason and Alexander,

I have tried to execute the C code but in MATLAB, however, I got totally different results! Any idea why?

I posted the MATLAB code below, please note that each epoch is saved to the prediction(i,j) array column by column

clear all,

X1 = [2.7810836, 1.465489372, 3.396561688, 1.38807019, 3.06407232, 7.627531214, 5.332441248, 6.922596716, 8.675418651, 7.673756466];

X2 = [2.550537003, 2.362125076, 4.400293529, 1.850220317, 3.005305973, 2.759262235, 2.088626775, 1.77106367, -0.2420686549, 3.508563011];
y = [0,0,0,0,0,1,1,1,1,1];

%initialize the variables
b0=0;
b1=0;
b2=0;
epochs =10 ; % epochs number ,, how many times it will repeat before selecting the final values of b0…bn
alpha=0.3; % the learning rate
prediction=[]; % an array for the predictions per each iteration per each data entry

for i = 1:epochs %the outer loop where it will work per each entry in the dataset

for j = 1:length (X1)

pred =(1/(1+ exp (- (b0+ (b1* X1(i)) + (b2*X2(i))))));

b0 = b0 + alpha * (y(i) – pred) * pred* (1 – pred) * 1;
b1 = b1 + alpha * (y(i) – pred) * pred* (1 – pred) * X1(i);
b2 = b2 + alpha * (y(i) – pred) * pred* (1 – pred) * X2(i);

display (b0);
display (b1);
display (b2);

prediction(i,j)=pred;
end

end
disp (‘____The Predictions are__________’);
for i= 1:length (X1)
pred =(1/(1+ exp (- (b0+ (b1* X1(i)) + (b2*X2(i))))));
disp (pred);
end

• Max August 26, 2019 at 1:46 am #

Solved it!

I had to replace the i in the inner loops with j

Error part:
pred =(1/(1+ exp (- (b0+ (b1* X1(i)) + (b2*X2(i))))));

b0 = b0 + alpha * (y(i) – pred) * pred* (1 – pred) * 1;
b1 = b1 + alpha * (y(i) – pred) * pred* (1 – pred) * X1(i);
b2 = b2 + alpha * (y(i) – pred) * pred* (1 – pred) * X2(i);

Correct part:
pred =(1/(1+ exp (- (b0+ (b1* X1(j)) + (b2*X2(j))))));
b0 = b0 + alpha * (y(j) – pred) * pred * (1 – pred) * 1;
b1 = b1 + alpha * (y(j) – pred) * pred * (1 – pred) * X1(j);
b2 = b2 + alpha * (y(j) – pred) * pred * (1 – pred) * X2(j);

• Jason Brownlee August 26, 2019 at 6:22 am #

Well done, happy to hear that!

• Jason Brownlee August 26, 2019 at 6:13 am #

Perhaps the difference in precision across the platforms/libraries? If so, this is common.

• Jason Brownlee February 17, 2017 at 9:59 am #

Hi Jack,

Yes, the process is repeated, enumerating each training example. The coefficients (b0, b1, b2) output from the previous iteration are used as inputs in the subsequent iteration.

Does that help? Is there a specific aspect I can make clearer?

I provide the fully working example and spreadsheet with my book if that helps.

10. Ntate Ndaba March 10, 2017 at 11:03 pm #

Hi Jason,

Thanks for this, it’s really helpful. I just have two questions.

1. About the meaning of probability,

In the beginning you said, ‘If the probability is > 0.5 we can take the output as a prediction for the default class (class 0)’; however the IF statement states that ‘prediction = IF (output 0.5 means that the instance belongs to class 0. May you please correct me :).

2. ‘At the end of epoch you can calculate error values for the model’

Do we do this by testing each instance in the training set using coefficients obtained from the last instance during training, and check how many are correctly classified vs those incorrectly classified to get the error error of the model ?.

• Jason Brownlee March 11, 2017 at 8:00 am #

Hi Ntate,

Yes, here the default class is class 1. If P > 0.5, then predict class 1.

Models are generally trained on a training dataset and evaluated on a test or validation dataset.

Does that help?

11. Ntate Ndaba March 12, 2017 at 11:06 am #

Hi Jason

Thank you for the feedback. Yes it helps. I love you articles.

Regards,
Ntate

12. Simon April 19, 2017 at 12:48 pm #

Hi Jason,

I noticed you assigned alpha as 0.3 and stated that a “good” value is between 0.1 – 0.3.

b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x

Can you define why the range between 0.1 – 0.3 are good values? also, is there another way to determine alpha?

regards,

• Jason Brownlee April 20, 2017 at 9:21 am #

Trial and error is the best away to configure alpha.

My suggestion is based on observations.

13. Rohan July 9, 2017 at 8:43 pm #

Thank you.Helepd me in understanding it easily

• Jason Brownlee July 11, 2017 at 10:16 am #

Thanks, I’m glad to hear that.

14. Monjur July 10, 2017 at 3:47 am #

I’ve some categorical and ordinal variables as independent variables. Do I need to normalise them for logistic regression? If yes, which normalisation?

• Jason Brownlee July 11, 2017 at 10:21 am #

I would recommend converting the categorical variables to integer encoded or one hot encoded first. Then consider standardizing or normalizing the real-valued variables.

15. Robert July 24, 2017 at 4:42 am #

Hey Jason,

I hope that you can help me out. I just loaded the dataset into a pandas dataframe and run over the values with the given functions in this article. But somehow I do not get the same results. Can you tell me why?

b0 = 0.0
b1 = 0.0
b2 = 0.0
alpha = 0.3
for i in range(10):
print(‘epoche ‘+str(i))
for j in range(10):
prediction = 1/(1+math.exp(-(b0+b1*df.X1[j]+0*df.X2[j])))
b0 = b0 + alpha*(df.Y[j]-prediction)*prediction*(1-prediction)*1
b1 = b1 + alpha*(df.Y[j]-prediction)*prediction*(1-prediction)*df.X1[j]
b2 = b2 + alpha*(df.Y[j]-prediction)*prediction*(1-prediction)*df.X2[j]
print(j,’b0:’+str(b0),’b1:’+str(b1),’b2:’+str(b2),’prediction:’,str(prediction))

epoche 0
0 b0:-0.0375 b1:-0.104290635 b2:-0.0956451376125 prediction: 0.5
1 b0:-0.0711363539015 b1:-0.153584354155 b2:-0.175098412628 prediction: 0.4525589342855867
2 b0:-0.0956211308566 b1:-0.2367484095 b2:-0.282838618223 prediction: 0.3559937888383731
3 b0:-0.12398808955 b1:-0.276123739244 b2:-0.335323741529 prediction: 0.3955015163005356
4 b0:-0.140423954937 b1:-0.326484419432 b2:-0.384718545948 prediction: 0.27487029790669615
5 b0:-0.122884891209 b1:-0.192704663377 b2:-0.336323669765 prediction: 0.06718893790753155
6 b0:-0.0812720241683 b1:0.0291935052754 b2:-0.24940992148 prediction: 0.24040302936619257
7 b0:-0.0461629882724 b1:0.27223920187 b2:-0.187229583516 prediction: 0.5301690178124752
8 b0:-0.0439592934139 b1:0.291357177347 b2:-0.187763028966 prediction: 0.9101629406368437
9 b0:-0.041234497312 b1:0.312266599052 b2:-0.178202910151 prediction: 0.8995147707414556
epoche 1
0 b0:-0.0854175615843 b1:0.189389803607 b2:-0.290893450483 prediction: 0.6957636219000934
1 b0:-0.126132084903 b1:0.129723102398 b2:-0.387066246971 prediction: 0.5478855806515489
2 b0:-0.168426127803 b1:-0.013931223349 b2:-0.573172450262 prediction: 0.5779785060018552
3 b0:-0.202118039235 b1:-0.0606979612511 b2:-0.635509909311 prediction: 0.4531965138273989
4 b0:-0.231317727075 b1:-0.150167916514 b2:-0.723263905585 prediction: 0.40417453404597653
5 b0:-0.192771357535 b1:0.143845720337 b2:-0.616904363818 prediction: 0.20153497961346012
6 b0:-0.16786327908 b1:0.2766665853 b2:-0.564880684243 prediction: 0.6397495977782179
7 b0:-0.162238550103 b1:0.315604315639 b2:-0.554918931099 prediction: 0.8516230388309892
8 b0:-0.160844460322 b1:0.327698628134 b2:-0.555256396538 prediction: 0.9292852140544192
9 b0:-0.158782126878 b1:0.34352447273 b2:-0.548020569702 prediction: 0.913238571936509
epoche 2
0 b0:-0.203070195453 b1:0.220355651541 b2:-0.660978927394 prediction: 0.6892441807751586
1 b0:-0.242672456027 b1:0.162318959562 b2:-0.754524420162 prediction: 0.5299288459930025
2 b0:-0.284900459421 b1:0.0188889410738 b2:-0.940340030239 prediction: 0.5765566602275273
3 b0:-0.317036444837 b1:-0.0257180623077 b2:-0.999798683361 prediction: 0.43568790553413944
4 b0:-0.346058130501 b1:-0.114642606031 b2:-1.08701772863 prediction: 0.4023126069119149
5 b0:-0.305303900258 b1:0.196211557251 b2:-0.974566120208 prediction: 0.22784879089424026
6 b0:-0.284135730542 b1:0.309089578586 b2:-0.930353714163 prediction: 0.6772107084944106
7 b0:-0.279392082974 b1:0.341927937662 b2:-0.921952412292 prediction: 0.8647793818147672
8 b0:-0.278250718608 b1:0.351829751376 b2:-0.922228700829 prediction: 0.9362537346812636
9 b0:-0.276418730838 b1:0.365887979368 b2:-0.915801056304 prediction: 0.9184600339928451
epoche 3
0 b0:-0.320829240308 b1:0.242378639814 b2:-1.02907170403 prediction: 0.6772464757372858
1 b0:-0.358962425423 b1:0.186494862307 b2:-1.11914705682 prediction: 0.5085926740259167
2 b0:-0.400784178807 b1:0.0444446970373 b2:-1.30317504761 prediction: 0.5681921310493052
3 b0:-0.431106414978 b1:0.00235530491457 b2:-1.35927786503 prediction: 0.4160301017601509
4 b0:-0.459481564543 b1:-0.0845882054421 b2:-1.4445538715 prediction: 0.39558638097213306
5 b0:-0.417358455079 b1:0.236707126821 b2:-1.32832516633 prediction: 0.24886389211209164
6 b0:-0.398407914119 b1:0.337759773107 b2:-1.28874455908 prediction: 0.6994895638042464
7 b0:-0.394265226466 b1:0.36643792905 b2:-1.28140759549 prediction: 0.8743265198068555
8 b0:-0.393309643975 b1:0.374728007212 b2:-1.28163891205 prediction: 0.9418454578963239
9 b0:-0.391663361645 b1:0.387361176894 b2:-1.27586282676 prediction: 0.9228889145563345
epoche 4
0 b0:-0.436106966113 b1:0.26375979738 b2:-1.38921788451 prediction: 0.6649919679911412
1 b0:-0.472655267001 b1:0.210198650866 b2:-1.47554954252 prediction: 0.4876100903262518
2 b0:-0.514052259808 b1:0.0695912110969 b2:-1.65770846209 prediction: 0.5600333525763922
3 b0:-0.542575863527 b1:0.029998447064 b2:-1.7104834132 prediction: 0.39712596044478815
4 b0:-0.570332853627 b1:-0.0550509779888 b2:-1.79390166134 prediction: 0.38920422454205744
5 b0:-0.52713220805 b1:0.274463294611 b2:-1.67469975148 prediction: 0.2708654841988902
6 b0:-0.510039576896 b1:0.365608746016 b2:-1.63899962439 prediction: 0.7183774008591356
7 b0:-0.506411991601 b1:0.390721056064 b2:-1.63257493987 prediction: 0.8829763460926162
8 b0:-0.505614321279 b1:0.397641180051 b2:-1.63276803085 prediction: 0.9470125245639989
9 b0:-0.504143617348 b1:0.408927003853 b2:-1.62760797344 prediction: 0.9272899884165315
epoche 5
0 b0:-0.548534355846 b1:0.285472649024 b2:-1.74082819456 prediction: 0.6531957979683319
1 b0:-0.583448848755 b1:0.234305830737 b2:-1.82330059378 prediction: 0.4675015748559041
2 b0:-0.624452043388 b1:0.095035950763 b2:-2.00372668579 prediction: 0.5528976488885597
3 b0:-0.651241447361 b1:0.0578503777005 b2:-2.0532929853 prediction: 0.379296457131698
4 b0:-0.678459443818 b1:-0.0255475318494 b2:-2.13509139263 prediction: 0.38367378819364967
5 b0:-0.634483421462 b1:0.309880951336 b2:-2.0137500149 prediction: 0.29456310937959235
6 b0:-0.618957686991 b1:0.392671018235 b2:-1.98132255018 prediction: 0.734570858647913
7 b0:-0.615773112207 b1:0.414716545178 b2:-1.97568246547 prediction: 0.8908395402598338
8 b0:-0.615108589413 b1:0.420481558613 b2:-1.97584332561 prediction: 0.9517573456578646
9 b0:-0.613801104729 b1:0.430514877661 b2:-1.97125593321 prediction: 0.9316021522525871
epoche 6
0 b0:-0.658065683935 b1:0.307411382371 b2:-2.0841543804 prediction: 0.6418716134977803
1 b0:-0.691328362642 b1:0.258665280242 b2:-2.16272498787 prediction: 0.4482960919573821
2 b0:-0.731971685172 b1:0.120617728064 b2:-2.34156753699 prediction: 0.5466747645320021
3 b0:-0.757102763343 b1:0.0857340276128 b2:-2.38806556841 prediction: 0.3624963002614126
4 b0:-0.783848933768 b1:0.00378182714664 b2:-2.46844599415 prediction: 0.3788558237431147
5 b0:-0.739460741679 b1:0.342354147839 b2:-2.34596733203 prediction: 0.3197321661982434
6 b0:-0.725177602262 b1:0.418518149617 b2:-2.31613518462 prediction: 0.747650886073883
7 b0:-0.722359808144 b1:0.438024601926 b2:-2.31114469182 prediction: 0.8977118003459105
8 b0:-0.721803648551 b1:0.442849519233 b2:-2.31127932063 prediction: 0.9559629146792477
9 b0:-0.72064049539 b1:0.451775273321 b2:-2.30719832447 prediction: 0.935626585073733
epoche 7
0 b0:-0.764713491327 b1:0.329204587117 b2:-2.41960813144 prediction: 0.630831196571219
1 b0:-0.796322040457 b1:0.282882594302 b2:-2.49427147796 prediction: 0.4298979150158073
2 b0:-0.836626296602 b1:0.145986702019 b2:-2.67162203546 prediction: 0.54103403758093
3 b0:-0.86017519983 b1:0.113299171441 b2:-2.71519269466 prediction: 0.34660715971969225
4 b0:-0.886491802455 b1:0.0326631977814 b2:-2.79428213772 prediction: 0.37448622204910614
5 b0:-0.842093735283 b1:0.371310840978 b2:-2.67177622766 prediction: 0.34584531374424465
6 b0:-0.828710920588 b1:0.442673914071 b2:-2.64382452257 prediction: 0.7572937776171192
7 b0:-0.826182553165 b1:0.460176782093 b2:-2.63934662288 prediction: 0.9034135590263473
8 b0:-0.825711234308 b1:0.464265670495 b2:-2.6394607144 prediction: 0.9595362539490104
9 b0:-0.82466874439 b1:0.472265484239 b2:-2.63580307284 prediction: 0.9391721159610058
epoche 8
0 b0:-0.868485382532 b1:0.350407750494 b2:-2.74755902977 prediction: 0.6198098107703024
1 b0:-0.898445083971 b1:0.306502126447 b2:-2.8183275918 prediction: 0.4121785664149056
2 b0:-0.938410924898 b1:0.170755682328 b2:-2.99418902301 prediction: 0.535591774903723
3 b0:-0.960450226361 b1:0.140163584958 b2:-3.03496658635 prediction: 0.3315041073863981
4 b0:-0.986352835601 b1:0.0607961169703 b2:-3.11281185262 prediction: 0.37028861216547454
5 b0:-0.942344793064 b1:0.396468835092 b2:-2.99138212281 prediction: 0.37223779515431543
6 b0:-0.92953081713 b1:0.464798608913 b2:-2.96461850958 prediction: 0.7634705770276337
7 b0:-0.927219758977 b1:0.480797132495 b2:-2.96052547844 prediction: 0.9078852161314516
8 b0:-0.926812709449 b1:0.484328457556 b2:-2.96062401238 prediction: 0.9624531770487973
9 b0:-0.925865930278 b1:0.491593810342 b2:-2.957302178 prediction: 0.9421224630749467
epoche 9
0 b0:-0.96935657639 b1:0.370642687688 b2:-3.06822668019 prediction: 0.6085681683087178
1 b0:-0.997678784367 b1:0.329136792905 b2:-3.13512727786 prediction: 0.39503800634778957
2 b0:-1.03728747577 b1:0.19460342917 b2:-3.30941714634 prediction: 0.53002748178722
3 b0:-1.05788693505 b1:0.166009933818 b2:-3.34753068441 prediction: 0.3170928534429609
4 b0:-1.08336985067 b1:0.087928437422 b2:-3.42411464294 prediction: 0.3660452797076954
5 b0:-1.0401081702 b1:0.417908255596 b2:-3.30474432179 prediction: 0.39826657846895
6 b0:-1.02756606041 b1:0.484788319168 b2:-3.27854853547 prediction: 0.7664481691662279
7 b0:-1.0254106916 b1:0.499709068196 b2:-3.27473124008 prediction: 0.9112042239475246
8 b0:-1.02505131574 b1:0.502826804247 b2:-3.27481823372 prediction: 0.9647626216394803
9 b0:-1.02417732025 b1:0.509533632813 b2:-3.27175176546 prediction: 0.9444604863065394

• Robert July 24, 2017 at 4:51 am #

I fixed it myself.

error: prediction = 1/(1+math.exp(-(b0+b1*df.X1[j]+0*df.X2[j])))
info: 0 = b2

correction: prediction = 1/(1+math.exp(-(b0+b1*df.X1[j]+b2*df.X2[j])))

Thx for this great article.

• Jason Brownlee July 24, 2017 at 6:57 am #

Machine learning algorithms are stochastic and give different results each time they are run on different platforms, even sometimes when the random seed is fixed.

16. vidya pati July 28, 2017 at 1:20 am #

I see all the logistic regression has below formula for updating weight

wj:=wj+η∑i=1n(y(i)−ϕ(z(i)))x(i

But in this post we have this formula

b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x

how is these two formula different??

17. NweWin August 31, 2017 at 10:13 pm #

First if all, thanks you so much for this blog. This really helps me to understand what and how regression works which is my thesis topic. I have more thing to know about How to predict the class ( 0 or 1) for new data (new x1 and new x2) in this example. Please kindly give me sample calculations for prediction of class value for new data by learning trained and valid datasets. Thanks you so much in advance.

• Jason Brownlee September 1, 2017 at 6:47 am #

Multiply each weight/coefficient by the input value and add the values together to get the prediction.

18. Jannah October 14, 2017 at 7:00 pm #

Hi Jason,

Just to say thank you so much for your awesome tutorial!
Now i clearly understand about implementing logistic regression in machine learning.

I shared your article on my FB 🙂

19. Searene October 30, 2017 at 8:14 am #

Wrote one using Scala, hopes it can help somebody.

package com.learn

object LogisticRegression {
def main(args: Array[String]): Unit = {
val input = Array(
(2.7810836, 2.550537003),
(1.465489372, 2.362125076),
(3.396561688, 4.400293529),
(1.38807019, 1.850220317),
(3.06407232, 3.005305973),
(7.627531214, 2.759262235),
(5.332441248, 2.088626775),
(6.922596716, 1.77106367),
(8.675418651, -0.2420686549),
(7.673756466, 3.508563011))
val output: Array[Double] = Array(0, 0, 0, 0, 0, 1, 1, 1, 1, 1)
val b: (Double, Double, Double) = (0, 0, 0)
val (b0, b1, b2) = getCoefficients(input, output, b, 0.3, 10)
}
def getCoefficients(
input: Array[(Double, Double)],
output: Array[Double],
b: (Double, Double, Double),
alpha: Double,
iterationCount: Int): (Double, Double, Double) = {
var remainingIterationCount = iterationCount
var b0 = b._1
var b1 = b._2
var b2 = b._3
while(remainingIterationCount > 0) {
remainingIterationCount -= 1
for(((x1, x2), y) <- input zip output) {
val prediction = getPrediction(b0, b1, b2, x1, x2)
b0 = getNextB(b0, 1, y, prediction, alpha)
b1 = getNextB(b1, x1, y, prediction, alpha)
b2 = getNextB(b2, x2, y, prediction, alpha)
}
println(s"b0: $b0, b1:$b1, b2: $b2") println(s"accuracy:${getAccuracy((b0, b1, b2), input, output)}")
}
(b0, b1, b2)
}
private def getPrediction(b0: Double, b1: Double, b2: Double, x1: Double, x2: Double): Double = {
1.0 / (1 + Math.exp(-1 * (b1 * x1 + b2 * x2)))
}
private def getNextB(b: Double, x: Double, y: Double, prediction: Double, alpha: Double): Double = {
b + alpha * (y – prediction) * prediction * (1 – prediction) * x
}
private def getAccuracy(b: (Double, Double, Double), input: Array[(Double, Double)], output: Array[Double]): Double = {
var correctOutputCount = 0
for(((x1, x2), y) <- input zip output) {
val output = if(getPrediction(b._1, b._2, b._3, x1, x2) < 0.5) 0 else 1
if(output == y) correctOutputCount += 1
}
correctOutputCount * 1.0 / input.length
}
}

• Jason Brownlee October 30, 2017 at 3:49 pm #

Very cool, thanks for sharing Searene!

20. Aminu Abdulsalami November 9, 2017 at 9:47 pm #

The tutorial was great. I was just wondering if i could get optimal values by applying a meta heuristic algorithm (particle swam optimization, cuckoo search algorithm) instead of gradient descent in the learning coefficients of logistic regression. What is your take on this…

• Jason Brownlee November 10, 2017 at 10:34 am #

Perhaps not optimal, but perhaps improved.

This would be a step above random search or grid search, and only suitable if you have the resources.

• Aminu Abdulsalami November 11, 2017 at 4:13 am #

21. Glimmer November 21, 2017 at 7:41 pm #

What software are you using for the analysis?

22. Aymen December 11, 2017 at 5:33 am #

you make a great job. Thank you

23. Chinnathambi A December 27, 2017 at 7:30 pm #

Hi sir,
How to use Logistic Regression for Machine Learning.
how to calculate simple Logistic Regression for Machine Learning

• Jason Brownlee December 28, 2017 at 5:22 am #

Does the above tutorial help?

24. loz January 26, 2018 at 9:55 am #

Hi there. Would it be the same concept if I had values from 1-24 (X axis) and as the values increased to 24, the Y axis increased to 1. I tried doing that but got funny results. You can check out my post here:

https://stats.stackexchange.com/questions/325101/error-in-calculating-the-logistic-regression-using-sgd

• Jason Brownlee January 27, 2018 at 5:50 am #

• Tarikh Chouhan January 27, 2018 at 9:38 pm #

What I mean is that if my data had numbers from 1 to 24 (representing the X axis) and numbers from 1-12 were 0 and numbers 13-24 where 1, would I use the same concept to calculate the sgd of this model?

25. JTB February 18, 2018 at 2:07 pm #

Above, Dylan asked how to obtain the line separating the two data sets. Here is one way to do this, which also gives some insight into the logistic regression method.

Imagine the line separating the two sets of points given in the example. This is the discriminant line. If a point lies on the line, the probability that it belongs to one set is the same as that of it belonging to the other set i.e., if it is on the discriminant line then the probability that it belongs to either set is 0.5.

We see from the logit function that it has the value of 0.5 when the exponent in the denominator is zero, i.e. when

B0 +B1*X1+B2*X2 is zero.

We have values for B0, B1 and B2, and we have two unknowns, i.e. the slope and intercept of the discriminant line. We merely substitute in a couple of convenient values for X1, compute the corresponding X2 and use freshman year algebra to compute the equation of the line. It is convenient to use X1=0 and X1=1 for this purpose. If we do the manipulations symbolically, we find that the X2 intercept is

-B0/B2

and the slope is

-B1/B2.

Therefore, for the values given as the result of ten epochs, the equation of the discriminant line is

x2 = 0.7718*X1 – 0.348

If you look at the graph and correlate points with their associated “predictions” from the logistic regression model, you will see that points closer to the discriminant line have values closer to 0.5 than those farther away, which makes sense.

26. Hilman May 21, 2018 at 4:15 am #

hi jason, i want to ask what value of Y is it? is it assumed its value is 0 or how?

• Jason Brownlee May 21, 2018 at 6:35 am #

Sorry, I don’t follow. What do you mean exactly?

27. Emiline May 24, 2018 at 4:14 pm #

Is the first prediction line always with coefficients initialized to 0? How does one guess a model that isn’t an initialization of 0?

• Jason Brownlee May 25, 2018 at 9:18 am #

Zero or small random numbers are a good starting point for SGD.

What do you mean exactly?

28. samreen May 31, 2018 at 11:35 pm #

how code implementation for a problem if a student has English marks greater than 40 belongs to class 1 otherwise 0 using logistic regression .

29. Priyanka August 8, 2018 at 7:30 pm #

Hi Jason ,

I have written this logistic regression algorithm in Java.I am getting proper output for all the 9 records out of 10.But for the 6th record I am getting as 0 instead of 1 as output. Below is the code

• Jason Brownlee August 9, 2018 at 7:36 am #

Nice work!

• Himanshu November 26, 2018 at 5:26 pm #

Hi Priyanka,

Could you please explain this work for multiple inputs.

Example: Could you please explain this for Titanic data.

30. Nor Azziaty September 15, 2018 at 2:23 pm #

Hi, I am using a Logistic Regression Model in Rapid Miner. I have tried to improve the accuracy of the logistic regression model but failed. Can you suggest any steps or ways how i can improve the accuracy (in confusion matrix).

31. Abhishek Vigg December 18, 2018 at 9:18 pm #

Hello
I am implementing a logistic regression model on a classification problem that deals with Churn Prediction. The final predictor indicates if a customer churns or not in terms of “Yes” or “No”. How can I find a threshold for such a logistic regression model that already predicts “Yes” or “No” but does not give out probabilities?

• Jason Brownlee December 19, 2018 at 6:33 am #

The model actually probabilities that are interpreted as labels. You can use the probabilities directly.

• KYT March 8, 2020 at 10:14 pm #

How do classify review texts with logistic regression?
How to calculate weight and bias?

• Jason Brownlee March 9, 2020 at 7:15 am #
• KYT March 15, 2020 at 3:26 pm #

Please classify this example with logistic regression (no coding).

Chinese Beijing Chinese, yes
Chinese Chinese Shanghai, yes
Chinese Macao, yes
Tokyo Japan Chinese, no
Chinese Chinese Chinese Tokyo Japan, ?

• Jason Brownlee March 16, 2020 at 5:52 am #

Sorry, I don’t have the capacity to prepare a custom example for you.

32. Aimal December 19, 2018 at 11:35 pm #

Great blog !
Thank you very much.

• Jason Brownlee December 20, 2018 at 6:25 am #

33. Harry February 1, 2019 at 11:07 am #

34. Claret February 20, 2019 at 3:02 am #

I’m have a problem understanding the process
I tried repeating the process, i started fine but did not have the same output as you
Here’s my table

35. Joe March 5, 2019 at 8:09 am #

Thank you so much Jason.
I was just using your data and tried the glm command in R with family of binomial.
I got a warning message like this:
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

summary(mylogit)
Call:
glm(formula = Y ~ X1 + X2, family = “binomial”, data = Logi)

Deviance Residuals:
Min 1Q Median 3Q Max
-1.292e-05 -2.110e-08 0.000e+00 2.110e-08 1.623e-05

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -37.18 797468.21 0 1
X1 15.87 62778.64 0 1
X2 -11.83 235792.12 0 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 1.3863e+01 on 9 degrees of freedom
Residual deviance: 4.9893e-10 on 7 degrees of freedom
AIC: 6

I then changed one of the Y values from 0 to 1 and it worked (of course not the same values as yours)

I am just curious why it did not work the first time. Any help is much appreciated. Thanks!

• Jason Brownlee March 5, 2019 at 2:20 pm #

I’m not sure, perhaps it was dependent on the specific data in the dataset?

36. Gopal April 2, 2019 at 12:10 am #

Hello Jason, Does separating hyper plane in logistic regression guarantees to be optimal one like in the case of SVM…?

• Jason Brownlee April 2, 2019 at 8:14 am #

Yes, under a framework of maximum likelihood.

But its based on the specific data, and such claims don’t mean much (to me) in the face of real results.

• Gopal April 30, 2019 at 2:07 am #

Thanks for the reply Jason. So with maximum likelihood method logistic regression is better when compare to svm as there is no need of support vectors so training will be fast and so prediction also..? please help here

• Jason Brownlee April 30, 2019 at 7:01 am #

37. Waseem Hassan May 1, 2019 at 2:11 am #

Sir Jason Brownlee wonderful blog !

I have a query. I have training dataset, i will train model using logistic regression, but how to use trained model for testing dataset ?

38. Chinmay May 17, 2019 at 12:13 am #

is there a relation between ‘c’ which is used in scikit-learn and learning rate

39. Dr. D. Lakshmi August 9, 2019 at 6:43 pm #

Dear Jason Brownlee ML Master,
I am really thankful for your great writings. You are awesome in writing and explaining things in understandable manner.

I request you to give a little explanation for the below equation for the parameters alpha/learning rate and x/intercept. If possible insight of the entire equation.
Old: b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x
Bit more explanation is required why do we choose alpha value between 01. to 0.3 and intercept x into 1.0.

Thank You.

• Jason Brownlee August 10, 2019 at 7:13 am #

The specific value were arbitrary for the example.

When using gradient descent to optimize logistic regression, it may be a good idea to test different alpha values.

Please don’t make videos from my blog posts.

40. Maxi August 30, 2019 at 10:30 pm #

Dear Jason,

Congrats on the well-written blog, I am amazed by how you simplify things and keep replying to comments on this blog. well done!

I have three mini questions that am hoping you have time to:

1- Would you please clarify that?
“variation of GD called SGD.In this variation, the gradient descent procedure described above is run but the update to the coefficients is performed for each training instance, rather than at the end of the batch of instances.”

2- Another step described here from the same post above:

“The first step of the procedure requires that the order of the training dataset is randomized. This is to mix up the order that updates are made to the coefficient”

Do we assume a randomization stage had already occurred on the dataset in the above tutorial?
Furthermore, if we are having unsequenced instances then what’s the point of randomizing already random instances?

3- What I knew (so far) is the difference between GD and SGD is that SGD takes several samples of the dataset and fits the model using those samples, while the GD go through the entire dataset updating the coefficients on each entry.

if the above sentence is valid, then shouldn’t we have (part) of the 10-entry dataset above used to fit the model?. or maybe call it a Logistic Regression using Gradient Descent?

• Jason Brownlee August 31, 2019 at 6:09 am #

Thanks.

Batch means updating at the end of the epoch and online means updating after every sample. Sometimes online is referred to as SGD. Sometimes SGD means the general method regardless of the update procedure.

It is a good idea to shuffle the training set prior to each epoch. Random order ensure we get different weight updates in a different order each iteration – avoids getting stuck and oscillating the weights in a worst case.

This might give you more ideas:
https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/

41. 01EG September 1, 2019 at 10:44 pm #

Dr. Brownlee, fanks a lot for your great tutorial!

42. Hkn Erdun October 24, 2019 at 1:20 am #

Do you have a working full C or C++ programme ?

• Jason Brownlee October 24, 2019 at 5:39 am #

I don’t have examples in C or cpp, sorry.

43. Max November 12, 2019 at 12:06 am #

Hello Jason,

Please forgive my ignorance but I am somewhat new to this and a little bit confused here,

So we are here using Stochastic Gradient Descent to estimate the coefficient values (b0 –bn)
Then what about the maximum likelihood estimate?
Is the SGD part of the max. likelihood estimate? or this is another, alternative, method?

• Jason Brownlee November 12, 2019 at 6:41 am #

No, we are “doing MLE”. SGD provides an (inefficient) way to navigate the search space for MLE.

A more efficient approach is to use a quadratic optimization algorithm, e.g. like the sklearn implementation.

44. Sam December 18, 2019 at 2:50 pm #

–Hi Jason,
Thank you for a very clear tutorial on Linear Logistic Regression.
I have a question. When I enter the data you have in the tutorial:

X1 X2 Y
2.7810836 2.550537003 0
1.465489372 2.362125076 0
……
in https://statpages.info/logistic.html
we get very different values for the betas (from what you have in the tutorial):

Variable Coeff. StdErr p O.R. Low — High
1 13.308016029.9345 0.9993 01984.5565 0.0000
2 -12.583472148.4146 0.9999 0.0000 0.0000
Intercept -24.868530306.4324 0.9999

Any explanation would be very highly appreciated.
Thanks

• Jason Brownlee December 19, 2019 at 6:17 am #

You’re welcome.

Yes, here we solve it using an inefficient optimization algorithm – sgd.

• Sami Khuri December 19, 2019 at 9:40 pm #

–Jason,
You mean it is OK to have one method (your tutorial) give:
b0 = -0.4066054641
b1 = 0.8525733164
b2 = -1.104746259

And the other (https://statpages.info/logistic.html):
Intercept (b0) = -24.87
b1 = 13.31
b2 = -12.58

Thanks.
–Sam

• Jason Brownlee December 20, 2019 at 6:45 am #

For learning purposes, sure.

45. Maxi January 15, 2020 at 5:53 am #

Hello Dr. Brownlee,

well done, this tutorial is really helping me a lot!

My question to you is:

looking into the coefficient update formula (b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x ) I wonder if this is the same one used by sklearn? if not, can you please highlight which one?
Also, is this the same formula they refer to it as the L1 &L2 regularization?

It would help me a lot if you can please recommend a reference.

Thanks
Max

46. Angelica Cruz February 25, 2020 at 6:19 am #

Hi!
I’m new to machine learning and I found your post very helpful! it wasn’t understanding the formula given by my instructor.

You should make it clearer that steps 1 and 2 must be done 10 times over the small dataset that we have!

47. Sac_Mit April 8, 2020 at 10:05 pm #

Hi Jason,

This is a beautiful post indeed!
Can you help me understand if,

b = b + alpha * (y – prediction) * prediction * (1 – prediction) * x

is applicable to tune coeffiecients for all linear models like logistic regression and if not how to do it for non linear and other complex models.

Also, How to best estimate alpha and what ways it can be tuned.

• Jason Brownlee April 9, 2020 at 8:03 am #

Thanks.

Yes, you can train a logistic model using this update equation.

Test different alpha values.

48. Dean July 16, 2020 at 4:21 am #

Hi Jason,
If my independent variables is categorical type, means either 1 or 0 (yes or no), is that possible use this method to calculate the coefficient?
Currently i have 4 independent variables, 3 is categorical another one is continuous.

• Jason Brownlee July 16, 2020 at 6:47 am #

You can, but the method does assume real valued inputs. Test it and compare to other methods.

49. Sumant July 20, 2020 at 12:40 am #

Hi Jason,

After how many iterations for b0, b1, b2 we should stop?

Also how we are checking the accuracy for a particular b0,b1,b2 by putting into the transformed equation and checking with the actual result. I am bit of confused here

• Jason Brownlee July 20, 2020 at 6:15 am #

When error stops improving on the training dataset or a hold out dataset.

50. Nitin September 13, 2020 at 2:27 am #

Nice one, how to implement GLM in python, any resources on it?

51. zero January 12, 2021 at 5:45 pm #

hi jason,

can you explain 10 epoch in excel sheet to get the value
b0 = -0.4066054641
b1 = 0.8525733164
b2 = -1.104746259

i tried but getting different values
plz do needful…

52. Yen February 19, 2021 at 12:41 pm #

Hi Jason, I saw one of your reply “When error stops improving on the training dataset or a hold out dataset”
Can u explain how should we calculate the “error”?

53. Saviour April 22, 2021 at 6:31 am #

Hello Jason,
I have just followed the code and I managed to load it successfully.
How could plot the boundary line of these dataset
Thankyou


text:
Mathematics » Exponents and Surds » Applications Of Exponentials

# Applications of Exponentials

## Applications of Exponentials

There are many real world applications that require exponents. For example, exponentials are used to determine population growth and they are also used in finance to calculate different types of interest.

## Example

### Question

A type of bacteria has a very high exponential growth rate at $$\text{80}\%$$ every hour.If there are $$\text{10}$$ bacteria, determine how many there will be in five hours, in one day and in one week?

### Exponential formula

$\text{final population} = \text{initial population} \times (1 + \text{growth percentage})^{\text{time period in hours}}$

Therefore, in this case: $\text{final population} = \text{10}(\text{1.8})^n$ where $$n =$$ number of hours.

### In $$\text{5}$$ hours

final population = $$\text{10}(\text{1.8})^{\text{5}} \approx \text{189}$$

### In $$\text{1}$$ day = $$\text{24}$$ hours

final population = $$\text{10}(\text{1.8})^{\text{24}} \approx \text{13 382 588}$$

### In $$\text{1}$$ week = $$\text{168}$$ hours

final population = $$\text{10}(\text{1.8})^{\text{168}} \approx \text{7.687} \times \text{10}^{\text{43}}$$

Note this answer is given in scientific notation as it is a very big number.

## Example

### Question

A species of extremely rare deep water fish has a very long lifespan and rarely has offspring. If there are a total of $$\text{821}$$ of this type of fish and their growth rate is $$\text{2}\%$$ each month, how many will there be in half of a year? What will the population be in ten years and in one hundred years?

### Exponential formula

$\text{final population} = \text{initial population} \times (1+\text{growth percentage})^{\text{time period in months}}$

Therefore, in this case: $\text{final population} = \text{821}(\text{1.02})^n$ where $$n =$$ number of months.

### In half a year = $$\text{6}$$ months

$$\text{final population} = \text{821}(\text{1.02})^6 \approx \text{925}$$

### In $$\text{10}$$ years = $$\text{120}$$ months

$$\text{final population} = \text{821}(\text{1.02})^{\text{120}} \approx \text{8 838}$$

### In $$\text{100}$$ years = $$\text{1 200}$$ months

$$\text{final population} = \text{821}(\text{1.02})^{\text{1 200}} \approx \text{1.716} \times \text{10}^{\text{13}}$$

Note this answer is also given in scientific notation as it is a very big number.

format: plain


text:
# CFD Code Snippets—1: 1D transient conduction, FTCS, Dirichlet BCs

This semester, I am teaching an elective UG course on CFD. For illustrating ideas concerning the title problem, I began by borrowing the Python code written by Prof. Lorena Barba (general URL: [^]; the step concerning diffusion: [^] )), and then altered it a bit. I am sharing my modified version here.

In particular, here is a summary of my changes: (i) The Dirichlet BCs are now applied (so that time-stepping can go on for an indefinitely long period), (ii) graph plotting is done right within the time-marching loop, (iii) three different initial conditions can be used, etc. Variable names have been changed, and in fact, even comments have been expanded/corrected. (For instance, I use the term “thermal diffusivity” instead of“viscosity,” and use the actual material property value for it.)

This code can be used for the laboratory component (aka “Term-Work”) for the final-year elective course on CFD, in the S. P. Pune University.

This is the same University that denies me approval as a Professor, now because they think that I don’t have the required amount of work-experience (i.e. 10 years of both industrial, research and teaching experience put together).

The thing is, I didn’t always work for companies with annual turnover of 15 crores or more.

According to the PR guy of my employers, this university is now asking for my previous employers (e.g. startups in India/abroad) to certify in writing on their original letterheads not only the fact that they had employed me, but also to identify whether they had had 15 crores or more of annual turnover during the times when they had employed me or not. The rule is new, but applies retrospectively, i.e., it is supposed to go back even to the late 1990s or early ’00s times, when I worked in the software startups, sometimes with barely 1 or 2 employees too in their initial stages.

Most of these companies are by now wound up (e.g. QueriSoft grew to 500+ people whereas I was may be 13th/14th employee in Pune; nirWANa Inc. and Imagine Tech. where I was the fourth employee, and the second employee etc.—with nirWANa growing to US $1 Million in VC funding (coming from Kanwal Rekhi) but only after I left (at which point it had grown to 60+ employees)). I cannot have their certificates either because they can no longer be contacted, or even if the VC funding was US$ 1 Million, the sales turnover was not, etc.

Thus, the quantum of my total experience, according to the certifiable madness of the “authorities” at the S. P. University of Pune, suddenly dwindles down.

This rule, thus, allows the S. P. University of Pune to happily debar me from becoming a Professor.

It also allows employers to employ people like me not following the UGC scale, but on a consolidated, lower-level, salary.

I do not know precisely who among the two parties wishes to emphasize the rule more, to short-sell me. However, here, I am inclined to believe that it is the former. After all, it has been the government-run COEP and the (now S.P.) University of Pune who have happily supported people like Dr. Ashok Ghatol and Dr. (!) G. K. Kharate.

If professors who have worked nowhere else but in the private engineering colleges affiliated to the university rules are to be applied the same rule, then many of them would disqualify. This contradiction apparently does not matter to the University “authorities.” The rule is retrospective in action, but only for the new applicants, as per what our PR gentleman told me.

Anyway, I am going to share my code, but let me do so with a forthright moral denunciation of those office-bearers of the S. P. Pune University (and/or Maharashtra government) who are (or have been) responsible for framing such idiotic, harassing, anti-life and overall morally putrid rules. (For all I know, there could even be a casteist angle to it all; after all, it’s S. P. Pune University!)

Anyway, here is the Python code. It’s been written in v. 2.7 of Python. You will also need numpy and matplotlib installed. Feel free to use any IDE; I use PyCharm Community Edition.

#################################################################
# 1D Transient Conduction, using FTCS, with Dirichlet BCs.
# Based on the code by Prof. Lorena Barba (12 Steps to NS).
# Shared with an explicit identification of how morally putrid
# the S. P. Pune University is.
# Version: 27-Jan-2016.

#################################################################
# Importing Routines
# Use numpy arrays for efficiency
import numpy
# Use a neat plotting library
from matplotlib import pyplot  as plt

#################################################################
#  Program Data. All units are in SI
#
# Domain size, in meters
domLen = 0.1
# Number of cells
nc = 20
# Number of nodes
nx = nc + 1
# Compute the width of each cell
dx = domLen / nc
# Thermal diffusivity. This value is for Alumninium, in m^2/s
alpha = 8.418e-5
# Physical time over which the simulation is to be run
totalTime = 10.0
# Temperatures in degree Celcius, to use in constructing
# the initial and boundary conditions
tempLevel_1 = 0.0
tempLevel_2 = 100.0

# Parrot out the main input data
print("Length of the domain, in m: %lf" % domLen)
print("Number of cells: %d. Number of nodes: %d" % (nc, nx))
print("Therefore, the cell-width, in m: %lf" % dx)
print("Thermal diffusivity, in m^2/s: %lf" % alpha)
print("Duration (in physical seconds) for the simulation: %lf" % totalTime)
print("Temperature levels (deg. C): Level 1: %lf, Level 2: %lf" % (tempLevel_1, tempLevel_2))
# Main program data over
#################################################################

# Compute the duration of each time-step, and the number of time-steps required
# Stability criterion: r = \alpha \dfrac{\Delta t}{(\Delta x)^2}
# r <= 0.5 for stability
r = .35
print("Assumed stability criterion value: %lf" % r)
# Time-interval between steps, in seconds
dt = r * dx ** 2 / alpha
print("Duration of each time-step in s, after applying the stability criterion: %lf" % dt)
# Compute the total no. of time-steps (an integer) to fit within the given time-interval
nt = int(totalTime / dt)
print("Number of time-steps required to step through: %d" % nt)
# No. of time-steps to skip for plotting
stepsToSkipInPlotting = 10

# Create the numpy arrays, initialized to zero, for holding
# the primary variable values (here, temperature)

# The main array, holds the values after time-marching through one step
T = numpy.zeros(nx)
# A temporary array, holds the values at the previous time-step
Tn = numpy.zeros(nx)
print(T)
#################################################################
#  Setting up of the initial condition:
#
#  Uncomment the required IC and keep others commented

#########################
# IC1: A single pulse in the middle
# TL = tempLevel_1
# TR = tempLevel_1
# T[ nx/2 ] = tempLevel_2

#########################
# IC2: A step function in the middle
# TL = tempLevel_2
# TR = tempLevel_2
# T[ nx/3.0 : 2.0*nx/3.0  ] = tempLevel_1

#########################
# IC3: Left half at tempLevel_1; right half at tempLevel_2
TL = tempLevel_1
TR = tempLevel_2
T[0: nx / 2] = TL
T[nx / 2 + 1: nx] = TR

#
# Setting up of the initial condition is over
#################################################################

# Create the x-axis vector, for plotting purposes
#   numpy.linspace(start, stop, num, endpoint=True, retstep=False, dtype=None)
#   Returns evenly spaced numbers over a specified interval.
xAxis = numpy.linspace(0, domLen, nx)
plt.title("1D Transient Conduction, using FTCS")
plt.xlabel("Distance, in m")
plt.ylabel("Temperature, in deg C")
plt.plot(xAxis, T, 'r')

print("Initial condition is:")
print(T)

#################################################################
#  Time-stepping, using FTCS
#
# Multiplying factor to be used at each time-step
MF = alpha * dt / dx ** 2

# Python range(nSize) returns an array of size nSize
# Thus, range(3) returns [0,1,2]
for n in range(nt):
# We save the values of T at time n, to Tn,
Tn = T.copy()
# and apply the Dirichlet boundary conditions
# to the solution at the time-step n
Tn[0] = TL
Tn[nx - 1] = TR

# Then, we advance time to (n+1) and set T with it
# A note on the range() function, and the index values.
# -- The index to the right-most node in the arrays
#    T and Tn is: nx-1. If we specify range( nx-1 ),
#    then i will at most become _equal_ to nx-2
#    i.e., one less than what we specified.
for i in range(1, nx - 1):
# This is the FTCS algorithm
T[i] = Tn[i] + MF * (Tn[i - 1] - 2.0 * Tn[i] + Tn[i + 1])

# We skip stepsToSkipInPlotting' number of steps, in between two plots
if n and not (n % stepsToSkipInPlotting):
# Uncomment the following line for erasing out any earlier plots
# plt.clf()
# Uncomment the following line if you want to print T to console
# print( T )
plt.plot(xAxis, T, 'k')
plt.pause(0.1)

# The solution after nt number of time-steps
print("The solution after %d steps i.e. %lf physical seconds is: " % (nt, nt * dt))
print(T)
plt.plot(xAxis, T, 'b')
plt.show()

print("done")
#################################################################
#################################################################

Some things to try:

1. Try different values of the parameter r. For instance, set r = 0.55 and see the instability develop.
2. Try mesh refinement.
3. Try changing material properties.
4. Try different initial conditions. For instance, try triangular profile, or a sine-wave profile.
5. Try setting a higher temperature, and keep the r’ parameter very close to 0.5. See whether it is the initial condition of a triangle or of a sharp pulse which first develops instability. Try to understand the fact that the von Neumann stability analysis is only linear, and therefore provides only an initial estimate for the stability criterion.
6. Compare in detail with the analytical solutions. For instance, the analytical solution for the initial triangular profile is worked out in Kreyszig.
7. If inclined towards programming, try using fft routines to work out analytical solutions, have the program save both the FTCS and the FFT-based solutions to output files, and then write short scripts to plot the errors as functions of, e.g., the shape of the initial profile, space (location), time-step, mesh-refinement, `r’ parameter, etc.

[May be one more editing pass, sometime later this week or so.]

[E&OE]


text:
# Worksheet on Trigonometric Angles

Practice the questions given in the worksheet on trigonometric angles to know the origin, development and necessity of trigonometry, different methods for measuring trigonometric angles, differentiate between trigonometric and geometric angles.

1. Express in degrees, minutes and seconds:

(i) 832’

(ii) 7312”

(iii) 375”

(iv) 271/12

(v) 72.04°

2. Find the circular measures:

(i) 60°

(ii) 135°

(iii) -150°

(iv) 27°

(v) 22° 30’

(vi) -67° 30’

(vii) 52° 52’ 30”

3. In a triangle one of the angles is 65° and the second one is π /12 find the sexagesimal and circular measures of the third angle.

4. The radius of a circle is 7 cm. Find the circular measure of the angle at the centre by an arc of this circle 5.5 cm in length.

5.The sum and difference of two angles are 135° and π /12 respectively. Find the sexagesimal and circular measures of the angles.

6. A rotating ray traces angle -5 π/12. State, in what direction the ray moves, and find how many complete revolutions it makes and how much more in degrees the ray rotates.

7. In an isosceles triangle ABC, ∠ABC contained by two equal side measures 45°. The bisector of ∠ABC meets AC at the point D. Find the circular measures of ∠ABD, ∠BAD, ∠CBD and ∠BCD.

8. ∠ABC of a right-angled triangle ABC is 90° and ∠BAC = 3 π/8. The perpendicular from the point B on AC meets AC at the point D. Mention the names of all the angles of ∆ABD and ∆BCD and write down their circular measures.

9. Produce the base BC of an equilateral triangle ABC to the point E, such that CE = BC. Join A, E. Now mention the names of all the angles of ∆ACE and write down their circular measure&

10. If π/3, 5π/6 and 90° be any three angles of a quadrilateral, find the fourth angle in terms of sexagesimal and circular systems.

Answers for the worksheet on trigonometric angles are given below to check the exact answers of the above question.

1. (i) 13° 52’

(ii) 2° 1’ 52”

(iii) 6’ 15”

(iv) 27° 5’

(v) 72° 2’ 24”

2. (i) π/3

(ii) 3 π/4

(iii) -5 π/6

(iv) 3 π/20

(v) π/8

(vi) -3 π/8

(vii) 47 π/160

3. 100°, 5π/9

4. π/4

5. 75°, 60° and 5π/12, π/3

6. 2 complete revolutions and 195° clockwise

7. ∠BAD = ∠BCD = 3π/8, ∠ABD = ∠CBD = π/8

8. ∠BAD = ∠DBC = 3π/8; ∠ABD = ∠BCD = π/8 and ∠ADB = ∠BDC = π/2

9. ∠EAC = ∠AEC = π/6; ∠ACE = 2π/3

10. 60°, π/3


text:
# Thread: one-to-one and/or onto?

1. ## one-to-one and/or onto?

Define h: Z -> Z by h(x) = 7|x| – 6

a. Is h one-to-one? Explain.

b. Is h onto? Explain.

I'm having a hard time finding an explanation for this one. I think it is one-to-one, but not onto. Is this correct? An explanation would be appreciated

2. Is it true that $h(1) = h( - 1)$? One-to-one??

Is is possible that $h(x)=-1$? Onto??

3. Hi, Plato. Thanks for the response. I guess I'm just confused by this.

Originally Posted by Plato
Is it true that $h(1) = h( - 1)$? One-to-one??
Yes, I think so...but I really can't explain why

Originally Posted by Plato
Is is possible that $h(x)=-1$? Onto??
I don't believe so...???

I guess if someone can help me think through this to determine why or why not.

4. Ok, I've been trying to think my way through this. This is what I think so far.

it is one-to-one, but not onto since x and –x map to same element |x|

Am I on the right track?

5. Originally Posted by relyt
Ok, I've been trying to think my way through this. This is what I think so far.

it is one-to-one, but not onto since x and –x map to same element |x|

Am I on the right track?
Hi relyt,
what is the definition of one-to-one and onto?

6. A function f is one-to-one if and only if whenever f(x) = f(y), x = y

A function f (from set A to B) is onto if and only for every y in B, there is at least one x in A such that f(x) = y

Ok, so

It is not one to one since $h(-4) = h(4)$

and I don't believe it would be onto then either

7. Originally Posted by relyt
Define h: Z -> Z by h(x) = 7|x| – 6

a. Is h one-to-one? Explain.

b. Is h onto? Explain.

I'm having a hard time finding an explanation for this one. I think it is one-to-one, but not onto. Is this correct? An explanation would be appreciated
not 1-1
$h(2)=h(-2)$

not onto
$\not \exists c \in \mathbb{Z}$ s.t. $h(c)=-50$


text:
# 7. FAQ¶

## 7.1. How can I set required environment variables?¶

The best practice to set environment variables (UFS_SCRATCH and UFS_INPUT) and source the NCEPLIBS provided shell script (setenv_nceplibs.sh|.csh) is to put them into the .bashrc (Bash shell) or .tcshrc (Tcsh shell) files. These files are executed when a user opens a new shell or logs in to the server (including compute nodes) so that their environment is set correctly.

BASH (edit ~/.bashrc):

export UFS_INPUT=/path/to/inputs
export UFS_SCRATCH=/path/to/outputs
source /path/to/nceplibs/bin/setenv_nceplibs.sh

BASH (edit ~/.tcshrc):

setenv UFS_INPUT /path/to/inputs
setenv UFS_SCRATCH /path/to/outputs
source /path/to/nceplibs/bin/setenv_nceplibs.csh

Note

The user might need to create ~/.bashrc or ~/.tcshrc file.

## 7.2. How can I see/check the steps in my workflow?¶

A good way to see what case.submit will do, is to use the preview_run command, which will output the environment for your run along with the batch submit and mpirun commands.

cd $CASEROOT ./preview_run  ## 7.3. How can I run an individual task in the existing workflow?¶ The CIME allows you to run a specific task in the workflow by supplying the --only-job parameter to the case.submit command. The following example will run only the preprocessing utility chgres_cube: cd$CASEROOT
./case.submit --only-job case.chgres

This will create the initial conditions for the model simulation using the raw input files that are provided by NOAA Operational Model Archive and Distribution System (NOMADS).

To run the simulation:

cd $CASEROOT ./case.submit --only-job case.run  If the user wants to define the first job submitted in a workflow, the --job parameter can be passed to the case.submit command. cd$CASEROOT
./case.submit --job case.run

In this case, two dependent jobs will be submitted: model simulation and post-processing.

## 7.4. How can I change the wall clock time and queue for specific tasks in the workflow?¶

These can be done by using the xmlchange command.

For example, the following command can be used to set the job wall clock time to 10 minutes for chgres_cube

cd $CASEROOT ./xmlchange JOB_WALLCLOCK_TIME=00:10:00 --subgroup case.chgres  The following command will change the job queue to bigmem for chgres_cube: cd$CASEROOT
./xmlchange JOB_QUEUE=bigmem --subgroup case.chgres

Note

Without the --subgroup option, the xmlchange command changes the job wall clock time for all submitted jobs.

## 7.5. How can I change the project account that will be used to submit jobs?¶

There are two ways to change project account that is used to submit job:

• Set PROJECT environment variable before creating case

• Use the xmlchange command to change the project account (please replace PROJECT ID with an appropriate project number).

cd $CASEROOT ./xmlchange PROJECT=[PROJECT ID]  Note A PROJECT environment variable setting will take precident over the case XML setting. ## 7.6. How do I change the processor layout for the UFS Weather Model?¶ The total number of processor used by the UFS Weather Model can be modified by using xmlchange command and editing the user_nl_ufsatm file. To query the default configuration of the processor layout: cd$CASEROOT
./pelayout

and to change the default processor layout:

cd $CASEROOT ./xmlchange NTASKS_ATM=150  This will set the total number of processors to 150, but the model configuration files (model_configure and input.nml) must be changed to be consistent with the total number of processors set by the xmlchange command. In this case, the following namelist options need to be modified accordingly: • layout: Processor layout on each tile. • ntiles: Number of tiles on the domain. For the cubed sphere, this should be 6, one tile for each face of the cubed sphere. • write_groups: Number of group for I/O tasks. • write_tasks_per_group: Number of I/O tasks for each group. The number of tasks assigned to a domain for UFS Medium-Range Weather Model must be equal to: $NTASKS\_ATM = layout_x * layout_y * ntiles + write\_tasks\_per\_group * write\_groups$ to have consistent model configuration with NTASKS_ATM defined above. user_nl_ufsatm can be changed as following: !---------------------------------------------------------------------------------- ! Users should add all user specific namelist changes below in the form of ! namelist_var = new_namelist_value ! Note - that it does not matter what namelist group the namelist_var belongs to !---------------------------------------------------------------------------------- layout = 3,8 write_groups = 1 write_tasks_per_group = 6  Note The model resolution also needs to divide evenly with the layout pair. For the given configuration (C96 resolution), $$96/3 = 32$$ and $$96/8 = 12$$. ## 7.7. How do I change the number of OPENMP threads?¶ The user may need to change the number of threads to reduce memory consumption for each compute node. This is especially true for high-resolution cases, and is already set by CIME for C768. This can be done using the following command: cd$CASEROOT
./xmlchange NTHRDS_ATM=4
./case.setup --reset
./case.build --clean-all
./case.build

Note

The model needs to be built again if threading is changed from 1. Setting NTHRDS_ATM does not require changes in the model configuration files. The job submission scripts handle it automatically and submit jobs using more compute nodes.

## 7.8. How do I restart the model?¶

To restart the model the xmlchange command can be used:

cd $CASEROOT ./xmlchange CONTINUE_RUN=TRUE ./case.submit  In this case, CIME makes the required changes to the model namelist files (model_configure and input.nml) and also copies the files from the RESTART to the INPUT directory. Note If there are restart files belonging to multiple time snapshots (i.e. with 20190829.060000., 20190829.120000. prefixes if written every 6-hours), CIME gets the latest one (the files with 20190829.120000. prefix) automatically. The restart interval can also be changed to a 6 hourly interval as follows: cd$CASEROOT
./xmlchange REST_OPTION=nhours
./xmlchange REST_N=6

Note

The default value of the restart_interval namelist option is zero (0), and the model writes a single restart file at the end of the simulation.

The following example demonstrates the 48 hour model simulation split into an initial 24-hour simulation with a cold start plus an additional 24-hour simulation with warm start.

The initial 24 hours simulation:

cd $CASEROOT ./xmlchange STOP_OPTION=nhours ./xmlchange STOP_N=24 ./case.submit  and restart the model for 24 hours simulation: cd$CASEROOT
./xmlchange CONTINUE_RUN=TRUE
./case.submit

Note

The restart run length can be changed using the xmlchange command and setting STOP_N and STOP_OPTION.

## 7.9. How do I change a namelist option for chgres_cube or the model?¶

To set model namelist options in CIME, edit the file user_nl_ufsatm in the case and add the change(s) as name-value pairs. For example:

!----------------------------------------------------------------------------------
! This file can be used to change namelist options for:
! - Chgres
! - UFS MR-Weather Model
! - NCEP Post
!
! Users should add all user-specific namelist changes below in the form of
!  namelist_var = new_namelist_value
!
! To change the namelist variables that are defined as multiple times under
! different namelist groups
!  namelist_var@namelist_group = new_namelist_value
!
! Following is the list of namelist variables that need to be accessed by
! specifying the namelist groups:
!
! alpha@nam_physics_nml
! alpha@test_case_nml
! avg_max_length@atmos_model_nml
! avg_max_length@gfs_physics_nml
! debug@atmos_model_nml
! debug@gfs_physics_nml
! icliq_sw@gfs_physics_nml
! icliq_sw@nam_physics_nml
! iospec_ieee32@fms_nml
! iospec_ieee32@fms_io_nml
! ntiles@fv_core_nml
! ntiles@nest_nml
! regional@chgres
! regional@fv_core_nml
!----------------------------------------------------------------------------------
do_skeb = T

Then run ./case.submit. This will update the namelist and submit the job.

If you want to review what you have done before you submit the case, you can run ./preview_namelists and then examine the namelist(s) in the run directory or the case subdirectory CaseDocs/.

Some variables are tied to xml in the case and can only be changed via the xmlchange command. Attempting to change them by editing the file user_nl_ufsatm may generate an error.

Warning

The user_nl_ufsatm file is also used to control namelist options for chgres_cube and NCEP-Post. Different namelist groups in the model namelist and the pre-, post-processing tools could have the same namelist variable. In this case, just using the namelist variable causes failures in the automated namelist generation. The following is the list of namelist variables that needs to be used along with their group name.

## 7.10. How do I turn on stochastic physics?¶

There are three types of stochastic physics supported with this release: SPPT, SHUM, and SKEB. They can be used together or separately, and their use is controlled by setting model namelist options DO_SPPT, DO_SHUM, DO_SKEB to true or false. These options are set to false by default for all supported compsets and physics suites.

In addition to the namelist variables that turn stochastic physics on or off, there are several variables that control the behavior of the physics. Those are explained in the Stochastic Physics User’s Guide.

In order to set variables DO_SPPT, DO_SHUM, DO_SKEB to true in the model namelist, as well as to set the values of the variables that customize the stochastic physics, please see FAQ entry How do I change a namelist option for chgres_cube or the model?

## 7.11. Can I customize the UPP output?¶

At this time the CIME workflow does not support the customization of the variables or levels output by UPP.

## 7.12. How do I find out which platforms are preconfigured for the MR Weather App?¶

Preconfigured machines are platforms that have machine specific files and settings scripts and should run the UFS MR Weather Application out-of-the-box (other than potentially needing to download input files). Preconfigured platforms are usually listed by their common site-specific name.

To see the list of preconfigured, out of the box platforms, issue the following commands:

cd $SRCROOT/cime/scripts ./query_config --machines  The output will contain entries like the following: cheyenne (current) : NCAR SGI platform, os is Linux, 36 pes/node, batch system is PBS (' os ', 'LINUX') (' compilers ', 'intel,gnu,pgi') (' mpilibs ', ['mpt', 'openmpi']) (' pes/node ', '36') (' max_tasks/node ', '36')  ## 7.13. What are the compsets and physics suites supported in this release?¶ There are two compsets supported in this release: GFSv15p2 and GFSv16beta, corresponding to the physics suites associated with the operational GFS v15 model and with the developmental physics for the future implementation of GFS v16. However, there are four physics suites supported for this release: GFSv15p2, GFSv15p2_no_nsst, GFSv16beta, and GFSv16beta_no_nsst. The difference between a suite and its no_nsst counterpart is that the no_nsst suites do not include the Near Sea Surface Temperature (NSST) ocean parameterization. Instead, they employ a simple ocean scheme (sfc_ocean) that keeps the sea surface temperature constant throughout the forecast. Compset GFSv15p2 can use either the GFSv15p2 suite or the GFSv15p2_no_nsst suite. Similarly, Compset GFSv16beta can use either the GFSv16beta suite or the GFSv16beta_no_nsst suite. The choice is made based on the format of the initial conditions file. When GRIB2 format is chosen, the non_nsst suites are used. When NEMSIO format is chosen, the suites with NSST are chosen. These differences are needed because the GRIB2 files do not have all the fields needed to initialize the operational NSST parameterization. ## 7.14. How can I change number of task used by chgres_cube or UPP (NCEP-Post)?¶ By default, CIME automatically sets number of tasks used by chgres_cube and NCEP-Post (UPP) based on the resolution of the created case using following logic: • chgres_cube It requires that number of task used by chgres_cube need to be divided evenly with the number of tiles (6). • C96: closest number of task to tasks_per_node, which can be divided by 6 • C192: closest number of task to tasks_per_node, which can be divided by 6 • C384: closest number of task to 2 * tasks_per_node, which can be divided by 6 • C768: closest number of task to 4 * tasks_per_node, which can be divided by 6 • UPP • C96: tasks_per_node • C192: tasks_per_node • C384: 2 * tasks_per_node • C768: 4 * tasks_per_node The number of tasks will increase along with the increased horizontal resolution due to the memory consumption of the pre-processing tool and tasks_per_node is defined for the each platform using MAX_MPITASKS_PER_NODE element (i.e. 36 for NCAR Cheyenne and 48 for TACC Stampede2). To change the values set automatically by CIME-CSS, the xmlchange command can be used: cd$CASEROOT
./xmlchange task_count=72 --subgroup case.chgres

This command will change the number of tasks used by chgres_cube to 72. If the user wants to change the number of task for NCEP-Post, the subgroup option needs to set to case.gfs_post.

## 7.15. How to change the filenames for input to chgres_cube?¶

By default, CIME uses pre-defined convention to define directory and file names for raw input to chgres_cube. In this case, 0.5-degree data in GRIB2 format is used from NCDC - Global Forecast System.

In the case of using 1.0-degree GRIB2 format data (with gfs_3_YYYYMMDD_00HH_000.grb2 naming convention), the user needs to download the file manually and place it under $DIN_LOC_IC/YYYYMM/YYYYMMDD. Then, the grib2_file_input_grid chgres_cube namelist variable needs to be modified by editing the user_nl_ufsatm file, which resides in the $CASEROOT directory. The following example is for the Dorian case:

!----------------------------------------------------------------------------------
! This file can be used to change namelist options for:
! - Chgres
! - UFS MR-Weather Model
! - NCEP Post
!
! Users should add all user-specific namelist changes below in the form of
!  namelist_var = new_namelist_value
!
! To change the namelist variables that are defined as multiple times under
! different namelist groups
!  namelist_var@namelist_group = new_namelist_value
!
! Following is the list of namelist variables that need to be accessed by
! specifying the namelist groups:
!
! alpha@nam_physics_nml
! alpha@test_case_nml
! avg_max_length@atmos_model_nml
! avg_max_length@gfs_physics_nml
! debug@atmos_model_nml
! debug@gfs_physics_nml
! icliq_sw@gfs_physics_nml
! icliq_sw@nam_physics_nml
! iospec_ieee32@fms_nml
! iospec_ieee32@fms_io_nml
! ntiles@fv_core_nml
! ntiles@nest_nml
! regional@chgres
! regional@fv_core_nml
!----------------------------------------------------------------------------------
grib2_file_input_grid = gfs_3_20190829_0000_000.grb2

Note

Please be aware that:
• Tests were not done with the AVN, MRF or analysis data.

• The date used in the directory naming must match the date used in file name and the RUN_STARTDATE in the case.

## 7.16. How can I run the UFS MR Weather App for another date without overriding my previous run?¶

Before running the App for a second date, you should save your previous run in another directory by moving that directory to a different location.

From the case directory do:

RUNDIR =  ./xmlquery RUNDIR --value
mv $RUNDIR$RUNDIR.forecastdate
`

## 7.17. How do I diagnose a failure with a high-resolution run?¶

One possible source of failure with high-resolution runs is lack of memory. To diagnose if this is the problem, try a low resolution run first.


text:
# Ask Uncle Colin: A Mess of Logs

Dear Uncle Colin,

I have to show that $$-\frac{x}{2} = \ln (\sqrt{1+e^x} - \sqrt{e^x }) + \ln (\sqrt{1+e^{-x}} + 1)$$. I can't get it anywhere near the right form!

- Proof Of It Not Coming - Any Reasonable Explanation?

Hi, POINCARE, and thanks for your message!

That's a bit of a mess - but with some careful book-keeping, it comes out ok!

Let's start by combining the logarithms to get $$\ln\br{\br{ \sqrt{1+e^x} - \sqrt{e^x }}\br{\sqrt{1+e^{-x}}+ 1}}$$.

Now we have something inside the log-bracket we can expand to get: $$\sqrt{ \br{1+e^{x}} \br{1+e^{-x}} } + \sqrt{1+e^x} - \sqrt{ \br{e^x} \br{1+e^{-x}}} -\sqrt{e^x}$$.

That can be expanded still further to get $$\sqrt{2 + e^x + e^{-x}} + \sqrt{1+e^x} - \sqrt{e^x+1} - \sqrt{e^x}$$.

Those two middle terms are equal and opposite, so the log-bracket is now $$\sqrt{e^x + 2 + e^{-x}} - \sqrt{e^x}$$.

I can rewrite that first square root by taking a 'factor' of $$\sqrt{e^x}$$ out, to make the whole bracket $$\sqrt{\br{e^x}\br{1 + 2e^{-x} + e^{-2x}}} - \sqrt{e^x}$$. Why do such a thing? It makes the second bracket in the first square root is $$\br{1+e^{-x}}^2$$ - so the whole bracket is $$\sqrt{e^x} \br{1 + e^{-x}} - \sqrt{e^x}$$.

Aha! That now becomes $$\sqrt{e^x} + e^{-\frac{1}{2}x} - \sqrt{e^x}$$, or $$e^{-\frac{1}{2}x}$$ - and it's all over bar the logging!

The entire right-hand side is $$\ln\br{ e^{-\frac{1}{2}x}}$$, which is $$-\frac{1}{2}x$$, as required. $$\blacksquare$$

Hope that helps!

- Uncle Colin

## Colin

Colin is a Weymouth maths tutor, author of several Maths For Dummies books and A-level maths guides. He started Flying Colours Maths in 2008. He lives with an espresso pot and nothing to prove.

#### Share

• ##### Mark Thornber

If you take a factor of sqrt(e^x) from the first term the two brackets become a difference of two squares and it drops out in a couple of nice lines!

• ##### Colin

That’s very neat! Thanks for sharing it 🙂

This site uses Akismet to reduce spam. Learn how your comment data is processed.


text:
1. ## Logarithms

Find the solution of the logarithmic equation
logx + log(x-9) = log(6x)

find the answer correct to four decimal places.
Thanks!

2. Originally Posted by Dan.I
Find the solution of the logarithmic equation
logx + log(x-9) = log(6x)

find the answer correct to four decimal places.
Thanks!
An almost identical question has been asked and answered here: http://www.mathhelpforum.com/math-he...ving-logs.html


text:
# a) Convert the following equations into matrix: x – y = 3 2x + 3y = 1

a) Convert the following equations into matrix:
$x–y=3$
$2x+3y=1$
You can still ask an expert for help

• Questions are typically answered in as fast as 30 minutes

Solve your problem for the price of one coffee

• Math expert for every subject
• Pay only if we can solve it

yunitsiL

Here the system of equations is given by
$x-y=3$
$2x+3y=1.$
This can be written as
$\left[1,2,-1,3\right]\left[x,y\right]=\left[3,1\right]$
Therefore the system can be written as AX = b, where $A=\left[1,2,-1,3\right],X=\left[x,y\right]\phantom{\rule{1em}{0ex}}\text{and}\phantom{\rule{1em}{0ex}}b=\left[3,1\right]$