There's more...

What do you think will happen if, instead of the threshold activation function, we use the sigmoid activation function?

You guessed right; firstly, we can use the TensorFlow optimizer to update weights. Secondly, the network will behave like the logistic regressor.