You're Correct!

For the majority of the putts you hit in a round of golf, you are looking to strike the ball with the middle of the club face, but there are times when you don’t.

One exception to this would be if your ball finished above the hole and the putt you were facing was not only downhill but a fast putt too. When you face this type of putt, there is a danger of hitting the ball too hard and the ball could run 10, 15 or even 20 feet past the hole. This would leave you with a longer up hill putt, which is not a total disaster but if the green is very fast you could potentially see the ball running off the green and into either a bunker or worst case scenario, a water hazard.

To avoid the potential for disaster and make sure you leave your scary downhill putt within 12 - 18 inches of the hole or even hole it, you should practice the following change to your set up:

Take your usual set up position and then move your feet away from the ball, making sure you allow the putter head to move towards you as you move your feet. You should see the toe (far side) of the putter come in line with the ball.
Once you are in this position, you need to make sure that the putting stroke you now make is the same as usual. This will see the impact of the ball in the toe of the putter, deadening the impact so that the ball travels with less speed and over spin than if you had hit your putt out of the middle of the putter face.
Now take your usual set up with the ball in the middle of the putter face. Hit the putt taking care to make the same length and speed of stroke. If the strokes are the same, you should see the second putt has travelled further having been hit with the middle of the putter face.

Find a hole with an obvious downhill putt and practice hitting the putts out of the toe of the putter. You will see a marked improvement in the distance control you have and you might even hole a couple too.

In a round of golf you will have very few putts that are straight and flat. Whether it’s a slight right to left putt, uphill and then downhill or has eight feet of break, the position on the putter face you want to strike the ball in can vary.

When playing on slower paced greens, the need to strike the ball in the centre of the putter face is no different than on fast greens. If you have a downhill putt you still need to adjusting the impact position on the putter face to maintain your distance control.

In an ideal world you would be looking to hit the majority of your putts from the middle of the putter face, even if the speed of the green is fast, although the faster the speed of the green, the more important it becomes to hit the ball softer on downhillers and this is when an off centre impact is vital to the success of the putt.