When using private nuget packages from github in Visual Studio, one has to provide credentials to the nuget package manager.
There's an option to remember your password. But apparently, that does not work. So you have to enter them everytime you open Visual Studio even when your solution isn't using the private nuget source. I found several solutions, one of them was storing the password as plaintext in the NuGet.Config. That's never a good idea. But it helped in digging deeper and with help of the nuget documentation I found a way to do it properly. The way to go is to create an entry in your NuGet.Config with the nuget.exe tool itself instead of editing it manually.
The steps below describe what you need to do:
First you need to download the latest nuget.exe from nuget.org

The next step is to cleanup your NuGet.Config
Remove old/outdated entries for nuget.pkg.github.com
Open “C:\Users\[USERNAME]\AppData\Roaming\NuGet\NuGet.Config” and remove optional outdated items for your private nuget package source.

Remove the package source "YourPriveNuGetOnGitHub" which should be your specific private nuget source.
Remove the package source credentials "YourPriveNuGetOnGitHub" which again should be your specific private nuget source.

Next generate the appropriate entry using nuget.exe
nuget.exe sources add -Name "NameYouLike" -Source "https://nuget.pkg.github.com/[YOUR GITHUB SUB]/index.json" -username "nuget.pkg.github.com\[YOUR GITHUB SUB]" -password "[YOUR PERSONAL GITHUB CREDENTIAL PASSWORD ]"