Sorry to tell you this, but yeah, just about any amount of cream and sugar that you put in there is going to screw up your fasting.
I made this point in another thread here, but I’ll bring it up here again. Studies have shown that so much as watching food being made and eaten can trigger an insulin response in people, which means your body begins to store fat instead of continually burning it. If just seeing food can do that, what do you think tasting sweetness can do? It’s not about the calories as much as it is about the hormones.
If you want to maximize your efforts, learn to take your coffee black. I drank extremely sweet coffee before, but I found that taking it black just took a bit of time to adjust to. Getting a better quality dark roast coffee can drastically help, as well.