My avatar has developed an issue the last few times I've used it where there's no power assist until the car is driven a bit of distance, roughly 3-4 miles. Bad check valve at the booster, or something else? The vacuum source is at the carb (Edeljunk) rather than the manifold which I've been meaning to change but haven't got to it yet.
Seems like it “building vacuum “. Can you tee into the line with a gauge and see how long it takes to stabilize? If not the diaphragm, maybe the hose is collapsing? Check valve (I thought) was there to keep vacuum in case of a power loss.
Is the booster still in the floor, or firewall mounted? Assume 7"? If in the floor, condensation may have built up in the booster. If the engine is running fine, sounds like the hose is suspect.
The booster is one of those 'generic' 7" chrome units, firewall mounted. This generally only happens if the car sits for an extended time, like 8 hours or more. As soon as it gets some temperature showing on the gauge the problem goes away.
If it's hooked to the ported vacuum on your carb, that would explain it; it should be connected to manifold vacuum.
From the symptoms you describe, it sounds like the check valve or the internal actuating assembly has a problem. Have you tried the pump down test on the check valve?
From watching a few videos, I suspect I may have a bad booster. So the next trick is, what do I have ?... It could be the check valve, I will check that first.