Sounds like an oscillator that is on all the time and/or a 'stuck' note.
I don't own a Pulse, but...
Does the unit have any sort of auto-calibrate / auto-tune function? Have you tried it?
Or a manual-tuning-reference oscillator like an old Moog? Is it on?
If not, or if that doesn't work, sounds like it's in need of service and a tune-up by a qualified tech.
On some traditional analog synth architectures, the oscillators really are 'on all the time', and if something goes wrong with the filter / amp / envelope sections, it can do that.
If you max out the polyphony (play more notes at the same time in different ranges than the synth has notes of available polyphony) does that 'always on' note change, or is it always at that same frequency?