There are several reasons for this.
Firstly, a note does not bear such emotional significance as colour does. Colour can cause a steady mood, for example, blue makes one pessimistic, while yellow excites.
Secondly, a colour can be either warm or cold. The warmest one is orange, contrasted to blue, the coldest one. A note does not have such a meaning, irrespective of its pitch.

It is also to be noted, that the names of notes and their frequency in Hertz have been changing in time. In 1826 in London, the standard for La note was set at 422 Hz, and in 1845 it increased to 455 Hz. In France La note was equal to 435 Hz, while in the US it was approximately 460 Hz. Now the most countries acknowledge the standard frequency of 440 Hz, accepted at the Shtudgard conference in 1834.
Thus, if Do note corresponded to violet (as suggested by Newton), then it would be another colour now. At the same time, musical composition can be played in different tonalities and create an impression, the sad song will remain sad, and merry one will always be merry.

The next elements in the music that follow the notes are intervals, which key properties are similar to colours.

Why the diagram represents less colours than the rainbow colour count?

What does a consonance chord and a consonance intervals mean?

Why the intervals are to be given the numerical values of semitones?

Why the interval of 3 semitones is compared with the blue colour and not with yellow one, for instance?
How the multitude of colours we see is expressed in music?

creativelab.org.ua