Phase noise and the choice of the first IF in a spectrum analyzer

Many of you may have heard about "phase noise" but do you thoroughly understand what this is all about.
I also was not aware of the relevance before I started measuring the performance of the my home build spectrum analyzer.

The 10.7MHz resolution filter (third IF filter) I'm using  has a -50dB width of about 60KHz and a -90dB width of 100kHz when measured on a VNA.
When sweeping this third IF filter in the SA while using a first IF at 2.6GHz a very different filter picture appears

The staircase at the center is caused by the discrete steps of the fractional PLL used for the sweep
From 20kHz offset and -40dB down there are side skirts and even a shoulder at 150kHz from center. (the peak at 120kHz is leakage) where neither the side skirts or the shoulders are visible on the VNA.
These skirts and shoulders are caused by the phase noise of the LO's. Not all energy is in the single intended output frequency but there is also noise generated that reduces when farther away from the intended frequency.
A standard way to measure this phase noise is to remove the first mixer and use the first LO as test signal and scan this LO and use a log frequency scale as can be seen in below plot

What you see is a upper side band scan, the lower side band scan should and actually does looks the same (apart from the small leakage peak)
The horizontal scale is the frequency in MHz from the the LO frequency. The sweep of the first LO is still done lineair so the lower frequencies have less measurement points compare to the higher frequencies.
The first point at 0.01MHz is the full LO signal normalized at 0dB and the first point with offset is at 0.1MHz away from the LO. You can not see the resolution filter details (as can be seen in the first picture in this post) as there are insufficient points in this scan at low frequencies but the noise fall-of when further away from the LO signal is clearly visible till about 10MHz where the phase noise goes below the SA noise floor of -105dB
A practical implication of this phase noise is when you have a strong (0dB) signal 300kHz away from a weak signal the noise floor of the SA will increase from -105dB (right part of scan) to -80dB so the sensitivity of the SA is reduced in the near presence of strong signals.
Do keep in mind that what you see is actually a result of 3 LO's  (first IF at 2.6GHz, second IF at 110Mhz and third IF at 10.7MHz)  so you can not simply attribute all to one LO but the bandwidth of the first and second IF will impact contribution of the second and third LO. This still needs more investigation.

Now what has this to do with the choice of the first IF of the SA? 
Phase noise is caused by noise in the steering of the VCO in the PLL of the LO. If you have a high first IF you need a high output frequency from the LO, in this case of a ADF4351 and a first IF at 2.6GHz  no output dividers are being used. When using a lower first IF frequency (say 110MHz) and using the same ADF4351 the output divider will be 16 and this will reduce the phase noise.
To check this I measured the close phase noise when a first IF of 110MHz is used and you get below picture (first IF at 110MHz and second IF at 10.7MHz and no third IF)

The shoulders have moved down with about 25dB, still not as good as the VNA measurement.
The VNA measurement was done at 10.7Mhz so phase noise is expected to have less impact in the VNA measurement
 
But what about the far out phase noise?
As you can see in below graph the far out phase noise also has gone down.

The divide by 16 of the output of the PLL has increased the steepness of the fall-of of the phase noise.
The phase noise at an offset of 300kHz is at the level of the noise floor so a strong (0dB) signal 300kHz way from a weak signal will have no impact on the noise floor of the SA when using the first IF of 110MHz instead of 2.6GHz

All this implies when building your own SA you should not blindly go for the highest possible first IF. You have to understand the impact of the PLL in the LO's you use and their phase noise and the output divider in relation to the selected IF frequencies. In general having a high first IF will introduce more phase noise and this makes your SA less sensitive in the presence of strong signals. Its probably better to choose the first IF low enough for most measurements and use a down converter for the odd measurement where you have to go higher.