Simple Way to Customize Encoder Settings

We recently noticed a clever post on the TeamLiquid forum, claiming to increase the performance by 400% by using a replacement DLL for  the x264 encoder.

Well, let’s not get into the numbers, but the post was right on the money. We did have some threading issues with our x264 DLL build, which resulted in a non-optimal use of multiple cores. It’s been fixed now and the encoder has been upgraded to the latest core version.

After upgrading to the latest version, as of May 13, 2011, available in tier 3 DO NOT attempt to replace any DLLs.

Now that we are on the topic, let’s further try to discuss how users themselves can change encoder settings. XSplit uses the world’s best h.264 encoder also known as the X264 encoder, an open source encoder for which we obtained a commercial license earlier this year.  X264 supports a number of settings and users are able to change the default settings used by XSplit, but this is NOT advised unless you know what you are doing.

There are two ways the settings can be customized and this blog will just cover the simpler and  more direct way.

You will notice that there is a combo box that says “preset” as part of most stream plugins …see image below.

If you click on the combo box you will see that there are a number presets which corresponds to presets (configurations) that are predefined in X264.  The preset “veryfast” pretty much corresponds to the XSplit Default which has been retained mostly for historical reasons …so if you  try “Veryfast” then you will probably not see any difference.

The presets correspond to a number of highly technical internal settings. The slower the preset, the higher the compression, and the more encoder features used – but caution is advised as moving to a preset slower preset than “Veryfast” requires a pretty powerful machine.

The presets generally do not include settings like rate control, key frame rate, etc. The default key frame rate used by the encoder is 250 and if you wanted to change that, you could enter the following text in the combo box.

veryfast&ex:keyint:300

Let’s say you now wanted to use a key frame rate of 300 and to also force the H.264 profile to baseline, which is required by some mobile devices. You would then need to enter the following:

veryfast&ex:keyint:300&ex:profile:baseline

If you need to apply custom rate control parameters then you would first have to ensure to avoid conflicts with the XSplit settings, so in this case you would set the Quality to “not set” and delete any numbers you may have set in then VBV-buffer and VBV-maxrate edit boxes.

We will cover rate control very closely in an upcoming blog post. It’s something that few users really understand but understanding of this can become a great benefit.

You can read much more about x264 settings here.

(But be careful, do not experiment and then complain afterwards)

Final note: While outside the scope of this post, please note that we have used the term key frame rate in the above text. This is a term that some people would be familiar with, but it is perhaps a slightly wrong technical term in this context. The Keyint parameter in X264 does not necessarily mean that a full frame is sent every 300 frames. The Keyint parameter just denotes the maximum number of frames between IDR frames – separating “groups of pictures” from where the encoder is restricted to make any reference outside the group.