Magento 2 FPC and Evolved Caching

Magento 2 has finally seen it's general availability release as of late last year, and the rebuilt platform brings with it some valuable new additions such as ajax add to cart, editable admin grid rows and perhaps most significantly a full page cache for both community and enterprise versions of the platform.  These are great additions as it means the core team has really been listening to what customers want and have built into the new version features which are real goto extension for Magento 1.

I have really enjoyed working with the Magento 2 codebase and I'll be honest, it really does feel light years ahead of Magento 1.  Removal of the Mage 'god' class in favour of the functionality offered by the object manager and other more minor (but still significant) steps such as no more code pools and all extension files residing in the same directory location are really sensible steps.  As well as listening to their customers, it seems like the core team have also really learnt from how Magento 1 was developed, used, and customised by third parties.  Magento 2 feels like a really solid, defined platform to develop from with many areas that were considered to be open for discussion by many developers in Magento 1 being clearly defined in Magento 2.

If anything, you could argue that some of the functionality that is offered by the object manager has gone a bit too far, i.e. is basically overkill purely because at the moment it seems to cater only for real edge cases and might never prove that useful to third party developers.  However these opinions may change as the platform matures and people get familiar with the full breadth of the codebase and what can be done with it.  Ultimately, we developers certainly can't complain about Magento giving us more rather than less options, we just need to be sure we are using the right tools for the right application.

If you are interested in getting to grips with the object manager in Magento 2 then Alan Storm has a great series you can have a look at.

So finally getting to the point of the post, the addition of built in full page cache for Magento 2 creates a dilemma for third party extension developers (such as us) who have developed our own caching extensions for Magento 1.  For us, we have to make a choice, do we port our caching solution to Magento 2 or does the built in solution negate the need for this?  Unfortunately there is very little documentation covering the functionality of the full page cache in Magento 2, so we have instead been diving into the codebase in order to answer the above question.

It appears the core team have taken steps to make full page caching easier for their built in solution.  They have made some noteworthy changes in Magento 2 such as category filtering now being restricted only to URL (where as it was also stored to the session in Magento 1), and storing the frontend form key to a cookie for quick and easy access (where it was only found again in the session for Magento 1).  So removing dependency on the session when serving cached content seems to have been to some extent the order of the day.

When reviewing the full page cache logic my point of greatest interest was where and how the request was intercepted to serve cached content as opposed to the framework processing the request as normal.  I expected to find an equivalent of the early request_processors hook in point used by the enterprise full page cache solution in Magento 1, but when investigating it seems that there is no equivalent in Magento 2.  The core team have stuck firmly within the confines of the framework as it stands and used plugins to intercept the request, deciding at this point whether to serve cached content or send the request into the framework as normal.

My reason for being most interested in this intercept point is because full page caching id of course all about how fast you can deliver cached content to the browser, with the TTFB defining how long it takes for the page to start rendering in the browser - the longer this takes, the 'slower' the page is for the user.  The choice of using a plugin is why I am rather disappointed with the performance of the built in full page cache for Magento 2 - it just isn't all that fast.

It's doing the right thing in development terms - using a plugin to intercept the request is not outside of the boundaries set by the framework, and is traceable and easy to find by third party developers as it's not using something else entirely (such as request_processors) to serve content, but the performance it gives is just pretty average.  I wonder if any of the core team wanted to implement a faster solution with an earlier hook in point?  We will probably never know.

The problem with using plugins is that in order to do this you need to instantiate the object manager, and this pretty much means instantiating the entire framework.  Doing the equivalent of this in Magento 1 was slow, but Magento 2 is that much more complex and so takes even longer.  An example - in my local development environment I see a TTFB of around 33% with the full page cache - so pages are served in around 6 seconds with caching disabled, and 2 seconds with caching enabled (using built in cache rather than Varnish).  While this is a significant improvement, and production environments would have a faster setup with this process completing quicker, the percentage improvement would still be the same.

My issue here is that with Evolved Caching under Magento 1 you are looking at less that 1% of the time for TTFB (see the benchmarks) which is possible because full page cached content is served entirely outside of the Magento framework making it massively faster.  There just really isn't any comparison between the two with around 33% being rather disappointing compared to less than 1%.

So to answer the original question, should we port Evolved Caching to Magento 2?  Considering the performance we are seeing with the built in full page cache, I think the answer has to be yes.  The challenge is building it in such a way that it gives equivalent performance to it's Magento 1 counterpart, yet falls within best development practices for Magento 2.

Keep your eyes open, but the Magento 2 version of Evolved Caching is under development and we will let you know when it's released.

Get huge data savings on your cache with Evolved Caching

If you run a store with a lot of products, or have a multi store setup then you are likely to have a pretty large full page cache.  Well from version 1.9.6 of Evolved Caching you can slash the size of your cache by 85% by compressing your cached data.  Here's the savings we saw during testing while introducing an overhead of just 2-3 milliseconds:

• Uncompressed cached page data size - 44KB
• Compressed cached page data size - 6.6KB
• Saving - 85%

If you want to significantly reduce the size of your full page cache then grab your copy today!

Why should you buy Evolved Caching?

If you are on the search for a full page caching solution for your Magento store you should you will probably have seen there are a wide range of solutions with more on the way seemingly all the time.  With Evolved Caching being at the higher end of the price range why should you pick it over other solutions?

Evolved Caching offers not only the best performance in the market place, but it's also a significantly more advanced solution than competing products.  In performance terms Evolved Caching is comparable to Varnish so you are likely to achieve the same time to first byte (TTFB).  If you want to use Varnish, then you can integrate directly and have both Varnish and Evolved Caching serve your cached pages.  Evolved Caching will also allow Varnish to cache many more pages than other Varnish only extension solutions - PageCache powered by Varnish for instance stops serving cached content altogether as soon as you add a product into the cart.

Evolved Caching is able to achieve the same TTFB as Varnish because of the way in which it serves cached content compared to other solutions. Most pull the cached content, then populate the cached page with dynamic data (i.e. mini cart, header links) before delivering anything to the browser.  Doing this however means you have to enter the Magento framework to generate the cached content and this is always going to be slow as every time you do this you need to run many thousands of lines of code and execute a large number of database requests.  This has a direct impact of how long it takes the server to deliver content to the browser and therefore your TTFB.

Evolved Caching is different because it delivers the cached content to the browser immediately giving an excellent TTFB and then populates the page with dynamic content after it has been delivered to the browser. It can populate the page with dynamic content in 2 ways, either via an AJAX request, or via BigPipe. It was the first solution to use AJAX, and the only solution to offer BigPipe as an option.

Whichever method you pick you are going to have the cached page content delivered to the browser quicker and therefore achieve a better TTFB.  To give you an idea of how much quicker, looking at some of the demo's for other solutions you might see the server taking something like around 30% of the time to generate HTML for each page (say 600ms without caching, 200ms with).  If you have a look at our benchmarks you will see you are looking at less than 0.5% of the time to generate HTML for each page.

So performance and TTFB is one reason to buy Evolved Caching over other solutions, but the benefits cover much more than just performance.  Evolved Caching has a totally different approach to keeping the cache up to date. Many other solutions rely on a cache purge and crawl cycle to refresh the cache - so this means that the entire cache is cleared and rebuilt on probably a daily, or more regular interval.  When making updates to your sites content you may not see those changes reflected until this cache purge and crawl cycle has been completed.  Evolved Caching is totally different in it's approach as the aim right from the start has been to make it a very much a hands off solution that just works while keeping your cache always primed and up to date. So just the relevant cached pages are rebuilt for instance when saving a product, category or CMS page. It also caters for caching scenarios that other solutions don't like rebuilding the cache for time limited design changes which are store wide, or just apply to a product, category or CMS page. The same is true for other time limited features such as product special prices.

Basically once a cached page has been created, there should never be a reason to have to delete it, and it should always be kept up to date automatically. Having said that, should you find for whatever reason you do need to clear your full page cache Evolved Caching does include functionality to crawl your site to create a good basic cache as a starting point (product, category and CMS pages are created). It also allows you to regenerate your entire cache should you, for instance, want to make a site wide design change that affects every page, i.e. in the header or footer.

Another reason to favour Evolved Caching over other solutions is the options it gives you in how you want to store your cached content.  You can store to files, APC, Memcached or Redis with memory based cache storage being faster on average than files.  Although some solutions offer different caching backends, many don't and only cache to files.

Evolved Caching was also developed to make it easy to configure for your install.  Other solutions require some development knowledge to holepunch any but standard blocks from caching - so you need to create an XML configuration file defining extra blocks to exclude.  Evolved Caching instead provides a frontend URL helper argument, ?shownames.  Append this to any frontend URL and the block names will be displayed, just add any of these names into the extension config in admin and that's it - no development knowledge required. You can equally add in pages by URL to be excluded from caching from admin, i.e. cart, checkout, customer login area etc.

These things are really just touching the surface of what the extension can do compared to competing, you can read a lot more in our documentation.

When to use Magento full page caching

Caching, and particularly full page caching in Magento is a hotly debated topic.  If you have a developer working on your store or you are a developer, you probably know the phrase "have you cleared the cache" very well.  You probably also know the famous phrase

"There are only two hard things in Computer Science: cache invalidation and naming things."

Put together the pieces and you can quickly ascertain that caching is a tricky area to deal with and something that should be carefully considered, and properly implemented for your Magento store.

So with this being such a difficult area to address, should you even bother implementing any kind of full page caching solution for your store?  Well the short answer is yes, simply because by caching pages you reduce the work the server has to do with the end result being web pages get served to your customers faster.  So even though the hardware your site is running on has not changed, you still 'boost performance' by using caching - and I put that inside quotes because actually the server is not working any faster, there is just less work it needs to do to serve a page.

The long answer is yes, but...  To implement caching properly you need to first and foremost have a good foundation for your store.  This means that your hosting must be sufficiently powerful to run your store well with no caching enabled.  This is a key point that many people miss when thinking about a full page caching solution for their store.  Full page caching is not a quick fix for poor store performance, it's a tool that should be implemented on a store that is already performing well, in order to take that performance to the next level.

When thinking about which of the wide range of full page caching solutions you want to implement, you should consider that Magento is a complex framework, and that caching Magento's pages effectively is not a straight forward task.  You should carefully consider and research the available solutions.  Firstly go for a solution that is mature in terms of development - so it's not a recently launched product (believe me there will be bugs!), and has a good number of releases over it's development cycle.  At the time of writing this, our own solution Evolved Caching has been in constant development since February 2013 with 58 releases, so it's a mature and highly capable product which can be used on any store, however much it has been developed.

Also review the capabilities of the full page caching solution you are considering.  Although there are a lot of options out there, remember that Magento is highly complex software which presents a very wide range of possible caching scenarios.  Unfortunately you will probably find that many of the solutions don't cater for many of these scenarios, for instance, does the cache get updated when a time limited design on the store, category or product page starts or ends? (yes Evolved Caching does!)

You should also definitely contact the extension developers for the solution you are interested in to ask them any questions you may have.  From the responses you get back, you can often get a good impression of the quality of the extension they produce.  If they can explain technical details in a clear, informative way that a non developer can easily understand, then they probably have a pretty good idea of what they are doing and therefore a good product.  Remember this quote?

Well that applies here too.  If the developers can't explain what the extension does in simple terms then they probably don't have a great understanding of the concepts themselves, in turn the product is unlikely to be as good as it should be so steer clear.

Finally, get a trial!  As a company we truly believe that if you are asking people to part with a reasonable amount of money to get hold of your code, you should give them the opportunity to trial the software first - so that's exactly what we do.  If a company refuses to allow you to trial their software before purchasing it then you might want to think about the confidence that means they might have in their product.

If you have any questions about anything here, or want to get in touch with us about Evolved Caching, you can easily contact us from our store where you can also find complete documentation about our own full page caching solution, Evolved Caching.

Evolved Caching 1.9.0 released

Version 1.9.0 of our flagship Magento extension, full page caching solution Evolved Caching has just been released.  1.9.0 marks the 12th release since version 1.8.0 and again a lot has changed.  We have added support for a wider range of setups (for instance supporting installs in a subdirectory), added better and more complete admin notifications to improve usability, greatly increased the number of scenarios under which the cache is refreshed (for instance on a frontend sale, and when taking admin action like creating orders and credit memos) and implemented a number of other functionality improvements and minor bug fixes.

The most noteworthy addition to the extension however is the use of BigPipe technology, new to 1.9.0.  This replaces the previous non AJAX dynamic content serving strategy and we believe Evolved Caching to be the first full page caching extension to implement it.  This greatly improves performance when compared to the previous non AJAX strategy and means that now both AJAX and non AJAX methods are closely comparable in performance terms.

BigPipe Update Strategy

Using BigPipe strategy brings a major advantage.  When using AJAX updating for dynamic content, if you exclude a block from caching which relies on certain JavaScript functionality, you may find this functionality breaks.  While you can already resolve these problems by using the evolved:loaded JavaScript event Evolved Caching fires, by switching to BigPipe you can now resolve these issues without any technical knowledge and still keep outstanding performance.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching 1.8.0 released

Version 1.8.0 of our flagship Magento extension, full page caching solution Evolved Caching has just been released.  As ever a lot has changed since version 1.7.0, and a number of fixes for occasionally occurring bugs are now in place.  More importantly however are two major new features added for this release.

The first major addition is the automatic re-caching of pages for scheduled design changes, and product special prices.  If you have configured a store wide, category based, or product based design change to run for a limited time, the relevant cached pages will now be cleared and re-cached automatically at the start, and end of this period.  The same is also true when scheduling a time limited special price for a product.  So now you can be sure your cache will always be up to date, even during seasonal promotions.

Generate Cache button

The second major addition is a feature which many competing full page cache extension developers offer only as a separate, paid extension - a cache crawler.  The Full Page Cache Entries grid now features a new button, Generate Cache.  Once generated, by using automatic cache clearing and warming, the cache really keeps itself up to date.  However when there is no cache, whether that's because Evolved Caching has just been installed, or because the cache needed to be cleared for some reason, the Generate Cache button lets you immediately build up a good basic cache for the store.  The crawler will cache all CMS, category and product pages for the store, so a good percentage of pages a customer visits will be cached right from the outset.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching 1.7.0 released

We have just released version 1.7.0 of our flagship Magento extension, full page caching solution Evolved Caching.  A lot has been added since version 1.6.0 including making both the extension and Varnish integration fully compatible with Magento CE 1.8+, adding the ability to fully manage your Varnish cache from within Magento, allowing URL parameters to be ignored by the cache, and when using AJAX based dynamic content updating you can now define holding HTML to be displayed before the dynamic content has been collected from the server.

You may have heard about many full page caching solutions, including Varnish, running into problems with the frontend form key which has been introduced with Magento CE 1.8.  The addition of this key if not fixed effectively breaks the store as you can no longer add products to the cart.  Some caching solutions which have 'fixed' the problem resolve it by stripping out the extra level of security offered the inclusion of this form key.  Evolved Caching resolves the issue both in it's own cache, and in Varnish whilst maintaining the extra security offered by the form key.

Another important addition since 1.6.0 is the ability to ignore URL parameters when building the cache.  You may for instance have a number of landing pages on your site where you use tracking parameters in the URL to analyse traffic.  As these parameters have no bearing on how the page is built, ignoring them means that extra cached content is not created due to their inclusion.

A couple of exciting new additions relate to Varnish integration, and holding HTML.  You can now actively manage your Varnish cache from within admin, and it will be automatically cleared and warmed at the same time as the Evolved Caching entries.  This also extends to managing image, CSS and JS content cached by Varnish.  This makes Evolved Caching the most capable Varnish cache integrated solution available for Magento CE 1.8 and above.

Setting block holding HTML

Finally, another major addition is the ability to define holding HTML for excluded blocks.  This allows you to set HTML to be displayed on screen while the dynamic block content is being collected from the server via AJAX.  Great if you want default content to be displayed as soon as the page loads.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Speeding up Magento

If you are a Magento store owner there's a good chance that one of the things that is pretty high on your todo list is reducing the time it takes your stores pages to load.  If you are a developer you have no doubt been asked to impart your knowledge on how this is best achieved on more than one occasion.

A quick search will soon reveal a multitude of pages and discussions talking about how to improve the speed of your store, with loading times often featuring heavily in any arguments against using Magento.  It's true that Magento is complex software, and is also a heavy application for a web server to run.  Until you are experienced it can also seem mind bogglingly complex from a development point of view.

However over the years of developing Magento I have come to find the framework to be the best developed, most capable ecommerce platform in the market place hands down.  Once you have mastered it it's also extremely intuitive and amazingly versatile to develop - it quite comfortably stands up in it's own right as a fully fledged MVC framework.

So even though the versatility you will find in Magento brings a down side in the form of potentially slow page loading times, rest assured that with so much scope for development it's certainly not a problem which can't be addressed with very effective results.

You have probably come across this post after doing some research into exactly how you might achieve faster page loading times, and that probably also means you have come across the near endless multitude of guides and suggestions on the topic.  You will find some good pointers, and tips like minifying JS/CSS, enabling Gzip compression and installing an opcode cache such as APC or XCache are all worthwhile, but in my experience pretty much all of the solutions in these guides don't actually make a great deal of noticeable difference to page loading times.

The reality is that for every request Magento processes, thousands of lines of code need to be executed.  This takes both time and processing power and until each request completes and the resulting HTML response has been sent, the user browsing the site won't see any page content.  While tweaking the server setup and Magento configuration might improve overall performance by probably no more than 5%, the process of Magento building each page accounts for at least 95% of the total work the server has to do to deliver the entire page and all assets (images, JS, CSS) to the browser.

Many of the guides and suggestions out there don't even make, let alone address this fundamental point, and rather cloud the issue with endless server and Magento tweaks which would take a huge number of hours to complete if you were to follow them all, but result in just a very modest improvement in performance.

If however you want to see a real improvement in page loading times, then there are really only three areas you need to concentrate your efforts.  Be warned though that all three of these areas are likely to require investment, so if you are only interested in free solutions, this probably isn't the post for you.  If however you have some money to put into it then read on...

Server Hardware

The first area you need to look at it is the hardware your store is running on, so your web hosting package.  This is of course the foundation of your entire store setup and anything that you build on this foundation can only ever be as good as the hosting will allow.  For instance if you are trying to run a Magento store on a cheap shared hosting package, then you can tweak the store, and spend as much time and money on optimising it as you like, but you will never achieve good performance because the raw processing power is just not there in the first place.

So review your hosting and make sure it is up to standard.  Which hosting package you choose directly relates to how much traffic you expect to see on the store as the higher the traffic, the more capable the server will have to be as it will be fulfilling more simultaneous requests.  Always steer clear of shared hosting when it comes to Magento, and instead go for a suitable VPS or dedicated hosting package.  As for which specific package to go for, well that entirely depends on your store setup, and your sites traffic.  If you are running a smaller, low traffic store, then you can probably get away with a mid to low range VPS package, if however you are running a high traffic store with lots of products, you will almost definitely want a dedicated server package.  As your site grows you will discover that the possibilities for Magento hosting continue to scale up to complex, multi server, load balanced setups - but if your store is at that level, you probably don't need to read this post.

To determine the best hosting package for your store I would recommend contacting someone with good knowledge on the subject.  There are many companies who offer Magento specific hosting and should be happy to advise, or a good Magento developer will also be able to give you some pointers on a suitable package to go for.

If you want to get in touch with us for some advice on which hosting package is best suited for your store, then just drop us an email to support@husseycoding.co.uk with the following information:

- Store address
- Summary of typical monthly traffic
- Geographical location of main target audience

We will be happy to have a look at your site and give you some free advice on hosting packages which suit your setup.

Time for first request to complete

When viewing a web page the first request that is sent to the web server is a request for the complete page HTML, and once this has been completed everything else relating to the page can be requested, so the images, CSS, JS and so on.

In the case of Magento, fulfilling this first request means initialising the Magento application, routing the request to the relevant controller in the framework, generating the layout for the page, and then finally rendering the page out to HTML from the layout.  This resulting HTML is then returned in the server response to the browser, at which point the browser can display the page HTML and begin sending requests for images, JS and CSS files.

Each step in generating the pages HTML takes time, and as a whole quite a lot of time.  When I talk about rendering out a page accounting for at least 95% of the work the server has to do for a page and all assets to be delivered to the browser, it's this complete process I'm talking about.  The rest of the requests which come after this are just for files so all the server has to do is deliver each file - no processing of the request is required.

So it makes sense that this first request for the page HTML is the one that could do with improving, and that's exactly what all of the Magento full page cache extensions, and also Varnish aim to do.  I'm sure you are aware there are quite a few Magento full page cache solutions out there, with varying levels of performance.

As already mentioned, the process of rendering out a Magento page is time consuming, and as soon you need to enter the Magento framework to fulfil a request you immediately significantly increase the time it takes for that request to complete.  This is the main difference between many of the full page cache solutions available, and our own solution, Evolved Caching.  Although all of the full page caching solutions available do actually cache the full page HTML, the vast majority of the solutions also need to enter the Magento framework in order to fulfil the cached page request, and this in turn means that the speed at which this request can be fulfilled becomes entirely dependant on the speed of your Magento install (which I'm sure you will agree to a large extent defeats the object of the exercise).  Evolved Caching operates much the same as Varnish and fulfils this initial request for the page HTML entirely outside of the Magento framework giving vastly improved performance when compared to other solutions.

In order to add any dynamic user specific content to the site, Varnish has ESI, which requires tags to be put in relevant places in the site code.  The major down side of using ESI with Magento is that there will often be multiple parts of the page which need to have dynamic content added, and for each of these, a separate request must be sent in to Magento.  This of course kicks off the slow process of initialising the framework and rendering out an HTML response.  In addition to this, Varnish will not deliver the page to the browser until all the ESI content has been collected, and although ESI responses can also be cached, the content Magento produces is just too dynamic to make this a realistic possibility.  So by using Varnish and ESI you might actually end up sending 2 or 3, or more requests into Magento before the page is delivered to the browser, thus resulting in page load times which are actually slower than without Varnish!

Evolved Caching on the other hand delivers the page HTML immediately after it is pulled from cache, and then fulfils the requirement for dynamic content on the page via a single AJAX request while other requests for things like images, JS and CSS are still running.  Although this request does have to enter the Magento framework, the request is quite a bit 'lighter' than a standard request, and is also only made after content has already been delivered to the browser thus eliminating the waiting time of other full page cache solutions, and also Varnish + ESI.

Evolved Caching was the first Magento full page cache extension to introduce this innovation and has a large amount of equally valuable functionality on top of this.  It also integrates directly with Varnish by way of it's caching key cookie and this greatly increases the ability of Varnish to cache and fulfil requests for your store.

So in summary reducing the time it takes for the server to fulfil this first request for the full page HTML is exactly what full page cache extensions, and also Varnish do - but the key to this being really successful is to not enter the Magento framework before content is delivered as this will immediately have a major impact on the performance of the cache.

If you are interested in Evolved Caching, then you can find a summary of functionality on Magento Connect, read the full guide here, and purchase the extension from our store here.

Requests per page

The final area which you need to look at is the least easily addressed, particularly if your store is already developed and themed.  That's certainly not to say that it can't still be looked at, it's just more easily done when the site is initially being built.

When viewing a page, multiple requests are sent to the server before that page fully loads, in fact every separate item on the page is requested individually from the server, so every image, JS file, CSS file and of course the full page HTML as we have learned above, are all received as a response to an individual request for that content.

If you look at a typical web page using the net panel of Firebug, or if you don't have that, a site such as GTmetrix, then you are able to see just how many requests have to be fulfilled in order for that page to fully load (if you are looking at GTmetrix, look at the waterfall diagram).  Each item in the waterfall which describes the loading of the page represents an individual request which has been made to the server for content.

This is significant because a browser can only fulfil a certain number of requests in parallel.  At the time of writing, in modern browsers this is along the lines of 6-8 - so 6-8 of the requests shown in the waterfall diagram can be completed simultaneously, but no more.  When the maximum possible number of requests for your browser are all in progress, further requests must be queued and become idle until an existing request completes, therefore making room for a queued request to start.  When looking at the waterfall diagram, the left hand most block of colour for each request represents the time that request is queueing and idle, and you will probably see that all requests after a certain point are queued.

This queueing is time that is entirely wasted, and no queueing would result in a significant reduction in the time taken for pages to fully load.  So one way to alleviate this is to reduce the number of requests per page.  Merging of CSS and JS is a good, easy way of going some way towards doing this, and for a Magento store will often mean reducing perhaps 25+ inidividual requests down to just 2, one for merged JS, and the other for merged CSS.

There is some discussion surrounding whether or not to use JS/CSS merging within Magento as the JS/CSS files which are required on each page tends to differ a little between different areas of the site, and this results in different merged JS/CSS being created wherever this is the case.  The end result is that your browser ends up needing to download a new JS/CSS merged file for each of these different areas but the actual file content contains very little that differs from what was perhaps already downloaded for another area of the site.  Without merging only the new JS/CSS files that differs for each area would need to be downloaded rather than the whole lot again (in merged form).  The argument against merging however ignores the fact that without merging the number of requests for a page will likely increase by at least 20, and as long as you have sensible file sizes for your images, request queueing has a much greater impact on page loading times than the amount of data which must be downloaded for each page (merged CSS/JS combined should only account for perhaps 30k of data, around the same as 1 small image).  In case you have come across any discussions like this, my recommendation is to definitely enable JS and CSS merging.

So that does a pretty good job of reducing the number of requests for JS and CSS content, however it's likely that there are still quite a few more requests that can potentially be reduced down, particularly for images.  Typically Magento themes use quite a high number of images and while things like product and category images can't realistically be worked on to this end, it's well worth reviewing what images are to be found in common areas of your store such as the header, footer and sidebars.  There is very likely scope here for reducing the number of images required by creating suitable sprites.  A sprite is simply a single image which contains several images within it.  This means you can effectively download a number of images in a single request, and then use CSS to position the sprite within an element to show just the image you require.  That way you can have a number of different images in different places on the site, but only require one server request to obtain them all.  A bit of common sense is of course required in balancing file size vs number of requests when using sprites.

As well as thinking about sprites also consider using more CCS3 styles.  All modern browsers now support CSS3, and support in older browser is getting better all the time as new browsers continue to be released and older ones fall away.  As time goes on of course, older browsers that don't support CSS3 will disappear entirely.  Using a few simple CSS3 styles can really help you cut down the number of images you need to use, and styles like border-radius address design problems which historically have created the need for quite a large number of individual images (a box with rounded corners for instance).  Using CSS3 these kinds of things can be achieved using no images at all, meaning the only request sent to the server is for the CSS file containing the styles.

Working on the images and considering the CSS3 styles to use is the reason why I say this is easier done at the initial design and build phase where the possibility of using sprites and styles can be taken into account - this is of course significantly easier than reworking an existing theme.

Going back to talking about the browser limit of 6-8 simultaneous requests at a time, it's worth noting that this limit is applicable on a domain basis, so if you had a page from domain1.com and domain2.com both loading at the same time in your browser, both pages could send 6-8 simultaneous requests, making a total of 12-16 simultaneous requests sent by the browser.  This is exactly the functionality which a CDN leverages.  As CDN's reside under a different domain to the main domain you see in the address bar, it allows a browser to send an additional 6-8 simultaneous requests for every CDN domain used.  The result of course is a reduction in requests queueing, and therefore a decrease in the time it takes for a pages content to load entirely.

Getting a CDN setup for your store isn't an entirely straight forward process, and something you should probably approach your hosting provider about if you don't have the skills in house, however if you have done all you can to reduce the number of images served by creating sprites and using CSS3 styles, then it's a good option to look at.



I hope that this 3 point guide has proved informative, and if you want to get in touch with us regarding the performance of your store, just drop us an email to support@husseycoding.co.uk with your site address and we will be happy to evaluate your stores performance for free and come back to you with some pointers on further steps you could take.  You can also assess your store using our profiler for some straight forward, clear, and specific advice on improving the performance of your store.

Evolved Caching 1.6.0 released

Recently released version 1.6.0 of our flagship Magento extension, full page caching solution Evolved Caching, again adds some major new additions to the extension.  Version 1.5.0 added the ability to view cached page from admin, a choice of cache warming strategy and the ability to lock the cache to reduce the chance of accidental deletion. 

Version 1.6.0 adds some important, behind the scenes maintenance features such as automatic cleaning up of user specific cached files, and two new admin options to firstly give you the option of using either AJAX, or non AJAX based dynamic content updating, and secondly add the ability to cache product, category and cms pages on save regardless of whether content already exists in the cache.

Adding the option to update dynamic content on the page without using AJAX gives you an additional way of resolving any potential JavaScript issues which may arise from using AJAX based updating where JS onload event handlers are being used.

AJAX dynamic content updating option

The option to warm pages even if they do not already exist in the cache means that cached content can immediately be created for instance when adding a new product, category or cms page.  The Always Warm Pages admin option adds this functionality.

Always Warm Pages admin option

You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching and Varnish

Documentation has now moved to our store, it will no longer be updated here!

Evolved Caching is a highly performance orientated, advanced full page caching solution for Magento developed by Hussey Coding.  It offers performance and functionality that we truly believe make it the premier choice for full page caching in the market place.

As well as being an extremely effective caching solution in it's own right, Evolved Caching also integrates directly with Varnish by way of it's caching key cookie.  This cookie is generated for every page as you navigate through your site and it contains a unique hash which describes many things about the way that page is displayed, such as the current store, the currency, any category or layered navigation filtering applied, whether the page is secure or insecure, the browser user agent, the tax rate according to the customers address, and of course the URL.

Varnish is designed to cache by URL, but this results in an incomplete solution in the case of Magento as quite a few aspects which govern the way a page is displayed are stored in a users session and won't necessarily appear in the URL at all (category page filtering is one example of this).  This means that Varnish can end up serving the wrong content to the user as it doesn't account for, amongst other things, session data.

The caching key cookie however is built using logic which considers all of the different ways in which this kind of data can effect a page, it's the same logic in fact Evolved Caching uses to store it's own cached content.  So with Varnish using the unique hash contained in the cookie, you have an effective, complete Varnish caching solution driven by the complex logic Evolved Caching already has built in.

As well as the caching key cookie meaning that you don't need to take on the significant task of providing Varnish with enough data to cache properly, it also tells Varnish which pages it should and shouldn't cache according to the excluded pages you configure in admin.

The end result of all this is that your Varnish configuration file only needs to be relatively few lines as all Varnish needs to do is cache and serve content according to the content of the cookie.

Although this post includes the sample .vcl files below, we won't cover the initial installation and setup of Varnish here.  Installation is straightforward and covered for a number of different Linux distributions in the Varnish documentation.  Initial setup is again simple and for many configurations, nothing more than is outlined under the installation exercise here will be required.

With Varnish installed and setup, you then need to edit your Varnish configuration file, which unless you have changed it during setup, can be found here on your server.

/etc/varnish/default.vcl

Again the Varnish documentation has plenty of information about building a VCL in case you want to modify the configuration below, but we expect for you to be able to use the sample .vcl as it is.

So all you need to do is open default.vcl at the location above (or the one you specified if you changed it), delete it's contents and copy and paste into it the first sample .vcl below for Varnish 3, or the second sample below for Varnish 4.


Varnish 3

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    if (req.request == "BAN_HTML") {
        ban("obj.http.Evolved-Type == html");
        error 200 "ban_html";
        return (error);
    } else if (req.request == "BAN_IMAGES") {
        ban("obj.http.Evolved-Type == image");
        error 200 "ban_images";
        return (error);
    } else if (req.request == "BAN_CSSJS") {
        ban("obj.http.Evolved-Type == cssjs");
        error 200 "ban_cssjs";
        return (error);
    } else if (req.request == "PURGE_SINGLE") {
        return (lookup);
    } else {
        if (req.url ~ "/form_key/" && req.http.Cookie ~ "evolved_formkey") {
            set req.url = regsub(req.url, "/form_key/[^/]*/", "/form_key/" + regsub(req.http.Cookie, ".*evolved_formkey=([^;]+).*", "\1") + "/");
        }
        if (req.restarts == 0) {
            if (req.http.x-forwarded-for) {
                set req.http.X-Forwarded-For =
                    req.http.X-Forwarded-For + ", " + client.ip;
            } else {
                set req.http.X-Forwarded-For = client.ip;
            }
        }
        if (req.http.Cookie !~ "evolved_key") {
            return (pass);
        }
        if (req.request != "GET" &&
            req.request != "HEAD" &&
            req.request != "PUT" &&
            req.request != "POST" &&
            req.request != "TRACE" &&
            req.request != "OPTIONS" &&
            req.request != "DELETE") {
            return (pipe);
        }
        if (req.request != "GET" && req.request != "HEAD") {
            return (pass);
        }
        if (req.http.Authorization) {
            return (pass);
        }
        return (lookup);
    }
}

sub vcl_hash {
    if (req.url ~ "\.(png|gif|jpg|jpeg|swf|css|js)$" && req.http.Cookie ~ "evolved_key") {
        hash_data(req.url);
    } else if (req.http.Cookie ~ "evolved_key") {
        hash_data(regsub(req.http.Cookie, ".*evolved_key=([^;]+).*", "\1"));
    }
    return (hash);
}

sub vcl_hit {
    if (req.request == "PURGE_SINGLE") {
        purge;
        error 200 "purge_single";
        return (error);
    }
}

sub vcl_fetch {
    if (req.http.Cookie ~ "evolved_key") {
        unset beresp.http.Set-Cookie;
        if (req.url ~ "\.(css|js)$") {
            set beresp.http.Evolved-Type = "cssjs";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        } else if (req.url ~ "\.(png|gif|jpg|jpeg|swf)$") {
            set beresp.http.Evolved-Type = "image";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        } else {
            set beresp.http.Evolved-Type = "html";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        }
    }
    return (deliver);
}

sub vcl_deliver {
    unset resp.http.Evolved-Type;
}

Varnish 4

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    if (req.method == "BAN_HTML") {
        ban("obj.http.Evolved-Type == html");
        return (synth(200, "ban_html"));
    } else if (req.method == "BAN_IMAGES") {
        ban("obj.http.Evolved-Type == image");
        return (synth(200, "ban_images"));
    } else if (req.method == "BAN_CSSJS") {
        ban("obj.http.Evolved-Type == cssjs");
        return (synth(200, "ban_cssjs"));
    } else if (req.method == "PURGE_SINGLE") {
        return (purge);
    } else {
        if (req.url ~ "/form_key/" && req.http.Cookie ~ "evolved_formkey") {
            set req.url = regsub(req.url, "/form_key/[^/]*/", "/form_key/" + regsub(req.http.Cookie, ".*evolved_formkey=([^;]+).*", "\1") + "/");
        }
        if (req.http.Cookie !~ "evolved_key") {
            return (pass);
        }
    }
    return (hash);
}

sub vcl_purge {
    return (synth(200, "purge_single"));
}

sub vcl_hash {
    if (req.url ~ "\.(png|gif|jpg|jpeg|swf|css|js)$" && req.http.Cookie ~ "evolved_key") {
        hash_data(req.url);
    } else if (req.http.Cookie ~ "evolved_key") {
        hash_data(regsub(req.http.Cookie, ".*evolved_key=([^;]+).*", "\1"));
    }
    return (lookup);
}

sub vcl_backend_response {
    if (bereq.http.Cookie ~ "evolved_key") {
        unset beresp.http.Set-Cookie;
        if (bereq.url ~ "\.(css|js)$") {
            set beresp.http.Evolved-Type = "cssjs";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        } else if (bereq.url ~ "\.(png|gif|jpg|jpeg|swf)$") {
            set beresp.http.Evolved-Type = "image";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        } else {
            set beresp.http.Evolved-Type = "html";
            if (beresp.ttl <= 0s) {
                set beresp.ttl = 120s;
            }
        }
    }
    return (deliver);
}

sub vcl_deliver {
    unset resp.http.Evolved-Type;
}

Once you have done that, check that the backend default section at the top is correct and points at the reconfigured web server (as set during the initial Varnish setup), and also edit the three instances of 120s in the vcl_fetch/vcl_backend_response section at the bottom to however long you want HTML, CSS/JS and images to remain in the Varnish cache.  This is set at the default of 120 seconds in the above sample .vcl, but you will very likely want to increase it.  How much you increase it by really depends on your caching strategy and the size of cache you want to make available to Varnish, but the longer you hold items in cache, the larger your cache is going to be.  A side note is that you can use values other than seconds so the following would for instance tell Varnish to hold items in cache for a week.

set beresp.ttl = 1w;

A couple of notes to add, the first being that the above samples are Varnish 3 and Varnish 4 configuration files rather than 2.1, a number of key things change after version 2.1 which means that these sample .vcl files will not work with version 2.1.

The second note is to say that the above sample .vcl has been updated to work with changes introduced in Magento 1.8 CE to use form keys on the frontend. Combined with version 1.6.6 of Evolved Caching, it resolves this issue in your Varnish cache.

Finally, if you want to make use of the Varnish cache deleting and warming rules inside this .vcl, you must use version 1.6.9 or greater of Evolved Caching.

The only other thing to point out is that we recommend you use Evolved Caching 1.5.2 or later when it's going to be in conjunction with Varnish.  If you have already purchased a license and have an earlier version than this we we will be happy to send you a more recent version free of charge, just drop us an email at sales@husseycoding.co.uk. Also note that since version 1.6.0 of Evolved Caching you have the option of using non AJAX based dynamic content updating. When using Varnish together with Evolved Caching you can only use AJAX based dynamic content updating, if you disable this your dynamic content will not be correctly displayed.

We highly recommend you thoroughly test this setup in a staging/development environment first, only then installing to your main production environment.

Finally if you would like assistance with configuring Varnish to work with your copy of Evolved Caching, then please contact us at support@husseycoding.co.uk and we will be happy to help.

If you want more information on Evolved Caching, then have a look at the full guide here.  You can also find a summary of functionality, chat to us about it, and purchase it from our store here.

Evolved Caching 1.5.0 Released

We have just released Evolved Caching 1.5.0, and this release adds several great new features to the extension.  Version 1.3.9 added automatic cache warming, which recreates the cache for your product, category and cms pages when you save them, this release adds 3 new features.  The first is cache viewing.  This adds an extra column to the Full Page Cache Entries admin page with a link you can click to open the cached page contents in a new browser window, so what you see here is exactly the same as the customer sees.

Full Page Cache Viewing

The second feature that has been added is a choice of cache warming strategy.  You may now pick between cache being warmed immediately on save, or via cron after a maximum of 5 minutes in the case of the immediate option impacting save times.

Warming Strategy Choice

Finally the third added feature is cache locking.  This allows you to retain the full page cache when clearing the standard cache using the 'Flush Magento Cache' button under Cache Management - useful if you want to clear the standard Magento cache but don't want to lose the full page cache.

Cache Locking

You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching 1.3.9 Released

Evolved Caching 1.3.9 has just been released, marking the addition of another highly useful feature.  Version 1.3.0 added the incredibly useful Full Page Cache Entries management area where you can see, and visually manage your full page cache entries.  This release expands on the earlier feature of automatic cache clearing with the addition of automatic cache warming.

This feature allows you to automatically rebuild cache entries that have been cleared with automatic cache clearing when saving a product, category or cms page.  The process is intelligent and respects your caching strategy, so only cache entries which actually exist and have not expired will be rebuilt.  This means that if you are using APC or Memcached storage and have set the cache to expire, then only valid, current pages will be refreshed and you won't be recreating old expired pages which could push more relevant, current entries out of the cache.

Automatic Cache Warming

The automatic cache clearing feature means that you don't have to worry about customers seeing expired cache pages, and the addition of automatic cache warming now means that they will always see a super fast, cached page, even after you have made changes.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching 1.3.0 Released

The release of Evolved Caching 1.3.0 marks another major update to the Magento extension.  While version 1.2.0 added important features which allowed for a truly effective caching strategy, version 1.3.0 takes management of the full page cache to a new level.

Most full page caching solutions give you no visual representation of what actual content is cached, but Evolved Caching 1.3.0 adds a new admin interface where you can view, filter and manage all full page cache content.  This easy interface gives you real time, granular control over the content of your full page cache, and allows you to:

• View all cached content as individual entries
• Apply advanced filtering to locate particular cache entries
• Delete specific cache entries (no need to delete the entire cache anymore)
• Show when cached content has expired or been deleted

Full page cache entries interface

So no more flying blind with your Magento full page caching, this update gives you confidence and complete control over your stores full page caching.

As well as adding this high level of caching control, Evolved Caching 1.3.0 also adds another highly useful feature - automatic cache clearing.  This optionally allows you to clear relevant full page cache content when saving product, category and cms pages in admin.  Only the relevant cache content will be cleared, so for instance when saving a category or cms page, just the cache for that category or cms page will be cleared leaving all other full page cache intact.  When saving a product page, automatic cache clearing will clear full page cache content for every instance of that product page (for instance if you can access the product the from different categories resulting in different URL's for the same product).  It will also clear the cache for any categories that product is assigned to.

In addition to this, you can also choose to automatically clear the standard block HTML cache type when saving product, category and cms pages, so no more invalidated cache warnings as soon as you hit the save button.

The inclusion of automatic cache clearing gives you the freedom to just get on and work with your store without having to continually clear the cache to make your changes visible on the frontend.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching 1.2.0 Released

We have just released version 1.2.0 of our advanced Magento full page caching solution, Evolved Caching.  This release adds a host of new features and we truly believe that this extension is now the premier choice for Magento full page caching, addressing all of the shortcomings regularly found in competing extensions.

The extension already caters for multiple stores and multiple currencies,  and even takes into account tax according to a logged in users location.  It can also cache by user agent allowing separate caching of mobile stores under the same domain, and has the ability to output headers which are normally lost when using full page caching.

Another great feature is it's integration with Varnish.  The extension will run all the logic for you in deciding how a page should be cached and can generate a cookie containing a unique hash relating to that page.  All Varnish needs to do is pull the full page HTML and store it against the value in the cookie, massively simplifying your Varnish configuration and completely removing all of the usual headaches you come across when caching Magento pages.  So even if Varnish is serving most of your pages, this is still a must have extension purely because of the logic it runs and the way it integrates with, and greatly simplifies your Varnish configuration.

On top of these existing features, version 1.2.0 also has these valuable additions:

• APC caching support
• Cache expiration for Memcached and APC to for a much better memory based caching strategy
• Dynamic updating of category page prices for complex, user specific pricing strategies
• Dynamic updating of category page product stock status
• Dynamic updating of category page review summary blocks
• Dynamic updating of product page review summary block

It's generally accepted that no Magento full page caching solution is complete and does not fully cater for the breadth of variation that can occur when thinking about things like having a complex, customer group based pricing structure, or having category and product pages correctly keep track of the stock status.  Indeed if you run some simple tests with other full page caching solutions you will very quickly find them lacking, many even with the common task of correctly keeping track of product stock.

Well this isn't the case anymore, you can purchase Evolved Caching confident in the knowledge that it will not only deliver a massive performance increase, but will also be guaranteed to cater, in full, for your store configuration whatever it is.


You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.

Evolved Caching Magento Extension Launched

Hussey Coding is proud to announce the launch of its new Magento extension Evolved Caching.  Evolved Caching is a highly performance orientated, advanced full page caching solution for Magento developed by Hussey Coding.

The extension combines incredible performance, a simple install unmatched by other full page caching solutions, and huge versatility through easy backend configuration options.  Unlike alternative full page caching solutions, you don't need to be a developer to install and configure this extension for complete full page caching with dynamic content loading.

Evolved Caching works out of the box with multi store, multi currency installs.  It can also handle caching mobile stores running under the same domain as the main store by allowing you to cache by user agent.

Configuring the updating of dynamic, user specific content is made easy with URL helper arguments and simple admin configuration options - you don't need to touch any code.

Block display URL helper

Admin configuration options

Evolved Caching will give your site a massive boost in performance, benchmarks show server processing times drop from 2650ms to just 10ms to generate the full page HTML, and still only 194ms after loading all dynamic content (which happens after page load via AJAX).  Siege multiple concurrent user benchmarks show the server jump from just 10.39 transactions a second to 138.89 - well over 13 times as many.

You can read about the full capabilities of Evolved Caching including in depth usage and full benchmarks here.

The extension is compatible with Magento 1.4 and above.  You can view all Hussey Coding extensions including this one on Magento Connect, and you can purchase Evolved Caching here.