Jump to content


Photo

Suggestion - Cache Query but not Block


  • Please log in to reply
13 replies to this topic

#1 alexp999

alexp999

    Advanced Member

  • Visitors
  • PipPipPipPip
  • 291 posts

Posted 19 April 2012 - 03:28 AM

I think there should be an option to cache the query used in feed blocks, not just the whole block.

For example, lets say you have some latest content feed blocks like latest posts, downloads, articles, comments, etc. You want these cached to reduce server load (my page loads start to lag having say 4 of these types of blocks on a page not cached).

The problem is, the block needs to be processed on a per member basis to set the correct timezone for the user.

So I think it would be great to have an option to cache the query used for the block, without having to cache the html output. That way you could set the cache query to say 5 mins, to reduce server load, and then just process the html template on demand using the cached query.
  • Grumpy, bfarber, Alfa1 and 1 other like this

#2 Marcher Technologies

Marcher Technologies

    $life=FALSE;$code=TRUE;$time--;

  • +Clients
  • 11,762 posts

Posted 22 July 2012 - 08:16 PM

I think there should be an option to cache the query used in feed blocks, not just the whole block.

For example, lets say you have some latest content feed blocks like latest posts, downloads, articles, comments, etc. You want these cached to reduce server load (my page loads start to lag having say 4 of these types of blocks on a page not cached).

The problem is, the block needs to be processed on a per member basis to set the correct timezone for the user.

So I think it would be great to have an option to cache the query used for the block, without having to cache the html output. That way you could set the cache query to say 5 mins, to reduce server load, and then just process the html template on demand using the cached query.

I like the idea... just wanted to note... there is still a query regardless of caching the result set query seperately... the HTML template is stored/pulled in/from the database all the same, and no cache file is generated for it.... though again, this still would be helpful.... and me, I would be similarly pleased with having those same html templates compiled and cached... I have 0 need for a query to be run for custom html blocks I have not gone and put in myself.... and caching the compiled template code to a file would similarly reduce extraneous queries, without actually having to go and cache the resulting html.... which honestly i've never been fond of... caching the html out direct to the database kills language abstraction amongst other bits.

#3 bfarber

bfarber

    RBT-KS

  • IPS Management
  • 28,676 posts

Posted 23 July 2012 - 10:25 AM

I like the idea... just wanted to note... there is still a query regardless of caching the result set query seperately... the HTML template is stored/pulled in/from the database all the same, and no cache file is generated for it.... though again, this still would be helpful.... and me, I would be similarly pleased with having those same html templates compiled and cached... I have 0 need for a query to be run for custom html blocks I have not gone and put in myself.... and caching the compiled template code to a file would similarly reduce extraneous queries, without actually having to go and cache the resulting html.... which honestly i've never been fond of... caching the html out direct to the database kills language abstraction amongst other bits.


In our profiling, we've often found that database skins (requiring "an extra query") outperform the regular cached template files in many cases, because disk i/o is slow. Caching these templates to some sort of flat file isn't necessarily faster than querying the templates at run time.
  • Marcher Technologies likes this

Brandon Farber
Development Manager / Senior Support

If it sounds like fun, it's not allowed on the bus!

php5_zce_logo_new.gif     

Invision Power Services, Inc.


#4 alexp999

alexp999

    Advanced Member

  • Visitors
  • PipPipPipPip
  • 291 posts

Posted 23 July 2012 - 11:13 AM

Ive since written my own plugin using the IP.Content blocks, basically:
  • Cached Feed Block, grabs all the latest content, etc (resource intensive query), and the output is a serialized $records array.
  • My custom plugin then grabs the feed block output from above (non intensive query) and unserializes it.
  • The template for my plugin is the generic feed template (modified slightly for my site), and shows the cached content, but generated on every page load to have the correct time for the user (and time difference, i.e. 5 mins ago).
It works as the original (cached) query only outputs timestamps, so my block later on which isnt cached can always display the correct time and zone for the user.

Thats why I started this thread, I think IP.Content should have a caching system for the query. It basically means I dont have to do resource intensive queries every page load, its dramatically sped up my site.
  • bfarber and Marcher Technologies like this

#5 Marcher Technologies

Marcher Technologies

    $life=FALSE;$code=TRUE;$time--;

  • +Clients
  • 11,762 posts

Posted 02 August 2012 - 04:20 AM

In our profiling, we've often found that database skins (requiring "an extra query") outperform the regular cached template files in many cases, because disk i/o is slow. Caching these templates to some sort of flat file isn't necessarily faster than querying the templates at run time.

But a hard file has not the low limits on storage SQL has.... there is an upper on that 'full' cache. I have hit it before and had to adjust the server.... and recache all blocks/templates to fix the data truncation when it hits field limits/max insert.

#6 bfarber

bfarber

    RBT-KS

  • IPS Management
  • 28,676 posts

Posted 02 August 2012 - 08:23 AM

Everything has limits (including hard drives). That doesn't change my statement, however. :)

Brandon Farber
Development Manager / Senior Support

If it sounds like fun, it's not allowed on the bus!

php5_zce_logo_new.gif     

Invision Power Services, Inc.


#7 Marcher Technologies

Marcher Technologies

    $life=FALSE;$code=TRUE;$time--;

  • +Clients
  • 11,762 posts

Posted 08 August 2012 - 03:54 PM

Just a side FYI to the OP, the data result set stored would have to be cleaned, and I mean heavily, to be reasonably stored. Even serialized, the results array from a single feed block pulling say, 10 rows is HUGE.

#8 alexp999

alexp999

    Advanced Member

  • Visitors
  • PipPipPipPip
  • 291 posts

Posted 08 August 2012 - 04:13 PM

Just a side FYI to the OP, the data result set stored would have to be cleaned, and I mean heavily, to be reasonably stored. Even serialized, the results array from a single feed block pulling say, 10 rows is HUGE.


I've got 4 blocks with 10 rows each stored serialized and it is still considerably quicker than no caching at all.

#9 Marcher Technologies

Marcher Technologies

    $life=FALSE;$code=TRUE;$time--;

  • +Clients
  • 11,762 posts

Posted 08 August 2012 - 04:35 PM

I've got 4 blocks with 10 rows each stored serialized and it is still considerably quicker than no caching at all.

I never said there would be no great benefit, it just needs implementing, and a further abstraction layer of feed blocks to separate the method executeBlock into two methods, one that gathers the data and returns the rows, and the other returning the template fed either cached or new rows.
It has IMMENSE benefits, I razed a statistics applications weight down to board index no hooks vanilla weight doing this.

#10 alexp999

alexp999

    Advanced Member

  • Visitors
  • PipPipPipPip
  • 291 posts

Posted 08 August 2012 - 04:40 PM

I misuderstood what you meant by heavy cleaning then :/


#11 Marcher Technologies

Marcher Technologies

    $life=FALSE;$code=TRUE;$time--;

  • +Clients
  • 11,762 posts

Posted 10 August 2012 - 05:19 PM

I misuderstood what you meant by heavy cleaning then :/

I assume you are not storing the member data loaded for each result in the feed then, basically, only store what is actually used or such a cache goes into the 1mb+ for 1 feed block scale, taking into consideration the ability to add custom fields, as many as you wish on x database, and a user-defined limit, and the array to be serialized can get unwieldy.

#12 alexp999

alexp999

    Advanced Member

  • Visitors
  • PipPipPipPip
  • 291 posts

Posted 11 August 2012 - 02:40 AM

I havent stripped anything from the array, its serialized the entire $records array.

I might look into stripping what I dont need. I imagine its only going to affect how quickly the server reaches max capacity as I cant see how the page can load any faster than it already does.

#13 GreenLinks

GreenLinks

    Spam Happy

  • +Clients
  • 831 posts

Posted 20 October 2012 - 04:41 AM


In our profiling, we've often found that database skins (requiring "an extra query") outperform the regular cached template files in many cases, because disk i/o is slow. Caching these templates to some sort of flat file isn't necessarily faster than querying the templates at run time.


I disagree with this specifically how HDD speeds have been improved lately. We use enterprise level SSD's on our live servers and no database query will read data faster than filesystem. Especially if you have a pretty busy board and big database.


Reducing query number is very critical for big boards. So even a query that inserts html output is a BAD query.


  • Alfa1 likes this
Mert Gokceimam
CTO / Greenlinks Bv
http://www.420network.com

#14 bfarber

bfarber

    RBT-KS

  • IPS Management
  • 28,676 posts

Posted 22 October 2012 - 09:43 AM

It is simply a statement from experience.  We run a hosting company and host many many large sites, after all.

 

For the record, I was also not stating we would or would not be doing anything specific.  I was only making a comment relevant to the current discussion. :)


Brandon Farber
Development Manager / Senior Support

If it sounds like fun, it's not allowed on the bus!

php5_zce_logo_new.gif     

Invision Power Services, Inc.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users