InfoQube IM - Community

Submitted by Jon on 2010/06/03 12:18

I am not sure that this is the proper forum for this, but here goes. I know that Pierre has been steadily improving performance, but for me large databases just seem very sluggish. This has nothing to do with filters, conditional formatting, or auto-assign features. For example, one of my DBs has a grid with a lot of entries (>5000) and exceeds 9 levels deep. Scrolling is a nuisance and refreshing takes forever. I have repaired and compacted the database with little discernable improvement. Do others see this too?

Jon

P.S. I may be overloading IQ through my own misuse. If so, tell me, but I thought IQ was built for this sort of flexibility. In its early iterations, Ultra Recall was (for me) very slow, but its performance has improved remarkably.

Comments

Hi Jon,

We can separate performance in the following categories:

Startup time
Open / refresh a simple grid with few items
Open / refresh a grid full of items, many levels deep with context parents, etc
Grid navigation (scrolling once loaded)
Grid editing, date and pop-up list display
Properties pane update
HTML pane update
Search

Can you comment on each of these, as to which one are the most problematic ?

Thanks !

[quote=Pierre_Admin]

Hi Jon,

We can separate performance in the following categories:

Startup time
Open / refresh a simple grid with few items
Open / refresh a grid full of items, many levels deep with context parents, etc
Grid navigation (scrolling once loaded)
Grid editing, date and pop-up list display
Properties pane update
HTML pane update
Search

Can you comment on each of these, as to which one are the most problematic ?

Thanks !

[/quote]

Start up time is MUCH longer as the database grows. You summarized my database with description #3. Grid navigation is sluggish and seems to proceed in fits and starts. The grid takes a long time to catch up (sych) with the scroll bar.I experience #5 a lot too. The property pane (#6) is slow to update, although this seems much improved from several versions ago. Sorry, but I generallly don't use the html pane, so I am not qualified to make an informed comment on this. Search (#8) is not too bad.

Does this help?

Jon

Yes, it helps. Some more for you :

Startup time. It typically consists of 3 parts. Can you comment on each ?
1. Program start
2. IQBase open
3. Grid open
Simple grids: how are they performing ? (Create a new grid and start adding items)
Complex grid full of items : This should be significantly improved with the new Ecco-style grid
Grid navigation: How many items are in the grid ? Is it dependant on the number of items in the grid ? Or is it only dependant on the number of items in the IQBase ?
Editing : Does it seem dependant on the number of items in the grid ? (You can apply an AlphaFilter to quickly hide a large proportion of items)
Auto-assign, inheritance and equations can impact performance, especially if used in highly populated fields. Do you have many of those ?

[quote=Pierre_Admin]

Yes, it helps. Some more for you :

Startup time. It typically consists of 3 parts. Can you comment on each ?
1. Program start
2. IQBase open
3. Grid open
Simple grids: how are they performing ? (Create a new grid and start adding items)
Complex grid full of items : This should be significantly improved with the new Ecco-style grid
Grid navigation: How many items are in the grid ? Is it dependant on the number of items in the grid ? Or is it only dependant on the number of items in the IQBase ?
Editing : Does it seem dependant on the number of items in the grid ? (You can apply an AlphaFilter to quickly hide a large proportion of items)
Auto-assign, inheritance and equations can impact performance, especially if used in highly populated fields. Do you have many of those ?

[/quote]

1. If by program start you mean the splash screen until I see the actual (but as yet unpopulated screen) program, it seems a little long, but not much more than I have observed with smaller databases. Once the program is displayed, the actual database takes a long time to open. Now this may be because the opening grid contains >5000 items. The grids seem to open pretty slowly if they contain a lot of items. Small grids open much more quickly, although they open slower than similar sizeed grids in smaller databases.

2. Doing anything in this larger database takes longer than in small databases, but I can create a new grid quickly as long as it does not contain a significant number of items. If I have the source point to large numbers of items, it is s-l-o-w.

3. I agree that over the last several versions there have been improvements to performance, but it is not what I would call Ecco-style fast. But it is much better.

4. It appears to be grid-dependent. Smaller grids respond more quickly.

5. Yes, once I get to the item, editing is faster. But scrolling is like nails on a blackboard for me. Filtering on the item is the quicker way to go, but sometimes you need to scroll to see what is in the item's neighborhood.

6. Nope. In this database, none of these are used.

Is this useful?

Jon

My largest grid has 2200 items

Loading is not quick but grand

It takes on average 20 seconds to expand all items from about 50 visible items (with a 'fresh' copy of IQ)

For the last month or two, my default ('upgraded') install has major problems starting up (20 to 30 seconds @ full CPU, well, one full core - EDIT/ this happens whether it is opening with a file or not /edit) and display problems when opening grids (the menubar disappears & reappears). I don't have these last two problems with a fresh install so will have to look at importing my settings & see what happens.

Have others with performance problems tried out the file in a 'fresh' copy of IQ ?

[quote=Tom]

Have others with performance problems tried out the file in a 'fresh' copy of IQ ?

[/quote]

A fresh copy (sample DB) is pretty fast, without anything added... but the problem is the sample DB is of no use to me (I need all my functions, formatting, etc.), not to mention all my items.

In any case, all the problems I mentioned were tested both with a sample DB and my own.

[quote=Armando]

[quote=Tom]

Have others with performance problems tried out the file in a 'fresh' copy of IQ ?

[/quote]

A fresh copy (sample DB) is pretty fast, without anything added... but the problem is the sample DB is of no use to me (I need all my functions, formatting, etc.), not to mention all my items.

In any case, all the problems I mentioned were tested both with a sample DB and my own.

[/quote]

Armando, I meant with a fresh install of IQ, not a different or a sample database... my point was that at least one problem disappeared for me when I moved to a fresh copy of IQ - but that may be a lot more difficult for you (I only have customised shortcuts and background colours)

[quote=Tom]

[quote=Armando]

[quote=Tom]

Have others with performance problems tried out the file in a 'fresh' copy of IQ ?

[/quote]

A fresh copy (sample DB) is pretty fast, without anything added... but the problem is the sample DB is of no use to me (I need all my functions, formatting, etc.), not to mention all my items.

In any case, all the problems I mentioned were tested both with a sample DB and my own.

[/quote]

Hi Tom,

Everytime there's a new release I overwrite the old one in IQ's folder. But maybe you mean transferring all my current items to a new DB ? Normally, it shouldn't change anything -- I usually do a repair after each new release, just to make sure. And if it did change something, transferring everything would be extremely tedious. It would take hours, if not days... 1- considering the current slowness of copy paste, 2- the creation/reconfigurion of all the auto-assign rules, equations and conditional format for 135 fields, 3- the reconfiguration of 60 grids, 4- the DB options, etc.

Hi Armando,

I have been doing the exact same as you "Everytime there's a new release I overwrite the old one in IQ's folder"

so,

what I mean is to create a completely new install of IQ (somewhere/anywhere - not related to any previous installs*) and open your database in this new install and see do you have all the same problems.

I suspect your problems are mainly to do with the size/complexity of your database, but I'm just throwing this idea out there because, in my example above, changing my IQ installation resolved at least one issue

* I'm not sure do multiple installs of IQ share stuff in "documents & settings"

[quote=Tom]

[/quote]

This is definitely strange. I suspect a corrupted menu.dat or dock.dat file is the problem. You can delete dock.dat no problem. If you delete menu.dat, you'll loose your customized menus and shortcuts. You can't save your customized menus, but there is now an export/import for shortcuts.

See InfoQube Program Install and Set-up

HTH

[quote=Pierre_Admin]

[quote=Tom]

[/quote]

Re the CPU usage:

unfortunately removing both of those didn't help. I was thinking I'd go through the settings and add them to my new portable install and see do any cause a problem but that's down my ToDo list a bit (also that I suspect it's not to do with my settings but probably some corruption of something like you thought)

Re the menu problems:

this is actually happening in my new 'install' so I think it's a display bug. Initially it didnt happen - or it happened so quickly I didnt notice - but it's very noticable with bigger files/grids - where the grid takes more time to open.

Description:

Click on a grid to open it,
the menubar disappears;
the grid tabs shift up to the menubar space;
when the new grid-tab shows, the menubar reappears and grid tabs move back down.

Armando - do you know did you report this at some stage (I think maybe?), I can add to mantis if not there and if approved

PS I had only noticed the export for shortcuts due to this move Pierre - that's great to have that !

I get all that Tom describes.

1- High CPU usage on starting and opening Grid. This never bothered me too much as usually start IQ once in a while and leave it open. Total time from double clicking on the Icon and seeing my first grid : about 20s. Yes, it isn't super quick.

2- Menu problem : yes, this was always like that. It feels a bit clunky. If fixing this makes everything else faster, sure, why not. But if it doesn't, I'd leave it for later.

P.S. : There seems to be some other performance problems affecting all other item manipulation operations (creating, deleting, pasting : all of these take a tremendous time in my DB, in most grids). And, yes, as I've described in other threads, these performance problems depend on having equations, inheritance, etc. or not... But, to put it bluntly, if I can't use these, there's absolutely no point in using IQ as far as I'm concerned : equations and auto-assign stuff is what makes it useful to me.

IMO, to be able to create a truly meaningful project management DB (for example) many things need to be automated -- even more than what I have now.

I'm one of those affected by performance issues daily.

DB is large, I guess, but not that much : 263 MB (for comparison : my Outlook dB is 1 GB)

There are least 2 or 3 threads around where I pinpointed performance issues (some with very specific examples, even steps, others less precise), and some became mantis issues :

Performance problems with auto assign rules, inheritance, item deletion. : auto-assign rules affect performance by a very significant margin. Unless you use them extremely sparsely. In fact, auto-assign rules affect performance more than normal row equations for some reasons. Inheritance and hierarchical calculations have a pretty big impact too.

Will the number of parents an item have affect performance ? : I got no answer for this one, but yes multiple parents affect performance : properties pane display, and hierarchical calculations especially.

When properties pane "fetches data" to populate certain sections (Parents & Siblings), properties refresh is very slow. : has to do with sibblings and parents -- but not related to previous issue... Well, I don't think so. This affects the properties pane display so much that I have to turn it off most of the time. [EDIT : note that to notice that performance problem, you need to move slowly enough from item to item so that the properties has the time (or almost) to refresh]

What are the differences in performance between different types of filtering ? (+ a few comments on important missing features) : I did some test on my own for this one, and yes, looking for strings inside text fields affects performance by a significant margin.

Performance problem *when Gantt is on* and hierarchy is collapsed : if grid shows empty rows it flickers for a long while. : This is a specific problem when the gantt chart is ON, and parents collapsed.

And some other users commentS :

performance in regard to filtering,&hoisting

Copying into IQ - performance issues : where some performance issues are mentioned, like the fact that (in my DB at least) when deleting an item, it takes a while for the deletion dialog to just pop up. This is reproduceable every time I have to delete something. There's also a discussion (the main one) about copy-paste... Still a big issue here.

Now, about Pierre's Specific points :

1- Startup time

- DB loads fairly slowly
(I must say that of all the performance issues, this is the one I can live with that since IQ is usually started only a few times a week, and then left ON, unless it crashes or needs to be restarted for various reasons. But others might of course feel otherwise.)

2- Open / refresh a simple grid with few items

- Pretty slow. 4s for the scratch displaying a small tree of items (9), 13 columns, mostly numbers and YN fields.

3- Open / refresh a grid full of items, many levels deep with context parents, etc

- Big grids refresh very slowly. About 9s to open, 7s to refresh.

4- Grid navigation (scrolling once loaded)

- I don't have problems with grid navigation using the up/down arrows IF the properties pane is closed. If the properties pane is opened, it's very choppy unless I leave my finger on the arrow so that the properties pane doesn't have the time to show anything.
Scrolling using the croll bar isn't a problem

5- Grid editing, date and pop-up list display

- Pop up list can be on the slow side when they are built arround special SQL requests.
- Grid editing is ok, but adding items to grid if there are auto-assign rules and inheritance involved can be very slow... (2s)

6- Properties pane update

See When properties pane "fetches data" to populate certain sections (Parents & Siblings), properties refresh is very slow..

7- HTML pane update

- Pretty good here. A tad onger for MHT content (not embedded), but IMO there are more pressing performance problems elsewhere.

HTH

None of my links appear in previous post... Isn't the syntax [node:nodeID] ? What did I do wrong ??

>None of my links appear in previous post... Isn't the syntax [node:nodeID] ? What did I do wrong ??

Input format was set to PHP code. Should be Filtered HTML. Fixed

Thanks...!

I wonder how I managed to change the input format without noticing it...

I think I found something...

With prerelease 25, I've been noticing the time to pull up a grid is approximately 5 seconds, which seemed very slow. Starting up a database is taking

5-10 seconds, depending on the database size.

I used prerelease 24, and found the same task took 1 second on the same database. Starting a database took a couple of seconds also.

I then took Tom's advice, and "reinstalled", putting a fresh copy of .prerelease 25 to another folder. The speed returned!

But then I copied the 47 extra icons I had (Armando's set, and maybe some of my own) back into the stock IQ folder of 70 icons. The speed decreased dramatically as before! Pierre, is there something in the coding that doesn't like more than the 70 icons???

Wow ! Great debugging Keith

I can't think of anything but will experiment and come back early next week

If you've got some spare time (!), can you try to remove 47 stock icons and copy Armando's 47 extra ones to see if it is (1) the icon set or (2) the number of icons which is the issue.

hmmm... I removed all icons from the icon folder and I've seen no speed improvement.

My project grid (pretty big) takes about 7.5s to open with our without the icons.

[However, we know that too many icons can affect overall windows performance -- GDI resource problem -- as I wrote it in an old thread. In any case, I tried again, just to make sure : removed all icons from the icon folder. No improvement.]

[quote=Pierre_Admin]

If you've got some spare time (!), can you try to remove 47 stock icons and copy Armando's 47 extra ones to see if it is (1) the icon set or (2) the number of icons which is the issue.

[/quote]

I found something unexpected--I had in the icons subdirectory the 3 themes files, *.cjstyles, which are fairly large 288-560k. Once I removed those, the speed was back. (I wonder how those got in there?)

For grins,I put a 7 megabyte movie file in there. IQ didn't start up, at least in a few minutes.

I then removed that big file, and copied over a few gif files--with 859 gif files in there, ranging in size from 56 bytes, to 986,552, the speed went down considerably.

I went down to 400 gif files, all under 1k, and the speed was back.

My guess is the sizes, more than the qty of files.

Can you check if the coding only reads in the .gif files, or does it read everything in that folder?

Interesting... I never tried with anything else than icons.

Even though removing or adding about 100 icons didn't do anything in my DB, you're right about the size aspect. This will affect performance.

Do you remember this thread (see my last post) : [SOLVED !] Can't open more than six-seven grids at a time -- 7th or 8th grid freezes or crashes IQ

-- comment : http://www.sqlnotes.net/drupal5/index.php?q=node/1067#comment-3952

That's why I avoid putting 100s of icons in the icon folder.

Pierre said in that old thread that icons should be loaded differently. I guess it should've been put in Mantis.

[quote=Armando]

Do you remember this thread (see my last post) : [SOLVED !] Can't open more than six-seven grids at a time -- 7th or 8th grid freezes or crashes IQ

-- comment : http://www.sqlnotes.net/drupal5/index.php?q=node/1067#comment-3952

Pierre said in that old thread that icons should be loaded differently. I guess it should've been put in Mantis.

[/quote] I remember that thread, but there was something else going on at about the same time--I remember crashing out at 24 grids, but there was some other reason...I can't seem to find that. (If only we had a printer friendly view for the forums, then use jibz script to make a chm of forum, for searching purposes, etc)

Let's see what Pierre comes up with Monday, and then see if it's appropriate to Mantis request load icons on demand, etc.

Or for that matter, what is the end state for icons,--will there eventually be a larger/more colorful ones available, etc.

You don't have any non-gif files in your icons folder, do you?

[quote=KeithB]

Let's see what Pierre comes up with Monday, and then see if it's appropriate to Mantis request load icons on demand, etc.

Or for that matter, what is the end state for icons,--will there eventually be a larger/more colorful ones available, etc.

You don't have any non-gif files in your icons folder, do you?

[/quote]

I added 2 conditions to load a file in the \icons\ folder:

File must be an image type : "JPG", "JPEG", "BMP", "GIF", "PNG", "ICO", "TIF", "TIFF"
File size must be less than 32KB

This should solve this issue

Fixed in 0.9.25Pre-Rel26