InfoQube IM - Community

Submitted by Armando on 2009/01/08 23:23

... and searching is now almost impossible (hence : database unusable)

In between December 20, and January 5, my database went from 161 mb to 1.26 gb.

This is weird...

During that period, I added 156 items to the database. Some with more content then others... But 1 gb ?? No, that's impossible.

Searching has become almost impossible (so this seems to indicate that a big database, with the current architecture, might not be easy to work with).

(I tried a repair... Nothing... Compacting doesn't do much either. Closing SQLNotes crashed my computer once.... But will have to do some more testing during the weekend -- going to bed now.)

Otherwise, Is there a way to... bring the database back to a manageable size -- where it was before -- without to much messing around ?

Thanks.

(I'm starting to wonder if IQ's web clipper would download videos if there are videos on a web page... That's the only thing I can think of.

Comments

You can open the ItemHTMLSize field and sort by size. For large items, you can do HTMLMenu > Edit > Delete database copy. You can also check the .files folder for large MHT files and check the corresponding item in the database

[edit] There is a new setting regarding this: Tools > Options > This File > Data storage: Save a copy of the HTML/MHT file in the database. If you can uncheck this. The only impact is that if the HTML/MHT file is not found, the content will not be accessible. simply copy the .files sub-folder with your IQBase.

I haven't been able to solve the problem. My largest MHT file is 2 mb, Which is really not that big.

I have tried to delete the database copy for the largest files... But that doesn't seem to change anything.

I'll see what I'll try next.... Maybe just reverting to the 3 weeks old backup as I didn't do much serious work since then. But that doesn,t seem to be the best solution.

When I backup my database, the compression ratio is 98%!! This is not normal. This database clearly has some structural problems.

Unfortunately, I there's way too much private stuff in there to even consider sending it for analysis. So there. I'm kinda stuck with a sick database.

Is there a way out of this trap ??? I need to be able to function again...

Because I really really need to get back to work, I started to transfer some data from one database to another and... it's fairly nightmarish.

Parent linking is not transferred (only children/subitems of selected items, in the copy paste process), etc. And so I realize that if things go bad in such a way, there are no easy import/export solution from one database to another (it will certainly take me many hours, which I can't afford)

Nope. It won't work. It would take me so much time to get it back as it was 3-4 weeks ago + all the work done during this time. There's no way.

So I'll be waiting for a real solution.

I'm sorry to hear that. For what it's worth, I had my database become corrupt recently, and the only thing that saved me was a backup, a few day's old backup, that was in an old directory (I'd moved my IQ stuff to c:\IQ). It's a long shot, but did you check for any trace of more backups? Do you have all the backups turned on?

(I'm paranoid about backing up all the ways possible, with at least 1 manual backup daily--of course I have to clear out the backup directory occasionally (like soon, since it's 10gig)

Were you just trying to export the 156 items, or everything? Is it possible to use the 3-wk old backup, and just export the 156 items out of the probably-corrupt database, and pull them into the backup? Then you could run another IQ session, pulling up the corrupt database, to check for the linking and stuff? Enough to get running, and save the old corrupt database to look at later, like I've done?

That's the only ideas I have. Hopefully Pierre will check in soon and help.

Thanks Keith.

I backup everything everyday, so the problem is not the backup per se. The problem is that the only valid database is 3-4 weeks old... because it took me a while to realize that "quick searching" had become slow has hell.

I tried copying and pasting all "newly" modified and created items (actually much much more than 156 -- 156 is only for newly created items and doesn't take into account all modified items during that period).

That doesn't work well, of course : some items get duplicated, others don't appear under the proper parent, etc. etc. Nightmare.

As for "pulling up" or importing data from another database, I havce no idea how to do that.

So what I'm doing at the moment -- because I'm pretty dependent on IQ -- is trying to work with my "corrupt" database. Don't really have the choice -- I've been avoiding using it for a few days, and I now need to get some work done.

Unfortunately I'm having other problems like :

Don't know where this one is coming from.

Problem found and solved!!!!!!

Ah!

I investigated the above sum error further. And I found out that only one of my field was corrupt : a number field had been converted to a text field. Impossible to convert it back for some mysterious reasons. So I deleted it, recreated it, and bingo. My data base is now 164 mb.

How did I discover that the field was corrupt ? I noticed that the field's content was filled with "0", infinitely. That's what was taking all the space. Everytime I was opening grids with that field, and made modifications, calculations were updated, and items associated with this field were filling up -- well, that's my interpretation.

Anyhow. Time to sleep. I'm relieved.

[quote=Armando]

Problem found and solved!!!!!!
...
Anyhow. Time to sleep. I'm relieved
[/quote]

a relief for everyone I'm sure

To everybody : I don't know why this happened, but will check it out with Pierre soon.

I've had a few computer Blue Screens of death in the last 2 months, so, possibly, one of them affected something in the database at some point.

Anyhow... we'll see...

Interestingly, like I said, the problem seemed/was related to a text field trying to display number calculations results (parents sums and others products) and was easily fixed. The DB was growing because data was indefinitely added to the field each time calculations were updated.

Still, the trouble you had copying items to a new database is a bit troubling. I can easily see reasons for wanting to do this and it would be nice to think it could be done well. Also, any database needs some sort of health check capability...computers will crash. I assume SQLNotes uses other safe database practices such as not keeping records open, etc.

Still, it would be nice to think one could copy items to a new database gracefully. Seems like you were having troubles with that. Also, is there a way to check the database health after a crash?

I wish Pierre had posted something in this thread. I mean, what's more basic than the data file. If that is in danger of easy corruption, it's gotta get solved or backups have to be improved or copying data has to be improved. Something.... yikes.

Jan Rifkinson

Ridgefield CT USA

HP Blackbird Vista Ultimate SP-1

Hi Jan,

Before posting, I wanted to meet with Armando (we both live in Montreal) to examine the database. What he observed was never seen before, so I wanted to check it out first. I think that all users can report that the your data is very safe with InfoQube.

Thanks, Pierre. To show you how paranoid I am, IQ is making backups every 10 mins.... going to a terrabyte drive. Question: can user put limit on backups, i.e. example: 10 backups, then IQ starts to delete the 11th, i.e. oldest? If not, is this doable or do you think this is a lousy idea?

Jan Rifkinson

Ridgefield CT USA

HP Blackbird Vista Ultimate SP-1

Ok. I sounded nervous and everything, because I was under a lot of stress and I really needed my database to work well.

Database problems are not fun...

And yes, data transfer from one database to another could be easier.

However, as Pierre said, IQ database corruption seems very unlikely.

And my case might not even be a case of real database corruption, but just... my own fault (combined, maybe, with some other factor).

Please read on...

I took the last hour and a half to examine backups since September (I didn't have the time to do it before), and... the guilty field was already a text field back then ! So I might have made that mistake (creating a text field instead of a number field) right at the beginning, but... didn't start having problem until recently, for some strange reasons.

The weird thing is that the formula in that field (a product between to variables) started to add "text" data (zeros) to it only after December 20th... and I got the user code error popup only recently. It's also weird that a number formula would work with a text field... Maybe a vb function was slightly changed between different IQ versions ? Don't know.