[Clipart] [support.osuosl.org #5206] big file loss (openclipart)

[Jon Phillips]

Thanks Rudy! We have successfully restored the old db from THURSDAY. I
sent emails to all artists who uploaded artwork from the 22 until now
and asked to re-upload please. 

We have a dump and backups of all data (db dump a few minutes before
restore and like 15 uploaded files on-disk) between the 22 until 26, but
will be difficult to update so I think our best option is to encourage
update of any changes in this timeframe.

I apologize for this much and hope we can not have this happen again in
the future.

Jon

On Mon, 2008-08-25 at 15:41 +0000, Rudy Grigar via RT wrote:
> On Mon Aug 25 06:12:00 2008, jon at rejon.org wrote:
> > On Sun, 2008-08-24 at 22:51 +0000, Lance Albertson via RT wrote:
> > > On Sat Aug 23 13:06:52 2008, jon at rejon.org wrote:
> > > > Something doesn't look right...anyone have the last count?
> > > > 
> > > > OSUOSL? Has anyone moved things around on the Open Clip Art Library
> > > > site in the last day or so? We are getting reports of lost files...
> > > > 
> > > > What backups do we have for openclipart.org as well? Maybe need to
> > > > review backups from a week ago or a few days...I haven't synced in a
> > > > few with the test.openclipart.org
> > > > 
> > > > Thanks all...working to track this down.
> > > 
> > > I'm not sure if this is related or not, but we have been getting an odd
> > > email from one of your cronjobs lately. I finally took a look at it just
> > > now, and noticed that the permissions on three of the files (the
> > > filelist* ones for snapshot creation) were wrong and weren't being
> > > updated correctly. It seems as though this has been broken since
> > > February. I just fixed it, but haven't ran the script yet. If you'd
> > > like I can do that for you and see how it goes. Also, I could add you
> > > or someone else onto the cronjob so that you could see the errors too.
> > > 
> > > As for the database server, we did encounter a deadlock situation on
> > > Friday which required us to restart the daemon. We never did figure out
> > > the cause but we had no errors in the logs following the restart, so I
> > > doubt we had any database corruption.
> > 
> > can you point to the logs...I'll stay in osuosl irc as well to pinpoint
> > in realtime...
> > 
> > > Where exactly are the files at that are missing? This ticket has been a
> > > bit vague on where they should be. It would make tracking this problem
> > > down a lot easier.
> > > 
> > > Thanks!
> > 
> > Jon
> > 
> > > > On Fri, 2008-08-22 at 23:21 -0700, Greg Bulmash wrote:
> > > > > Jon,
> > > > > 
> > > > > Perhaps this was what Angelo was referring to.  I just went to the
> > > > > site, and there are 2 files dated today, then it next file is dated
> > > > > Feb 5th.
> > > > > 
> > > > > John Olsen wrote:
> > > > > > What has happened to the files uploaded after Feb. 5, 08?  There
> > > > > > are  two August files after them but then all the rest of the last
> > > > > > 6 months  is gone.  Was this a rollback as it shows a larger
> > > > > > number of unchecked  files and files needed cleanup then the
> > > > > > recent numbers had and with  about 2000 less files!
> > > 
> > > 
> 
> Jon,
> 
> I believe the database issue happened either with the deadlock or while
> diagnosing the deadlock we had with mysql on Friday.  We do keep nightly
> backups of all of our databases, as well as binary logs.  With your
> permission I can restore our backup from Thursday evening (these are
> taken at roughly 9PM PST) to get the lost data restored.
> 
> To be safe it'd be a good idea to take the site offline for a few
> minutes while the restore is taking place, I am also on IRC (basic`) if
> you would like to coordinate things further from there.
> 
> I am extremely sorry for the trouble and confusion this may have caused
> you and your project, and for not being around this weekend to take care
> of it sooner.
> 
> Sincerely-
> --
> Rudy Grigar
> OSUOSL
> 
-- 
Jon Phillips
San Francisco, CA + Guangzhou + Beijing
GLOBAL +1.415.830.3884
CHINA +86.1.360.282.8624
jon at rejon.org
http://www.rejon.org
IM/skype: kidproto
Jabber: rejon at gristle.org
IRC: rejon at irc.freenode.net