[Clipart] [Fwd: Re: [support.osuosl.org #5206] big file loss (openclipart)]

[Jon Phillips]

Cool

Jon

-------- Forwarded Message --------
> From: John Crockford <john52 at crockford.org>
> To: Jon Phillips <jon at rejon.org>
> Subject: Re: [support.osuosl.org #5206] big file loss (openclipart)
> Date: Mon, 25 Aug 2008 10:42:30 -0700
> 
> Thank you very much. I just logged into my account and found everything 
> there.
> 
> John Crockford
> 
> On Aug 25, 2008, at 10:05 AM, Jon Phillips wrote:
> 
> Thanks Rudy! We have successfully restored the old db from THURSDAY. I
> sent emails to all artists who uploaded artwork from the 22 until now
> and asked to re-upload please.
> 
> We have a dump and backups of all data (db dump a few minutes before
> restore and like 15 uploaded files on-disk) between the 22 until 26, but
> will be difficult to update so I think our best option is to encourage
> update of any changes in this timeframe.
> 
> I apologize for this much and hope we can not have this happen again in
> the future.
> 
> Jon
> 
> On Mon, 2008-08-25 at 15:41 +0000, Rudy Grigar via RT wrote:
> > On Mon Aug 25 06:12:00 2008, jon at rejon.org wrote:
> >> On Sun, 2008-08-24 at 22:51 +0000, Lance Albertson via RT wrote:
> >>> On Sat Aug 23 13:06:52 2008, jon at rejon.org wrote:
> >>>> Something doesn't look right...anyone have the last count?
> >>>>
> >>>> OSUOSL? Has anyone moved things around on the Open Clip Art Library
> >>>> site in the last day or so? We are getting reports of lost files...
> >>>>
> >>>> What backups do we have for openclipart.org as well? Maybe need to
> >>>> review backups from a week ago or a few days...I haven't synced in a
> >>>> few with the test.openclipart.org
> >>>>
> >>>> Thanks all...working to track this down.
> >>>
> >>> I'm not sure if this is related or not, but we have been getting an 
> >>> odd
> >>> email from one of your cronjobs lately. I finally took a look at it 
> >>> just
> >>> now, and noticed that the permissions on three of the files (the
> >>> filelist* ones for snapshot creation) were wrong and weren't being
> >>> updated correctly. It seems as though this has been broken since
> >>> February. I just fixed it, but haven't ran the script yet. If you'd
> >>> like I can do that for you and see how it goes. Also, I could add you
> >>> or someone else onto the cronjob so that you could see the errors 
> >>> too.
> >>>
> >>> As for the database server, we did encounter a deadlock situation on
> >>> Friday which required us to restart the daemon. We never did figure 
> >>> out
> >>> the cause but we had no errors in the logs following the restart, so 
> >>> I
> >>> doubt we had any database corruption.
> >>
> >> can you point to the logs...I'll stay in osuosl irc as well to 
> >> pinpoint
> >> in realtime...
> >>
> >>> Where exactly are the files at that are missing? This ticket has 
> >>> been a
> >>> bit vague on where they should be. It would make tracking this 
> >>> problem
> >>> down a lot easier.
> >>>
> >>> Thanks!
> >>
> >> Jon
> >>
> >>>> On Fri, 2008-08-22 at 23:21 -0700, Greg Bulmash wrote:
> >>>>> Jon,
> >>>>>
> >>>>> Perhaps this was what Angelo was referring to.  I just went to the
> >>>>> site, and there are 2 files dated today, then it next file is dated
> >>>>> Feb 5th.
> >>>>>
> >>>>> John Olsen wrote:
> >>>>>> What has happened to the files uploaded after Feb. 5, 08?  There
> >>>>>> are  two August files after them but then all the rest of the last
> >>>>>> 6 months  is gone.  Was this a rollback as it shows a larger
> >>>>>> number of unchecked  files and files needed cleanup then the
> >>>>>> recent numbers had and with  about 2000 less files!
> >>>
> >>>
> >
> > Jon,
> >
> > I believe the database issue happened either with the deadlock or while
> > diagnosing the deadlock we had with mysql on Friday.  We do keep 
> > nightly
> > backups of all of our databases, as well as binary logs.  With your
> > permission I can restore our backup from Thursday evening (these are
> > taken at roughly 9PM PST) to get the lost data restored.
> >
> > To be safe it'd be a good idea to take the site offline for a few
> > minutes while the restore is taking place, I am also on IRC (basic`) if
> > you would like to coordinate things further from there.
> >
> > I am extremely sorry for the trouble and confusion this may have caused
> > you and your project, and for not being around this weekend to take 
> > care
> > of it sooner.
> >
> > Sincerely-
> > --
> > Rudy Grigar
> > OSUOSL
> >
> -- 
> Jon Phillips
> San Francisco, CA + Guangzhou + Beijing
> GLOBAL +1.415.830.3884
> CHINA +86.1.360.282.8624
> jon at rejon.org
> http://www.rejon.org
> IM/skype: kidproto
> Jabber: rejon at gristle.org
> IRC: rejon at irc.freenode.net
> 
-- 
Jon Phillips
San Francisco, CA + Guangzhou + Beijing
GLOBAL +1.415.830.3884
CHINA +86.1.360.282.8624
jon at rejon.org
http://www.rejon.org
IM/skype: kidproto
Jabber: rejon at gristle.org
IRC: rejon at irc.freenode.net