[immodule-qt] Re: Please restart apache2 to free permanently locked svn mutex

[YamaKen]

At Tue, 7 Sep 2004 10:45:57 +1000,
daniel at freedesktop.org wrote:
> 
> On Mon, Sep 06, 2004 at 07:52:32PM +0900, YamaKen wrote:
> > At Sun, 5 Sep 2004 22:10:59 +1000,
> > daniel at freedesktop.org wrote:
> > > 
> > > On Thu, Sep 02, 2004 at 12:30:15AM +0900, YamaKen wrote:
> > > > Hello admins, please help our project.
> > > > 
> > > > I'm a developer of project "immodule for Qt" on fdo, and we are
> > > > suffering strange deadlock problem on the svn repository. Please
> > > > restart apache2 to free the permanently locked svn mutex.
> > > 
> > > Done -- sorry for the delay.
> > 
> > Thank you. But the trouble is remaining, please help again.
> > 
> > An invalid apache2 process that locks our svn repository is
> > still living. Please kill it:
> 
> A killall -9 apache2 later, I think it should be OK.

Our svn repository has been successfully restored with some
investigation and surgery after your operation. Thank you for
cooperation.

I've written a recovery procedure memo. Refer it when same
problem attacks other projects in future.

----------------------------------------------------------------
* Problem

If a svn repository is suffering the problem, any processes that
attempt to access to the repository will be waiting forever,
even if the process is the low-level deadlock-detecting tool of
Berkeley DB db_deadlock.

In such situation, there is a permanently locked mutex on mapped
shared file /svn/immqt/db/__db.001 and so on. But the owner of
mutex is already died accidencialy without freeing the mutex. It
is not a usual transactional deadlock. Therefore, we cannot
unlock, recover or dump information in the database with
ordinary administration tools in subversion or Berkeley DB.


* How to identify the problem

Run following command (rewtite '/svn/immqt/db/' with your
repository).

$ fuser /svn/immqt/db/*

It will shows many of pending processes such as apache2 or
svnserve.distrib if the repository is suffering the problem.
In such case, kill pending proccess first.

Once pending proccess are killed and fuser(1) is not showing
such processes, try following command.

$ strace db_deadlock -v -a e -h /svn/immqt/db

If the command does not exit and repeats following message, the
repository is having the problem.

select(0, NULL, NULL, NULL, {0, 25000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 25000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 25000}) = 0 (Timeout)
...


* How to restore the repository

If the repository is having the problem, we must perform
unlocking in improper way. Try following steps.

(1) tar cvzf svndb-immqt.tar.gz /svn/immqt
(2) tar xvzf svndb-immqt.tar.gz
(3) ensure nobody is accessing to the database extracted in (2)
(4) build Berkeley DB with db-4.2.52-mutexbuster.diff. don't
    install it!

  tar xvzf db-4.2.52.tar.gz
  cd db-4.2.52
  patch -p1 <db-4.2.52-mutexbuster.diff
  cd build_unix
  ../dist/configure --disable-shared --enable-static --enable-debug
  make

(5) run db_deadlock in the build directory with gdb. It releases
    3 permanetly locked mutexes

  gdb ./db_deadlock
  (gdb) r -v -a e -h /svn/immqt/db
  db_deadlock: running at Tue Sep  7 11:10:32 2004
  
  Program exited normally.

(6) if (5) is not exit normally, another mutex may be
    locked. interrupt gdb and find the locked mutex. Once the
    mutex found, clear the mutex as same as in
    db-4.2.52-mutexbuster.diff.

(7) dump svn database

  svnadmin recover somewhere/svn/immqt/
  svnadmin dump somewhere/svn/immqt/ >svn-immqt.dump

(8) omit dead repository and reconstruct new one with the dumped
    flie. don't use the repository recovered in (5).
----------------------------------------------------------------


-------------------------------
YamaKen  yamaken at bp.iij4u.or.jp
-------------- next part --------------
A non-text attachment was scrubbed...
Name: db-4.2.52-mutexbuster.diff
Type: application/octet-stream
Size: 1214 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/immodule-qt/attachments/20040909/0867f93c/attachment.obj