no such metadata / no such ref errors on the command line

[Dan Nicholson]

On Fri, Apr 6, 2018 at 2:33 AM, Michael Gratton <mike at vee.net> wrote:
> On Fri, Apr 6, 2018 at 5:20 PM, Alexander Larsson <alexl at redhat.com> wrote:
>>
>>
>> Hmm, i wonder what that commit is from, can you run:
>>
>>  grep -r 3f97e8db7dffa626bd19b4a763b6a371698c58054877215c00bf1b1d4d00db50
>> /var/lib/flatpak/repo/refs/
>>
>> And see if you have a local ref pointing to it?
>
>
> Yup, it's there:
>
>> mjg at payens:~$ grep -r
>> 3f97e8db7dffa626bd19b4a763b6a371698c58054877215c00bf1b1d4d00db50
>> /var/lib/flatpak/repo/refs/
>>
>> /var/lib/flatpak/repo/refs/remotes/gnome-apps-nightly/appstream/x86_64:3f97e8db7dffa626bd19b4a763b6a371698c58054877215c00bf1b1d4d00db50

We hit this issue a bunch at Endless (mostly self-inflicted because we
decided to share an ostree repo between flatpak and ostree). I wrote a
nasty script to find missing objects, mark commits as partial (so
ostree pull descends the whole commit), and then pulls them after
figuring out what ref/remote they point to.

https://github.com/endlessm/eos-meta/blob/master/eos-tech-support/eos-fix-ostree-repo

I think it would work if you pass it --repo=/var/liib/flatpak/repo,
but I've only ever tested it on Endless.

I intended to add the "find missing objects and mark associated
commits as partial" feature to ostree fsck, but ETIME.

If you can't run that script, the way I used to debug these failures
was pretty horrible.

# Run fsck verbose so it prints out the objects as it's going
ostree fsck --repo=$repo --verbose

# Eventually it fails with a missing object. Look back in the output
until you find the commit it was traversing. Mark that commit as
partial. Assuming the commit checksum is in $checksum:
touch $repo/objects/${checksum::2}/${checksum:2}.commitpartial

# Figure out what ref that commit points to
for ref in $(ostree --repo=$repo refs); do
  ostree --repo=$repo rev-parse $ref
done | grep $checksum

Then pull that remote/ref again.