[PATCH RFC]: Generic Linux multi-domain PCI handling

[David S. Miller]

From: Ian Romanick <idr at us.ibm.com>
Date: Tue, 03 Jan 2006 09:39:21 -0800

> Ironically, the first part of the year for me is going to be occupied
> working on some similar stuff.  There are some pretty significant
> problems with 32-bit X-servers on 64-bit pSeries hardware.  In addition
> to non-zero PCI domains, devices in domain 0 get mapped above
> 0x7fffffff.  This causes problems when doing an mmap of /dev/mem in
> 32-bit mode. :(

Not if you end up using mmap64() properly.  We had a similar problem
with a tool that mmap()'d network device registers in order to test
PCI PIO latency.  And the fix was simply to add "-D_FILE_OFFSET_BITS=64"
to the build command line.  You can probably do something similar.

> This is similar to the approach that I was going to take.  However, I
> was going to go a bit further.  Rather than using /proc, I was going to
> add a new version of pciFindNext that iterated over the entries in
> /sys/bus/pci/devices.  That way only devices that actually exist get
> iterated.  The guess-and-check method used in the existing pciFindNext
> implementations has always bothered me.  Of course, if /sys doesn't
> exist it would fall back to the older method.

That's a great way to do this.  And if you'll go that far, it's probably
time to just do the right thing and actually use libpci.a from pciutils.
We're reimplementating a lot of code already in there that knows
perfectly well how to decode the various /proc/bus/pci and /sys/bus/pci
incantations depending upon what kernel version and what filesystems
are mounted.

> The second part would be to wrap the existing PCI structures and enums
> (e.g., PCITAG) with a slightly higher-level structure.  Rather than
> mapping by offset, drivers really want to say, "Map this resource of
> this device."  Since, on 64-bit platforms, /sys has a separate file for
> each BAR, we can mmap that file with a zero offset instead of mapping
> /dev/mem with a > 32-bit offset.

See above why the > 32-bit offset mmap is not a problem.  You can do
it properly, you just have to tell the build environment that you
want to pass in a 64-bit offset to mmap().