[ghns] Re : about SoC proposal at Blender Foundation on GHNS + questions

Wed Apr 9 15:31:33 PDT 2008

Hi

I thank you for your clear and detailed reply.
I hope that you do no mind if I think aloud.
In this mail, I think over the 3 protocol option you Frank said (within the"
== ... of my reflexion ===" boundaries) and then I pose some questions to
you and tell which api option I would prefer.

=== start of my reflexion ===
*as to option 1.* with autogenerated feeds located at php page.
you say that is doesn't support searching, and a workaround for this would
be to create lots of ids. For example, I see on
http://www.kde-look.org/help/ghns.php the following variation of themes :
soundthemes,mousethemes,kdmthemes,amarokthemes,plasmathemes,iconthemes.
What I would consider doing if ever I used this 1st option would be to have
lots of id names for example for materials :
materialswood,materialsmetal,materialsground,materialsglass,materialsnature...
one problem is we'd need a materials feed that would lead to this list of
materials.
Because I do not want people in Blender to see all the different materials
types from the beginning. User will want to download more kinds of data
block than just materials : models, scripts, compositing nodes, logic
bricks, armature setups, textures.
I think that in the end, we would need a lot of Ids which could make the
feed page too long to download when running Blender's assets manager. Unless
we cache that page locally and refresh once a day on the Blender's plug in
launch.
There would not be a big problem from the web interface point of view to
have as many ideas, as what kde-looks.org shows is that the web interface
categories map to the different ids or vice-versa.

(+) on the client side :
- direct access to what to download with little xml browsing required
- script is easy to implement
- no authentication needed
- load distributed over many servers around the globe

(-) on the client side :
- searching has to be done by the client script
- if users submit data, they'll have to way < 1 hour to see it up online
through the client script
- there needs do exist some file online or offline that tells which category
has which sub-categories=id names. This is awkward to have all the
subcategories on the same root in the feeds list though that's not a
problem.
- there's no nb of items per page limit compared to the REST api (10 items
max / page and possibility to request which page to display by its number)
so this makes the client potentially download a lot (ie. imagine there a
lots of glass materials). Getting the N first items of a list would be ok
(ie : close the http xml download right from the moment when know we have
downloaded N elements), but as to the next N elements one would have to use
http resume support.

*as to option 2.*
(+) on the client side :
- can get content/info without having for the script to browse through xml
hyperlinks, just make SOAP calls.
- if we use a SOAP library things are very easy
I've found something in the dive into python book with wsdl
http://www.diveintopython.org/soap_web_services/google.html which makes
method calling extremely easy if a wsdl file exists, one problem is that
SOAPpy used there isn't in
- one button click really corresponds to one web SOAP call.
- voting and searching implemented.
- sorting is not done by the client.

(-) on the client side :
- authentication needed
- as to the SOAP capabilities on Python's side : Python's standard
distribution (and there's no soap library at all in the python's standard
distrib I think)... so I'd have to include the whole SOAPpy package inside
Blender releases which would push them to be "unofficial" because I doubt
that such an inclusion would be accepted. Or I'd implement the SOAPing with
xml.minidom and libhttp myself. Maybe, the blender script could, not contain
this package, and on the first run oblige the user to click on a "download
SOAPpy package now" which would make the script download and install the
package inside Blender's .blender/scripts/ directory.
I should talk about that to the Blender developpers maybe. I think from at
first thought that : it preferable not to increase the size of Blender
builds too much; it would be nice that the script (some scripts are packaged
with official Blender builds but they rarely - I think - depend on external
libraries) could work right way one run, not requiring to download a soap
library.
- broadband internet connection is required

*as to option 3.*
(+)
- easy to implement. Coding the client could be very evident.
- no real need for libs not present in a standard python distribution.
- one click equals one internet call
- not really a need for a higher speed connection compared to SOAP.
- sorting is not done by the client
- voting and search implement.

(-)
- authentication required

=== end of my reflexions ==

The requirement to authenticate for using a SOAP/REST interface is not a
burdain to users to me. What makes me a little reluctant to it though is
that login + password are sent in clear. Instead, why not send the user a
generated image with "garbled"/morphed text and ask him to type the
corresponding text ? Doing that kind of translation is maybe something that
bots can't do and this could spare people from signing up.
I think that having to sign up for uploading contents would be a good thing,
so that what's posted is related to some user with with all his/her details
at hand for visitors.

"If someone sends 100 search requests a second my server is down." do you
mean for just one person sending 100 requests ? Or do you mean, that if
several people start flooding with searches at the same time, this will you
bring your server down ?

How do you submit content ? on the ghns feed I can't find any upload url
(just a nouploadurl attribute...), and for the REST api contents I can fetch
I see no upload url attribute at all. I don't know as to the SOAP interface.
Would that be compulsory to use the web interface form to upload things :
http://www.kde-look.org/content/add.php ?

I'm interested in the REST API more than in the SOAP one because of the
external libraries requirement issue. So if I were to tell you to set
something up I would request the platform to have that first API.
However, please do not setup something for blender yet. I've just written to
the bf-committers (bf=blender foundation) list. I'll let you know about the
outcome of the replies.
The mail I posted is here :
http://lists.blender.org/pipermail/bf-committers/2008-April/020741.html

Sincerely,
Jonathan-david Schroder

On Wed, Apr 9, 2008 at 4:59 PM, Frank Karlitschek <karlitschek at kde.org>
wrote:

> Hi Jonathan,
>
> sorry for the confusion.
> We are talking about three different interfaces here.
>
> 1. GHNS v1.0
> I´m generating static xml files and distribute them, together with the
> download files over the kde mirror network. This is done every hour.
> The files are accessed from the KDE Application over the host
> download.kde.org which is doing roundrobin via DNS.
>
> You find the feeds here:
> http://www.kde-look.org/help/ghns.php
>
> This is used by most of the KDE applications at the moment.
> The benefit of this solution is that the load is distributed over lots of
> high capacity servers all over the globe. Access to the content is possible
> without authentication because a deny of service attack is very difficult.
>
> This could be used for blender too. But I´m not sure if we are allowed to
> use the kde mirrors for blender content.
>
> 2. GHNS v2.0
> The new Specification done my Josef. Also called DXS.
> It is using SOAP. It is quite powerful but also heavyweight on the server.
> The only serverside implementation I know is the one from Josef.
> I also plan to support the interface in the future but it is not
> implemented yet.
>
>
>
> 3. openDesktop.org REST
> The REST interface done by me. You find the details here:
> http://api.opendesktop.org
>
> The interface is more powerfull than the GHNS v1.0 It is possible to
> browse all the content from the sites and supports voting. I will also
> implement searching in the near future.
>
> I can setup a new community website for blender in less than a week
> together with a REST api for you. I can also integrate this website into the
> openDesktop.org network so it is getting promotion and users fast.
>
> The problem is that this REST interface is vulnerable for deny of service
> attacks or script kiddies. If someone sends 100 search requests a second my
> server is down.
> Because of that access to this interface is only possible for
> authenticated users or from dedicated IPs.
>
> Google, Yahoo, Amazon and eBay are doing the same because of the same
> reason.
>
>
> So I´m not sure what to suggest.
>
> Interface 1 has the best performance. But it only supports downloading and
> no searching. And you need server infrastructure.
>
> Interface 2 hast the most features. But I´m not sure if Josefs
> Implementation is ready and if there is a community system. It also has the
> deny of service problem.
>
> Interface 3 is only usable from authenticated users. I´m not sure if this
> makes sense to require for blender users.
>
>
> What do you think?
>
>
> Best Regards
> Frank
>
>
>
>
>
> On 09.04.2008, at 15:35, Jonathan-David SCHRODER wrote:
>
> > Hi again,
> > please forgive me for my last message that was missing words here and
> > there.
> > Thank you for you answer Josef.
> > Ok I'll try to show the Bf (blender foundation) that I'm already
> > tinkering with the scripts (this is what I've started doing, for real )and
> > that I'm getting advice on this list.
> >
> > I have studied some xml pages fetched thanks to the api.
> > Things look good. On this I have question, how is that possible to
> > submit new files to the server through the api ? Submit doesn't belong the 3
> > functions list from api.kde-look.org .
> > Also, what's the difference between the data fetched thanks to the api
> > from : api.kde-look.org , and the data fetchable by browsing through xml
> > pages, starting from the ones mentionned here :
> > http://www.kde-look.org/help/ghns.php ?
> > I don't understand much how to find again the same categories listed on
> > http://api.opendesktop.org/V1/CATEGORIES, and on
> > http://www.kde-look.org/help/ghns.php .
> >
> > I've had a look at :
> > http://svn.ggzgamingzone.org/trac.cgi/browser/trunk/python/lib/sdlnewstuff.py?rev=head .
> > It uses the http://www.kde-look.org/help/ghns.php interface with the
> > providers.xml files. To me, this interface could also be called a REST
> > interface somehow, because there's no SOAP, and it uses the download xml
> > -then- go to one of the hyperlinks steps which resemble what I've read about
> > REST.
> > I haven't been able to run the python script because my pygame install
> > on linux is weird (doing apt-get install pygame does install pygame but does
> > not setup the pythonpath to add pygame's folfers... looking on the web, this
> > seems to be an old deb. package bug while I'm using a much newer ubuntu
> > distribution...). Changing my pyhonpath the way I could guess does make
> > importing pygame work partly, but I still have problems finding some parts
> > of the pygame library... Maybe I'll purge the sdlnewstuff script from any
> > pygame related code and try to run it, though reading the code is enough to
> > understand what this script does with the ghns interface.
> >
> > I'd be ready to rewrite some python script using the REST interface,
> > maybe complementing what I could see with sdlnewstuff and making a python
> > only library to be used by any kind of software.
> >
> > Sincerely,
> > Jonathan-david Schröder
> >
> > On Wed, Apr 9, 2008 at 11:38 AM, Josef Spillner <spillner at kde.org>
> > wrote:
> > Am Dienstag 08 April 2008 03:14:57 schrieb Jonathan-David SCHRODER:
> > > I've "met" on #blendercoders a blender developper interested in my
> > proposal
> > > (he had access to the SoC admin side for the blender foundation)
> > because he
> > > worked using mysql for blender files data blocks.
> > > He wants to talk more about the proposal with me in a few days.
> >
> > You can tell him that you could easily get advice on GHNS-specifics on
> > this
> > list even if someone from Blender becomes a mentor.
> >
> > > By then... I could have tried to use khotnewstuff, and maybe some
> > shell
> > > scripts I've seen on the repos to know that I can make the GHNS system
> > work
> > > somehow and start discovering more about it.
> >
> > It will give a positive view if you already tinker with the scripts and
> > servers prior to the application rating deadline.
> >
> > Josef
> >
> > P.S. You're always on CC during the whole mail thread. Are you
> > subscribed to
> > it now? We could drop the CC then.
> >
> >
> >
> >
> > --
> > http://www.jaxtr.com/myselfhimself_______________________________________________
> > ghns mailing list
> > ghns at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/ghns
> >
>
> --
> Frank Karlitschek
> karlitschek at kde.org
>
>
>
>
>

-- 
http://www.jaxtr.com/myselfhimself
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freedesktop.org/archives/ghns/attachments/20080410/6b442515/attachment.htm 