[ghns] Re : about SoC proposal at Blender Foundation on GHNS + questions

Wed Apr 9 16:20:14 PDT 2008

Hi !

Do you think that there could be more than 1 category level ?
The REST api seems to show that there's only one level, and so does
the feed view with all the xml files.
If you want I could try and digg into the code myself and see how I
could implement that with you... Or maybe you'll hophop my friend sit
back, you've got other work to do.

This is in some way, customizing work that I'd be supposed to do as
part of my SoC. So please tell me, and I shall study and work on the
code on SVN (where is the repository actually ?)
I'd need more than 1 level of category (that is to say 2 for now : )
but this number may change).
If I could customize something else, that would be the possibility to
put search terms/regexps in the REST api.
I should see if I need to add search criteria.

I think I'll setup some wiki page somewhere on my personal web site
domain first with what I'd need for the web site.

Sincerely,
Jonathan Schröder

On Thu, Apr 10, 2008 at 12:31 AM, Jonathan-David SCHRODER
<myselfhimself at free.fr> wrote:
> Hi
>
> I thank you for your clear and detailed reply.
> I hope that you do no mind if I think aloud.
> In this mail, I think over the 3 protocol option you Frank said (within the"
> == ... of my reflexion ===" boundaries) and then I pose some questions to
> you and tell which api option I would prefer.
>
> === start of my reflexion ===
> as to option 1. with autogenerated feeds located at php page.
> you say that is doesn't support searching, and a workaround for this would
> be to create lots of ids. For example, I see on
> http://www.kde-look.org/help/ghns.php the following variation of themes :
> soundthemes,mousethemes,kdmthemes,amarokthemes,plasmathemes,iconthemes.
>  What I would consider doing if ever I used this 1st option would be to have
> lots of id names for example for materials :
> materialswood,materialsmetal,materialsground,materialsglass,materialsnature...
> one problem is we'd need a materials feed that would lead to this list of
> materials.
>  Because I do not want people in Blender to see all the different materials
> types from the beginning. User will want to download more kinds of data
> block than just materials : models, scripts, compositing nodes, logic
> bricks, armature setups, textures.
>  I think that in the end, we would need a lot of Ids which could make the
> feed page too long to download when running Blender's assets manager. Unless
> we cache that page locally and refresh once a day on the Blender's plug in
> launch.
>  There would not be a big problem from the web interface point of view to
> have as many ideas, as what kde-looks.org shows is that the web interface
> categories map to the different ids or vice-versa.
>
> (+) on the client side :
> - direct access to what to download with little xml browsing required
> - script is easy to implement
> - no authentication needed
> - load distributed over many servers around the globe
>
> (-) on the client side :
> - searching has to be done by the client script
> - if users submit data, they'll have to way < 1 hour to see it up online
> through the client script
> - there needs do exist some file online or offline that tells which category
> has which sub-categories=id names. This is awkward to have all the
> subcategories on the same root in the feeds list though that's not a
> problem.
>  - there's no nb of items per page limit compared to the REST api (10 items
> max / page and possibility to request which page to display by its number)
> so this makes the client potentially download a lot (ie. imagine there a
> lots of glass materials). Getting the N first items of a list would be ok
> (ie : close the http xml download right from the moment when know we have
> downloaded N elements), but as to the next N elements one would have to use
> http resume support.
>
> as to option 2.
> (+) on the client side :
> - can get content/info without having for the script to browse through xml
> hyperlinks, just make SOAP calls.
> - if we use a SOAP library things are very easy
>  I've found something in the dive into python book with wsdl
> http://www.diveintopython.org/soap_web_services/google.html which makes
> method calling extremely easy if a wsdl file exists, one problem is that
> SOAPpy used there isn't in
>  - one button click really corresponds to one web SOAP call.
> - voting and searching implemented.
> - sorting is not done by the client.
>
> (-) on the client side :
> - authentication needed
> - as to the SOAP capabilities on Python's side : Python's standard
> distribution (and there's no soap library at all in the python's standard
> distrib I think)... so I'd have to include the whole SOAPpy package inside
> Blender releases which would push them to be "unofficial" because I doubt
> that such an inclusion would be accepted. Or I'd implement the SOAPing with
> xml.minidom and libhttp myself. Maybe, the blender script could, not contain
> this package, and on the first run oblige the user to click on a "download
> SOAPpy package now" which would make the script download and install the
> package inside Blender's .blender/scripts/ directory.
> I should talk about that to the Blender developpers maybe. I think from at
> first thought that : it preferable not to increase the size of Blender
> builds too much; it would be nice that the script (some scripts are packaged
> with official Blender builds but they rarely - I think - depend on external
> libraries) could work right way one run, not requiring to download a soap
> library.
>  - broadband internet connection is required
>
> as to option 3.
> (+)
> - easy to implement. Coding the client could be very evident.
> - no real need for libs not present in a standard python distribution.
>  - one click equals one internet call
> - not really a need for a higher speed connection compared to SOAP.
> - sorting is not done by the client
> - voting and search implement.
>
> (-)
> - authentication required
>
> === end of my reflexions ==
>
> The requirement to authenticate for using a SOAP/REST interface is not a
> burdain to users to me. What makes me a little reluctant to it though is
> that login + password are sent in clear. Instead, why not send the user a
> generated image with "garbled"/morphed text and ask him to type the
> corresponding text ? Doing that kind of translation is maybe something that
> bots can't do and this could spare people from signing up.
>  I think that having to sign up for uploading contents would be a good
> thing, so that what's posted is related to some user with with all his/her
> details at hand for visitors.
>
>
> "If someone sends 100 search requests a second my server is down." do you
> mean for just one person sending 100 requests ? Or do you mean, that if
> several people start flooding with searches at the same time, this will you
> bring your server down ?
>
> How do you submit content ? on the ghns feed I can't find any upload url
> (just a nouploadurl attribute...), and for the REST api contents I can fetch
> I see no upload url attribute at all. I don't know as to the SOAP interface.
>  Would that be compulsory to use the web interface form to upload things :
> http://www.kde-look.org/content/add.php ?
>
> I'm interested in the REST API more than in the SOAP one because of the
> external libraries requirement issue. So if I were to tell you to set
> something up I would request the platform to have that first API.
>  However, please do not setup something for blender yet. I've just written
> to the bf-committers (bf=blender foundation) list. I'll let you know about
> the outcome of the replies.
> The mail I posted is here :
> http://lists.blender.org/pipermail/bf-committers/2008-April/020741.html
>
> Sincerely,
> Jonathan-david Schroder
>
>
>
>
> On Wed, Apr 9, 2008 at 4:59 PM, Frank Karlitschek <karlitschek at kde.org>
> wrote:
> > Hi Jonathan,
> >
> > sorry for the confusion.
> > We are talking about three different interfaces here.
> >
> > 1. GHNS v1.0
> > I´m generating static xml files and distribute them, together with the
> download files over the kde mirror network. This is done every hour.
> > The files are accessed from the KDE Application over the host
> download.kde.org which is doing roundrobin via DNS.
> >
> > You find the feeds here:
> >
> > http://www.kde-look.org/help/ghns.php
> >
> > This is used by most of the KDE applications at the moment.
> > The benefit of this solution is that the load is distributed over lots of
> high capacity servers all over the globe. Access to the content is possible
> without authentication because a deny of service attack is very difficult.
> >
> > This could be used for blender too. But I´m not sure if we are allowed to
> use the kde mirrors for blender content.
> >
> > 2. GHNS v2.0
> > The new Specification done my Josef. Also called DXS.
> > It is using SOAP. It is quite powerful but also heavyweight on the server.
> > The only serverside implementation I know is the one from Josef.
> > I also plan to support the interface in the future but it is not
> implemented yet.
> >
> >
> >
> > 3. openDesktop.org REST
> > The REST interface done by me. You find the details here:
> > http://api.opendesktop.org
> >
> > The interface is more powerfull than the GHNS v1.0 It is possible to
> browse all the content from the sites and supports voting. I will also
> implement searching in the near future.
> >
> > I can setup a new community website for blender in less than a week
> together with a REST api for you. I can also integrate this website into the
> openDesktop.org network so it is getting promotion and users fast.
> >
> > The problem is that this REST interface is vulnerable for deny of service
> attacks or script kiddies. If someone sends 100 search requests a second my
> server is down.
> > Because of that access to this interface is only possible for
> authenticated users or from dedicated IPs.
> >
> > Google, Yahoo, Amazon and eBay are doing the same because of the same
> reason.
> >
> >
> > So I´m not sure what to suggest.
> >
> > Interface 1 has the best performance. But it only supports downloading and
> no searching. And you need server infrastructure.
> >
> > Interface 2 hast the most features. But I´m not sure if Josefs
> Implementation is ready and if there is a community system. It also has the
> deny of service problem.
> >
> > Interface 3 is only usable from authenticated users. I´m not sure if this
> makes sense to require for blender users.
> >
> >
> > What do you think?
> >
> >
> > Best Regards
> > Frank
> >
> >
> >
> >
> >
> >
> >
> > On 09.04.2008, at 15:35, Jonathan-David SCHRODER wrote:
> >
> > >
> > >
> > >
> > > Hi again,
> > > please forgive me for my last message that was missing words here and
> there.
> > > Thank you for you answer Josef.
> > > Ok I'll try to show the Bf (blender foundation) that I'm already
> tinkering with the scripts (this is what I've started doing, for real )and
> that I'm getting advice on this list.
> > >
> > > I have studied some xml pages fetched thanks to the api.
> > > Things look good. On this I have question, how is that possible to
> submit new files to the server through the api ? Submit doesn't belong the 3
> functions list from api.kde-look.org .
> > > Also, what's the difference between the data fetched thanks to the api
> from : api.kde-look.org , and the data fetchable by browsing through xml
> pages, starting from the ones mentionned here :
> http://www.kde-look.org/help/ghns.php ?
> > > I don't understand much how to find again the same categories listed on
> http://api.opendesktop.org/V1/CATEGORIES, and on
> http://www.kde-look.org/help/ghns.php .
> > >
> > > I've had a look at :
> http://svn.ggzgamingzone.org/trac.cgi/browser/trunk/python/lib/sdlnewstuff.py?rev=head
> . It uses the http://www.kde-look.org/help/ghns.php interface with the
> providers.xml files. To me, this interface could also be called a REST
> interface somehow, because there's no SOAP, and it uses the download xml
> -then- go to one of the hyperlinks steps which resemble what I've read about
> REST.
> > > I haven't been able to run the python script because my pygame install
> on linux is weird (doing apt-get install pygame does install pygame but does
> not setup the pythonpath to add pygame's folfers... looking on the web, this
> seems to be an old deb. package bug while I'm using a much newer ubuntu
> distribution...). Changing my pyhonpath the way I could guess does make
> importing pygame work partly, but I still have problems finding some parts
> of the pygame library... Maybe I'll purge the sdlnewstuff script from any
> pygame related code and try to run it, though reading the code is enough to
> understand what this script does with the ghns interface.
> > >
> > > I'd be ready to rewrite some python script using the REST interface,
> maybe complementing what I could see with sdlnewstuff and making a python
> only library to be used by any kind of software.
> > >
> > > Sincerely,
> > > Jonathan-david Schröder
> > >
> > > On Wed, Apr 9, 2008 at 11:38 AM, Josef Spillner <spillner at kde.org>
> wrote:
> > > Am Dienstag 08 April 2008 03:14:57 schrieb Jonathan-David SCHRODER:
> > > > I've "met" on #blendercoders a blender developper interested in my
> proposal
> > > > (he had access to the SoC admin side for the blender foundation)
> because he
> > > > worked using mysql for blender files data blocks.
> > > > He wants to talk more about the proposal with me in a few days.
> > >
> > > You can tell him that you could easily get advice on GHNS-specifics on
> this
> > > list even if someone from Blender becomes a mentor.
> > >
> > > > By then... I could have tried to use khotnewstuff, and maybe some
> shell
> > > > scripts I've seen on the repos to know that I can make the GHNS system
> work
> > > > somehow and start discovering more about it.
> > >
> > > It will give a positive view if you already tinker with the scripts and
> > > servers prior to the application rating deadline.
> > >
> > > Josef
> > >
> > > P.S. You're always on CC during the whole mail thread. Are you
> subscribed to
> > > it now? We could drop the CC then.
> > >
> > >
> > >
> > >
> > > --
> > >
> > > http://www.jaxtr.com/myselfhimself
> _______________________________________________
> > > ghns mailing list
> > > ghns at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/ghns
> > >
> >
> >
> >
> >
> > --
> > Frank Karlitschek
> > karlitschek at kde.org
> >
> >
> >
> >
> >
>
>
>
> --
> http://www.jaxtr.com/myselfhimself

-- 
http://www.jaxtr.com/myselfhimself