This problem is much less severe now that Plone uses tinyMCE in the newer versions, but we still run into problems with older documents created in Kupu on older versions of Plone.
Case in point, yesterday I dumped the content of such a document to a file and cleaned it up. This resulted in a reduction in file size of more than 90%.
-rw-r--r-- 1 izak izak 3.2M 2011-09-15 15:52 /tmp/before.html -rw-r--r-- 1 izak izak 205K 2011-09-15 16:09 /tmp/after.html
One thing that TinyMCE definitely doesn't handle as well as Kupu, is 3.2M documents, so we can no longer ignore the MSWord bloat. I wrote the following bit of code to make the cleanup easier. It uses Elementtree.
import sys
from lxml import etree
from lxml.etree import HTMLParser
parser = HTMLParser()
fp = open(sys.argv[1], 'r')
tree = etree.parse(fp, parser)
fp.close()
xslt = etree.XML("""\
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:template match="comment()" />
<xsl:template match="style" />
<xsl:template match="link" />
<xsl:template match="@*|node()">
<xsl:copy><xsl:apply-templates select="@*|node()"/></xsl:copy>
</xsl:template>
</xsl:stylesheet>""")
transform = etree.XSLT(xslt)
newtree = transform(tree)
print str(newtree)
I hope this is useful to someone.
]]>This is a somewhat opinionated post on security practices used with projects that use relational databases.
My number one gripe of course is the use of "trust" in pg_hba.conf. If you use "trust", your thinking needs readjustment. I think the traditional first step in these situations is to admit that you have a problem. Hi, my name is (fill in your name here) and I use "trust" in my postgresql authentication setup. I don't care about security, I just want it to work.
You don't want to use "trust". It means that anyone is who they say they are. If my mail server has a remotely exploitable vulnerability, as did Exim last year, and someone manage to get shell access to that unprivileged user, they can connect to postgresql and demand to be called god. And it shall be granted.
If you don't want to be bothered with understanding how this works, then use a password mechanism such as md5. For example, on most Linux systems you want this in your pg_hba.conf:
local all postgres ident local all all md5 host all all 127.0.0.1/32 md5 host all all ::1/128 md5
You need to keep that one ident line so the cron job that does the vacuuming will continue to work. Everything else will require a password, so this will work just like MySQL.
I do not like this setup though. Why you may ask? Because I do not like embedding login details in configuration files. Login details are supposed to be there for security purposes. Storing those details in a file, a file that is most likely readable by the application server you expose to the big bad world, provides little security. It also means that you cannot check that configuration file into your source version control system as is without exposing potentially sensitive information to anyone who has access to it, and if you are anything like me, you've done this accidentally at least once. Storing a non-functional configuration file means that each time you check it out, you have to modify a configuration file before it will work. Not big problems I'll admit, but completely avoidable.
This is why I like "ident" authentication. In its default setup, it says simply that if the username of the logged in user, as seen from the operating system's point of view, is the same as the database user he claims to be, then access is granted. This works fairly well for development. Simply set the owner of your database to be the same as your unix user name, and no login details is necessary.
local all postgres ident local all all ident host all all 127.0.0.1/32 md5 host all all ::1/128 md5
At this point I want to explain how the lines in pg_hba.conf work. The first word in the line specifies whether this line applies to local connections, through the local unix socket, or to network connections from remote machines. The second column contains a database name, or "all" for all databases. The third column contains a user name. This is followed by an address and a netmask for "host" authentication, and finally by an authentication method, of which there are several. These lines are evaluated from top to bottom, and whichever one matches first is used for authentication. Order is therefore important, generic rules need to come after specific ones. In the above setup, all users connecting through the local unix socket need to connect with a username that match their unix user name (ident). All users connecting through the network need a username and password (md5).
Using "ident" on its own will require the unix user and the postgresql user to match. But there is a way you can allow users to connect as a different user while still using "ident". I'm told that this is what is most confusing to new users of postgresql.
You set this up in a file called pg_ident.conf. This file contains maps, it maps unix users to postgresql users. It is a simple three-column file:
# MAPNAME SYSTEM-USERNAME PG-USERNAME devel john zope devel jane zope devel peter zope
Combined with a pg_hba.conf like this
local all postgres ident local myprojectdb all ident map=devel local all all ident host all all 127.0.0.1/32 md5 host all all ::1/128 md5
This will allow john, jane and peter to log into myprojectdb with the username "zope".
However, complex setups that require setup in pg_ident.conf can also be avoided. If your application server runs as the "zope" user, then simply create a zope user in postgresql, and grant that user the required access in the relevant databases. End of story.
It also means the connection string in your configuration file is now
postgresql://@:5432/myprojectdb
Instead of
postgresql://john:b00j4h@:5432/myprojectdb
And that means you can check it straight into source control without exposing sensitive login details.
]]>For the most part I try not to bother with low-level SQL, even though I'm quite proficient in doing the occasional complex join operation. But occasionally you just want to delete a whole bunch of records quickly. In this specific instance, I just wanted to delete large numbers of users stored in a postgresql database and used in zope by means of pas.plugins.sqlalchemy.
I tried to do it the obvious way.
from pas.plugins.sqlalchemy import model
from z3c.saconfig import named_scoped_session
Session = named_scoped_session("pas.plugins.sqlalchemy")
session = Session()
session.execute(delete(model.User.__table__))
It didn't work. The records were all still there. So I turned statement logging on in the database to see what it was doing, and I noticed that at the end of the transaction, it always calls ROLLBACK. Google suggested that this sneaky low-level modification of the database does not mark the session as dirty, and therefore it is not committed. Eventually I dumbed down my search string and found this solution, which I will now (re)share with the world. If you are using zope.sqlalchemy, all you need to do is this:
from zope.sqlalchemy import mark_changed mark_changed(session)
Its even documented on the project page on pypi.
]]>There are a few people out there who provide such packages, myself included, but often they do not provide packages for all architectures, of which there are now several combinations. Just calculate the cross-product of lucid, maverick, natty and debian squeeze and i386 and amd64 as a start. Sure, debootstrap is your friend and someone like me could build them all if I wanted to, but I tend to stop working once I've scratched my own itches.
Here then, by popular request from my co-workers, is how you build your own. This package is based off the python 2.4.6 package that was in Debian Lenny, and was only adapted to remove most of the documentation, because it didn't build properly and you can just install the python2.6 versions if you must, and to deal with the multi-arch feature that is now in Natty.
Start by pointing your browser at http://public.upfronthosting.co.za/debian/sources/. Download the .orig.tar.gz file, and the .dsc and .diff.gz file that correspond with your distribution. Place them in a convenient spot, personally I prefer $HOME/debian/python2.4.
You will need to install a few things to get started
apt-get install build-essential fakeroot
Then unpack your package and check the build dependencies
dpkg-source -x python2.4_2.4.6-1+natty1.dsc cd python2.4-2.4.6 dpkg-checkbuilddeps
Install any listed dependencies, then compile your package
dpkg-buildpackage -b -uc -rfakeroot
The -uc option will skip any signing of the resulting binary packages. This is okay, since this is for local consumption only. If you insist on signing them, you'll have to set up gpg keys and update debian/changelog. Exact details of this is beyond the scope of this post, and frankly, beyond the time limits of my own memory as well.
When the building completes, you will have a number of debian packages in the parent directory, that is $HOME/debian/python2.4 if you followed my example. These can be installed with dpkg -i, or with gdebi. For the most part:
sudo dpkg -i python2.4-minimal_2.4.6-1+natty1_i386.deb python2.4_2.4.6-1+natty1_i386.deb python2.4-dev_2.4.6-1+natty1_i386.deb
The natty version build-depends on dpatch. This is simply because I was too lazy to convert the patch to work without it, and because an additional patch was required to deal with its multi-arch nature, or simply, the fact that zlib is not where it used to be previously.
Also note that even though a -doc package is produced, its largely useless.
Also, for the impatient who just need a binary package and happen to run an i386 architecture, you can add one of these to /etc/apt/sources.list
deb http://public.upfronthosting.co.za/debian/lucid-i386 / deb http://public.upfronthosting.co.za/debian/maveric-i386 / deb http://public.upfronthosting.co.za/debian/natty-i386 /]]>
This is not a long post. Its just something we had to do this morning, something that was necessary because some software on the windows platform has trouble with very long urls.
We noticed that the majority of urls on the site ends in resolveuid/someuid, but that they are very long because the folder structure is both deeply nested and have long and descriptive names. What we wanted to do is to shorten them simply to /resolveuid/someuid. But we didn't want to modify the content, because it would just break again the next time its opened in TinyMCE.
The solution is simple, because we were already using xdv.
<xsl:template match="@href[starts-with(., 'resolveuid/') or contains(., '/resolveuid/')]"> <xsl:attribute name="href">/resolveuid/<xsl:value-of select="substring-after(., 'resolveuid/')" /></xsl:attribute> </xsl:template>
This has to be placed in your rules file directly inside the <rules> tag. You cannot place it in any file that is included using xinclude, as I wanted to do, that does not work. It does now work because xincluded rules end up in nested <rules> tags, and the xsl stylesheet that generates the final stylesheet only looks for inline xsl in xdv:rules/xsl:*, that is, the topmost level.
]]>For example, if you named a table "order", you need to quote it because order is also a reserved word in SQL:
Postgresql: select * from "order" MySQL: select * from `order`
Triggers are a controversial feature that many people prefer not to use, but they can be useful for validation. But not if you run an older version of MySQL, and they may turn out to be useless even in the newer versions. I am told this issue has been fixed in the latest versions, but we often have to deploy on older versions so I include this little gripe. Although you can "make things happen" upon an insert, update or delete, you cannot reject the data by raising some kind of error. The only way to do this is to do something illegal in your trigger to effectively crash the transaction, and then the error message does not match the crime at all.
Its hard to write stored procedures or triggers (and therefore useful aggregate functions) for MySQL, because the languages at your disposal are so limited. Postgresql does spoil you with a plethora of choices, but even just one slightly-more-powerful-than-SQL yet slightly-higher-level than C language would help a lot.
Yes, it has auto_increment. Yes, you can implement your own makeshift sequences, but then you have to use locking to ensure mutual exclusion to make sure two clients do not get the same sequence value.
When I define a column in a table, I can specify a default for the column. In Postgresql this can be any function, and in the normal auto-increment use case you use a function to get the next available value from a sequence. In MySQL, there are two special cases: NOW and auto_increment. All other default values must be constants.
The documentation calls this a feature. MySQL will pick a representative value from the group if you don't tell it how to group a selected column. This allows people to write queries that appear to work, and they might even rely on the selected representative value without realising that there is no explicit way to know how it was picked. If you explicitly use an aggregate function in your where clause, you know exactly what to expect. Explicit is better than implicit. And besides, these queries are incompatible with other databases and hinders portability.
This is where things become zopeish. On a recent project using ore.alchemist I spent an hour trying to figure out why acquisition does not work and I end up with a ghost object from the database instead of a template somewhere higher up. It turns out that when I asked for context/index_html, and due to an omission in my code that did not check for numeric keys, MySQL was asked to find a user with USERID='index_html'. Because USERID is an integer column, MySQL correctly raised a warning (which disappeared among many other lines in the log file), assumed that what I really meant is USERID=0, and returned that row instead. Obvious errors like this should not pass with just a warning but with a very loud and clear error.
I don't know if this might be limited to older versions or to the open source versions, but during development we noticed that MySQL only uses one core on the CPU. Its faster, but only when it runs on a single-core single-cpu system using the MyISAM engine. On a recent project, I found that Postgresql can easily outperform MySQL (with InnoDB) on a more powerful machine.
We have a slightly ageing, but still very serviceable server that we wanted to put out in the 'just-look-after-some-media-pasture' for its final connected days. We had to add some additional drive-capacity to it and ran into a problem when we wanted to add the third SATA-drive, since there are only two slots on the motherboard.
We had a RoHS SATA4000 Serial ATA 4-channel PCI card floating around with nothing to do and I thought it could make itself handy by helping 'iggy' (our pet-name for the old server) to handle more SATA drives. I plugged it into the PCI-slot that looked eager to accept the SATA4000, plugged the additional SATA-drive into it and switched on the server expectantly.
Alas, it booted up to a point where it asked for me to press Ctr+S or F4 to configure some RAID facility, showed the SATA-drive id of the drive I connected to it ... and then it just hangs there ... no matter how many times I press Ctrl+S or F4.
So I rebooted (Ctrl+Alt+Del) with the intention to go fiddle around in the BIOS. The server has an Intel motherboard - Intel Server Board SE7210TP1 - and one has to press F2 to gain access to the BIOS. As soon as the machine started rebooting, I started pressing F2 continuously, but I might as well not have been there - I was ignored flatly and the machine proceeded exactly as before and obstinately went to the 'configure RAID' point and stayed there.
After many attempts including Googling for motherboard manuals, SATA4000 card manuals, downloading the pdf's and reading (okay ... scanning!) them, I asked an experienced sysadmin-friend who suggested the first, simple and obvious break-through step: "Remove the SATA4000" card altogether and THEN try to access the BIOS.
Voila! F2 to access BIOS suddenly worked! Since I thought that it was maybe trying to boot from the other drive, I had a look at the Boot Sequence and Boot Device settings first. I limited them to CDROM and Boot Hard Drive, disabling the possibility of using any of the other drives for booting. I also had a look at the PCI-settings but did not notice anything (at the time) that screamed 'Change Me!' at me. I saved the settings ... shut the machine down ... added the SATA4000 again ... and rebooted expectantly ... DAMN! ... same problem.
Okay ... out with the SATA4000 again ... back into the BIOS ... gravitated to the PCI-configuration settings again (Advanced->PCI Configuration) and asked my friend what are these 'Slot 1 etc Option ROM Enabled/Disabled' all about? And what is 'Slot 6'? As I was asking this I looked at the actual motherboard and noticed that the slots are named on the board and the slot where I add the SATA4000, is marked 'Slot 6' on the board.
That's when the penny dropped - "Disable the ROM option" on Slot 6 is what my friend, Izak suggests immediately. Apparently there is a ROM on the card that the machine is stubbornly 'listening' to upon bootup ... hence the compulsion to RAID ... :) This setting instructs the machine to ignore whatever a ROM on a card in that slot, is trying to say/instruct.
After that BIOS change everything went smoothly and according to plan and 'iggy' is now proudly looking after media with lotsa SATA drives in its belly!
I have an old, but still stately Thinkpad Z60m that is still serving me just fine. Yeah, yeah okay ... I suppose I can also be described as old an stately!
Anyway, I love using the trackpoint on the Thinkpad - in fact, I love it so much that I will never buy another laptop or netbook unless it has a trackpoint. One tweak needed to make the trackpoint-experience perfect for me, is to be able to scroll with the trackpoint whilst keeping the middle-button down.
This functionality, however, is not enabled by default in my Ubuntu installs. I have recently had a stutter from my harddrive and rather replaced it. That also gave me the opportunity to install Ubuntu 10.04 LTS - the Lucid Lynx.
Immediately after the install I was confronted with why I love my middle-button scroll functionality - because I did not have it!
Here is how I got it back:
I got this information originally from: http://www.thinkwiki.org/wiki/How_to_configure_the_TrackPoint#xorg.conf.d
The above will not work in Ubuntu 10.10 (maverick) - I tried ... :)
BUT ... they have acknowledge our need/desires by bringing it closer to the surface for us noobs. In maverivk, you simply:
1. Go to the Ubuntu Software Centre
2. Search for 'pointing devices'
3. Click on 'install' for 'pointing devices'
4. After install has completed go to System->Preferences->Pointing devices
5. Select 'Use Wheel Emulation'
6. Select 'button 2'
7. Select both vertical and horizontal scroll
8. Click 'Ok'
Voila!!
]]>I decided to reconfigure my AP to use WPA instead of WEP. While I was fiddling with the web based configuration a Windows laptop somewhere in my house was connected to the AP. I foolishly enabled both WPA and WEP on my AP, and the Windows wi-fi driver (for a Broadcom 4306 on an HP notebook) threw a fit and disconnected the notebook.
I realized my mistake and enabled only WPA, but another machine had issues so I went back to my original WEP settings. So the HP should just work, right?
No. It could connect to the network with trouble. This seemed strange because the signal strength was as good as always. After a while it disconnected me, and then slowly reconnected. Figuring that I broke something in Windows I rebooted in Ubuntu since I just needed to do some web browsing. Imagine my surprise when Ubuntu exhibited the same behaviour. My other WEP connected machines were fine but this notebook had somehow mysteriously been broken on a hardware level.
After struggling in Ubuntu for hours I went back to Windows - still broken. I downloaded a recent Broadcom driver, installed it and it all worked! Even better, rebooted in Ubuntu and it worked there as well! This means the Windows driver did something to firmware on the wi-fi adapter when it encountered that first WEP / WPA error.
Moral of the story: disable all connectivity to an AP completely before reconfiguring its connection method.
]]>The PC boots up and immediately goes bezerk, telling you there is spyware on the machine and you had better run a full scan pronto. Then this rather official looking program pops up and tells you you have about two dozen files infected by as many root kits, trojans and virii. It will remove those at a cost of $50. Internet posts suggests you might be charged arbitrary amounts ranging in the hundreds for this little favour.
This is the story of a Rogue aka fake Antivirus program. It is incredibly sneaky. First things first, it does this to your registry:
HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\System\DisableTaskManager = 1
This makes it impossible to get into the task manager and kill anything. In addition, trying to start anything, be it taskmgr, regedit or a web browser, results in a popup telling you that the application is infected and cannot run. It completely killed off the two other antivirus programs on the machine, McAfee and AVG, and stopped Adobe Flash from updating itself. This is of course an attempt to stop you from using any kind of tool that might get you out of this situation, under the guise that doing anything at this point other than running a full scan (which will cost you a random amount) is too dangerous.
But the programmer made a small mistake that is helpful here. By the time you read this, it might no longer work. When the dialog box pops up, don't click OK. Leave it there. In this position, the rogue app is blocked and does not kill apps that you start. Now run regedit and fix your registry (just delete the above key), then hit ctrl+alt+del and start the task manager. Not that it is going to help much, but it feels sort of good.
It doesn't help because this irritating piece of buffalo dung has taken over what is supposedly a valid windows binary called wscntfy.exe, and it cannot be killed. This is a surprise to me, since I have often lamented the crazy KillProcess stuff that does not use signals that can be caught. Something must be supervising this and messing us around.
If you delete wscntfy.exe, it will be replaced. You cannot overwrite it, for windows does not allow files to be modified while they are open. You could boot into your favourite linux-on-cd rescue system and work away at it, but in my case this machine had sufficiently weird hardware that Linux could not find it's root filesystem after switching to protected mode. Nice.
So I took the easy way out. Download Malwarebyte's anti-malware program. Run it and let it clean up. Do not interact with any of the dialog boxes of the rogue program.
Why do I write this? Because I know this blog is syndicated in at least one place where a couple of readers might still run windows. Because it is interesting in the way it would be interesting to Doctor Gregory House. Because there needs to be more good search engine findable links to a solution. And because I wasted the better part of an afternoon on this.
Normal programming, the kind that never speaks a kind word about a certain Os from Redmond, will resume shortly.
]]>This is a follow-up on my previous post about upfront.diet. Although we have done some work around folder contents and references we decided to take a step back, document Plone's catalog usage and propose alternative fat free implementations for comment by other developers. Below is a list of features that currently use the catalog. We consider each of them and whether they can work without the catalog and propose an alternative implementation where possible.
Portal tabs query the catalog for top level folders only. This can easily be replaced by using upfront.diet's foldercontents adapter that fetch objects directly. This adapter indexes allowed roles and users directly on the adapted container and uses this index to only return children a user has access to. This adapter is already used in upfront.diet's folder_contents implementation.
We already have a successful implementation for folder_contents but some of the other folder views (folder_listing, folder_summary_view and folder_tabular_view) still need to be fixed. The catalog query in use by these templates need to be replaced by iterating over the contents of the foldercontents adapter.
Replacing the navigation portlet with an implementation that doesn't use the catalog could be significantly slower if not thought through carefully. The existing navigation portlet can render multiple levels deep and it renders all the children of all the parent folders in the path of the current folder. Before one burdens the catalog with indexes and metadata to support the navigation portlet, one should reconsider fetching objects to build the tree and determine exactly when this becomes too costly and if caching can be used to achieve good performance. If it turns out that cataloging is indeed required, one should consider using a separate catalog to support navigation. Having a separate catalog ensures that we don't end up with a bloated catalog like we currently have and makes it possible to clear the portal_catalog without breaking navigation. It makes more sense to separate catalogs that are meant for content searching from catalogs that are required to support specific use cases like navigation.
The next step is to benchmark a navigation portlet that fetches objects where the navigation tree has no more than a 100 objects at each level. A navigation portlet that renders more than a 100 objects don't really make any sense and anything more than a 100 is most probably unusable.
There is a significant overlap between the current implementation for the navigation portlet and the site map in that they both use the NavtreeQueryBuilder to build a tree and one could optimise them using the same strategy.
The review portlet asks the workflow tool for the worklists of all workflows. In a standard Plone site all the workflows have a worklist named reviewer_queue and they all query the catalog for content in the pending state. It could be argued that review_state is an important index for site wide searches in any case and at least this attribute must be indexed in the portal_catalog. This is definitely a convenient conclusion but I'm not going to accept it just yet. An alternative implementation might be to use a real queue of objects that need to be reviewed. This has a couple of advantages over indexing review_state in the catalog in that you don't need to index review_state on all objects and the queue will always be small since objects will be popped from the queue as they are processed. The reviewer queue can be a simple persistent object possibly using one of the queue classes in zc.queue. Besides the queue, event subscribers are necessary to add pending objects to and remove processed objects from the queue.
The calendar portlet queries the catalog using the CatalogTool.catalog_getevents method. This method still has a comment that states "XXX: this method violates the rules for tools/utilities: it depends on a non-utility tool" so we have an opportunity to fix this as well. The comment isn't entirely clear but I assume it is referring to the dependency on the catalog tool. This method queries the catalog on the portal_type, review_state, start and end indexes. The calendar clearly needs to index start and end date of events. If one only indexes portal types and review states configured in the calendar tool, you don't need indexes for portal_type and review_state. If you change the calendar configuration you can always re-index content in your portal to include other portal_types or additional review states.
A possible fix would be to add an utility, named IEventIndexer, that indexes start and end date for content types listed in the calendar tool configuration. An additional utility, IQueryEvents, can be used to query for events. The ZODB versions of these utilities can index and query a ZCatalog instance.
The search performed by smart folders are clearly site wide content searches so it makes perfect sense to use the portal_catalog in this case. The Collection configlet already allows one to customise which fields must be enabled for smart folders so we don't really need to develop anything to allow searching on fewer indexes. Note that the News and Events smart folders installed by Plone only requires the portal_type, review_state and start indexes, should you want to reduce the number of indexes in your catalog.
The site search clearly needs the portal_catalog, but hopefully we require a lot fewer indexes in the catalog. Live search and the regular site search query the SearchableText, portal_type and path indexes. The query on portal_type is used to restrict results to "user friendly types" which are types configured in the search configlet. If one has a portal where one restricts the number of types that can be searched, it probably doesn't make sense to index types that are not searched anymore. Given that we use the portal_catalog only for content searches and not to support functions listed above we can safely stop indexing these types without breaking the rest of the Plone user interface. One should probably make this configurable by adding an option reading "Don't index types that are excluded from searches" on the search configlet.
By using Simple References we removed the dependency on the reference catalog and this has proven to perform significantly better in some of our reference-heavy applications. However, we still depend on the UID catalog and are considering an approach that doesn't require any catalog for references. ZODB already support references between objects, by simply assigning a persistent object to an attribute of an existing persistent object eg. folder.item = object. The only reason ZODB references are not used for Archetypes references is that most applications require the reference to refer to the object in its original context. If you access a ZODB reference (like folder.item) it will wrap 'item' in the context of 'folder', regardless of its original context. Luckily Zope3 introduced the idea of a location (see ILocation in zope.location.interfaces) for objects with a structural location. This location of an object can be determined by reading the __parent__ attribute of an object and its parents. This means that we can develop a Archetypes reference field that use ZODB references to reference objects but still return the objects in their original location by wrapping it in their original location. Currently Plone containers don't implement ILocation so we will need to investigate what needs to be done to fix this - the interface suggests it is straightforward to implement.
Knowing what pieces of Plone depend on the portal catalog is already helpful if you are developing for Plone or trying to optimise it. The above list is definitely not an exhaustive list of catalog dependencies but covers the most common ones. Please tell us if we've missed something obvious. In no particular order, we will continue to implement the alternatives suggested above and will gladly incorporate suggestions or improvements.
]]>First things first: I get upset when people bitch and moan about Archetypes. It has solved a lot of problems for Plone folks over the years so we should respect that. Even so it is getting on a bit so enter Dexterity.
I like the name Dexterity. Dexterity has always been my favorite attribute in D&D. You can turn an Elf into a tank with a 19 DEX and presumably I can fashion my apps in a similarly robust way.
So what is the best part of Dexterity for me? It is unobtrusive and blindingly fast. It does not depend on a request being present everywhere, so it is much easier to do things programmatically. It works well and is fully documented.
What do I dislike? It forces me to open my eyes to other exciting technologies like z3c.form. So I really dislike myself for not having looked at those technologies earlier. There is no major learning curve to Dexterity - it's all hidden in the related set of tools. But the reward for finally learning z3c.form and friends is that I can use them outside of Plone as well.
]]>Over the years I have tried very hard to unpack the truth about Plone's performance problems and many other developers have done so too. A few products have seen the light to address the speed of Plone of which CacheFu, experimental.catalogqueryplan, archetypes.schematuning and SimpleReferenceField are the ones that I use often. Besides these products, the plone core has seen many optimisations over the years but naturally, and unfortunately, some new performance bugs are introduced a long the way. Generally though we have enough tools to save the day but I don't think that we have done enough to solve Plone's performance problems and with that the perception that Plone is slow.
There is something that complicates how this perception is formed, specifically the fact that Plone is not only a CMS, it is a development platform as well. If it wasn't, I think it wouldn't be any good in it's primary role as a CMS. With tools like ArchgenXML and paster, developers can churn out new content types and Plone products faster than Mcdonalds slides hamburgers across the counter. Yes, they are equally unhealthy! These content types inherit too much fat from the existing Plone environment and add even more fat by design. A lot of the fat can be attributed to Plone's obsession with the catalog and its irrational fear of waking up objects. The truth is, you can't use Plone if you empty your portal catalog: folder contents won't work, navigation will break, portlets will be empty and there won't be any breadcrumbs to help you run from the Grimm brothers' evil old witch. We really don't need all content types to be indexed, to have metadata, workflow, history and complicated references. In short, you don't need all content types to have all the features. This is the biggest problem with developing for Plone. Unless you know your way around 10 million lines of code and 10 years of history really well, you will find it surprisingly hard to make simple content types with less features. In other development frameworks you typically start with nothing and add the features as you need them. For example, nothing is searchable by default, you add the index when required. I believe that the same is possible with Plone, and that such an approach will serve it much better in the long run. This brings me to upfront.diet.
upfront.diet is an experimental product that will explore the ways that Plone can be turned in a development platform that lets you add features as needed, rather than taking them away. By default no content type should require being indexed or being part of a workflow. To make this work, we will rewrite the existing content listings to not use the catalog but rather list the contents of the folder using the traditional ObjectManager API using objectIds or objectValues. To filter the content for a specific role or user, we will store allowedRolesAndUsers as a local keyword index on the folder itself. From this point on we will carefully consider any additional index requirements imposed by searching and sorting and rather than installing all these indexes by default, we will allow adding them through configuration.
upfront.diet will patch existing content types to use SimpleReferenceField instead of the default Archetypes ReferenceField to remove the dependency on the reference catalog. In our experience the reference catalog is a major performance bottleneck on content types with as little as two or three references.
Both the portal_catalog and reference_catalog are often responsible for conflict errors in a Plone site so avoiding them for simpler operations that don't require them must be a step in the right direction. I hope that one can end up with an extremely light weight portal_catalog that only contains indexes that are required for site-wide content searches. This should greatly reduce conflict errors and catalog bloat. Additionally moving to an implementation where modifying content don't involve a single tool but modify attributes in a local context instead is a much better fit when using the ZODB.
Let's hope this diet works and we can make a toast on Plone's health. Prost!
]]>
After a nine month experiment with pair programming, we decided that it is not a good fit for us. Here's why:
To achieve the same level of quality and knowledge transfer that pair programming promises, while being more flexible at the same time, we practice the following:
Developers generally don't like documenting what lies ahead of them, but they have to fight against the urge to just jump in and start coding. I think it is essential for developers to learn to express the design of a system in a narrative, both verbally and in writing. This helps them to design better systems and increases the maintainability of their software. In short, I think good developers are good story tellers.
]]>Varnish has a feature called backend probing, which I still think is a very unfortunate name. You basically tell it what url to hit on the backend, and it will periodically hit that url and check the responsiveness of the backend.
The idea is to use backend probing on the zope backends, so that when you restart, you do it in such a way that you always have enough healthy backends left to carry the load while the other ones restart and warm their caches.
This is different to what squid does. Squid does an ICP probe on the backend, but this is where the problem starts. When zope starts, ICP becomes responsive long before that zope instance is ready to serve content, causing squid to pass requests to cold backends. It also seems that the newly started zope instance responds faster, and squid passes the request to whichever backend responds first to the ICP query.
So what you get is a website that is partially non-responsive while you have n-1 perfectly healthy backends. Varnish was going to solve all that.
But varnish has an odd way of checking the backend. It sends the request, then it half-closes the connection (it closes the writing side, read the man page for shutdown system call if you're interested) and waits for the response on the open read side. Zope, or more specifically python's asyncore module, cannot distinguish between a half-close and a full close, and it shuts down the connection without sending a response.
Depending on how you interpret HTTP1.1, zope actually does the right thing when it closes the connection. Under "8.1.4 Practical Considerations" in RFC2616 it says:
When a client or server wishes to time-out it SHOULD issue a graceful close on the transport connection. Clients and servers SHOULD both constantly watch for the other side of the transport close, and respond to it as appropriate. If a client or server does not detect the other side's close promptly it could cause unnecessary resource drain on the network.
As I understand it, a "graceful close" means to close your end of the connection, ie, half-close. In other words, half-closing means you're timing out.
I then took the shutdown() call out of the code and backend probing started to work. I also posted the entire reasoning and the thread on zope-dev to varnish-dev and one of their developers indicated that the shutdown() call will be removed.
The thread on the varnish list can be found here
]]>